RFC 6249

Metalink/HTTP: Mirrors and Hashes

📅 published June 2011, from ietf

Okay bestie, here's the title block tea ☕ This is RFC 6249, Metalink/HTTP: Mirrors and Hashes, a Standards Track doc from the IETF (ISSN 2070-1721). It came out in June 2011. The authors are A. Bryan, N. McNab, T. Tsujikawa, P. Poeml (MirrorBrain), and H. Nordstrom. It doesn't obsolete or update anything. 💅

Abstract ✨

Okay so this doc is all about Metalink/HTTP: Mirrors and Cryptographic Hashes, and it's shipped as HTTP header fields 🎀 It's a different way to get the info that usually lives in the Metalink XML-based download description format. Metalink/HTTP packs up multiple download locations (mirrors), Peer-to-Peer stuff, cryptographic hashes, digital signatures, and more, all using standards that already exist for HTTP header fields. Clients can use this to make file transfers more robust and reliable, and the normative requirements for Metalink/HTTP clients and servers are spelled out right here. Iconic 🙌

Status of This Memo 🌸

This is an Internet Standards Track document.

This doc is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication. If you want more on Internet Standards, check Section 2 of RFC 5741.

For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc6249.

Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, since they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents 🧚

1. Introduction ....................................................3
   1.1. Example Metalink Server Response ...........................4
   1.2. Notational Conventions .....................................4
   1.3. Terminology ................................................5
2. Requirements ....................................................5
3. Mirrors / Multiple Download Locations ...........................7
   3.1. Mirror Priority ............................................8
   3.2. Mirror Geographical Location ...............................8
   3.3. Coordinated Mirror Policies ................................8
   3.4. Mirror Depth ...............................................9
4. Peer-to-Peer / Metainfo .........................................9
   4.1. Metalink/XML Files ........................................10
5. Signatures .....................................................10
   5.1. OpenPGP Signatures ........................................10
   5.2. S/MIME Signatures .........................................10
6. Cryptographic Hashes of Whole Documents ........................11
7. Client / Server Multi-Source Download Interaction ..............11
   7.1. Error Prevention, Detection, and Correction ...............15
        7.1.1. Error Prevention (Early File Mismatch Detection) ...15
        7.1.2. Error Correction ...................................16
8. IANA Considerations ............................................16
9. Security Considerations ........................................17
   9.1. URIs and IRIs .............................................17
   9.2. Spoofing ..................................................17
   9.3. Cryptographic Hashes ......................................17
   9.4. Signing ...................................................17
10. References ....................................................18
   10.1. Normative References .....................................18
   10.2. Informative References ...................................19
Appendix A. Acknowledgements and Contributors .....................20

1. Introduction ✨

Okay so Metalink/HTTP is an alternative and complementary way to represent Metalink info, which usually shows up as an XML-based document format [RFC5854]. It tries to give you as much functionality as the Metalink/XML format, but it does it using standards that already exist, like Web Linking [RFC5988], Instance Digests in HTTP [RFC3230], and Entity Tags (also known as ETags) [RFC2616]. It's used to list info about a file you're about to download: lists of multiple URIs (mirrors), Peer-to-Peer info, cryptographic hashes, and digital signatures. 🦋

Identical copies of a file are often sitting in multiple spots on the Internet, over a bunch of protocols (like FTP, HTTP, and Peer-to-Peer). Sometimes users get shown a list of these download locations (mirrors) and have to manually pick one based on geography, priority, or bandwidth. That spreads the load across servers, and it should also boost throughput and resilience. 🌈 But sometimes servers are slow, outdated, or unreachable, and you can't tell until the download has already started. Users rarely have enough info to pick the best server, so they usually just grab the first one on the list, which might not be ideal for them, and then one server ends up getting way too much traffic. 😭 Using a suboptimal mirror can make people cancel and restart the download to hunt for a better source. Also, errors during transmission can corrupt the file, and there's no easy way to fix those files. For big downloads, that's a nightmare. Honestly, any of the many things that can go wrong during a download leads to major frustration. 💀

Some popular sites automate mirror selection with DNS load balancing. The goal is to roughly balance load between servers and send clients to nearby servers, hoping that improves throughput. And yes, DNS load balancing can balance long-term server load pretty well, but it's way less effective at giving users the best throughput when the bottleneck is the network instead of the server. 🙈

This document describes a mechanism that lets mirrors pay off automatically and more effectively. All the info about a download, including mirrors, cryptographic hashes, digital signatures, and more, can be sent in coordinated HTTP header fields, which we'll call a "Metalink". This Metalink hands the knowledge of the download server (and its mirror database) over to the client. Clients can fall back to other mirrors if the current one has a problem, so the client can work its way to a successful download even when things are rough. All of this happens without complicated user interaction, and the download can be way more reliable and efficient. 👑 Compare that to a traditional HTTP redirect to a mirror, which only tells you one link to one server, and HTTP has no provision for handling failures. Also, to get better load distribution across servers and potentially faster downloads, Metalink/HTTP supports multi-source downloads, where portions of a file come from multiple mirrors (and, optionally, Peer-to-Peer) at the same time. ✨

When a client connects to a Metalink/HTTP server, it gets info about other sources of the same resource and a cryptographic hash of the whole resource. Then the client can request chunks of the file from the different sources and schedule them to maximize the download rate. 🚀

This example is a quick Metalink server response with an ETag, mirrors, Peer-to-Peer info, Metalink/XML, an OpenPGP signature, and a cryptographic hash of the whole file:

Etag: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
Link: <http://www2.example.com/example.ext>; rel=duplicate
Link: <ftp://ftp.example.com/example.ext>; rel=duplicate
Link: <http://example.com/example.ext.torrent>; rel=describedby;
type="application/x-bittorrent"
Link: <http://example.com/example.ext.meta4>; rel=describedby;
type="application/metalink4+xml"
Link: <http://example.com/example.ext.asc>; rel=describedby;
type="application/pgp-signature"
Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO
DYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ==

1.2. Notational Conventions 📝

This specification describes conformance of Metalink/HTTP.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, [RFC2119], as scoped to those conformance targets. Those are the official key words, so pay attention to them 💅

1.3. Terminology 💜

Here are the terms, as used in this document, and what they mean:

  • Metalink server: HTTP server that provides a Metalink in HTTP response header fields.
  • Metalink : A collection of HTTP response header fields from a Metalink server, which is the reply to a GET or HEAD request from a client and which includes Link header fields listing mirrors and Instance Digests listing a cryptographic hash.
  • Link header field: HTTP response header field, defined in [RFC5988], that can list mirrors and, potentially, other download methods for obtaining a file, along with digital signatures.
  • Instance Digest: HTTP response header field, defined in [RFC3230], that contains the cryptographic hash of a file, which is used by the Metalink client to verify the integrity of the file once the download has completed.
  • Entity Tag or ETag: HTTP response header field, defined in [RFC2616], that, if synchronized between the Metalink server and mirror servers, allows Metalink clients to provide advanced features.
  • Mirror server: Typically, FTP or HTTP servers that "mirror" the Metalink server, i.e., provide identical copies of (at least some) files that are also on the mirrored server.
  • Metalink clients: Applications that process Metalinks and use them to provide an improved download experience. They support HTTP and could also support other download protocols like FTP or various Peer-to-Peer methods.
  • Metalink/XML: An XML file that can contain similar information to an HTTP response header Metalink, such as mirrors and cryptographic hashes. 🧚

2. Requirements ✨

okay so first things first 💅 in this doc, "Metalink" means Metalink/HTTP, which is basically mirrors and cryptographic hashes living in HTTP header fields 🎀. "Metalink/XML" is the XML format from [RFC5854] 🌸.

Metalink resources use Link header fields [RFC5988] to show a list of mirrors when a client asks for the resource 🦋. Metalink servers MUST include the cryptographic hash of a resource via Instance Digests in HTTP [RFC3230]. The algorithms for the Instance Digest field are registered in the IANA registry called "Hypertext Transfer Protocol (HTTP) Digest Algorithm Values" at <http://www.iana.org/> 📋. This doc puts limits on how those algorithms get used. SHA-256 and SHA-512 were added to the registry by [RFC5843] ✨. Metalinks contain whole file hashes (see Section 6), and they MUST include SHA-256, as specified in [FIPS-180-3]. They MAY also include other hashes 🔥.

Metalink servers are HTTP servers with one or more Metalink resources 🧚. They MUST support the Link header fields for listing mirrors and MUST support Instance Digests in HTTP [RFC3230]. Metalink servers MUST return the same Link header fields and Instance Digests on HEAD requests 💜. Metalink servers and their preferred mirror servers MUST all share the same ETag policy. Metalink servers and their normal mirror servers SHOULD all share the same ETag policy. (See Section 3.3 for the definition of "preferred" and "normal" mirror servers 👑.) It's up to the administrator of the Metalink server to tell the mirror administrators the details of the shared ETag policy, so the mirrors can be configured to match 🫶. Having the same ETag policy means ETags are synced across servers for mirrored resources. In other words, byte-for-byte identical files get the same ETag on the mirrors that they have on the Metalink server. For example, it's better to derive an ETag from a cryptographic hash of the file contents than from server-unique filesystem metadata 🙈. Metalink servers SHOULD offer Metalink/XML documents that contain cryptographic hashes of parts of the file (and other information) if error recovery is desirable 😭.

Mirror servers are typically FTP or HTTP servers that "mirror" another server 🍓. They provide identical copies of (at least some) files that are also on the mirrored server. Mirror servers SHOULD support serving partial content. HTTP mirror servers SHOULD share the same ETag policy as the originating Metalink server. HTTP mirror servers SHOULD support Instance Digests in HTTP [RFC3230] using the same algorithm as the Metalink server. Optimally, HTTP mirror servers will share the same ETag policy and support Instance Digests in HTTP ✨. Mirror servers that share the same ETag policy and/or support Instance Digests in HTTP using the same algorithm as a Metalink server are known as preferred mirror servers 👑.

Metalink clients use the mirrors that a Metalink server provides in Link header fields [RFC5988], but they're restricted to using the mirrors from the initial Metalink server they contacted 🚫. If Metalink clients find Link header fields [RFC5988] from mirrors that in turn list mirrors, or from a Metalink server listing itself as a mirror, they MUST discard such Link header fields [RFC5988] to prevent a possible infinite loop 🌀. Metalink clients MUST support HTTP and SHOULD support FTP [RFC0959] 🎀. Metalink clients MAY support BitTorrent [BITTORRENT] or other download methods 🦋. Metalink clients SHOULD switch downloads from one mirror to another if a mirror becomes unreachable. Metalink clients MAY support multi-source, or parallel, downloads, where portions of a file are downloaded from multiple mirrors at the same time (and, optionally, from Peer-to-Peer sources) 🚀. Metalink clients MUST support Instance Digests in HTTP [RFC3230] by requesting and verifying cryptographic hashes. Metalink clients SHOULD support error recovery by using the cryptographic hashes of parts of the file listed in Metalink/XML files. Metalink clients SHOULD support checking digital signatures 🔐.

3. Mirrors / Multiple Download Locations 🪞

Mirrors are specified with the Link header fields [RFC5988] and a relation type of "duplicate" as defined in Section 8 🎀.

The following list contains OPTIONAL attributes, which are defined elsewhere in this document:

  • depth : mirror depth (see Section 3.4).
  • geo : mirror geographical location (see Section 3.2).
  • pref : a preferred mirror server (see Section 3.3).
  • pri : mirror priority (see Section 3.1).

This example shows a brief Metalink server response with two mirrors only 💖:

Link: <http://www2.example.com/example.ext>; rel=duplicate;
pri=1; pref
Link: <ftp://ftp.example.com/example.ext>; rel=duplicate;
pri=2; geo=gb; depth=1

Some organizations have a lot of mirrors, so it's up to the organization to configure how many Link header fields the Metalink server provides 🧠. That could be a random selection or a hard-coded limit based on network proximity, file size, server load, or other factors 🌈.

3.1. Mirror Priority 🥇

Entries for mirror servers MAY have a "pri" value to designate the priority of a mirror. Valid ranges for the "pri" attribute are from 1 to 999999. Mirror servers with a lower value of the "pri" attribute have a higher priority, while mirrors with an undefined "pri" attribute are considered to have a value of 999999, which is the lowest priority. For example, a mirror with "pri=10" has higher priority than a mirror with "pri=20". Metalink clients SHOULD use mirrors with lower "pri" values first, but depending on other conditions, they MAY decide to use other mirrors 💅.

This is purely an expression of the server's preferences. It's up to the client what it does with this information, particularly when it comes to how many servers to use at any one time 🫶.

3.2. Mirror Geographical Location 🗺️

Entries for a mirror server MAY have a "geo" value, which is an [ISO3166-1] alpha-2 two-letter country code for the geographical location of the physical server the URI is used to access 🌍. A client MAY use this information to select a mirror, or set of mirrors, that is geographically near (if the client has access to such information), with the aim of reducing network load at inter-country bottlenecks ✨.

3.3. Coordinated Mirror Policies 🤝

There are two types of mirror servers: preferred and normal 👑. Entries for preferred HTTP mirror servers have a "pref" value, and entries for normal mirrors don't. Preferred mirror servers are HTTP mirror servers that MUST share the same ETag policy as the originating Metalink server, or, if the ETag is not used, MUST provide an Instance Digest using the same algorithm as the Metalink server. Preferred mirrors make it possible for Metalink clients to detect early on, before any data is transferred, whether the file requested matches the desired file 🔍. This early file mismatch detection is described in Section 7.1.1. Normal mirrors do not necessarily share the same ETag policy or support Instance Digests using the same algorithm as the Metalink server. FTP mirrors are considered "normal", since they don't emit ETags or support Instance Digests 🙃.

3.4. Mirror Depth 🌳

Some mirrors can mirror single files, whole directories, or multiple directories 📁.

Entries for mirror servers can have a "depth" value, where "depth=0" is the default. A value of 0 means that only that file is mirrored and that other URI path segments are not. A value of 1 means that the file and all other files and URI path segments contained in the rightmost URI path segment are mirrored. For values of N, N-1 URI path segments closer to the Host are mirrored. A value of 2 means one URI path segment closer to the Host is mirrored, and all files and URI path segments contained are mirrored. For each higher value, another URI path segment closer to the Host is mirrored 🪜.

This example shows a mirror with a depth value of 4 🧚:

Link: <http://www2.example.com/dir1/dir2/dir3/dir4/dir5/example.ext>;
rel=duplicate; pri=1; pref; depth=4

In the above example, four URI path segments closer to the Host are mirrored, from /dir2/ and all files and directories included 💜.

4. Peer-to-Peer / Metainfo 🫧

Entries for metainfo files, which describe ways to download a file over Peer-to-Peer networks or otherwise, are specified with the Link header fields [RFC5988], a relation type of "describedby", and a type parameter that indicates the MIME type of the metadata available at the URI. Since metainfo files can sometimes describe multiple files, or the filename MAY not be the same on the Metalink server and in the metainfo file but MAY still have the same content, an OPTIONAL "name" attribute can be used 🌸.

The following list contains an OPTIONAL attribute, which is defined in this document:

  • name : a file described within the metainfo file.

This example shows a brief Metalink server response with .torrent and .meta4 🦋:

Link: <http://example.com/example.ext.torrent>; rel=describedby;
type="application/x-bittorrent"; name="differentname.ext"
Link: <http://example.com/example.ext.meta4>; rel=describedby;
type="application/metalink4+xml"

Metalink clients MAY support the use of metainfo files for downloading files 🔥.

Metalink/XML files for a given resource MAY be provided in a Link header field, as shown in the example in Section 4. Metalink/XML files are specified in [RFC5854], and they are particularly useful for providing metadata such as cryptographic hashes of parts of a file, which lets a client recover from errors (see Section 7.1.2) 🩹. Metalink servers SHOULD provide Metalink/XML files with partial file hashes in Link header fields, and Metalink clients SHOULD use them for error recovery 😭.

5. Signatures ✍️

5.1. OpenPGP Signatures 🔐

OpenPGP signatures [RFC3156] of requested files are specified with the Link header fields [RFC5988], a relation type of "describedby", and a type parameter of "application/pgp-signature" 💌.

This example shows a brief Metalink server response with OpenPGP signature only:

Link: <http://example.com/example.ext.asc>; rel=describedby;
type="application/pgp-signature"

Metalink clients SHOULD support the use of OpenPGP signatures 🫶.

5.2. S/MIME Signatures 📨

Secure/Multipurpose Internet Mail Extensions (S/MIME) signatures [RFC5751] of requested files are specified with the Link header fields [RFC5988], a relation type of "describedby", and a type parameter of "application/pkcs7-mime" 🎀.

This example shows a brief Metalink server response with S/MIME signature only:

Link: <http://example.com/example.ext.p7m>; rel=describedby;
type="application/pkcs7-mime"

Metalink clients SHOULD support the use of S/MIME signatures 💖.

6. Cryptographic Hashes of Whole Documents 🧮

If the Metalink servers don't provide Instance Digests, the Link header fields pertaining to this specification MUST be ignored 🚨.

This example shows a brief Metalink server response with ETag, mirror, and cryptographic hash 🌈:

Etag: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
Link: <http://www2.example.com/example.ext>; rel=duplicate
Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO
DYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ==

7. Client / Server Multi-Source Download Interaction ✨

okay bestie, so a Metalink client starts the download with a totally normal HTTP [RFC2616] GET request to the Metalink server 💖 and it MAY use a range limit if it wants to.

GET /distribution/example.ext HTTP/1.1
Host: www.example.com

The Metalink server answers with the data and these header fields 🎀

HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 14867603
Content-Type: application/x-cd-image
Etag: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
Link: <http://www2.example.com/example.ext>; rel=duplicate; pref
Link: <ftp://ftp.example.com/example.ext>; rel=duplicate
Link: <http://example.com/example.ext.torrent>; rel=describedby;
type="application/x-bittorrent"
Link: <http://example.com/example.ext.meta4>; rel=describedby;
type="application/metalink4+xml"
Link: <http://example.com/example.ext.asc>; rel=describedby;
type="application/pgp-signature"
Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO
DYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ==

Another option is that the client begins with a HEAD request to the Metalink server, so it can find mirrors through the Link header fields. Then it can skip ahead and make the following decisions for every available mirror server it finds in those Link header fields 🦋

After that, the client follows up with a GET request to the mirrors it wants 🌸

From the Metalink server response, the client learns some or all of this metadata, all while it's already starting to receive the object 🧚

  • Mirror locations, with optional attributes describing the mirror's priority, whether it shares the ETag policy of the originating Metalink server, geographical location, and mirror depth.
  • Instance Digest, which is the whole file cryptographic hash.
  • ETag.
  • Object size from the Content-Length header field.
  • Metalink/XML, which can include partial file cryptographic hashes to repair a file.
  • Peer-to-Peer information.
  • Digital signature.

Next, the Metalink client asks a preferred mirror server for a range of the object. It does this with If-Match conditions so it can check the ETag 💅

GET /example.ext HTTP/1.1
Host: www2.example.com
Range: bytes=7433802-
If-Match: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
Referer: http://www.example.com/distribution/example.ext

Metalink clients SHOULD use preferred mirrors, if possible, because they allow early file mismatch detection, as described in Section 7.1.1 🔥 Preferred mirrors have coordinated ETags, as described in Section 3.3, and Metalink clients SHOULD use If-Match conditions based on the ETag to quickly spot out-of-date mirrors. They do this by using the ETag from the Metalink server response. Metalink clients SHOULD also use partial file cryptographic hashes, as described in Section 7.1.2 and if they're available, to check whether the mirror server returned the correct data 🫶

Optimally, the mirror server also includes an Instance Digest in its response to the client's GET request, and the client can use that to catch a mismatch early 👀 Metalink clients MUST reject individual downloads from mirrors that support Instance Digests if the Instance Digest from the mirror does not match the Instance Digest reported by the Metalink server, and the same algorithm is used. If normal mirrors are used, a mismatch can't be detected until the completed object is verified. Errors in transmission and substitutions of incorrect data on mirrors, whether deliberate or accidental, can be detected with error correction as described in Section 7.1.2.

Here, the preferred mirror server has the correct file (the If-Match conditions match) and replies with a 206 Partial Content HTTP status code plus the right "Content-Length", "Content-Range", ETag, and Instance Digest header fields. In this example, the mirror server responds with data to the request above 🌈

HTTP/1.1 206 Partial Content
Accept-Ranges: bytes
Content-Length: 7433801
Content-Range: bytes 7433802-14867602/14867603
Etag: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO
DYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ==

Now for the parallel part 🚀 Metalink clients MAY start a bunch of parallel range requests, one per selected mirror server other than the first one, using mirrors from the Link header fields with the "duplicate" relation type. Metalink clients MUST limit the number of parallel connections to mirror servers, ideally by watching how the aggregate throughput changes as connections are opened. It would be pointless to keep blindly opening connections once the path bottleneck is full 🙄 After establishing a new connection, a Metalink client SHOULD monitor whether the aggregate throughput goes up across all the connections that are part of the download. During that period, the client SHOULD NOT open additional connections. If the aggregate throughput has increased, the client MAY open an additional connection and repeat these steps. Otherwise, the client SHOULD NOT open a new connection until an established one closes. Metalink clients SHOULD use the location of the original GET request in the "Referer" header field for these range requests 💜

The Metalink client can figure out the size and number of ranges to request from each server. It bases this on the type and number of mirrors, plus the performance it has observed from each mirror 📊 Range requests put an overhead on servers, so clients need to be aware of that and not abuse them 🙅‍♀️ When downloading a particular file, Metalink clients MUST NOT make more than one concurrent request to each mirror server they download from.

Metalink clients SHOULD close all but the fastest connection if any range requests generated after the first request end up with a complete response instead of a partial one. This can happen because some mirrors might not support HTTP ranges. Closing the rest only makes sense if the goal is the fastest transfer 🏎️ Metalink clients MAY also monitor mirror conditions and dynamically switch between mirrors to get the fastest download possible. Similarly, Metalink clients SHOULD abort extremely slow or stalled range requests and finish the request on other mirrors. If all the ranges have finished except the final one, the Metalink client can split that final range into multiple range requests to other mirrors so the transfer finishes faster ⭐

If the first request was a GET, no Range header field was sent, and the client decides later that it will issue a range request, then the client SHOULD close the first connection once it catches up with the other parallel range requests for the same object. This means the first connection was sacrificed 😭 Metalink clients can use a HEAD request first, if possible. That way the client can find out whether there are any Link header fields, and then the range-based requests go to the mirror servers without sacrificing a first connection 🧠

Metalink clients MUST reject individual downloads from mirrors where the file size does not match the file size reported by the Metalink server. No exceptions, bestie 🚫

If a Metalink client doesn't support certain download methods (like FTP or BitTorrent) that a file is available from, and there are no available download methods the client does support, then the download has no way to complete 💀

Metalink clients MUST verify the cryptographic hash of the file once the download has completed. If the cryptographic hash offered by the Metalink server with Instance Digests doesn't match the cryptographic hash of the downloaded file, see Section 7.1.2 for a possible way to repair the errors 🛠️

If the download can't be repaired, it's considered corrupt. The client can try to re-download the file 🔁

Metalink clients that support verifying digital signatures MUST verify digital signatures of requested files if they're included. Digital signatures MUST validate back to a trust anchor, as described in the validation rules in [RFC3156] and [RFC5280] 🔐

## 7.1. Error Prevention, Detection, and Correction 🌟

Error prevention, or early file mismatch detection, is possible before file transfers happen, using the file sizes, ETags, and Instance Digests that Metalink servers provide 📏 Error detection needs Instance Digests so it can catch transfer errors after the transfers have completed. Error correction, or download repair, is possible with partial file cryptographic hashes 🩹

Heads up: cryptographic hashes from Instance Digests are base64 encoded, while the ones from Metalink/XML are hexadecimal 🔤

7.1.1. Error Prevention (Early File Mismatch Detection) 🕵️‍♀️

In HTTP terms, the merging of ranges from multiple responses SHOULD be verified with a strong validator. In this context, that's either an Instance Digest or a shared ETag from that Metalink server, one that matches the Instance Digest or ETag provided by a preferred mirror server 💖 In most cases it's enough for the Metalink server to provide the mirrors and Instance Digest information. But operation is more robust and efficient if the mirror servers implement a shared ETag policy or Instance Digests too 🦋 There's no need to specify how the ETag is generated. It just needs to be shared between the Metalink server and the mirror servers ✨ The benefit of mirror servers returning an Instance Digest is that the client can detect mismatches early even when ETags aren't used. Mirrors that support both a shared ETag and Instance Digests do provide value, but just one is enough for early mismatch detection 🎀 If a mirror server provides neither a shared ETag nor an Instance Digest, early mismatch detection isn't possible unless the file length also differs 🙈 Finally, errors can still be detected after the download completes, when the cryptographic hash of the merged response gets verified 🔍

ETags can't be used to verify the integrity of the received content. If the ETag from the mirror server matches the ETag from the Metalink server, then the Metalink client assumes the responses are valid for that object 🤷‍♀️

Here's the guarantee: a mismatch will be detected using only the shared ETag from a Metalink server and a mirror server. Metalink clients will detect an error if the ETags don't match, which stops accidental merges of ranges from different versions of files that share the same name 🛑

A shared ETag or Instance Digest can't strictly protect against malicious attacks, or against server or network errors that replace content. An attacker can make a mirror server seem to respond with the expected Instance Digest or ETags even when the file contents have been modified. The same goes for various system failures, which could also return bad data (aka corrupted files) 😱 So the Metalink client has to rely on the Instance Digest returned by the Metalink server in the first response to verify the downloaded object as a whole. To verify the individual ranges, which might have come from different sources, see Section 7.1.2 🔐

7.1.2. Error Correction ✨

Okay bestie, partial file hashes are how you catch errors during the download 🔍 Metalink servers SHOULD provide Metalink/XML files with partial file hashes in Link header fields as specified in Section 4.1, and Metalink clients SHOULD use them for error correction 💖

An error in transfer or a substitution attack gets caught when the hash of the object doesn't match the Instance Digest from the Metalink server 🚨 If that happens, the client SHOULD fetch the Metalink/XML (if available). That file may have partial file hashes, which lets you figure out which mirror server sent the bad data 🕵️‍♀️ Metalink clients SHOULD use the Metalink/XML data to work out which ranges of the downloaded data can be recovered and which ones need to be fetched again 🔁

Other methods can be used for error correction too. For example, some other metainfo files also include partial file hashes that can be used to check for errors 🌸

8. IANA Considerations 🎀

Accordingly, IANA has made the following registration to the "Link Relation Types" registry at http://www.iana.org/. 📝

  • Relation Name: duplicate
  • Description: Refers to a resource whose available representations are byte-for-byte identical with the corresponding representations of the context IRI.
  • Reference: This specification.
  • Notes: This relation is for static resources. That is, an HTTP GET request on any duplicate will return the same representation. It does not make sense for dynamic or POSTable resources and should not be used for them.

9. Security Considerations 🔐

Okay, this section is serious, so let's go 🫶

9.1. URIs and IRIs

Metalink clients handle URIs and Internationalized Resource Identifiers (IRIs). See Section 7 of [RFC3986] and Section 8 of [RFC3987] for security considerations related to their handling and use 🔗

9.2. Spoofing

There is potential for spoofing attacks where the attacker publishes Metalinks with false information. 💀

In that case, this could deceive unaware downloaders into downloading a malicious or worthless file. Metalink clients are advised to prevent loops, possibly from a mirror server to a Metalink server and back again, in Section 2. As with all downloads, users should only download from trusted sources 🚫 Also, malicious publishers could attempt a distributed denial-of-service attack by inserting unrelated URIs into Metalinks. [RFC4732] contains information on amplification attacks and denial-of-service attacks 😭

9.3. Cryptographic Hashes

Currently, some of the digest values defined in Instance Digests in HTTP [RFC3230] are considered insecure. These include the whole Message Digest family of algorithms, which are not suitable for cryptographically strong verification. Malicious people could provide files that appear to be identical to another file because of a collision; i.e., the weak cryptographic hashes of the intended file and a substituted malicious file could match 🙈

9.4. Signing

Metalinks SHOULD include digital signatures, as described in Section 5 ✍️

Digital signatures provide authentication and message integrity, and enable non-repudiation with proof of origin 👑

10. References 📚

10.1. Normative References

[BITTORRENT]  Cohen, B., "The BitTorrent Protocol Specification",
              BITTORRENT 11031, February 2008,
              <http://www.bittorrent.org/beps/bep_0003.html>.

[FIPS-180-3]  National Institute of Standards and Technology (NIST),
              "Secure Hash Standard (SHS)", FIPS PUB 180-3,
              October 2008.

[ISO3166-1]   International Organization for Standardization, "ISO
              3166-1:2006.  Codes for the representation of names of
              countries and their subdivisions -- Part 1: Country
              codes", November 2006.

[RFC0959]     Postel, J. and J. Reynolds, "File Transfer Protocol",
              STD 9, RFC 0959, October 1985.

[RFC2119]     Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC2616]     Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
              Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext
              Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.

[RFC3156]     Elkins, M., Del Torto, D., Levien, R., and T. Roessler,
              "MIME Security with OpenPGP", RFC 3156, August 2001.

[RFC3230]     Mogul, J. and A. Van Hoff, "Instance Digests in HTTP",
              RFC 3230, January 2002.

[RFC3986]     Berners-Lee, T., Fielding, R., and L. Masinter,
              "Uniform Resource Identifier (URI): Generic Syntax",
              STD 66, RFC 3986, January 2005.

[RFC3987]     Duerst, M. and M. Suignard, "Internationalized Resource
              Identifiers (IRIs)", RFC 3987, January 2005.

[RFC5280]     Cooper, D., Santesson, S., Farrell, S., Boeyen, S.,
              Housley, R., and W. Polk, "Internet X.509 Public Key
              Infrastructure Certificate and Certificate Revocation
              List (CRL) Profile", RFC 5280, May 2008.

[RFC5751]     Ramsdell, B. and S. Turner, "Secure/Multipurpose
              Internet Mail Extensions (S/MIME) Version 3.2 Message
              Specification", RFC 5751, January 2010.

[RFC5854]     Bryan, A., Tsujikawa, T., McNab, N., and P. Poeml, "The
              Metalink Download Description Format", RFC 5854,
              June 2010.

[RFC5988]     Nottingham, M., "Web Linking", RFC 5988, October 2010.

10.2. Informative References

[RFC4732]     Handley, M., Ed., Rescorla, E., Ed., and IAB, "Internet
              Denial-of-Service Considerations", RFC 4732,
              December 2006.

[RFC5843]     Bryan, A., "Additional Hash Algorithms for HTTP
              Instance Digests", RFC 5843, April 2010.

Appendix A. Acknowledgements and Contributors 💜

Thanks to the Metalink community, Alexey Melnikov, Julian Reschke, Mark Nottingham, Daniel Stenberg, Matt Domsch, Micah Cowan, David Morris, Yves Lafon, Juergen Schoenwaelder, Ben Campbell, Lars Eggert, Sean Turner, Robert Sparks, and the HTTPBIS Working Group 🫶

Thanks to Alan Ford and Mark Handley for spurring us on to publish this document ✨

This document is dedicated to Zimmy Bryan, Juanita Anthony, and Janie Burnett 🌈

Authors' Addresses 🦋

Anthony Bryan
Pompano Beach, FL
USA

EMail: anthonybryan@gmail.com
URI:   http://www.metalinker.org


Neil McNab

EMail: neil@nabber.org
URI:   http://www.nabber.org


Tatsuhiro Tsujikawa
Shiga
Japan

EMail: tatsuhiro.t@gmail.com
URI:   http://aria2.sourceforge.net


Dr. med. Peter Poeml
MirrorBrain
Venloer Str. 317
Koeln  50823
DE

Phone: +49 221 6778 333 8
EMail: peter@poeml.de
URI:   http://mirrorbrain.org/~poeml/


Henrik Nordstrom

EMail: henrik@henriknordstrom.net
URI:   http://www.henriknordstrom.net/
📦 where this page comes from