uri(3tcl)

📦
version 2.0 · 2 versions
, in debian trixie, ubuntu questing, ubuntu resolute, ubuntu stonking

🦋 uri — URI utilities 🦋

(this is the manual page for uri(3tcl), aka Tcl Uniform Resource Identifier Management — yes bestie, we're doing URIs today and it's gonna be iconic)

💖 NAME 💖

uri — URI utilities. That's it, that's the whole name. Slay.

✨ SYNOPSIS ✨

package require Tcl 8.5 9

package require uri ?1.2.8?

uri::setQuirkOption option ?value?

uri::split url ?defaultscheme?

uri::join ?key value?...

uri::resolve base url

uri::isrelative url

uri::geturl url ?options...?

uri::canonicalize uri

uri::register schemeList script

🌸 DESCRIPTION 🌸

Okay so this package does two things. Two. Like, a whole duality moment. Let's get into it.

One: it hands you a bunch of commands for manipulating URLs/URIs and for fetching the data they point at. When it comes to fetching, this package peeks at the URL/URI you asked for, figures out what it is, and then dispatches it to the right package (http, ftp, …) to do the actual retrieval. Currently those commands are defined for the schemes http, https, ftp, mailto, news, ldap, ldaps and file. And the package uri::urn adds the scheme urn.

Two: it gives you regular expressions for a whole lineup of registered URL/URI schemes. The registered schemes right now are ftp, ldap, ldaps, file, http, https, gopher, mailto, news, wais and prospero. And again — the package uri::urn adds the scheme urn. 💅

The commands of this package conform to RFC 3986 (https://www.rfc-editor.org/rfc/rfc3986.txt) — with one exception, a loophole that comes out of RFC 1630 and is described in RFC 3986 Sections 5.2.2 and 5.4.2. The loophole lets a relative URI include a scheme if it's the same scheme as the base URI it's being resolved against. And like, RFC 3986 itself recommends avoiding this usage. So, noted, bestie. 👀

🎀 COMMANDS 🎀

  • uri::setQuirkOption option ?value? — this one is an accessor command for a bunch of "quirk options". It has the exact same semantics as the command set: call it with one argument and it reads an existing value; call it with two arguments and it writes a new value. The value of a "quirk option" is boolean — false asks for conformance with RFC 3986, while true asks for the quirk. Head to the section QUIRK OPTIONS for the discussion of the different options and what they're for.
  • uri::split url ?defaultscheme? — takes a url, decodes it, and then returns a list of key/value pairs suitable for array set, containing all the constituents of the url. If the scheme is missing from the url, it defaults to the value of defaultscheme if that was specified — or http otherwise. Right now the schemes http, https, ftp, mailto, news, ldap, ldaps and file are supported by the package itself. Wanna expand that range? Section EXTENDING has you covered.

The set of constituents of a URL (= the set of keys in the returned dictionary) depends on the scheme of the URL. Which means the only key that's always present is scheme. For the following schemes the constituents and their keys are known:

  • ftp — user, pwd, host, port, path, type, pbare. The pbare is optional.
  • http(s) — user, pwd, host, port, path, query, fragment, pbare. The pbare is optional.
  • file — path, host. The host is optional.
  • mailto — user, host. The host is optional.
  • ldap(s) — host, port, dn, attrs, scope, filter, extensions
  • news — either message-id or newsgroup-name.

For discussion of the boolean pbare, see the options NoInitialSlash and NoExtraKeys over in QUIRK OPTIONS.

Heads up: the constituents come back as slices of the argument url, with no removal of percent-encoding ("url-encoding") and no other adaptations. Notably, on Windows® the path in scheme file is not a valid local filename. Go look at EXAMPLES for more on that. 😭
  • uri::join ?key value?... — takes a list of key/value pairs (the kind uri::split generates, for example) and returns the canonical URL they represent. Currently the schemes http, https, ftp, mailto, news, ldap, ldaps and file are supported by the package itself. Again: see section EXTENDING for how to widen that range.

The arguments are expected to be slices of a valid URL, with percent-encoding ("url-encoding") and any other necessary adaptations already in place. And notably, on Windows the path in scheme file is not a valid local filename. EXAMPLES has the receipts.

  • uri::resolve base url — resolves the specified url relative to base, in conformance with RFC 3986. In other words: a non-relative url comes back unchanged, whereas for a relative url the missing parts get taken from base and prepended to it. The result of this operation is what gets returned. For an empty url, the result is base without its URI fragment (if it had one). This command is available for the schemes http, https, ftp, and file.
  • uri::isrelative url — figures out whether the specified url is absolute or relative. This command is available for a url of any scheme. Versatile queen. 👑
  • uri::geturl url ?options...? — decodes the specified url and then dispatches the request to the package appropriate for the scheme found in the URL. It assumes the package handling that scheme either has the same name as the scheme itself (including possible capitalization) followed by ::geturl, or — if that fails — just has the same name as the scheme itself (including possible capitalization). It further assumes that whatever package got loaded provides a geturl-command in the namespace of the same name as the package itself. That command gets called with the given url and all the given options. Currently geturl does not handle any options itself.
Note: file-URLs are an exception to the rule above. They get handled internally. 🙈

It is not possible to specify the results of this command. Those depend entirely on the geturl-command for whichever scheme the request got dispatched to.

  • uri::canonicalize uri — returns the canonical form of a URI. The canonical form is the one where relative path specifications, i.e. "." and "..", have been resolved. This command is available for all URI schemes that have uri::split and uri::join commands. It returns a canonicalized URI if the URI scheme has a path component (i.e. http, https, ftp, and file). For schemes that do have uri::split and uri::join commands but have no path component (i.e. mailto, news, ldap, and ldaps), the command returns the uri unchanged. Chill behavior, honestly.
  • uri::register schemeList script — registers the first element of schemeList as a new scheme, and the remaining elements as aliases for that scheme. It creates the namespace for the scheme and executes the script inside that new namespace. The script has to declare variables containing regular expressions relevant to the scheme. At minimum, the variable schemepart has to be declared, because that one is used to extend the variables that keep track of all the registered schemes. Don't skip that part bestie, it's load-bearing. 💀

🌈 SCHEMES 🌈

On top of the commands above, this package provides regular expressions for recognizing URLs across a bunch of URL schemes.

For each supported scheme there's a namespace with the same name as the scheme itself, sitting inside the namespace uri, and it contains the variable url — whose contents are a regular expression that recognizes URLs of that scheme. Additional variables may hold regular expressions for parts of URLs for that scheme. Very organized. Very demure.

The variable uri::schemes contains a list of all the registered schemes. Currently those are ftp, ldap, ldaps, file, http, https, gopher, mailto, news, wais and prospero. ✨

🧚 EXTENDING 🧚

Widening the range of schemes that uri::split and uri::join support is easy, because neither command actually handles the request itself — they dispatch it to another command in the uri namespace, using the scheme of the URL as the criterion. Delegation! Main character energy!

Specifically, uri::split and uri::join call Split[string totitle <scheme>] and Join[string totitle <scheme>] respectively.

Providing those split and join commands is enough to extend the commands uri::canonicalize and uri::geturl (the latter being subject to the availability of a suitable package with a geturl command). But in contrast — to extend the command uri::resolve to a new scheme, the command itself has to be modified. No shortcuts there, sorry.

And to extend the range of schemes for which pattern information is available, use the command uri::register.

An example of a package that provides both commands and pattern information for a new scheme is uri::urn, which adds the scheme urn. 🫶

✨ QUIRK OPTIONS ✨

okay bestie, a "quirk option" is giving boolean: value false requests conformance with RFC 3986, while true requests use of the quirk. Use command uri::setQuirkOption to access the values of quirk options. 💅

Quirk options are useful both for allowing backwards compatibility when a command specification changes, and for adding useful features that are not included in RFC specifications. The following quirk options are currently defined:

  • NoInitialSlash — This quirk option concerns the leading character of path (if non-empty) in the schemes http, https, and ftp.

RFC 3986 defines path in an absolute URI to have an initial /, unless the value of path is the empty string. For the scheme file, all versions of package uri follow this rule. The quirk option NoInitialSlash does not apply to scheme file.

For the schemes http, https, and ftp, versions of uri before 1.2.7 define the path NOT to include an initial /. When the quirk option NoInitialSlash is true (the default), this behavior is also used in version 1.2.7. To use instead values of path as defined by RFC 3986, set this quirk option to false.

This setting does not affect RFC 3986 conformance. If NoInitialSlash is true, then the value of path in the schemes http, https, or ftp, cannot distinguish between URIs in which the full "RFC 3986 path" is the empty string "" or a single slash "/" respectively. The missing information is recorded in an additional uri::split key pbare.

The boolean pbare is defined when quirk options NoInitialSlash and NoExtraKeys have values true and false respectively. In this case, if the value of path is the empty string "", pbare is true if the full "RFC 3986 path" is "", and pbare is false if the full "RFC 3986 path" is "/".

Using this quirk option NoInitialSlash is a matter of preference.

  • NoExtraKeys — This quirk option permits full backward compatibility with versions of uri before 1.2.7, by omitting the uri::split key pbare described above (see quirk option NoInitialSlash). The outcome is greater backward compatibility of the uri::split command, but an inability to distinguish between URIs in which the full "RFC 3986 path" is the empty string "" or a single slash "/" respectively — i.e. a minor non-conformance with RFC 3986.

If the quirk option NoExtraKeys is false (the default), command uri::split returns an additional key pbare, and the commands comply with RFC 3986. If the quirk option NoExtraKeys is true, the key pbare is not defined and there is not full conformance with RFC 3986.

Using the quirk option NoExtraKeys is NOT recommended, because if set to true it will reduce conformance with RFC 3986. The option is included only for compatibility with code, written for earlier versions of uri, that needs values of path without a leading /, AND ALSO cannot tolerate unexpected keys in the results of uri::split.
  • HostAsDriveLetter — When handling the scheme file on the Windows platform, versions of uri before 1.2.7 use the host field to represent a Windows drive letter and the colon that follows it, and the path field to represent the filename path after the colon. Such URIs are invalid, and are not recognized by any RFC. When the quirk option HostAsDriveLetter is true, this behavior is also used in version 1.2.7. To use file URIs on Windows that conform to RFC 3986, set this quirk option to false (the default).
Using this quirk is NOT recommended, because if set to true it will cause the uri commands to expect and produce invalid URIs. The option is included only for compatibility with legacy code.
  • RemoveDoubleSlashes — When a URI is canonicalized by uri::canonicalize, its path is normalized by removal of segments "." and "..". RFC 3986 does not mandate the removal of empty segments "" (i.e. the merger of double slashes, which is a feature of filename normalization but not of URI path normalization): it treats URIs with excess slashes as referring to different resources. When the quirk option RemoveDoubleSlashes is true (the default), empty segments will be removed from path. To prevent removal, and thereby conform to RFC 3986, set this quirk option to false.

Using this quirk is a matter of preference. A URI with double slashes in its path was most likely generated by error, certainly so if it has a straightforward mapping to a file on a server. In some cases it may be better to sanitize the URI; in others, to keep the URI and let the server handle the possible error.

🫶 BACKWARD COMPATIBILITY 🫶

To behave as similarly as possible to versions of uri earlier than 1.2.7, set the following quirk options:

  • uri::setQuirkOption NoInitialSlash 1
  • uri::setQuirkOption NoExtraKeys 1
  • uri::setQuirkOption HostAsDriveLetter 1
  • uri::setQuirkOption RemoveDoubleSlashes 0

In code that can tolerate the return by uri::split of an additional key pbare, set

  • uri::setQuirkOption NoExtraKeys 0

in order to achieve greater compliance with RFC 3986.

🌈 NEW DESIGNS 🌈

For new projects, the following settings are recommended:

  • uri::setQuirkOption NoInitialSlash 0
  • uri::setQuirkOption NoExtraKeys 0
  • uri::setQuirkOption HostAsDriveLetter 0
  • uri::setQuirkOption RemoveDoubleSlashes 0|1

👑 DEFAULT VALUES 👑

The default values for package uri version 1.2.7 are intended to be a compromise between backwards compatibility and improved features. Different default values may be chosen in future versions of package uri.

  • uri::setQuirkOption NoInitialSlash 1
  • uri::setQuirkOption NoExtraKeys 0
  • uri::setQuirkOption HostAsDriveLetter 0
  • uri::setQuirkOption RemoveDoubleSlashes 1

🍓 EXAMPLES 🍓

A Windows® local filename such as "C:\Other Files\startup.txt" is not suitable for use as the path element of a URI in the scheme file.

The Tcl command file normalize will convert the backslashes to forward slashes. To generate a valid path for the scheme file, the normalized filename must be prepended with /, and then any characters that do not match the regexp bracket expression

[a-zA-Z0-9$_.+!*'(,)?:@&=-]

must be percent-encoded.

The result in this example is "/C:/Other%20Files/startup.txt" which is a valid value for path.

% uri::join path /C:/Other%20Files/startup.txt scheme file

file:///C:/Other%20Files/startup.txt

% uri::split file:///C:/Other%20Files/startup.txt

path /C:/Other%20Files/startup.txt scheme file

On UNIX® systems filenames begin with / which is also used as the directory separator. The only action needed to convert a filename to a valid path is percent-encoding.

💖 CREDITS 💖

Original code (regular expressions) by Andreas Kupries. Modularisation by Steve Ball, also the split/join/resolve functionality. RFC 3986 conformance by Keith Nash.

😭 BUGS, IDEAS, FEEDBACK 😭

This document, and the package it describes, will undoubtedly contain bugs and other problems. Please report such in the category uri of the Tcllib Trackers [http://core.tcl.tk/tcllib/reportlist]. Please also report any ideas for enhancements you may have for either package and/or documentation.

When proposing code changes, please provide unified diffs, i.e the output of diff -u.

Note further that attachments are strongly preferred over inlined patches. Attachments can be made by going to the Edit form of the ticket immediately after its creation, and then using the left-most button in the secondary navigation bar.

⭐ KEYWORDS ⭐

fetching information, file, ftp, gopher, http, https, ldap, mailto, news, prospero, rfc 1630, rfc 2255, rfc 2396, rfc 3986, uri, url, wais, www

💅 CATEGORY 💅

Networking

and that's the whole uri(3tcl) page, bestie! 🌸 This iconic little manual came from tcllib 1.2.8, uri(3tcl) — the footer didn't hand us a date, but the source/version is right there. slay, hydrate, and set your quirk options responsibly. 💖✨

📦 where this page comes from