uri_string(3erl)

💅 uri_string — URI processing functions ✨

uri_string(3erl) — Erlang Module Definition — okay so this is the page, bestie, and we're gonna read it like a group chat. 🫶

🌸 NAME 🌸

uri_string — URI processing functions. That's the whole name section, and honestly it ate. 💖

🦋 DESCRIPTION 🦋

This module is literally packed with functions for parsing and handling URIs (RFC 3986) and form-urlencoded query strings (HTML 5.2). Parsing and serializing non-UTF-8 form-urlencoded query strings is also supported, btw (HTML 5.0). 👑

A URI is an identifier consisting of a sequence of characters matching the syntax rule named URI in RFC 3986.

The generic URI syntax is a hierarchical sequence of components referred to as the scheme, authority, path, query, and fragment:

URI         = scheme ":" hier-part [ "?" query ] [ "#" fragment ]
hier-part   = "//" authority path-abempty
               / path-absolute
               / path-rootless
               / path-empty
scheme      = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )
authority   = [ userinfo "@" ] host [ ":" port ]
userinfo    = *( unreserved / pct-encoded / sub-delims / ":" )

reserved    = gen-delims / sub-delims
gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"
sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
            / "*" / "+" / "," / ";" / "="

unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~"

The way a URI is interpreted depends only on the characters used — not on how those characters are represented in a network protocol. That's the vibe check. ✨

The functions implemented by this module cover these use cases:

• Parsing URIs into their components and returning a map — parse/1
• Recomposing a map of URI components into a URI string — recompose/1
• Changing inbound binary and percent-encoding of URIs — transcode/2
• Transforming URIs into a normalized form — normalize/1, normalize/2
• Composing form-urlencoded query strings from a list of key-value pairs — compose_query/1, compose_query/2
• Dissecting form-urlencoded query strings into a list of key-value pairs — dissect_query/1
• Decoding percent-encoded triplets in a URI map or a specific component of a URI — percent_decode/1
• Preparing and retrieving application specific data included in URI components — quote/1, quote/2, unquote/1

There are four different encodings hanging out during the handling of URIs:

• Inbound binary encoding in binaries
• Inbound percent-encoding in lists and binaries
• Outbound binary encoding in binaries
• Outbound percent-encoding in lists and binaries

Functions with a uri_string() argument accept lists, binaries, and mixed lists (lists with binary elements) as input type. All of the functions except transcode/2 expect input as lists of unicode codepoints, UTF-8 encoded binaries, and UTF-8 percent-encoded URI parts ("%C3%B6" corresponds to the unicode character "ö"). 💅

Unless otherwise specified, the return value type and encoding are the same as the input type and encoding. So: binary input gives binary output, list input gives list output, but mixed input gives list output. Slay.

In case of lists there is only percent-encoding. In binaries, however, both binary encoding and percent-encoding shall be considered. transcode/2 is the function that converts between the supported encodings — it takes a uri_string() and a list of options specifying inbound and outbound encodings. 🔥

RFC 3986 does not mandate any specific character encoding, and it is usually defined by the protocol or surrounding text. This library takes the same assumption: binary and percent-encoding are handled as one configuration unit, and they cannot be set to different values. Periodt.

Quoting functions are intended to be used by a URI producing application during the component preparation or retrieval phase to avoid conflicts between data and characters used in URI syntax. Quoting functions use percent encoding, but with different rules than, for example, during execution of recompose/1. It is your responsibility, bestie, to provide quoting functions with application data only and use their output to combine a URI component. For example, quoting functions can be used for constructing a path component with a segment containing '/' character which should not collide with '/' used as a general delimiter in the path component. 🎀

💖 DATA TYPES 💖

error() = {error, atom(), term()}

Error tuple indicating the type of error. Possible values of the second component are:

• invalid_character
• invalid_encoding
• invalid_input
• invalid_map
• invalid_percent_encoding
• invalid_scheme
• invalid_uri
• invalid_utf8
• missing_value

The third component is a term providing additional information about the cause of the error. 👀

uri_map() =

#{fragment => unicode:chardata(),
  host => unicode:chardata(),
  path => unicode:chardata(),
  port => integer() >= 0 | undefined,
  query => unicode:chardata(),
  scheme => unicode:chardata(),
  userinfo => unicode:chardata()}

Map holding the main components of a URI. 💖

uri_string() = iodata()

List of unicode codepoints, a UTF-8 encoded binary, or a mix of the two, representing an RFC 3986 compliant URI (percent-encoded form). A URI is a sequence of characters from a very limited set: the letters of the basic Latin alphabet, digits, and a few special characters. 🦋

🔥 EXPORTS 🔥

allowed_characters() -> [{atom(), list()}]

This is a utility function meant to be used in the shell for printing the allowed characters in each major URI component, and also in the most important character sets. Please note that this function does not replace the ABNF rules defined by the standards — these character sets are derived directly from those aforementioned rules. For more information, see the Uniform Resource Identifiers chapter in stdlib's Users Guide. 🌈

compose_query(QueryList) -> QueryString

Types:

QueryList = [{unicode:chardata(), unicode:chardata() | true}]
QueryString = uri_string() | error()

Composes a form-urlencoded QueryString based on a QueryList, a list of non-percent-encoded key-value pairs. Form-urlencoding is defined in section 4.10.21.6 of the HTML 5.2 specification and in section 4.10.22.6 of the HTML 5.0 specification for non-UTF-8 encodings. 👑

See also the opposite operation dissect_query/1.

Example:

1> uri_string:compose_query([{"foo bar","1"},{"city","örebro"}]).
"foo+bar=1&city=%C3%B6rebro"
2> uri_string:compose_query([{<<"foo bar">>,<<"1">>},
2> {<<"city">>,<<"örebro"/utf8>>}]).
<<"foo+bar=1&city=%C3%B6rebro">>

compose_query(QueryList, Options) -> QueryString

Types:

QueryList = [{unicode:chardata(), unicode:chardata() | true}]
Options = [{encoding, atom()}]
QueryString = uri_string() | error()

Same as compose_query/1, but with an additional Options parameter that controls the encoding ("charset") used by the encoding algorithm. There are two supported encodings: utf8 (or unicode) and latin1. 💅

Each character in the entry's name and value that cannot be expressed using the selected character encoding is replaced by a string consisting of a U+0026 AMPERSAND character (&), a "#" (U+0023) character, one or more ASCII digits representing the Unicode code point of the character in base ten, and finally a ";" (U+003B) character. 😭

Bytes that are out of the range 0x2A, 0x2D, 0x2E, 0x30 to 0x39, 0x41 to 0x5A, 0x5F, 0x61 to 0x7A, are percent-encoded (U+0025 PERCENT SIGN character (%) followed by uppercase ASCII hex digits representing the hexadecimal value of the byte). 🍓

See also the opposite operation dissect_query/1.

Example:

1> uri_string:compose_query([{"foo bar","1"},{"city","örebro"}],
1> [{encoding, latin1}]).
"foo+bar=1&city=%F6rebro"
2> uri_string:compose_query([{<<"foo bar">>,<<"1">>},
2> {<<"city">>,<<"東京"/utf8>>}], [{encoding, latin1}]).
<<"foo+bar=1&city=%26%2326481%3B%26%2320140%3B">>

dissect_query(QueryString) -> QueryList

Types:

QueryString = uri_string()
QueryList =
    [{unicode:chardata(), unicode:chardata() | true}] |
error()

Dissects an urlencoded QueryString and returns a QueryList, a list of non-percent-encoded key-value pairs. Form-urlencoding is defined in section 4.10.21.6 of the HTML 5.2 specification and in section 4.10.22.6 of the HTML 5.0 specification for non-UTF-8 encodings. 🫶

See also the opposite operation compose_query/1.

Example:

1> uri_string:dissect_query("foo+bar=1&city=%C3%B6rebro").
[{"foo bar","1"},{"city","örebro"}]
2> uri_string:dissect_query(<<"foo+bar=1&city=%26%2326481%3B%26%2320140%3B">>).
[{<<"foo bar">>,<<"1">>},
 {<<"city">>,<<230,157,177,228,186,172>>}]

normalize(URI) -> NormalizedURI

Types:

URI = uri_string() | uri_map()
NormalizedURI = uri_string() | error()

Transforms a URI into a normalized form using Syntax-Based Normalization as defined by RFC 3986. 💖

This function implements case normalization, percent-encoding normalization, path segment normalization, and scheme based normalization for HTTP(S) with basic support for FTP, SSH, SFTP, and TFTP. 🌸

Example:

1> uri_string:normalize("/a/b/c/./../../g").
"/a/g"
2> uri_string:normalize(<<"mid/content=5/../6">>).
<<"mid/6">>
3> uri_string:normalize("http://localhost:80").
"http://localhost/"
4> uri_string:normalize(#{scheme => "http",port => 80,path => "/a/b/c/./../../g",
4> host => "localhost-örebro"}).
"http://localhost-%C3%B6rebro/a/g"

normalize(URI, Options) -> NormalizedURI

Types:

URI = uri_string() | uri_map()
Options = [return_map]
NormalizedURI = uri_string() | uri_map() | error()

Same as normalize/1, but with an additional Options parameter that controls whether the normalized URI shall be returned as an uri_map(). There is one supported option: return_map. 👑

Example:

okay so we're literally mid-normalize/2 example, here's the rest of the receipts:

1> uri_string:normalize("/a/b/c/./../../g", [return_map]).
#{path => "/a/g"}
2> uri_string:normalize(<<"mid/content=5/../6">>, [return_map]).
#{path => <<"mid/6">>}
3> uri_string:normalize("http://localhost:80", [return_map]).
#{scheme => "http",path => "/",host => "localhost"}
4> uri_string:normalize(#{scheme => "http",port => 80,path => "/a/b/c/./../../g",
4> host => "localhost-örebro"}, [return_map]).
#{scheme => "http",path => "/a/g",host => "localhost-örebro"}

---

parse(URIString) -> URIMap

Types:

• URIString = uri_string()
• URIMap = uri_map() | error()

Parses an RFC 3986 compliant uri_string() into a uri_map() that holds the parsed components of the URI. If parsing fails, you get an error tuple back, bestie.

Also see the opposite operation recompose/1.

Example era:

1> uri_string:parse("foo://user@example.com:8042/over/there?name=ferret#nose").
#{fragment => "nose",host => "example.com",
  path => "/over/there",port => 8042,query => "name=ferret",
  scheme => foo,userinfo => "user"}
2> uri_string:parse(<<"foo://user@example.com:8042/over/there?name=ferret">>).
#{host => <<"example.com">>,path => <<"/over/there">>,
  port => 8042,query => <<"name=ferret">>,scheme => <<"foo">>,
  userinfo => <<"user">>}

---

percent_decode(URI) -> Result

Types:

• URI = uri_string() | uri_map()
• Result = uri_string() | uri_map() | {error, {invalid, {atom(), {term(), term()}}}}

Decodes all percent-encoded triplets in the input that can be both a uri_string() and a uri_map(). Note that this function performs raw decoding and it shall be used on already parsed URI components. Applying this function directly on a standard URI can effectively change it — no cap.

If the input encoding is not UTF-8, an error tuple is returned.

Example era:

1> uri_string:percent_decode(#{host => "localhost-%C3%B6rebro",path => [],
1> scheme => "http"}).
#{host => "localhost-örebro",path => [],scheme => "http"}
2> uri_string:percent_decode(<<"%C3%B6rebro">>).
<<"örebro"/utf8>>

⚠️ Warning: Using uri_string:percent_decode/1 directly on a URI is not safe. This example shows that after each consecutive application of the function the resulting URI will be changed. None of these URIs refer to the same resource.

3> uri_string:percent_decode(<<"http://local%252Fhost/path">>).
<<"http://local%2Fhost/path">>
4> uri_string:percent_decode(<<"http://local%2Fhost/path">>).
<<"http://local/host/path">>

---

quote(Data) -> QuotedData

Types:

• Data = QuotedData = unicode:chardata()

Replaces characters out of unreserved set with their percent encoded equivalents.

Unreserved characters defined in RFC 3986 are not quoted.

Example era:

1> uri_string:quote("SomeId/04").
"SomeId%2F04"
2> uri_string:quote(<<"SomeId/04">>).
<<"SomeId%2F04">>

⚠️ Warning: Function is not aware about any URI component context and should not be used on whole URI. If applied more than once on the same data, might produce unexpected results.

---

quote(Data, Safe) -> QuotedData

Types:

• Data = unicode:chardata()
• Safe = string()
• QuotedData = unicode:chardata()

Same as quote/1, but Safe allows the user to provide a list of characters to be protected from encoding.

Example era:

1> uri_string:quote("SomeId/04", "/").
"SomeId/04"
2> uri_string:quote(<<"SomeId/04">>, "/").
<<"SomeId/04">>

⚠️ Warning: Function is not aware about any URI component context and should not be used on whole URI. If applied more than once on the same data, might produce unexpected results.

---

recompose(URIMap) -> URIString

Types:

• URIMap = uri_map()
• URIString = uri_string() | error()

Creates an RFC 3986 compliant URIString (percent-encoded), based on the components of URIMap. If the URIMap is invalid, an error tuple is returned.

Also see the opposite operation parse/1.

Example era:

1> URIMap = #{fragment => "nose", host => "example.com", path => "/over/there",
1> port => 8042, query => "name=ferret", scheme => "foo", userinfo => "user"}.
#{fragment => "nose",host => "example.com",
  path => "/over/there",port => 8042,query => "name=ferret",
  scheme => "foo",userinfo => "user"}

2> uri_string:recompose(URIMap).
"foo://example.com:8042/over/there?name=ferret#nose"

---

resolve(RefURI, BaseURI) -> TargetURI

Types:

• RefURI = BaseURI = uri_string() | uri_map()
• TargetURI = uri_string() | error()

Convert a RefURI reference that might be relative to a given base URI into the parsed components of the reference's target, which can then be recomposed to form the target URI.

Example era:

1> uri_string:resolve("/abs/ol/ute", "http://localhost/a/b/c?q").
"http://localhost/abs/ol/ute"
2> uri_string:resolve("../relative", "http://localhost/a/b/c?q").
"http://localhost/a/relative"
3> uri_string:resolve("http://localhost/full", "http://localhost/a/b/c?q").
"http://localhost/full"
4> uri_string:resolve(#{path => "path", query => "xyz"}, "http://localhost/a/b/c?q").
"http://localhost/a/b/path?xyz"

---

resolve(RefURI, BaseURI, Options) -> TargetURI

Types:

• RefURI = BaseURI = uri_string() | uri_map()
• Options = [return_map]
• TargetURI = uri_string() | uri_map() | error()

Same as resolve/2 but with an additional Options parameter that controls whether the target URI shall be returned as an uri_map(). There is one supported option: return_map.

Example era:

1> uri_string:resolve("/abs/ol/ute", "http://localhost/a/b/c?q", [return_map]).
#{host => "localhost",path => "/abs/ol/ute",scheme => "http"}
2> uri_string:resolve(#{path => "/abs/ol/ute"}, #{scheme => "http",
2> host => "localhost", path => "/a/b/c?q"}, [return_map]).
#{host => "localhost",path => "/abs/ol/ute",scheme => "http"}

---

transcode(URIString, Options) -> Result

Types:

URIString = uri_string()
Options =
    [{in_encoding, unicode:encoding()} |
     {out_encoding, unicode:encoding()}]
Result = uri_string() | error()

Transcodes an RFC 3986 compliant URIString, where Options is a list of tagged tuples, specifying the inbound (in_encoding) and outbound (out_encoding) encodings. in_encoding and out_encoding specifies both binary encoding and percent-encoding for the input and output data. Mixed encoding, where binary encoding is not the same as percent-encoding, is not supported. If an argument is invalid, an error tuple is returned.

Example era:

1> uri_string:transcode(<<"foo%00%00%00%F6bar"/utf32>>,
1> [{in_encoding, utf32},{out_encoding, utf8}]).
<<"foo%C3%B6bar"/utf8>>
2> uri_string:transcode("foo%F6bar", [{in_encoding, latin1},
2> {out_encoding, utf8}]).
"foo%C3%B6bar"

---

unquote(QuotedData) -> Data

Types:

• QuotedData = Data = unicode:chardata()

Percent decode characters.

Example era:

1> uri_string:unquote("SomeId%2F04").
"SomeId/04"
2> uri_string:unquote(<<"SomeId%2F04">>).
<<"SomeId/04">>

⚠️ Warning: Function is not aware about any URI component context and should not be used on whole URI. If applied more than once on the same data, might produce unexpected results.

---

footer tea: source/version is Ericsson AB, stdlib 4.3.1.3, on the uri_string(3erl) page — and the date didn't show up in this footer, bestie, but periodt anyway 🎀💅✨