VMOD_STD(3)

✨ VMOD_STD ✨

okay bestie, welcome to vmod_std — the Varnish Standard Module, aka the manual page that lives in VMOD_STD(3) in your Library Functions Manual and is genuinely about to become your new hyperfixation. 🎀

---

NAME

vmod_std — Varnish Standard Module 💖

---

SYNOPSIS

no cap, here's the whole roster, copied letter-for-letter because I would never fumble the signature of my girls:

import std [as name] [from "path"]

REAL random(REAL lo, REAL hi)

REAL round(REAL r)

VOID collect(HEADER hdr, STRING sep)

STRING querysort(STRING)

STRING toupper(STRING s)

STRING tolower(STRING s)

STRING strstr(STRING s1, STRING s2)

BOOL fnmatch(STRING pattern, STRING subject, BOOL pathname, BOOL noescape, BOOL period)

STRING fileread(STRING)

BLOB blobread(STRING)

BOOL file_exists(STRING path)

BOOL healthy(BACKEND be)

INT port(IP ip)

DURATION duration([STRING s], [DURATION fallback], [REAL real], [INT integer])

BYTES bytes([STRING s], [BYTES fallback], [REAL real], [INT integer])

INT integer([STRING s], [INT fallback], [BOOL bool], [BYTES bytes], [DURATION duration], [REAL real], [TIME time])

IP ip(STRING s, [IP fallback], BOOL resolve, [STRING p])

REAL real([STRING s], [REAL fallback], [INT integer], [BOOL bool], [BYTES bytes], [DURATION duration], [TIME time])

TIME time([STRING s], [TIME fallback], [REAL real], [INT integer])

STRING strftime(TIME time, STRING format)

VOID log(STRING s)

VOID syslog(INT priority, STRING s)

VOID timestamp(STRING s)

BOOL syntax(REAL)

STRING getenv(STRING name)

BOOL cache_req_body(BYTES size)

VOID late_100_continue(BOOL late)

VOID set_ip_tos(INT tos)

VOID rollback(HTTP h)

BOOL ban(STRING)

STRING ban_error()

TIME now()

DURATION timed_call(SUB)

---

DESCRIPTION

so vmod_std is packed with the basic functions that are literally part and parcel of Varnish 🫶 — but for reasons of architecture they fit better in a VMOD. it's giving "we live here now." 💅

---

NUMERIC FUNCTIONS

REAL random(REAL lo, REAL hi)

spits out a random real number somewhere between lo and hi. ✨

this one runs on the "testable" random generator inside varnishd, which is what lets deterministic tests happen (peep the debug.srandom CLI command). 👀

⚠️ bestie, do not use this for cryptographic applications. she is not that kind of girl. 💀

Example:

set beresp.http.random-number = std.random(1, 100);

REAL round(REAL r)

rounds the real r to the nearest integer — but halfway cases get rounded away from zero (go check round(3) if you want the receipts). 🌸

---

STRING FUNCTIONS

VOID collect(HEADER hdr, STRING sep=", ")

collapses a bunch of hdr headers into one long header. the default separator sep is the standard comma separator you'd use when collapsing headers, plus a lil extra whitespace for pretty printing. 🎀

⚠️ be careful when you collapse headers, okay? specifically, collapsing Set-Cookie is gonna give you unexpected results on the browser side. not the vibe you want. 😭

🚨 using hdr from obj.http triggers a VCL failure. periodt.

Examples:

std.collect(req.http.accept);
std.collect(req.http.cookie, "; ");

STRING querysort(STRING)

sorts the query string, purely for cache normalization purposes. iconic behavior honestly. ✨

Example:

set req.url = std.querysort(req.url);

STRING toupper(STRING s)

converts the string s to UPPERCASE. screaming, but make it text. 🔥

Example:

set beresp.http.scream = std.toupper("yes!");

STRING tolower(STRING s)

converts the string s to lowercase. soft launch energy. 🦋

Example:

set beresp.http.nice = std.tolower("VerY");

STRING strstr(STRING s1, STRING s2)

returns a string starting at the first place where s2 shows up inside s1 — or an empty string if s2 is nowhere to be found. 🙈

heads up: the comparison is case-sensitive. no lowercase shortcuts here.

Example:

if (std.strstr(req.url, req.http.restrict)) {
        ...
}

this one checks whether the content of req.http.restrict occurs anywhere in req.url. 💅

BOOL fnmatch(STRING pattern, STRING subject, BOOL pathname, BOOL noescape, BOOL period)

BOOL fnmatch(
   STRING pattern,
   STRING subject,
   BOOL pathname=1,
   BOOL noescape=0,
   BOOL period=0
)

shell-style pattern matching 🐚 — returns true if subject matches pattern, where pattern can have wildcard characters like * or ?.

the match gets executed by the fnmatch(3) implementation on your system. the rules for pattern matching on most systems include the following:

• * matches any sequence of characters
• ? matches a single character
• a bracket expression such as [abc] or [!0-9] is interpreted as a character class according to the rules of basic regular expressions (not pcre2(3) regexen), except that ! is used for character class negation instead of ^.

if pathname is true, then the forward slash character / is only matched literally, and never matches *, ? or a bracket expression. otherwise, / may match one of those patterns. by default, pathname is true.

if noescape is true, then the backslash character \ is matched as an ordinary character. otherwise, \ is an escape character, and matches the character that follows it in the pattern. for example, \\ matches \ when noescape is true, and \\ when false. by default, noescape is false.

if period is true, then a leading period character . only matches literally, and never matches *, ? or a bracket expression. a period is leading if it's the first character in subject; if pathname is also true, then a period that immediately follows a / is also leading (as in /.). by default, period is false.

🚨 std.fnmatch() invokes VCL failure and returns false if either pattern or subject is NULL — for example, if an unset header is specified. don't do that to her.

Examples:

# Matches URLs such as /foo/bar and /foo/baz
if (std.fnmatch("/foo/\*", req.url)) { ... }

# Matches URLs such as /foo/bar/baz and /foo/baz/quux
if (std.fnmatch("/foo/\*/\*", bereq.url)) { ... }

# Matches /foo/bar/quux, but not /foo/bar/baz/quux
if (std.fnmatch("/foo/\*/quux", req.url)) { ... }

# Matches /foo/bar/quux and /foo/bar/baz/quux
if (std.fnmatch("/foo/\*/quux", req.url, pathname=false)) { ... }

# Matches /foo/bar, /foo/car and /foo/far
if (std.fnmatch("/foo/?ar", req.url)) { ... }

# Matches /foo/ followed by a non-digit
if (std.fnmatch("/foo/[!0-9]", req.url)) { ... }

---

FILE(SYSTEM) FUNCTIONS

STRING fileread(STRING)

reads a text file and hands you back a string with the content. 🍓

the entire file gets cached on the first call, and subsequent calls will return this cached content — even if the file has changed in the meantime. she said "I remember you as you were." 😭

for binary files, use std.blobread() instead. 🦋

Example:

synthetic("Response was served by " + std.fileread("/etc/hostname"));

⚠️ consider that the entire contents of the file appear in the string that gets returned — including newlines that may result in invalid headers if std.fileread() is used to form a header. in that case, you may need to modify the string, for example with regsub() (see vcl(7)):

set beresp.http.served-by = regsub(std.fileread("/etc/hostname"), "\R$", "");

BLOB blobread(STRING)

reads any file and returns a blob with the content. 💜

the entire file is cached on the first call, and subsequent calls will return this cached content, even if the file has changed in the meantime. same memory situation as her sister up there. 🎀

BOOL file_exists(STRING path)

returns true if path or the file pointed to by path exists, false otherwise. simple as that, bestie. ✨

Example:

if (std.file_exists("/etc/return_503")) {
        return (synth(503, "Varnish is in maintenance"));
}

---

TYPE INSPECTION FUNCTIONS

BOOL healthy(BACKEND be)

returns true if the backend be is healthy. 🩺✨

INT port(IP ip)

returns the port number of the IP address ip. 🌈

always returns 0 for a *.ip variable when the address is a Unix domain socket. 💅

🎀 TYPE CONVERSION FUNCTIONS 🎀

okay bestie, these ones are alllll the same shape, they're literally siblings:

TYPE type([arguments], [fallback TYPE])

You give it exactly one argument (the fallback is optional, it doesn't count as the argument, don't be fooled), and it gets converted into that TYPE. Slay.

If the conversion fails? You get fallback back. No fallback specified? Then the VCL fails. Periodt, that's the whole deal.

🌸 DURATION duration([STRING s], [DURATION fallback], [REAL real], [INT integer])

DURATION duration(
   [STRING s],
   [DURATION fallback],
   [REAL real],
   [INT integer]
)

Hands you back a DURATION built from a STRING, REAL or INT argument.

When it's a STRING s, bestie, s must be quantified with one of these units: ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks) or y (years). No unit, no deal 💅

real and integer arguments? Those are read as seconds.

If the conversion of an s argument flops, you get fallback if you provided one — otherwise a VCL failure is triggered, sorry babe 😭

Conversions from real and integer never fail. Iconic.

Only one of the s, real or integer arguments may be given — if you sneak in more than one, VCL failure, no cap.

Examples:

set beresp.ttl = std.duration("1w", 3600s);
set beresp.ttl = std.duration(real=1.5);
set beresp.ttl = std.duration(integer=10);

💖 BYTES bytes([STRING s], [BYTES fallback], [REAL real], [INT integer])

BYTES bytes(
   [STRING s],
   [BYTES fallback],
   [REAL real],
   [INT integer]
)

Returns BYTES from a STRING, REAL or INT argument. Same energy as duration, different units.

A STRING s argument can be quantified with a multiplier: k (kilo), m (mega), g (giga), t (tera) or p (peta). Go big or go home 🫶

real and integer arguments are taken as bytes.

If the conversion of an s argument fails, fallback comes through if you gave one, otherwise VCL failure. Standard.

Other conversions can also fail if the argument can't be represented — like it's negative, too small or too large. Again: fallback if provided, else VCL failure, so have your fallback ready bestie.

real arguments get rounded down.

And again, only one of the s, real or integer arguments may be given or VCL failure gets triggered.

Example:

std.cache_req_body(std.bytes(something.somewhere, 10K));
std.cache_req_body(std.bytes(integer=10*1024));
std.cache_req_body(std.bytes(real=10.0*1024));

✨ INT integer([STRING s], [INT fallback], [BOOL bool], [BYTES bytes], [DURATION duration], [REAL real], [TIME time])

INT integer(
   [STRING s],
   [INT fallback],
   [BOOL bool],
   [BYTES bytes],
   [DURATION duration],
   [REAL real],
   [TIME time]
)

Returns an INT from a STRING, BOOL or other quantity. So many options, main character energy fr fr.

If the conversion of an s argument fails, fallback if provided, otherwise VCL failure.

A bool argument comes back as 0 for false and 1 for true. This conversion never fails ✨

For a bytes argument, you get the number of bytes. This conversion never fails either.

A duration argument gets rounded down to the number of seconds and returned.

A real argument gets rounded down and returned.

For a time argument, you get the number of seconds since the UNIX epoch (1970-01-01 00:00:00 UTC).

duration, real and time conversions may fail if the argument can't be represented because it's too small or too large. If that happens, fallback if provided, or VCL failure — you know the drill by now 💅

Only one of the s, bool, bytes, duration, real or time arguments may be given or a VCL failure will be triggered.

Examples:

if (std.integer(req.http.foo, 0) > 5) {
        ...
}

set resp.http.answer = std.integer(real=126.42/3);

🦋 IP ip(STRING s, [IP fallback], BOOL resolve=1, [STRING p])

Converts the string s to the first IP number returned by the system library function getaddrinfo(3). If conversion fails, fallback is returned or a VCL failure happens.

The IP address includes a port number that you can find with std.port() — and it defaults to 80. You can set that default port to something else with the p argument. But heads up: it gets overridden if s contains both an IP address and a port number or service name.

When s contains both, the syntax is either address:port or address port. And if the address is a numerical IPv6 address it must be wrapped in brackets, like [::1] 80 or [::1]:http. The fallback can also contain both an address and a port, but its default port is always 80.

If resolve is false, getaddrinfo(3) is called using AI_NUMERICHOST and AI_NUMERICSERV to avoid network lookups, which depend on the system's getaddrinfo(3) or nsswitch configuration. This makes "numerical" IP strings and services cheaper to convert. Cheaper = better, we love efficiency 🎀

Example:

if (std.ip(req.http.X-forwarded-for, "0.0.0.0") ~ my_acl) {
        ...
}

🌈 REAL real([STRING s], [REAL fallback], [INT integer], [BOOL bool], [BYTES bytes], [DURATION duration], [TIME time])

REAL real(
   [STRING s],
   [REAL fallback],
   [INT integer],
   [BOOL bool],
   [BYTES bytes],
   [DURATION duration],
   [TIME time]
)

Returns a REAL from a STRING, BOOL or other quantity.

If the conversion of an s argument fails, fallback if provided, otherwise VCL failure.

A bool argument comes back as 0.0 for false and 1.0 for true.

For a bytes argument, the number of bytes is returned.

For a duration argument, the number of seconds is returned.

An integer argument is returned as a REAL.

For a time argument, you get the number of seconds since the UNIX epoch (1970-01-01 00:00:00 UTC).

None of these conversions other than s will fail. Obsessed.

Only one of the s, integer, bool, bytes, duration or time arguments may be given or a VCL failure will be triggered.

Example:

if (std.real(req.http.foo, 0.0) > 5.5) {
        ...
}

💜 TIME time([STRING s], [TIME fallback], [REAL real], [INT integer])

TIME time([STRING s], [TIME fallback], [REAL real], [INT integer])

Returns a TIME from a STRING, REAL or INT argument.

For a STRING s argument, these formats are supported:

"Sun, 06 Nov 1994 08:49:37 GMT"
"Sunday, 06-Nov-94 08:49:37 GMT"
"Sun Nov  6 08:49:37 1994"
"1994-11-06T08:49:37"
"784111777.00"
"784111777"

real and integer arguments are taken as seconds since the epoch.

If the conversion of an s argument fails, or a negative real or integer argument is given, fallback is returned if provided, or a VCL failure will be triggered.

Examples:

if (std.time(resp.http.last-modified, now) < now - 1w) {
        ...
}

if (std.time(int=2147483647) < now - 1w) {
        ...
}

🧚 STRING strftime(TIME time, STRING format)

Formats the time argument with the format argument using strftime(3) and returns the result for the UTC (historically GMT) timezone.

The empty string is returned if formatting fails — but fair warning, an empty string can also just be a valid result. So don't read too much into it bestie 🙈

Example:

set req.http.iso = std.strftime(now, "%Y%m%dT%H%M%SZ");
# e.g. 20210521T175241Z

---

📝 LOGGING FUNCTIONS 📝

⭐ VOID log(STRING s)

Logs the string s to the shared memory log, using vsl(7) tag SLT_VCL_Log.

Example:

std.log("Something fishy is going on with the vhost " + req.http.host);

💀 VOID syslog(INT priority, STRING s)

Logs the string s to syslog tagged with priority. That priority is formed by ORing the facility and level values together. Peep your system's syslog.h file for the possible values.

Notice: unlike VCL and other functions in the std vmod, this function will not fail VCL processing for workspace overflows: for an out of workspace condition, the std.syslog() function has no effect. So it just... does nothing. Silent treatment 😭

Example:

std.syslog(9, "Something is wrong");

This will send a message to syslog using LOG_USER | LOG_ALERT.

🍓 VOID timestamp(STRING s)

Drops a timestamp into the log with the current time, using the string s as the label. This is so useful for timing the execution of lengthy VCL subroutines, and it makes the timestamps that Varnish inserts automatically way more accurate. Like, actually accurate. Groundbreaking 🫶

Example:

std.timestamp("curl-request");

🎛️ CONTROL AND INFORMATION FUNCTIONS 🎛️

okay bestie, this is the part where std starts running things instead of just doing math, and honestly? it's giving main character energy fr fr 💅

---

• BOOL syntax(REAL) — returns true if your VCL version is at least REAL. that's it, that's the whole vibe check ✅
• STRING getenv(STRING name) — hands you back the environment variable name, or the empty string if it's not there. peek getenv(3) if you wanna know more 🌸

Example:

set req.http.My-Env = std.getenv("MY_ENV");

• BOOL cache_req_body(BYTES size) — caches the request body only if it's smaller than size. returns true if the body got cached, false if it didn't 🫶

so normally a request body can only be sent once, right? that's the whole problem. caching it means you can retry backend requests that have a request body — which is literally the usual situation with POST and PUT. lowkey a lifesaver.

Example:

if (std.cache_req_body(1KB)) {
        ...
}

Restricted to: vcl_recv.

• VOID late_100_continue(BOOL late) — this one controls when varnish reacts to a client request header of Expect: 100-continue ✨

varnish always generates a 100 Continue response if the client asks for it through that header while waiting for request body data.

but — and this is the plot twist — by default, that 100 Continue response is already generated immediately after vcl_recv returns, to reduce latencies, based on the assumption that the request body will be read eventually. assumptions!! 😭

calling std.late_100_continue(true) inside vcl_recv means the 100 Continue response will only be sent when needed. this may cause additional latencies for processing request bodies, but it's the correct behavior if you read RFC7231 strictly. no cap.

this function has no effect outside vcl_recv, and no effect after calling std.cache_req_body() or any other function that consumes the request body.

Example:

vcl_recv {
        std.late_100_continue(true);

        if (req.method == "POST") {
                std.late_100_continue(false);
                return (pass);
        }
        ...
 }

Restricted to: vcl_recv.

• VOID set_ip_tos(INT tos) — sets the Differentiated Services Codepoint (DSCP) / IPv4 Type of Service (TOS) / IPv6 Traffic Class (TCLASS) byte for the current session to tos. if the listen address is a Unix domain socket, this is silently ignored 🙈

pls note: setting the traffic class affects all requests on the same http1.1 / http2 TCP connection, and in particular, it is not removed at the end of the request. it lingers, bestie 💀

Example:

if (req.url ~ "^/slow/") {
        std.set_ip_tos(0);
}

Restricted to: client.

• VOID rollback(HTTP h) — restores the h HTTP headers back to their original state. undo button but for headers, iconic 🎀

Example:

std.rollback(bereq);

Restricted to: backend, vcl_recv, vcl_pass, vcl_hash, vcl_purge, vcl_miss, vcl_hit, vcl_deliver, vcl_synth.

• BOOL ban(STRING) — invalidates all objects in cache that match the given expression, using the ban mechanism. returns true if the ban succeeded and false if it didn't. error deets are available via std.ban_error(). so dramatic, so powerful 👑

the format of STRING is:

<field> <operator> <arg> [&& <field> <oper> <arg> ...]

• <field>:
  • string fields:
    • req.url: The request url
    • req.http.*: Any request header
    • obj.status: The cache object status
    • obj.http.*: Any cache object header

obj.status is treated as a string despite the fact that it is actually an integer. the audacity 😭

• duration fields:
  • obj.ttl: Remaining ttl
  • obj.age: Object age
  • obj.grace: The grace time of the object
  • obj.keep: The keep time of the object
  • obj.last_hit: Time since the last hit

obj.ttl, obj.age and obj.last_hit are relative to the submission time of the ban, such that they represent a fixed point in time despite being specified as a duration. fancy ✨

• <operator>:
  • for all fields:
    • ==: <field> and <arg> are equal
    • !=: <field> and <arg> are unequal

strings are compared case-sensitively. yes, capital letters matter, i don't make the rules 💅

• for string fields:
  • ~: <field> matches the regular expression <arg>
  • !~:<field> does not match the regular expression <arg>
• for duration fields:
  • >: <field> is greater than <arg>
  • >=: <field> is greater than or equal to <arg>
  • <: <field> is less than <arg>
  • <=: <field> is less than or equal to <arg>

• <arg>:
  • for string fields:

Either a literal string or a regular expression. note that <arg> does not use any of the string delimiters like " or {"..."} or """...""" used elsewhere in varnish. to match against strings containing whitespace, regular expressions containing \s can be used.

• for duration fields:

A VCL duration like 10s, 5m or 1h, see vcl(7)_durations

expressions can be chained using the and operator &&. for or semantics, use several bans. simple as that 🦋

the unset <field> is not equal to any string, such that, for a non-existing header, the operators == and ~ always evaluate as false, while the operators != and !~ always evaluate as true, respectively, for any value of <arg>. yeah it's a whole thing.

obj.last_hit can be used almost as a "last accessed" time, so, for example, ban obj.last_hit > 1d removes all objects which were last accessed more than one day ago. also, it can be used to remove objects stuck at request bans by issuing ban obj.last_hit > X, with X being slightly less than the time since the request ban. sneaky but valid.

obj.last_hit is based on an internal last LRU time, which might not be implemented by all storage engines. where it is not available, ban expressions using obj.last_hit evaluate to false, which means that the respective ban behaves as if it was not present. where implemented, the last LRU time might only get updated by the lru_interval parameter, which therefore is the maximum precision of obj.last_hit bans. so like, know your storage engine, bestie 🌈

• STRING ban_error() — returns a textual error description of the last std.ban() call from the same task, or the empty string if there either was no error or no std.ban() call. receipts, basically 🧾
• TIME now() — returns the current time. in contrast to the now built-in variable, every call returns a new value. so it actually moves, unlike some people 💀
• DURATION timed_call(SUB) — call the given SUB and return a high precision measurement of the execution time. stopwatch core ⏱️

---

🔍 SEE ALSO 🔍

• varnishd(1)
• vsl(7)
• fnmatch(3)
• strftime(3)

---

📜 COPYRIGHT 📜

like, legally required so here it is, verbatim, no edits, we respect licenses in this house 🫡

     Copyright (c) 2010-2017 Varnish Software AS
     All rights reserved.

     Author: Poul-Henning Kamp <phk@FreeBSD.org>

     SPDX-License-Identifier: BSD-2-Clause

     Redistribution and use in source and binary forms, with or without
     modification, are permitted provided that the following conditions
     are met:
     1. Redistributions of source code must retain the above copyright
        notice, this list of conditions and the following disclaimer.
     2. Redistributions in binary form must reproduce the above copyright
        notice, this list of conditions and the following disclaimer in the
        documentation and/or other materials provided with the distribution.

     THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
     ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
     IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
     ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
     FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
     DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
     OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
     HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
     LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
     OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
     SUCH DAMAGE.

---

and that's a wrap on the whole thing!! 🎀 you just read the entire VMOD_STD(3) page, top to bottom, and i'm so proud of you fr fr 💖 go forth and std. everything, bestie — you ate, periodt 👑✨ </output>