TOK(3)

💖 NAME 💖

tok — the space separated token stream routines, bestie ✨ This is TOK(3) from the Library Functions Manual, and we're about to make token streams make sense 💅

✨ SYNOPSIS ✨

#include <ast.h>

void* tokopen(char* string) char* tokread(void* tok) void tokclose(void*
tok)

int tokscan(char* string, char** next, const char* format, ...);

Sfio_t* tokline(const char* input, int flags, int* line);

🌸 DESCRIPTION 🌸

Okay so tokopen returns a pointer to a space separated token stream on the 0 terminated string string. tokread returns a pointer to the next space separated token in the token stream tok as returned by tokopen. 0 is returned when no tokens remain. tokread temporarily modifies string by inserting 0's to terminate each token. tokclose closes the token stream and restores string to its original state 🫶

tokscan scans the string string for tokens specified in format. It is a more forgiving sscanf(3). If next != 0 then it will point to the next unread character in string on return. The number of scanned tokens is returned. -1 is returned if string was not empty and format failed to match and tokens 😭

A space in format matches 0 or more space or tab characters. A newline in format eats the remainder of the current line in string. "...", '...' and \character quotes are interpreted. A quoted carriage-return is converted to newline. A newline in string is equivalent to end of string except when quoted. \newline is a line splice. Iconic. No cap 🎀

% in format prefixes format conversion characters; each conversion character corresponds to a tokscan argument following the format argument. The format conversions are:

• %c — A single char.
• %hd %d %ld — [short, int, long] base 10 integer.
• %hn %n %ln — [short, int, long] C-style base integer.
• %ho %o %lo — [short, int, long] base 8 integer.
• %s — String.
• %hu %u %lu — [short, int, long] C-style base unsigned integer.
• %v — The next two arguments are a pointer to a char** argument vector and the maximum number of elements in the vector.
• %hx %x %lx — [short, int, long] base 16 integer.

%s and %v data may also be counted length strings of the form (count:data) where count is the number of characters in data and the terminating ) may also be a tab, or the data may be (null) which represents the NULL string 🔥

tokline returns an sfio(3) stream to a file or string that splices \newline into single lines, allows "..." and '...' quotes to span newlines (done by translating quoted newline to carriage-return; tokscan above converts quoted carriage-return back to newline), and deletes # ... newline comments. This is done by pushing an sfio discipline onto a string or file stream. Seeks are disabled on the resulting stream 👑

• If flags == SF_READ then input is a file name.
• If flags == SF_STRING then input is a 0 terminated string.
• Otherwise input is an open Sfio_t* stream.
• If line != 0 then it points to a line count that is initialized to 0 and is incremented for each input line.

👑 SEE ALSO 👑

• sfio(3)

💖 source/version: Library Functions Manual; date: not listed on the page — that's the TOK(3) sign-off, bestie, no cap 🎀✨