STRTOK(3P)

💖 STRTOK — the POSIX girlies' guide to splitting strings into tokens 💖

ok so bestie, this is strtok(3P), from the POSIX Programmer's Manual. yes the actual IEEE one. she's iconic, she's a little cursed, let's get into it 🫶

🌸 PROLOG 🌸

sooo this manual page is part of the POSIX Programmer's Manual. lowkey important disclaimer: the Linux implementation of this interface may differ (so like, go read the corresponding Linux manual page for the actual Linux tea), or the interface might not even be implemented on Linux at all. 💀

---

✨ NAME ✨

strtok, strtok_r — split string into tokens. yes girl, that's the whole job. 🎀

---

🦋 SYNOPSIS 🦋

#include <string.h>

char *strtok(char *restrict s, const char *restrict sep);
char *strtok_r(char *restrict s, const char *restrict sep,
    char **restrict state);

not the restrict keywords doing the most, we love to see it 💅

---

💜 DESCRIPTION 💜

okay so for strtok(): the functionality described on this reference page is aligned with the ISO C standard. and if there's any conflict between the requirements described here and the ISO C standard, that conflict is unintentional — POSIX.1-2017 defers to the ISO C standard, periodt. no drama, the C standard wins. 👑

A sequence of calls to strtok() breaks the string pointed to by s into a sequence of tokens, each of which is delimited by a byte from the string pointed to by sep. The first call in the sequence has s as its first argument, and is followed by calls with a null pointer as their first argument. The separator string pointed to by sep may be different from call to call — she's flexible, she contains multitudes ✨

The first call in the sequence searches the string pointed to by s for the first byte that is not contained in the current separator string pointed to by sep. If no such byte is found, then there are no tokens in the string pointed to by s and strtok() shall return a null pointer. If such a byte is found, it is the start of the first token.

The strtok() function then searches from there for a byte that is contained in the current separator string. If no such byte is found, the current token extends to the end of the string pointed to by s, and subsequent searches for a token shall return a null pointer. If such a byte is found, it is overwritten by a NUL character, which terminates the current token. The strtok() function saves a pointer to the following byte, from which the next search for a token shall start.

Each subsequent call, with a null pointer as the value of the first argument, starts searching from the saved pointer and behaves as described above. it's giving "remember where we were" energy 🧚

The implementation shall behave as if no function defined in this volume of POSIX.1-2017 calls strtok().

⚠️ The strtok() function need not be thread-safe. ngl bestie that's the whole plot twist. 😭

The strtok_r() function shall be equivalent to strtok(), except that strtok_r() shall be thread-safe, and the argument state points to a user-provided pointer that allows strtok_r() to maintain state between calls which scan the same string. The application shall ensure that the pointer pointed to by state is unique for each string (s) being processed concurrently by strtok_r() calls. The application need not initialize the pointer pointed to by state to any particular value. The implementation shall not update the pointer pointed to by state to point (directly or indirectly) to resources, other than within the string s, that need to be freed or released by the caller. so like, no sneaky malloc vibes, you're safe 🫶

---

🌈 RETURN VALUE 🌈

Upon successful completion, strtok() shall return a pointer to the first byte of a token. Otherwise, if there is no token, strtok() shall return a null pointer.

The strtok_r() function shall return a pointer to the token found, or a null pointer when no token is found.

---

💀 ERRORS 💀

No errors are defined.

The following sections are informative. (yes bestie, that means the juicy stuff is optional reading, but we're reading it anyway 😌)

---

🍓 EXAMPLES 🍓

Searching for Word Separators

The following example searches for tokens separated by <space> characters.

#include <string.h>
...
char *token;
char line[] = "LINE TO BE SEPARATED";
char *search = " ";

/* Token will point to "LINE". */
token = strtok(line, search);

/* Token will point to "TO". */
token = strtok(NULL, search);

see how the second call is just NULL? she's remembering. main character energy fr 💅

Find First two Fields in a Buffer

The following example uses strtok() to find two character strings (a key and data associated with that key) separated by any combination of <space>, <tab>, or <newline> characters at the start of the array of characters pointed to by buffer.

#include <string.h>
...
char    *buffer;
...
struct element {
    char *key;
    char *data;
} e;
...
// Load the buffer...
...
// Get the key and its data...
e.key = strtok(buffer, " \t\n");
e.data = strtok(NULL, " \t\n");
// Process the rest of the contents of the buffer...
...

the " \t\n" separator string is doing SO much work and honestly? ate. 🔥

---

🌟 APPLICATION USAGE 🌟

💡 Note that if sep is the empty string, strtok() and strtok_r() return a pointer to the remainder of the string being tokenized. wild, right? empty separator = "no delimiters, just vibes".

The strtok_r() function is thread-safe and stores its state in a user-supplied buffer instead of possibly using a static data area that may be overwritten by an unrelated call from another thread. she brought her own storage, no sharing required 🎀

---

👑 RATIONALE 👑

The strtok() function searches for a separator string within a larger string. It returns a pointer to the last substring between separator strings. This function uses static storage to keep track of the current string position between calls. The new function, strtok_r(), takes an additional argument, state, to keep track of the current position in the string.

so basically strtok() has one (1) global memory and everyone's fighting over it, and strtok_r() said "not me, I brought my own." iconic behavior honestly 💖

---

🦄 FUTURE DIRECTIONS 🦄

None.

(that's it. nothing. the future is a blank page. we love a chill roadmap ✨)

---

💕 SEE ALSO 💕

The Base Definitions volume of POSIX.1-2017, <string.h>

---

📜 COPYRIGHT 📜

Portions of this text are reprinted and reproduced in electronic form from IEEE Std 1003.1-2017, Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, 2018 Edition, Copyright (C) 2018 by the Institute of Electrical and Electronics Engineers, Inc and The Open Group. In the event of any discrepancy between this version and the original IEEE and The Open Group Standard, the original IEEE and The Open Group Standard is the referee document. The original Standard can be obtained online at http://www.opengroup.org/unix/online.html .

Any typographical or formatting errors that appear in this page are most likely to have been introduced during the conversion of the source files to man page format. To report such errors, see https://www.kernel.org/doc/man-pages/reporting_bugs.html .

---

okay bestie that's the whole page!! this one's from the IEEE/The Open Group, 2017, and it lives in section 3P of the manual. mwah, go tokenize something responsibly 💅✨🫶