std::strtok(3)

std::strtok(3)

okay bestie, this is std::strtok(3) from the C++ Standard Libary man page — manual name folded in right here like a little name tag 🎀. She tokenizes strings, she has static state, and she is ready to spill. Let's get into it.

---

🎀 NAME 🎀

std::strtok - std::strtok (yes bestie, that's literally the name, she's not doing too much).

✨ Synopsis ✨

Defined in header <cstring>.

char* strtok( char* str, const char* delim );

Finds the next token in a null-terminated byte string pointed to by str. The separator characters are identified by the null-terminated byte string pointed to by delim.

This function is designed to be called multiple times to obtain successive tokens from the same string.

okay so here's the exact play-by-play, no cap:

• If str is not a null pointer, the call is treated as the first call to strtok for this particular string. The function searches for the first character which is not contained in delim.
• If no such character was found, there are no tokens in str at all, and the function returns a null pointer.
• If such character was found, it is the beginning of the token. The function then searches from that point on for the first character that is contained in delim.
• If no such character was found, str has only one token, and the future calls to strtok will return a null pointer.
• If such character was found, it is replaced by the null character '\0' and the pointer to the following character is stored in a static location for subsequent invocations.
• The function then returns the pointer to the beginning of the token.
• If str is a null pointer, the call is treated as a subsequent call to strtok: the function continues from where it left in previous invocation. The behavior is the same as if the previously stored pointer is passed as str.

🌸 Parameters 🌸

• str — pointer to the null-terminated byte string to tokenize.
• delim — pointer to the null-terminated byte string identifying delimiters.

💖 Return value 💖

Pointer to the beginning of the next token or a nullptr if there are no more tokens.

⚠️ Notes ⚠️

bestie, this function is destructive: it writes the '\0' characters in the elements of the string str. In particular, a string literal cannot be used as the first argument of std::strtok. do not pass a literal, I am begging 😭

Each call to this function modifies a static variable: is not thread safe. fr fr.

Unlike most other tokenizers, the delimiters in std::strtok can be different for each subsequent token, and can even depend on the contents of the previous tokens. she's flexible like that 💅.

🧚 Possible implementation 🧚

char* strtok(char* str, const char* delim)
{
    static char* buffer;

    if (str != nullptr)
        buffer = str;

    buffer += std::strspn(buffer, delim);

    if (*buffer == '\0')
        return nullptr;

    char* const tokenBegin = buffer;

    buffer += std::strcspn(buffer, delim);

    if (*buffer != '\0')
        *buffer++ = '\0';

    return tokenBegin;
}

Actual C++ library implementations of this function delegate to the C library, where it may be implemented directly (as in MUSL libc), or in terms of its reentrant version (as in GNU libc).

🌈 Example 🌈

// Run this code

#include <cstring>
#include <iomanip>
#include <iostream>

int main()
{
    char input[] = "one + two * (three - four)!";
    const char* delimiters = "! +- (*)";
    char* token = std::strtok(input, delimiters);
    while (token)
    {
        std::cout << std::quoted(token) << ' ';
        token = std::strtok(nullptr, delimiters);
    }

    std::cout << "\nContents of the input string now:\n\"";
    for (std::size_t n = 0; n < sizeof input; ++n)
    {
        if (const char c = input[n]; c != '\0')
            std::cout << c;
        else
            std::cout << "\\0";
    }
    std::cout << "\"\n";
}

🍓 Output: 🍓

"one" "two" "three" "four"
Contents of the input string now:
"one\0+ two\0* (three\0- four\0!\0"

👑 See also 👑

• strpbrk — finds the first location of any character from a set of separators (function).
• strcspn — returns the length of the maximum initial segment that consists of only the characters not found in another byte string (function).
• strspn — returns the length of the maximum initial segment that consists of only the characters found in another byte string (function).
• ranges::split_view — a view over the subranges obtained from splitting another view using a delimiter; (C++20); (class template).
• views::split — a view over the subranges obtained from splitting another view using a delimiter; (C++20); (range adaptor object).
• C documentation for strtok.

---

okay that's the whole page, bestie! source: http://cppreference.com, version/date: 2024.06.10, and it's std::strtok(3) — mwah 💋✨