NE_TOKEN(3)

💅 NE_TOKEN(3) — neon API reference, but make it a group chat

okay so bestie, welcome to the neon manual page for ne_token and ne_qtoken 💖 these two are the string tokenizers, they live in the neon API reference, and honestly they're little icons. let me put you on 🫶

---

✨ NAME ✨

ne_token, ne_qtoken — string tokenizers 🎀

---

🌸 SYNOPSIS 🌸

like, here's the include and the signatures, exactly as they are:

#include <ne_string.h>

char *ne_token(char **str, char sep);

char *ne_qtoken(char **str, char sep, const char *quotes);

that's the whole vibe check. two functions, both returning char *, both taking a char **str. slay ✨

---

🦋 DESCRIPTION 🦋

okay so here's the tea, bestie ☕ ne_token and ne_qtoken tokenize the string at the location stored in the pointer str.

every single time you call the function, it returns the next token, and it modifies the str pointer so it points to the remainder of the string — or NULL if there are no more tokens left in the string. periodt 💅

a token is delimited by the separator character sep. and if you're using ne_qtoken, any quoted segments of the string are skipped when it's searching for a separator 👀 a quoted segment is enclosed in a pair of one of the characters given in the quotes string. iykyk 🌈

ngl, important warning bestie 🚨 the string being tokenized is modified every time the tokenizing function is called — it replaces the next separator character with a NUL terminator. so it's giving in-place, main character energy, and your original string is not coming back the same 😭

---

🧚 EXAMPLES 🧚

the following function prints out each token in a comma-separated string list, which is modified in-place (told you it was in-place 💀):

static void splitter(char *list)
{
  do {
    printf("Token: %s\n", ne_token(&list, ','));
  while (list);
}

it's giving "print every token, keep going while list is still real" 🍓 and yes bb, that's character-for-character how it appears up top, no cap, i don't make the rules 🎀

---

👑 COPYRIGHT 👑

Copyright © 2001-2026 Joe Orton

---

anyway, that's the whole page bestie, go tokenize something iconic 💜

with love, from the neon manual — source/version neon, dated 03/20/2026, and the page itself is NE_TOKEN(3) 🦋✨