mtext_tok(3m17n)

📦
version 1.8.4 · 2 versions
, in debian trixie, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

💖 mtext_tok — bestie edition 💖

okay so this is mtext_tok from the m17n Library (that's man page section 3m17n, iykyk), and honestly? it's giving tokenizer energy and I'm obsessed. sit down, grab a boba, we're doing this together 🧋✨

✨ SYNOPSIS ✨

MText* mtext_tok (MText * mt, MText * delim, int * pos)

🌸 DESCRIPTION 🌸

So the whole vibe here is: we're hunting for a token inside an M-text. The mtext_tok() function searches for a token that firstly occurs after position pos in the M-text mt.

And what even is a token, bestie? A token means a substring where each character does not appear in the M-text delim. Like, delim is the bouncer at the club and any character on the list is NOT getting in 💅

⚠️ real talk though — note that the type of pos is not int but pointer to int. It's a pointer. Do not come for me, that's just what it is. int *, periodt.

🦋 RETURN VALUE 🦋

  • 💖 If a token is found — mtext_tok() copies the corresponding part of mt and returns a pointer to the copy. In this case, pos is set to the end of the found token. slay, we got one.
  • 💀 If no token is found — it returns NULL without changing the external variable merror_code. nothing happened, no drama, no error, just vibes.
  • 😭 If an error is detected — it returns NULL and assigns an error code to the external variable merror_code.

🔥 ERRORS 🔥

  • MERROR_RANGE

Copyright (C) 2001 Information-technology Promotion Agency (IPA)

Copyright (C) 2001-2011 National Institute of Advanced Industrial Science and Technology (AIST)

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License <http://www.gnu.org/licenses/fdl.html>.

okay that's the whole page, you ate it, I'm so proud 🫶 this was Version 1.8.4, dated Mon Sep 25 2023, and it lives in mtext_tok(3m17n). go forth and tokenize, main character 🌈⭐🎀

📦 where this page comes from