Camlinternal...Lib.LexerUtil(3)

💅 NAME

CamlinternalMenhirLib.LexerUtil — okay so bestie, there is literally no description here. That's it. That's the whole name section. The manpage said "no description" and honestly? Iconic of it. 💀

---

🦋 Module

It's the module CamlinternalMenhirLib.LexerUtil, and this page is part of the OCaml library docs. Keep that in your back pocket, iykyk.

---

✨ Documentation

So the module is literally:

Module LexerUtil
 : sig end

Yes queen, that's the vibe. Now let's get into the actual functions, because this is where things get good. 🫶

🌸 val init : string -> Lexing.lexbuf -> Lexing.lexbuf

init filename lexbuf sets up — initializes — the lexing buffer lexbuf so that whenever you read positions from it later, those positions point at the file filename. And then it hands lexbuf right back to you. Slay, returning it is the whole point. ✨

🎀 val read : string -> string * Lexing.lexbuf

read filename reads the entire contents of the file filename — the whole thing, no crumbs left — and gives you back a pair: that content as a string, plus a lexing buffer that's already been initialized based on that string. Two-for-one, we love to see it. 💖

⭐ val newline : Lexing.lexbuf -> unit

newline lexbuf bumps up the line counter that lives inside lexbuf. The catch: your lexer has to call it itself every single time it consumes a newline character. Do that and you keep a correct current line number inside lexbuf. Also, plot twist: it's literally synonymous with Lexing.new_line lexbuf. Same thing, different name. No cap. 🔥

🌈 val range : Lexing.position * Lexing.position -> string

range (startpos, endpos) — you give it a start position and an end position, and it prints out a textual description of that range. Details, bestie:

• The description is exactly one line long and ends with a newline character.
• It mentions the file name, the line number, and a range of characters on that line.
• ⚠️ The line number is only correct if newline has been used properly, as described above. Skip that and your line numbers are giving lies. 😭

👑 val tabulate : ('a -> bool) -> (unit -> 'a) -> unit -> 'a

tabulate is_eof lexer tabulates the lexer lexer. Which means: it runs that lexer right away, all the way through until it hits an EOF token, stores all the tokens in an array in memory, and then returns a brand-new lexer that reads tokens from that array whenever you invoke it. Pre-gaming your tokens, basically. 🍓

The rules though — read them, fr fr:

• The function lexer is not allowed to raise an exception.
• It must produce a finite stream of tokens: after a finite number of invocations, it has to return a token that is_eof identifies as an EOF token.

Both the original lexer lexer and the new lexer that tabulate is_eof lexer returns are functions of type unit -> 'a — where that 'a is most likely going to be instantiated with a triple of a token and two positions, per the revised lexer API described over in the module CamlinternalMenhirLib.Convert. Go check that one out, it's the main character of this whole saga. 💜

One thing to keep in mind: lexer raising an exception is a hard no, and an infinite token stream is a hard no — the whole tabulate trick depends on the stream ending. 🧚

---

Alright bestie, that's the whole page! This lil' doc drop came from OCamldoc, dated 2026-06-22 — so if you're reading this in the future, hi, hope your lexer is behaving. 🦋 Go forth and tokenize responsibly. 💅✨