Math::E...::Lexer(3)

💖 Math::E...::Lexer 💖

okay bestie, so this is Math::Expression::Evaluator::Lexer from the User Contributed Perl Documentation set — the full name is a whole mouthful so the man page abbreviates it to Math::E...::Lexer, and honestly? iconic of it. 🎀

---

✨ NAME ✨

Math::Expression::Evaluator::Lexer — Simple Lexer 🦋

---

🌸 SYNOPSIS 🌸

use Math::Expression::Evaluator::Lexer qw(lex);
# suppose you want to parse simple math expressions
my @input_tokens = (
    ['Int',             qr/[+-]?\d+/ ],
    ['Op',              qr{[+-/*]}   ],
    ['Brace_Open',      qr/\(/       ],
    ['Brace_Close',     qr/\)/       ],
    ['Whitespace',      qr/\s+/, sub { return; }],
     );
my $text = "-12 * (3+4)";
my $out_tokens = lex($text, \@input_tokens);
for (@$out_tokens){
    my ($name, $text, $pos) = @$_;
    print "Found Token $name: $text (string pos: $pos)\n";
}

like, look at her go 🫶 she takes a string and just shreds it into tokens, no cap.

---

💅 DESCRIPTION 💅

Math::Expression::Evaluator::Lexer is a simple lexer that breaks up a text into tokens, depending on the input tokens you provide. that's it. that's the whole job. she's not doing your taxes, she's just chopping your string into little pieces for you. 💖

---

👑 METHODS 👑

⭐ lex

The only exported routine is lex, which expects input text as its first argument and a array ref to list of input tokens. 🧚

Each input token consists of:

• a token name (which you can choose freely — go feral, name it whatever),
• a regex which matches the desired token,
• and optionally a reference to a functions that takes the matched token text as its argument. The token text is replaced by the return value of that function. If the function returns undef, that token will not be included in the list of output tokens.

⚠️ okay so the regex should either fail or match at least one character; zero-width matches utterly confuse the lexer, and are disallowed. don't do it. she will simply malfunction and you will have a bad time. 😭

lex() returns an array ref to a list of output tokens, each output token is a reference to a list which contains:

• the token name,
• the matched text,
• the string position (in characters, counted from the start of the input string, zero based),
• and the line number.

🙈 heads up bestie: lex() puts parentheses around the entire regex, so if you want to use backreferences, the numbering of the capturing group is changed. fr fr, adjust your \1s accordingly or you're gonna have a confusing day. 💀

---

🎀 COPYRIGHT AND LICENSE 🎀

Copyright (C) 2007 by Moritz Lenz, <http://moritz.faui2k3.org>, moritz@faui2k3.org.

This Program and its Documentation is free software. You may distribute it under the same terms as perl itself.

However all code examples are to be public domain, so you can use it in any way you want to. 🌈

---

and that's the whole page, bestie!! this one came to you courtesy of perl v5.42.0, dated 2025-07-25, and the man page itself goes by Math::E...::Lexer(3) up top. 💜✨ slay, go tokenize something. 🦋