flexc++api(3)

📦
version 2.19.00 · 6 versions
, in ubuntu stonking

💖 NAME 💖

flexc++api — this is the Application programmer’s interface of flexc++ generated classes page. It's the flexc++ API manpage, section 3, so now you know exactly where to find it again bestie 🫶

✨ DESCRIPTION ✨

Okay so. flexc++ was designed after flex(1) and flex++(1). Fr fr. And just like those two, flexc++ generates code that does pattern-matching on text, and it can run actions whenever certain regular expressions get recognized. It's giving lexer factory, no cap.

If you want the big-picture tour, go read flexc++(1). This page is all about the Application Programmer’s Interface of the classes flexc++ spits out, and it's organized into these sections:

  • 1. INTERACTIVE SCANNERS: how to create an interactive scanner
  • 2. THE CLASS INTERFACE: SCANNER.H: Constructors and members of the scanner class generated by flexc++
  • 3. NAMING CONVENTION: symbols defined by flexc++ in the scanner class.
  • 4. CONSTRUCTORS: constructors defined in the scanner class.
  • 5. PUBLIC MEMBER FUNCTION: public member declared in the scanner class.
  • 6. PRIVATE MEMBER FUNCTIONS: private members declared in the scanner class.
  • 7. SCANNER CLASS HEADER EXAMPLE: an example of a generated scanner class header
  • 8. THE SCANNER BASE CLASS: the scanner class is derived from a base class. The base class is described in this section
  • 9. PUBLIC ENUMS AND -TYPES: enums and types declared by the base class
  • 10. PROTECTED ENUMS AND -TYPES: enumerations and types used by the scanner and scanner base classes
  • 11. NO PUBLIC CONSTRUCTORS: the scanner base class does not offer public constructors.
  • 12. PUBLIC MEMBER FUNCTIONS: several members defined by the scanner base class have public access rights.
  • 13. PROTECTED CONSTRUCTORS: the base class can be constructed by a derived class. Usually this is the scanner class generated by flexc++.
  • 14. PROTECTED MEMBER FUNCTIONS: this section covers the base class member functions that can only be used by scanner class or scanner base class members
  • 15. PROTECTED DATA MEMBERS: this section covers the base class data members that can only be used by scanner class or scanner base class members
  • 16. FLEX++ TO FLEXC++ MEMBERS: a short overview of frequently used flex(1) members that received different names in flexc++.
  • 17. THE CLASS INPUT: the scanner’s job is completely decoupled from the actual input stream. The class Input, nested within the scanner base class handles the communication with the input streams. The class Input, is described in this section.
  • 18. INPUT CONSTRUCTORS: the class Input can easily be replaced by another class. The constructor-requirements are described in this section.
  • 19. REQUIRED PUBLIC MEMBER FUNCTIONS: this section covers the required public members of a self-made Input class

🦋 UNDERSCORES 🦋

Quick lore drop: starting with version 2.07.00, flexc++ reserved identifiers no longer end in two underscore characters — they end in one. Why? Because according to the C++ standard, identifiers with two or more consecutive underscore characters are reserved by the language 😭 so flexc++ had to stop doing that.

In practice this could mean you need some minor edits to existing source files that use flexc++’s facilities. Most likely it's just changing StartCondition__ into StartCondition_, and PostEnum__ into PostEnum_. That's it, that's the whole vibe check.

The complete list of affected names is:

Enums:

  • ActionType_, Leave_, StartConditon_, PostEnum_;

Member functions:

  • actionType_, continue_, echoCh_, echoFirst_, executeAction_, getRange_, get_, istreamName_, lex_, lop1_, lop2_, lop3_, lop4_, lopf_, matched_, noReturn_, print_, pushFront_, reset_, return_;

Protected data members:

  • d_in_ d_token_ s_finIdx_, s_interactive_, s_maxSizeofStreamStack_, s_nRules_, s_rangeOfEOF_, s_ranges_, s_rf_.

👑 1. INTERACTIVE SCANNERS 👑

An interactive scanner is characterized by the fact that scanning is postponed until an end-of-line character has been received, followed by reading all information on the line read so far. Yes, it waits for the newline like a polite queen. 💅

Flexc++ supports the %interactive directive), generating an interactive scanner. For everything below we're assuming Scanner is the name of the scanner class generated by flexc++.

Caveat bestie, listen up: generating interactive and non-interactive scanners should not be mixed, because their class organizations fundamentally differ, and several of the Scanner class’s members are only available in the non-interactive scanner. As the Scanner.h file contains the Scanner class’s interface, which is normally left untouched by flexc++, flexc++ cannot adapt the Scanner class when asked to change the interactivity of an existing Scanner class. Because of this, support for the --interactive option was discontinued at flexc++’s 1.01.00 release. RIP to that flag, it's gone gone.

The interactive scanner generated by flexc++ has the following characteristics:

  • The Scanner class is derived privately from std::istringstream and (as usual) publicly from ScannerBase.
  • The istringstream base class is constructed by its default constructor.
  • The function lex’s default implementation is removed from Scanner.h and is implemented in the generated lex.cc source file. It performs the following tasks:
    • If the token returned by the scanner is not equal to 0 it is returned as then next token;
    • Otherwise the next line is retrieved from the input stream passed to the Scanner’s constructor (by default std::cin). If this fails, 0 is returned.
    • A '\n' character is appended to the just read line, and the scanner’s std::istringstream base class object is re-initialized with that line;
    • The member lex_ returns the next token. This implementation allows code calling Scanner::lex() to conclude, as usual, that the input is exhausted when lex returns 0.

Here's an example of how such a scanner could be used — look at this, so iconic:

// scanner generated using ’flexc++ lexer’ with lexer containing
// the %interactive directive
int main()
{
    Scanner scanner;        // by default: read from std::cin

    while (true)
    {
        cout << "? ";       // prompt at each line

        while (true)        // process all the line’s tokens
        {
            int token = scanner.lex();

            if (token == ’\n’)  // end of line: new prompt
                break;

            if (token == 0)     // end of input: done
                return 0;

                                // process other tokens
            cout << scanner.matched() << ’\n’;
            if (scanner.matched()[0] == ’q’)
                return 0;
        }
    }
}

🌸 2. THE CLASS INTERFACE: SCANNER.H 🌸

By default, flexc++ generates a file Scanner.h containing the initial interface of the scanner class performing the lexical scan according to the specifications given in flexc++’s input file. The name of the file that gets generated can easily be changed using flexc++’s --class-header option. In this man-page we’ll stick to using the default name, because consistency is the main character here.

The file Scanner.h is generated only once, unless an explicit request is made to rewrite it (using flexc++’s --force-class-header option).

The provided interface is very light-weight, primarily offering a link to the scanner’s base class (see this manpage’s sections 8 through 16).

Many of the facilities offered by the scanner class are inherited from the ScannerBase base class. Additional facilities offered by the Scanner class are covered below. ✨

🎀 3. NAMING CONVENTION 🎀

All symbols that are required by the generated scanner class end in an underscore character (e.g., executeAction_). These names should not be redefined, okay? They're spoken for. As they are part of the Scanner and ScannerBase class, their scope is immediately clear and confusion with identically named identifiers elsewhere is unlikely. Ngl that's just good design.

Some member functions do not use the underscore convention. These are the scanner class’s constructors, or names that are similar or equal to names that have historically been used (e.g., length). Also, some functions are offered offering hooks into the implementation (like preCode). The latter category of function also have names that don’t end in underscores. So yes, the underscore thing is the rule but it also has exceptions — it's giving ✨ nuanced ✨.

💅 4. CONSTRUCTORS 💅

  • explicit Scanner(std::istream &in = std::cin, std::ostream &out = std::cout, bool keepCwd = true) — This constructor by default reads information from the standard input stream and writes to the standard output stream. When the Scanner object goes out of scope the input and output files are closed.

By default (keepCwd == true) the directory that was active when the scanner was constructed is made current again once the scanning process ends (i.e., when lex_(), see below, returns 0).

When keepCwd == false the scanner doesn’t reset its working directory to the directory that was current when the scanner was constructed. This may be convenient in situations where a program repeatedly calls switchStream(), followed by full lexical scans of the switched-to streams. Super useful when you're bouncing between streams fr fr.

With interactive scanners input stream switching or stacking is not available; switching output streams, however, is. So half the party is open 🫶

  • Scanner(std::string const &infile, std::string const &outfile, bool keepCwd = true) — This constructor opens the input and output streams whose file names were specified. When the Scanner object goes out of scope the input and output files are closed. If outfile == "-" then the standard output stream is used as the scanner’s output medium; if outfile == "" then the standard error stream is used as the scanner’s output medium.

The parameter keepCwd is used as with the previous constructor.

This constructor is not available with interactive scanners. Sad but true.

(more parts coming, don't go anywhere bestie 💖)

✨ 5. PUBLIC MEMBER FUNCTIONS ✨

  • int lex() — okay so this is THE main character function, bestie 🫶 lex does the lexical scanning of the input file that got specified at construction time (but also peek section 6.1. for the tea on intermediate stream-switching facilities, iykyk). It hands back an int that represents the token tied to the matched regular expression. If it returns 0, that's the signal for end-of-file, no more input, we're done 💀. Looking at its default implementation, it could be redefined by the user. Fun fact: lex's default implementation is literally just a lil call to lex_:
inline int Scanner::lex()
{
    return lex_();
}
⚠️ Caveat, and this one matters: with interactive scanners the lex function is defined in the generated lex.cc file. Once flexc++ has generated the scanner class header file, this scanner class header file isn't automatically rewritten by flexc++. So if, at some later stage, an interactive scanner must be generated, then that inline lex implementation has to be removed by hand from the scanner class header file 😭. And likewise, a lex member implementation (like the one above) must be provided by hand if a non-interactive scanner is required after first having generated files implementing an interactive scanner. Yes it's giving tedious. Yes it's real. Periodt.

🌸 6. PRIVATE MEMBER FUNCTIONS 🌸

  • int lex_() — this one is used internally by lex, and should not otherwise be used. Hands off, she's busy 💅
  • int executeAction_() — also used internally by lex, also should not otherwise be used. Same vibe, same rule 🙈
  • void preCode() — by default this function has an empty, inline implementation in Scanner.h. It can safely be replaced by a user-defined implementation 🎀. It gets called by lex_, right before it starts matching input characters against its rules: preCode is called by lex_ when lex_ is called, and also after having executed the actions of a rule which did not execute a return statement. The outline of lex_'s implementation looks like this:
int Scanner::lex_()
{
    ...
    preCode();

    while (true)
    {
        size_t ch = get_();            // fetch next char
        ...
        switch (actionType_(range))    // determine the action
        {
            ... maybe return
        }
        ... no return, continue scanning
        preCode();
    } // while
}
  • void postCode(PostEnum_ type) — by default this function has an empty, inline implementation in Scanner.h, and it can safely be replaced by a user-defined implementation 🦋. It gets called by lex_ just after a rule has been matched. The values of the enum class PostEnum_ tell you the characteristic of the matched rule. PostEnum_ has four values: PostEnum_::END, PostEnum_::POP, PostEnum_::RETURN, and PostEnum_::WIP. Slide over to section 10 for what each one actually means 👑
  • void print() — when the --print-tokens option or %print-tokens directive is used, this function gets called to display, on the standard output stream, the tokens returned and the text matched by the scanner generated by flexc++ ✨. Displaying is suppressed when the lex.cc file is (re)generated without using this directive. The function that actually shows the tokens (ScannerBase::print_) is called from print, which is defined in-line in Scanner.h. So calling ScannerBase::print_ can also super easily be controlled by an option controlled by the program that's using the scanner object 💖.

🎀 7. SCANNER CLASS HEADER EXAMPLE 🎀

Okay bestie, here's the whole thing laid out, character for character, because honestly seeing it is understanding it:

#ifndef Scanner_H_INCLUDED_
#define Scanner_H_INCLUDED_

// $insert baseclass_h
#include "Scannerbase.h"

// $insert classHead
class Scanner: public ScannerBase
{
    public:
        explicit Scanner(std::istream &in = std::cin,
                                std::ostream &out = std::cout);

        Scanner(std::string const &infile, std::string const &outfile);

        // $insert lexFunctionDecl
        int lex();

    private:
        int lex_();
        int executeAction_(size_t ruleNr);

        void print();
        void preCode();     // re-implement this function for code that must
                            // be exec’ed before the patternmatching starts

        void postCode(PostEnum_ type);
                            // re-implement this function for code that must
                            // be exec’ed after the rules’s actions.
};

// $insert scannerConstructors
inline Scanner::Scanner(std::istream &in, std::ostream &out)
:
    ScannerBase(in, out)
{}

inline Scanner::Scanner(std::string const &infile, std::string const &outfile)
:
    ScannerBase(infile, outfile)
{}

// $insert inlineLexFunction
inline int Scanner::lex()
{
    return lex_();
}

inline void Scanner::preCode()
{
    // optionally replace by your own code
}

inline void Scanner::postCode(PostEnum_ type)
{
    // optionally replace by your own code
}

inline void Scanner::print()
{
    print_();
}

#endif // Scanner_H_INCLUDED_

💜 8. THE SCANNER BASE CLASS 💜

By default, flexc++ generates a file Scannerbase.h containing the interface of the base class of the scanner class that flexc++ also generates 🫶. The name of the file that gets generated is easy to change using flexc++'s --baseclass-header option. In this man-page we're using the default name, for the sake of everyone's sanity.

The file Scannerbase.h is generated at each new flexc++ run. It contains no user-serviceable or extensible parts, so don't go poking around in there, she's not yours 😭. If you want to stop it from being rewritten, that's what flexc++'s --no-baseclass-header option is for 💅.

🦋 9. PUBLIC ENUMS AND -TYPES 🦋

  • enum class StartCondition_ — this strongly typed enumeration defines the names of the start conditions (aka the mini scanners 👑). It at least contains INITIAL, but when the %s or %x directives were used it also contains the identifiers of the mini scanners declared by those directives. And since StartCondition_ is a strongly typed enum, its values must be preceded by its enum name. Like, for example:
begin(StartCondition_::INITIAL);

🌈 10. PROTECTED ENUMS AND -TYPES 🌈

  • enum class ActionType_ — this strongly typed enumeration is for internal use only. Not for you, bestie 🙈
  • enum Leave_ — also internal use only. Same energy.
  • enum class PostEnum_ — values of this strongly typed enumeration get passed to the scanner's private member postCode, indicating the scanner's action after matching a rule. Here's what each value means:
  • PostEnum_::END: the function lex_ immediately returns 0 once postCode returns, indicating the end of the input was reached 🌸
  • PostEnum_::POP: the end of an input stream was reached, and processing continues with the previously pushed input stream. In this case the function lex_ doesn't return, it simply coontinues processing the previously pushed stream (yes, "coontinues" is what the original says, we respect the typo lore ✨)
  • PostEnum_::RETURN: the function lex_ immediately returns once postCode returns, returning the next token
  • PostEnum_::WIP: the function lex_ has matched a non-returning rule, and continues its rule-matching process

👑 11. NO PUBLIC CONSTRUCTORS 👑

There are no public constructors. Nope, zero, none, nada 💀. ScannerBase is a base class for the Scanner class generated by flexc++. ScannerBase only offers protected constructors. So you can't just go instantiating her directly, she's protected for a reason, bestie 🎀.

💖 12. PUBLIC MEMBER FUNCTIONS 💖

okay bestie, these are the ones you can actually call from outside. the whole public toolbox, no notes. 💅

  • std::string const &cwd() const — hands you the working directory that was chillin' as the active one back when the Scanner's constructor got called. it's ALSO the program's current working directory right after you construct a Scanner object, and when its lex member returns. so like, it's giving consistency. 🫶
  • bool debug() const — returns true if --debug or %debug was specified, otherwise false. that's it, that's the whole vibe check. ✨
  • bool interactiveLine() — bestie this one is only available with interactive scanners. what it does: literally all the remaining contents of the current interactive line buffer get yeeted, and the interactive line buffer gets refilled with the contents of the next input line. reach for this when you hit a condition that invalidates whatever's left on the line. after you call it, the next token your lexical scanner returns will be the first token on the next line. it returns true if the next line is available the next line is available and false otherwise. 🦋
  • std::string const &filename() const — returns the name of the file currently being processed by the scanner object. iconic. ✨
  • size_t length() const — returns the length of the text that was matched by lex. heads up: with flex++ this function was called leng. glow-up tbh. 👑
  • size_t lineNr() const — returns the line number of the currently scanned line. this one is always available (note: flex++ only offered a similar function, called lineno, after using the %lineno option). we just give it to you. no cap. 💜
  • size_t... no wait lmao, the next one:
  • std::string const &matched() const — returns the text matched by lex (note: flex++ offers a similar member called YYText). same energy, cuter name. 🌸
  • void setDebug(bool onOff) — switches debugging output on/off by passing the argument true or false. ⚠️ but real talk: switching debugging output on only has visible effects if the debug option was specified. so if you skipped the debug option, this is lowkey just vibes. 😭
  • void switchIstream(std::string const &infilename) — the currently processed input stream gets closed, and processing continues at the stream whose name you passed as the function's argument. 🚨 heads up: this is not a stack-operation — after processing infilename, processing does not return to the original stream. one way trip, bestie.

this member is not available with interactive scanners. 💀

  • void switchOstream(std::ostream &out) — the currently processed output stream is closed, and new output is written to out. simple, clean, slay. 🎀
  • void switchOstream(std::string const &outfilename) — the current output stream is closed, and output is written to outfilename. if this file already exists, it gets rewritten. fresh start energy. ✨
  • void switchStreams(std::istream &in, std::ostream &out = std::cout) — the currently processed input and output streams both get closed, and processing continues at in, writing output to out. 🚨 again: this is not a stack-operation — after processing in, processing does not return to the original stream.

this member is not available with interactive scanners. 🙈

  • void switchStreams(std::string const &infilename, std::string const &outfilename) — the currently processed input and output streams are closed, and processing continues at the stream whose name is given as the function's first argument, writing output to the file whose name is given as the function's second argument. that second file gets rewritten. and yes, once more with feeling: this is not a stack-operation — after processing infilename, processing does not return to the original stream.

if outfilename == "-" then the standard output stream is used as the scanner's output medium; if outfilename == "" then the standard error stream is used as the scanner's output medium.

okay and the original literally says that exact same thing AGAIN right after, and honestly? queen behavior. not gatekeeping a repeat 💅

If outfilename == "-" then the standard output stream is used as the scanner's output medium; if outfilename == "" then the standard error stream is used as the scanner's output medium.

this member is not available with interactive scanners. 🫶

🎀 13. PROTECTED CONSTRUCTORS 🎀

these are for the derived classes, iykyk. civilians stay out. ✋

  • ScannerBase(std::string const &infilename, std::string const &outfilename, bool keepCwd = true) — the scanner object opens and reads infilename, and opens (rewrites) and writes outfilename. it gets called from the corresponding Scanner constructor, so you don't summon it yourself. 🧚

by default (keepCwd == true) the directory that was active when ScannerBase was constructed is made current again once the scanning process ends — that's when Scanner::lex_() returns 0. 🌈

when keepCwd == false the scanner does not reset its working directory to the directory that was current when the ScannerBase was constructed. this can be convenient in situations where a program repeatedly calls switchStream(), followed by full lexical scans of the switched-to streams. main character optimization fr fr.

this constructor is not available for interactive scanners. 💀

  • ScannerBase(std::istream &in, std::ostream &out, bool keepCwd = true) — the in and out parameters are, respectively, the derived class constructor's input stream and output streams. 🦋

the parameter keepCwd is used exactly as with the previous constructor. ⭐

💖 14. PROTECTED MEMBER FUNCTIONS 💖

okay so bestie, heads up: every single member function whose name ends in an underscore is strictly for internal use. like, do not call them from your own user-defined members of the Scanner class, that's not your lane, it's giving private-but-technically-protected.

BUT — the ones below? those are safe for members of the generated Scanner class to call. slay. ✨

  • void accept(size_t nChars = 0) — accept(n) hands all but the first nChars characters of the current token back to the input stream, where they get rescanned once the scanner goes looking for its next match. So basically it matches nChars of the characters in the input buffer and rescans the rest. This function effectively sets length's return value to nChars (and ngl, a note for the flex++ girlies: with flex++ this function was called less).
  • void begin(StartCondition_ startCondition) — activates the regular expression rules associated with StartCondition_ startCondition. and because this enumeration is a strongly typed enum, you gotta spell out the StartCondition_ scope too. like so:
begin(StartCondition_::INITIAL);
  • void echo() const — the currently matched text (i.e., the text returned by the member matched) gets inserted into the scanner object's output stream. little echo, little vibe. 💅
  • void leave(int retValue) — okay so actions defined in the lexical scanner specification file may or may not return, and that frequently results in complicated or overlong compound statements that blur the readability of the spec file. By encapsulating the actions in a member function, readability is enhanced. however, frequently a compound statement is still required, as in:
regex-to-match  {
                    if (int ret = memberFunction())
                        return ret;
                }

The member leave removes the need for constructions like the above. leave can be called from within member functions encapsulating actions performed when a regular expression has been matched. It ends lex, returning retValue to its caller. The above rule can now be written like this:

regex-to-match  memberFunction();

and memberFunction could be implemented as follows:

void memberFunction()
{
    if (someCondition())
    {                           // any action, e.g.,
                                // switch mini-scanner
        begin(StartCondition_::INITIAL);

        leave(Parser::TOKENVALUE);    // lex returns TOKENVALUE
        // this point is never reached
    }

    pushStream(d_matched);      // switch to the next stream
                                // lex continues
}
⚠️ bestie READ THIS ⚠️ the member leave should only be called (indirectly) — usually nested — from actions defined in the scanner's specification s; calling leave outside of this context results in undefined behavior. 🫠 that's not a vibe, that's a crime scene.
  • void more() — the matched text is kept 💾 and will be prefixed to the text that gets matched at the next lexical scan.
  • std::ostream &out() — returns a reference to the scanner's output stream.
  • bool popStream() — closes the currently processed input stream. If the stream stack (cf. the member streamStack()) contains multiple elements, then the most recently pushed element is removed and processing continues with the then final element of the streamStack. If this switch was successfully performed, true is returned. If the stream stack contains only one element, then all information has been processed and false is returned. This member is called automatically at the end of the currently processed input stream. After completely processing the initial input file the member lex returns 0.

When the <<EOF>> pattern is used in the scanner's specification file, popStream is not automatically called. In that case it should explicitly be called in the <<EOF>> pattern's action block, returning 0 when it returns false. E.g.:

<<EOF>> {
            if (not popStream())
                return 0;
            cerr << "WIP on " << streamStack().size() <<
                "file(s)\n";
        }
  • void push(size_t ch) — character ch is pushed back onto the input stream. I.e., it will be the character that is retrieved at the next attempt to obtain a character from the input stream.
  • void push(std::string const &txt) — the characters in the string txt are pushed back onto the input stream. I.e., they will be the characters that are retrieved at the next attempt to obtain characters from the input stream. The characters in txt are retrieved from the first character to the last. So if txt == "hello" then the 'h' will be the character that's retrieved next, followed by 'e', etc, until 'o'. 🎀 order matters, she's a queue not a chaos gremlin.
  • void pushStream(std::istream &curStream) — this function pushes curStream on the stream stack.
🙈 this member is not available with interactive scanners.
  • void pushStream(std::string const &curName) — same, but the stream curName is opened first, and the resulting istream is pushed on the stream stack. If curName does not exist a std::exception is thrown. An exception is also thrown when the stream stack size exceeds (by default) 10 (the option max-depth can be used to change the default).
🙈 this member is not available with interactive scanners.
  • void redo(size_t nChars = 0) — this member acts like accept but its argument counts backward from the end of the matched text. All but these nChars characters are kept and the last nChar characters are rescanned. This function effectively reduces length's return value by nChars.
  • void setFilename(std::string const &name) — this function sets the name of the stream returned by filename to name.
  • void setMatched(std::string const &text) — this function stores text in the matched text buffer. Following a call to this function matched returns text.
  • StartCondition_ startCondition() const — returns the currently active start condition (mini scanner). 👑
  • std::vector<StreamStruct> const &streamStack() const — returns the vector of currently stacked input streams. Starting with flexc++ version 2.08.00 the size and content of the information in this vector has been changed. Starting with version 2.08.00 its size is at least 1. Its size is incremented at every pushStream call, and decremented at every poStream call.

The StreamStruct itself is a struct having only one documented member: std::string const &pushedName, containing the absolute pathname of the initial and pushed streams.

Internally, streamStack vector is used as a stack. The pathname of the file that was specified at the scanner's construction time is found at index position 0, and pathname of the file that's currently being processed is found at streamStack().back(). Switching input streams changes the pathname at streamStack().back(), and if the pathname of a stream cannot be determined (which happens when constructing a scanner from or switching to a std::istream) then the program's current working directory is not altered and pushedName contains (istream).

🙈 this member is not available with interactive scanners.

🧚 15. PROTECTED DATA MEMBERS 🧚

All protected data members are for internal use only, allowing lex_ to access them. All of them end in an underscore character. so like, d_-something-something, hands off bestie, that's her business 💅

🌸 16. FLEX++ TO FLEXC++ MEMBERS 🌸

for the girlies migrating over, here's your glow-up translation chart ✨ (old on the left, new on the right):

  • lineno() → lineNr()
  • YYText() → matched()
  • less() → accept()

like, same energy, new name, no cap. 🦋

🦋 17. THE CLASS INPUT 🦋

So flexc++ generates a file called Scannerbase.h, right? And that file defines your scanner class's base class — by default it's named ScannerBase (and bestie, that's the name we're gonna use all through this page, keep up 💅). Inside that base class lives a nested class called Input, and it has this exact interface:

class Input
{
    public:
        Input();
        Input(std::istream *iStream, size_t lineNr = 1);
        size_t get();
        size_t lineNr() const;
        size_t nPending() const;
        void setPending(size_t nPending);
        void reRead(size_t ch);
        void reRead(std::string const &str, size_t fmIdx);
        void close();
};

Every single member of this class is required — no skipping, no "eh I'll do it later" 🙈 — and together they offer an implementation level that sits between what ScannerBase is doing and the actual input file flexc++ is munching through.

And here's the chill part: by default, flexc++ already provides the implementation of all of Input's required members. So in most situations you can literally ignore this whole section. iconic behavior tbh ✨

But if you're feeling brave and wanna define and extend your own Input class, handing flexc++'s base class a custom one? Your rules file has to contain these two directives:

%input-inline         = "inline"
%input-interface      = "interface"
%input-implementation = "sourcefile"

(not me counting three lines after it said "two directives" 💀 anyway) Let's break it down:

  • inline — the name of a file holding the inline implementations of class Input's member functions.
  • interface — the name of a file holding class Input's interface.
  • sourcefile — where the non-inline implementations of class Input live.

Now, by default class Input is defined in ScannerBase's private section. But when you roll your own implementation, class Input gets declared and defined in ScannerBase's protected section instead — that way its members can also be reached by the derived class Scanner 🫶. And in that same situation, the (otherwise private) ScannerBase data member Input *d_input also gets declared in ScannerBase's protected section. team effort.

In the default implementation of class Input, some short one-line implementations of its members are defined inline right below the interface of class ScannerBase in the generated ScannerBase.h file. When you provide your own implementation, those members can still be defined inline — just put their implementations in a file (like "inputinline", for example) and point the %input-inline directive at it.

⚠️ When you're using the %input-inline directive, make sure the members you define are specified as inline Input:: class members inside class ScannerBase — and use the scanner class name you're actually using, okay?

So like, if the default scanner class is in play (peep the %class-name directive too), then the inline implementation of the member Input::lineNr could be:

inline size_t ScannerBase::Line::lineNr() const
{
    return d_lineNr;
}

And if you're using your own Input class but don't modify the default implementations of those inline members? Then just... use the defaults by simply not specifying the %input-inline directive. easy 💅

Okay bestie, listen up because this one's a real tip:

Avoid using standard extensions for the files you specify at the input-implementation, input-inline, and input-interface directives — it keeps the compiler from getting confused when you run program maintenance utilities. Why? Because the interface specifies the interface of class Input (like the one shown above), and that interface then gets inserted into ScannerBase's class as a nested class. So if you tuck those files into, say, a subdirectory input of the scanner's directory, and then specify something like:

%input-interface      = "input/interface"

...you nicely separate the interface and implementation of the scanner-class from the user-defined input class. chef's kiss 👑

The file sourcefile that you name at the %input-implementation directive contains the standard (non-inline) implementations of the members of your user-defined Input class.

And here's a rule you cannot break: all member functions of your self-defined class Input must be defined within class ScannerBase's class scope. For example, an implementation of the member Input::reRead(size_t ch) would use this mold:

void ScannerBase::Input::reRead(size_t ch)
{
    .... your implementation here
}

If your user-defined input class needs headers that class ScannerBase isn't already providing (sneak a peek at the top lines of a generated scannerbase.h file for a rundown of what's already #included), then drop the required #include specifications into the file named at the %baseclass-preinclude directive — or the corresponding option, whatever you vibe with. The generated scanner itself is self-supporting and doesn't need extra headers. 💖

Advice time 🎀 When you're implementing your own Input class, start out with the default class declaration and implementation as found in ScannerBase.h and lex.cc, and then just modify that default class into the class you actually need. Note that the interface of your class Input must at least offer the same members and constructors as the default provided class input — but it may totally define extra data members and/or member functions if your implementation needs them. (Also go read the next two sections for descriptions of class Input's constructors and required members, iykyk.)

Not me about to explain the stack, because this is genuinely the plot twist:

When the lexical scanner generated by flexc++ switches streams using the //include directive (see also section 2. FILE SWITCHING in the flexc++input(7) man page), the input stream that's currently being processed gets pushed onto an Input stack maintained by ScannerBase, and processing continues at the file named in the //include directive. Once that file is done, the previously pushed stream gets popped off the stack, and processing of the popped stream continues.

Which means... Input objects must be stack-able. Periodt. Its interface must always be designed in a way that satisfies this requirement 🦋

💜 18. INPUT CONSTRUCTORS 💜

  • Input() — the default constructor. ScannerBase uses it to prepare the stack for Input objects. It has to make sure a default (empty) Input object is in a valid state and can be destroyed. That's it, it serves no further purpose, no main character energy required. Input objects, however, must support the default (or overloaded) assignment operator. non-negotiable ✋
  • Input(std::istream *iStream, size_t lineNr = 1) — this constructor receives a pointer to a dynamically allocated istream object. The Input constructor should preserve this pointer when the Input object is pushed on and popped off the stack. A shared_ptr probably comes in handy here 👀. The Input object becomes the owner of the istream object — although its destructor is not supposed to destroy the istream object. Destroying stays the responsibility of the ScannerBase object, which calls the Input::close member (see below) when it's time to destroy (close) the stream.

The new input stream's line counter is set to lineNr, which by default is 1.

🌸 19. REQUIRED PUBLIC MEMBER FUNCTIONS 🌸

  • size_t get() — returns the next character to be processed by the lexical scanner. Usually it'll be the next character from the istream that was passed to the Input class at construction time. It is never called by the ScannerBase object for Input objects defined using Input's default constructor. It should return 0x100 once istream's end-of-file has been reached. 🏁
  • size_t lineNr() const — should return the (1-based) number of the istream object passed to the Input object. At construction time the istream has just been opened, so at that point lineNr should return 1. counting from one, always 💅
  • size_t nPending() const — should return the number of pending characters. That's the number of characters that were passed back to the Input object using its reRead members which were not yet retrieved again by its get member. lil queue of leftovers 🍓
  • void setPending(size_t nPending) — should remove nPending characters from the head of the Input object's pending input queue. The lexical scanner always passes the value received from nPending to setPending, without calling get in between. yes, they match. it's a whole thing.
  • void reRead(size_t ch) — if provided with a value smaller than 0x100, ch should be pushed back onto the istream, where it becomes the character next to be returned. Physically the character doesn't have to be pushed back though. The default implementation uses a deque onto which the character is pushed-front. Only when this deque is exhausted are characters retrieved from the Input object's istream. rewind era 🙈
  • void reRead(std::string const &str, size_t fmIdx) — the characters in str from fmIdx until the string's final character are pushed back onto the istream object so that the string's first character is retrieved first and the string's last character is retrieved last. order matters bestie ⭐
  • void close() — the istream object initially passed to the Input object is deleted by close, thereby not only freeing the stream's memory, but also closing the stream if the stream in fact was an ifstream. Note that Input's destructor should not destroy Input's istream object. clean up your own mess, diva ✨

🧚 FILES 🧚

Flexc++'s default skeleton files are in /usr/share/flexc++.

By default, flexc++ generates the following files:

  • Scanner.h: the header file containing the scanner class's interface.
  • Scannerbase.h: the header file containing the interface of the scanner class's base class.
  • Scanner.ih: the internal header file that's meant to be included by the scanner class's source files (like, it's included by lex.cc, see the next item's file 👀), and that should contain all declarations required for compiling the scanner class's sources.
  • lex.cc: the source file implementing the scanner class member function lex (and support functions), performing the lexical scan.

🌈 SEE ALSO 🌈

flexc++(1), flexc++input(7)

💀 BUGS 💀

  • Generating interactive and non-interactive scanners (see section 1. INTERACTIVE SCANNERS) cannot be mixed. pick a lane, respectfully.

This is free software, distributed under the terms of the GNU General Public License (GPL). 🫶

💖 AUTHOR 💖

Frank B. Brokken (f.b.brokken@rug.nl), Jean-Paul van Oosten (j.p.van.oosten@rug.nl), Richard Berendsen (richardberendsen@xs4all.nl) (until 2010).

and that's a wrap on the whole flexc++api(3) saga bestie, we ate 🎀 this page lives in flexc++ version 2.19.00, dated 2008-2026, and it's filed under flexc++api(3). go forth and slay your lexer, ily 🌸💜

📦 where this page comes from