CTAGS-OPTLIB(7)

📦
version 6.2.1 · 9 versions
, in alpine 3.24, fedora 43, fedora 44

✨ CTAGS-OPTLIB(7) ✨

okay bestie, welcome to ctags-optlib — aka the Universal Ctags parser definition language 💅 this is the manual page for the whole ctags-optlib thing, and yes it's a big one so buckle up 🎀

🌸 SYNOPSIS 🌸

ctags [options] [file(s)]
etags [options] [file(s)]

🦋 DESCRIPTION 🦋

So Exuberant Ctags, which is literally the ancestor of Universal Ctags, already let you define a whole new parser straight from the command line. Universal Ctags took that idea and extended and refined it ✨. In Universal Ctags we call such a parser an optlib parser — "opt" because the parser gets defined by a combo of command line options, and "lib" because an optlib parser can be way more than just an ad-hoc personal config, it's a whole library vibe 💖.

This man page is for the girlies who want to define an optlib parser. But fr, read ctags(1) of Universal Ctags first, no skipping 🫶.

These options are the ones for defining (or customizing) a parser:

  • --langdef=<name>
  • --map-<LANG>=[+|-]<extension>|<pattern>
  • --kinddef-<LANG>=<letter>,<name>,<description>
  • --regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/[<flags>]
  • --mline-regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/{mgroup=<N>}[<flags>]

And these options are for controlling the loading of parser definitions:

  • --options=<pathname>
  • --options-maybe=<pathname>
  • --optlib-dir=[+]<directory>

The design of options and notations for defining a parser in Exuberant Ctags was lowkey all about reducing how much you have to type. And ngl that's important when you wanna define (or customize) a parser fast ⚡.

Universal Ctags said "not me making that tradeoff" and focused on maintainability instead. The notation in Universal Ctags is more redundant than Exuberant Ctags', bestie — the newly introduced kind has to be declared explicitly, (long) names are approved over one-letter flags for specifying kinds, and the naming rules are stricter 😤. it's giving structure.

This man page only explains stable options and flags. Universal Ctags also has experimental options and flags whose names start with _. For docs on those, go visit the Universal Ctags web site at <https://ctags.io/> 🌈.

📁 Storing a parser definition to a file

Yes you can define a parser from the command line, but bestie, you do not wanna type that same command line every single time you need the parser 💀. So store the options for defining a parser into a file instead.

ctags loads files (the preload files) listed in the "FILES" section of ctags(1) at program startup. You can put your usually-needed parser definition in those files 🎀.

--options=<pathname>, --options-maybe=<pathname>, and --optlib-dir=[+]<directory> are for loading the optlib files you only need occasionally. Check the "Option File Options" section of ctags(1) for these options, iykyk ✨.

As explained in the "FILES" section of ctags(1), options for defining a parser get listed line by line in an optlib file. Prefixed white spaces are ignored 🙈. A line starting with # is treated as a comment. And escaping shell meta characters? not needed, periodt 💅.

Use .ctags as the file extension for an optlib file. You can define multiple parsers in one optlib file, but it's better to make one file per parser definition (organization is self-care, fr fr 🌸).

The --_echo=<msg> and --_force-quit=<num> options are for debugging optlib parsers 🐛.

🧚 Overview for defining a parser

  1. Design the parser

You need to know both the target language and the ctags concepts (definition, reference, kind, role, field, extra). About those concepts, ctags(1) of Universal Ctags may help you 💖.

  1. Give a name to the parser

Use the --langdef=<name> option. That <name> gets referred to as <LANG> in all the later steps, got it? ✨

  1. Give a file pattern or file extension for activating the parser

Use --map-<LANG>=[+|-]<extension>|<pattern>.

  1. Define kinds

Use the --kinddef-<LANG>=<letter>,<name>,<description> option. Universal Ctags introduces this option — Exuberant Ctags doesn't have it 😭. In Exuberant Ctags, a kind is defined as a side effect of specifying the --regex-<LANG>= option, so the user never even gets a chance to recognize how important the definition of a kind actually is. tragic, honestly.

  1. Define patterns

Use --regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/[<flags>] for a single-line regular expression. You can also use --mline-regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/{mgroup=<N>}[<flags>] for a multi-line regular expression 🔥.

As <kind-spec>, you can use the one-letter flag defined with the --kinddef-<LANG>=<letter>,<name>,<description> option.

👑 OPTIONS 👑

💜 --langdef=<name>

Defines a new user-defined language, <name>, to be parsed with regular expressions. Once it's defined, <name> may be used in other options that take language names ✨.

<name> must consist of alphanumeric characters, #, or + ([a-zA-Z0-9#+]+). The graph characters other than # and + are disallowed (or reserved). Some of them ([-=:{.]) are disallowed because they can make the command line parser of ctags confused 😵. The rest of them are just reserved for future extending of ctags.

all is an exception. all as <name> is not acceptable — it's a reserved word. See the description of the --kinds-(<LANG>|all)=[+|-](<kinds>|*) option in ctags(1) about how that reserved word gets used.

NONE is another exception. NONE as <name> is not acceptable either.

The names of built-in parsers are capitalized. When ctags evaluates an option in a command line and chooses a parser, ctags uses the names of parsers in a case-insensitive way. So giving a name that starts with a lowercase character does not help you avoid parser name confliction 😔. However, in a tags file, ctags prints parser names in a case-sensitive way; it prints a parser name exactly as specified in the --langdef=<name> option. So we recommend you give a name starting with a lowercase character to your private optlib parser 💅. With this convention, people can tell whether a tag entry in a tag file comes from a built-in parser or a private optlib parser. iconic.

🎀 --kinddef-<LANG>=<letter>,<name>,<description>

Define a kind for <LANG>. And do not get this confused with --kinds-<LANG> bestie, they are different things 🙈.

<letter> must be an alphabetical character ([a-zA-EG-Z]) other than "F". "F" has been reserved for representing a file since Exuberant Ctags.

<name> must start with an alphabetic character, and the rest must be alphanumeric ([a-zA-Z][a-zA-Z0-9]*). Do not use "file" as <name>. It's been reserved for representing a file since Exuberant Ctags.

Note that using a number character in a <name> violates the version 2 of tags file format, even though ctags accepts it. For more detail, see tags(5).

<description> comes from any printable ASCII characters. The exception is { and \. { is reserved for adding flags to this option in the future. So put \ before { to include { in a description. To include \ itself in a description, put \ before \.

Both <letter>, <name> and their combination must be unique in a <LANG>.

This option is newly introduced in Universal Ctags ✨. It reduces the typing when defining a regex pattern with --regex-<LANG>=, and it keeps the consistency of kind definitions within a language. slay.

The <letter> can be used as an argument for the --kinds-<LANG> option to enable or disable the kind. Unless the K field is enabled, the <letter> is used as the value in the "kind" extension field in tags output 💖.

The <name> surrounded by braces can be used as an argument for the --kind-<LANG> option. If the K field is enabled, the <name> is used as the value in the "kind" extension field in tags output ✨.

The <description> and <letter> are listed in --list-kinds output. All three elements of the kind-spec are listed in --list-kinds-full output. Don't use braces in the <description> — they'll be used as meta characters in the future 🌸.

🌈 --regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/[<flags>]

Define a single-line regular expression.

The /<line_pattern>/<name_pattern>/ pair defines a regular expression replacement pattern, similar in style to sed substitution commands, s/regexp/replacement/, with which to generate tags from source files mapped to the named language, <LANG> (case-insensitive; either a built-in or user-defined language) 💅.

The regular expression, <line_pattern>, defines an extended regular expression (roughly that used by egrep(1)), which is used to locate a single source line containing a tag, and it may specify tab characters using \t.

When a matching line is found, a tag will be generated for the name defined by <name_pattern>, which generally will contain the special back-references \1 through \9 to refer to matching sub-expression groups within <line_pattern> ✨.

The / separator characters shown in the parameter to the option can actually be replaced by any character. Note that whichever separator character you use will have to be escaped with a backslash (\) character wherever it's used in the parameter as something other than a separator 😤. The regular expression defined by this option is added to the current list of regular expressions for the specified language, unless the parameter is omitted — in which case the current list is cleared.

Unless modified by <flags>, <line_pattern> is interpreted as a POSIX extended regular expression. The <name_pattern> should expand for all matching lines to a non-empty string of characters, or a warning message will be reported — unless the {placeholder} regex flag is specified 🙈.

A kind specifier (<kind-spec>) for tags matching regexp may follow <name_pattern>, which will determine what kind of tag is reported in the kind extension field (see tags(5)).

<kind-spec> has two forms: the one-letter form and the full form.

The one-letter form is in the form of <letter>. It just refers to a kind <letter> defined with --kinddef-<LANG>. This form is recommended in Universal Ctags 💖.

The full form of <kind-spec> is in the form of <letter>,<name>,<description>. Either the kind <name> and/or the <description> can be omitted. See the description of the --kinddef-<LANG>=<letter>,<name>,<description> option about the elements ✨.

The full form is supported only for keeping compatibility with Exuberant Ctags, which does not have the --kinddef-<LANG> option. Supporting the form will be removed from Universal Ctags in the future ⚠️.

to be continued bestie, there's more where this came from 💖

okay so the <flags> situation has its own whole section, bestie — full tea in FLAGS FOR --regex-<LANG> OPTION, scroll down a lil. 👇

And if you wanna go absolutely feral on regex lore, ctags is working with the regexes described in the regex(5,7) man page, or the GNU info documentation for regex (like info regex). ✨

  • --list-regex-flags — dumps every single flag you're allowed to use inside the --regex-<LANG> option. window shopping but make it regex 💅
  • --list-mline-regex-flags — same energy, but it lists the flags for the --mline-regex-<LANG> option instead.
  • --mline-regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/{mgroup=<N>}[<flags>] — defines a multi-line regular expression. 🌈

It's basically the --regex-<LANG> option's sibling, except the pattern gets applied to the whole file's contents instead of going line by line. no cap, whole file.

Peep FLAGS FOR --mline-regex-<LANG> OPTION for the {mgroup=<N>} tea. and heads up: {mgroup=<N>} is mandatory, not optional, don't forget it 🫶

  • --_echo=<message> — prints <message> straight to the standard error stream. super handy for understanding (and debugging) the optlib loading feature of Universal Ctags. lowkey a lil debug bestie 🐛
  • --_force-quit[=<num>] — bails immediately when this option gets processed. if you pass <num>, that becomes the exit status. default is 0. also just here to help you debug the optlib loading feature of Universal Ctags. dramatic exit, we love her 💀

🎀 FLAGS FOR --regex-<LANG> OPTION 🎀

so you can stack more than one flag — <letter>|{<name>} — at the end of --regex-<LANG> to control exactly how Universal Ctags uses your pattern. main character behaviour honestly 👑

Exuberant Ctags was giving minimalist and used a <letter> for a flag. Universal Ctags said "let's make it readable" — so now a <name> wrapped in braces (the name form) works too, on top of <letter>. the name form just makes life easier for whoever's reading an optlib file later. considerate fr fr 💖

and here's the thing: most of the new flags added in Universal Ctags don't have a one-letter form at all. they're name-representation only. --list-regex-flags shows you the whole roster. 📋

  • basic (one-letter form b) — the pattern is interpreted as a POSIX basic regular expression.
  • exclusive (one-letter form x) — if a line matches this pattern, skip testing all the other patterns. great for dodging wasted CPU when you're parsing line comments. efficient queen ⚡
  • extend (one-letter form e) — the pattern is interpreted as a POSIX extended regular expression. this is the default.
  • pcre2 (one-letter form p, experimental) — the pattern is interpreted as a PCRE2 regular expression, as explained in pcre2syntax(3). ⚠️ only available if your ctags was built with the pcre2 library. check the output of --list-features to find out whether your build is built-with pcre2 or not.
  • icase (one-letter form i) — the regular expression gets applied in a case-insensitive manner. lowercase uppercase, doesn't care 💅
  • placeholder — don't emit a tag captured with a regex pattern. the replacement can be an empty string. peep the scope=... description below to see why this is so useful. it's giving secret agent 🕵️
  • postrun — match the pattern at the end of all the parsing processes, which includes:
    • running the built-in code for <LANG>,
    • applying --mline-regex-<LANG> patterns,
    • applying --_mtable-regex-<LANG> patterns, and
    • applying non-postrun --regex-<LANG>= patterns.

this flag is a whole vibe when you combine it with scope=intervaltab.

here's the lore: the built-in code processes source files line-by-line, delivered by the main part of ctags. the main part applies non-postrun --regex-<LANG>= patterns to a line just after handing that line to the built-in code. so non-postrun --regex-<LANG>= patterns literally cannot refer to the tags info that the built-in code finally extracts. 😭

and this is where postrun comes in to save the day. the main part never applies postrun --regex-<LANG>= patterns while delivering lines to the built-in code. instead, it applies the postrun patterns in batch after all lines have been delivered to the built-in code. so postrun patterns can refer to the tags information extracted by the built-in code. slay ✨

--mline-regex-<LANG> and --_mtable-regex-<LANG> have no {postrun} flag, because the main part always applies the patterns specified with those options after running the built-in code for <LANG>. built different.

also see the description of the scope=intervaltab flag. 👀

  • scope=intervaltab — use the interval table maintained by the main part of ctags to fill in the scope: field. this flag is the moment for extending a built-in parser with the --regex-<LANG>= option plus the postrun flag. 🫶

the interval table holds tag entries that have both line: and end: fields. those tag entries are stored in a table keyed by their line: and end: field pairs. so the table can answer queries like "Is there a tag entry that includes this line?" or "Which tag entry contains this line?" — it's giving oracle 🔮

the source line, where the postrun --regex-<LANG> pattern finds a language object, can be a key for those queries. the tag entry the table hands back gets set in the scope: field of the newly created tag entry for the language object.

you need the postrun flag so the built-in parser actually runs and stores those tag entries before the patterns specified with --regex-<LANG> get applied.

peep the example in "Using the interval table" too. 📖

  • scope=(ref|push|pop|clear|set|replace) — specifies what to do with the internal scope stack. let's get into it 🎀

a parser programmed with --regex-<LANG> has a stack (the scope stack) internally. you can use it for tracking scope information. the scope=... flag is what manipulates and utilizes that scope stack.

if {scope=push} is specified, a tag captured with --regex-<LANG> gets pushed onto the stack. and {scope=push} implies {scope=ref}. two for one.

you can fill the scope field (scope:) of a captured tag with {scope=ref}. if the {scope=ref} flag is given, ctags attaches the tag at the top to the tag captured with --regex-<LANG> as the value for the scope: field.

ctags pops the tag at the top of the stack when --regex-<LANG> with {scope=pop} matches the input line.

specifying {scope=clear} removes all the tags in the scope. specifying {scope=set} removes all the tags in the scope, and then pushes the captured tag just like {scope=push} does.

{scope=replace} does three things in sequence: first it does the same as {scope=pop}, then it fills the scope: field of the tag captured with --regex-<LANG>, and finally it pushes that tag onto the scope stack as if {scope=push} had been given. ⚠️ you cannot specify another scope action together with {scope=replace}.

and bestie, you do not want to specify {scope=pop}{scope=push} as an alternative to {scope=replace}. here's why: {scope=pop}{scope=push} fills the scope: field of the tag captured with --regex-<LANG> first, then pops the tag at the top of the stack, and pushes the captured tag onto the scope stack finally. the timing of when the end field gets filled is different between {scope=replace} and {scope=pop}{scope=push}. subtle but it matters 💀

sometimes you only want --regex-<LANG> for its side effects — using it purely to manipulate the stack and not to capture a tag at all. in that case, make the <name_pattern> component of the --regex-<LANG> option empty while specifying {placeholder} as a regex flag. for example, a non-named tag can be put on the stack by giving a regex flag "{scope=push}{placeholder}". sneaky, iconic.

you might be wondering what happens if a regex pattern with the {scope=ref} flag matches an input line but the stack is empty, or a non-named tag is sitting at the top. okay so: if the regex pattern contains a {scope=ref} flag and the stack is empty, the {scope=ref} flag is ignored and nothing gets attached to the scope: field.

if the top of the stack holds an unnamed tag, ctags searches deeper into the stack to find the top-most named tag. and if it hits the bottom of the stack without finding a named tag, the {scope=ref} flag is ignored and nothing gets attached to the scope: field. it just gives up gracefully 🙈

when a named tag on the stack is popped or cleared as the side effect of a pattern matching, ctags attaches the line number of the match to the end: field of that named tag.

and ctags clears all of the tags on the stack when it reaches the end of the input source file. the line number of the end gets attached to the end: field of the cleared tags. cleanup era 🧹

  • warning=<message> — prints the given <message> at WARNING level. ⚠️
  • fatal=<message> — prints the given <message> and exits. 💀

🌸 FLAGS FOR --mline-regex-<LANG> OPTION 🌸

  • mgroup=<N> — decides the location of the tag extracted with the --mline-regex-<LANG> option. 📍

<N> is the number of a capture group in the pattern, and it's what gets used to record the line number location of the tag. ⚠️ mgroup=<N> is not optional — you must add an mgroup=<N> flag, even if <N> is 0 (which means the start position of the whole regex pattern). yes, even zero. no skipping 🫶

  • scope=intervaltab — see the description of this flag back in FLAGS FOR --regex-<LANG> OPTION.

but unlike the --regex-<LANG> option, you don't have to specify the postrun flag here. one less thing to remember, we love that for you ✨

💅 EXAMPLES 💅

Okay bestie, this is the fun part — real-life glow-ups, actual .ctags files you can steal, and receipts of what ctags spits back out. Let's go. ✨

🌸 Perl Pod 🌸

This right here is the definition (pod.ctags) that gets used in ctags for parsing Pod files (perldoc.perl.org/perlpod.html). Look at her. She's beautiful. She's four kinddefs and four regexes and she's thriving.

--langdef=pod
--map-pod=+.pod

--kinddef-pod=c,chapter,chapters
--kinddef-pod=s,section,sections
--kinddef-pod=S,subsection,subsections
--kinddef-pod=t,subsubsection,subsubsections

--regex-pod=/^=head1[ \t]+(.+)/\1/c/
--regex-pod=/^=head2[ \t]+(.+)/\1/s/
--regex-pod=/^=head3[ \t]+(.+)/\1/S/
--regex-pod=/^=head4[ \t]+(.+)/\1/t/

Four heading levels, four kinds, one vibe. 💖

🦋 Using scope regex flags 🦋

So like, imagine we're writing a parser for a very tiny subset of Ruby. Baby Ruby. Ruby jr. Let's set it up.

the input source file (input.srb):

class Example
  def methodA
        puts "in class_method"
  end
  def methodB
        puts "in class_method"
  end
end

what we want from it

The parser should capture Example with the class kind, plus methodA and methodB with the method kind. And here's the tea: methodA and methodB should have Example as their scope. Also the end: fields of each tag should have proper values. No slacking on the fields, periodt. 💅

the optlib file (sub-ruby.ctags):

--langdef=subRuby
--map-subRuby=.srb
--kinddef-subRuby=c,class,classes
--kinddef-subRuby=m,method,methods
--regex-subRuby=/^class[ \t]+([a-zA-Z][a-zA-Z0-9]+)/\1/c/{scope=push}
--regex-subRuby=/^end///{scope=pop}{placeholder}
--regex-subRuby=/^[ \t]+def[ \t]+([a-zA-Z][a-zA-Z0-9_]+)/\1/m/{scope=push}
--regex-subRuby=/^[ \t]+end///{scope=pop}{placeholder}

command line and output:

$ ctags --quiet --fields=+eK \
--options=./sub-ruby.ctags -o - input.srb
Example input.srb       /^class Example$/;"     class   end:8
methodA input.srb       /^  def methodA$/;"     method  class:Example   end:4
methodB input.srb       /^  def methodB$/;"     method  class:Example   end:7

See how class:Example shows up on both methods? That's the scope doing its little dance. 🩰 The scope=push opens a scope, scope=pop closes it, and {placeholder} means the end lines don't get their own tag. Ate. 🔥

👑 Using the interval table 👑

Now let's extract kernel threads defined in the Linux kernel. Big girl territory, but we got this.

kthread_run is the function for creating and starting a kernel thread. For example, the kswapd kernel thread is created with the function like this (quoted from linux/mm/vmscan.c):

void kswapd_run(int nid)
{
                /* ... */
                pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
                /* ... */

kthread-v1.ctags illustrates the way to extract the name of kernel threads appearing at the third argument of kthread_run.

kthread-v1.ctags:

--kinddef-C=K,kernelThread,the name of kernel thread in Linux kernel
--regex-C=/kthread_run\([^"]+"([^"]+)"/\1/K/

With kthread-v1.ctags, ctags emits the following tags:

$ ctags --options=linux/kthread-v1.ctags  -o - vmscan.c
...
kswapd%d        vmscan.c        /^... = kthread_run(kswapd, pgdat, "kswapd%d"...$/;"    K
...

Hmm okay but lowkey we want more. Using the interval table, you can attach the name of the function where the kernel thread is defined. Main character upgrade. ✨

kthread-v2.ctags:

--kinddef-C=K,kernelThread,the name of kernel thread in Linux kernel
--regex-C=/kthread_run\([^"]+"([^"]+)"/\1/K/{postrun}{scope=intervaltab}

With kthread-v2.ctags, ctags emits the following tags:

$ ctags --options=linux/kthread-v2.ctags  -o - vmscan.c
...
kswapd%d        vmscan.c        /^... = kthread_run(kswapd, pgdat, "kswapd%d"...$/;"    K       function:kswapd_run
...

With the new .ctags file, function:kswapd_run is attached to the tag entry as its scope field. That's it, that's the glow-up. 💖🦋

🔎 SEE ALSO 🔎

The official Universal Ctags web site at:

https://ctags.io/

Also go read these, bestie, they're the whole friend group:

🫶 AUTHOR 🫶

Universal Ctags project https://ctags.io/ (This man page partially derived from ctags(1) of Executable-ctags)

Darren Hiebert < dhiebert@users.sourceforge.net > http://DarrenHiebert.com/

And that's a wrap on this whole thing, babe. 🌈 This page is version 6.2.1, filed under CTAGS-OPTLIB(7) — so if you ever need to cite it, that's the number, and no cap, you're gonna be so good at optlib now. Go write a parser. Slay. 👑💜

📦 where this page comes from