ctags-optlib(7)

📦
version 6.2.1 · 3 versions
, in ubuntu resolute, ubuntu stonking

💖 NAME 💖

ctags-optlib — the Universal Ctags parser definition language, bestie. Yes, that's a whole language. Sit down. 🫶

🦋 SYNOPSIS 🦋

ctags [options] [file(s)]
etags [options] [file(s)]

✨ DESCRIPTION ✨

Okay so, history lesson real quick: Exuberant Ctags — the ancestor of Universal Ctags, the OG — gave you a way to define a whole new parser straight from the command line. Universal Ctags took that idea and refined it, slay. The result is what we call an optlib parser. "opt" because the parser gets defined with a combo of command line options, and "lib" because an optlib parser can be way more than just some ad-hoc personal config you threw together at 2am. 💅

This man page is for the girlies who want to define an optlib parser. But fr, read ctags(1) of Universal Ctags first. Required reading, no skipping. 😭

These options are the ones for defining (or customizing) a parser:

  • --langdef=<name>
  • --map-<LANG>=[+|-]<extension>|<pattern>
  • --kinddef-<LANG>=<letter>,<name>,<description>
  • --regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/[<flags>]
  • --mline-regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/{mgroup=<N>}[<flags>]

And these are the options for controlling the loading of parser definitions:

  • --options=<pathname>
  • --options-maybe=<pathname>
  • --optlib-dir=[+]<directory>

So here's the tea ☕ on the design philosophy. The way Exuberant Ctags designed its options and notation for defining a parser? It was all about reducing how much you have to type. Which makes sense — if you wanna define or customize a parser quickly, fewer keystrokes is a whole mood.

But Universal Ctags? Different priorities, bestie. Universal Ctags is all about maintainability. The notation is lowkey more redundant than Exuberant's, and that's on purpose:

  • a newly introduced kind has to be declared explicitly
  • (long) names are approved over those one-letter flags specifying kinds
  • the naming rules are stricter

This man page only covers stable options and flags, btw. Universal Ctags also has experimental options and flags whose names start with _. For the docs on those, go visit the Universal Ctags web site at <https://ctags.io/>. 🌈

🎀 Storing a parser definition to a file 🎀

Okay so, yeah, you can define a parser right from the command line, but bestie… you do not wanna be typing that same command line every single time you need the parser. Not the vibe. So: store your options for defining a parser in a file. Periodt.

ctags loads files (the preload files) listed in the "FILES" section of ctags(1) when the program starts up. So if there's a parser definition you need usually, just drop it in those files. Easy. 🫶

Then --options=<pathname>, --options-maybe=<pathname>, and --optlib-dir=[+]<directory> are for loading the optlib files you only need occasionally. Check the "Option File Options" section of ctags(1) for those.

And as explained in the "FILES" section of ctags(1), options for defining a parser get listed line by line in an optlib file. Prefixed white spaces? Ignored, don't worry about it. A line starting with # is treated as a comment. And ngl the best part — escaping shell meta characters is not needed. 🙈

Use .ctags as the file extension for an optlib file. You can totally define multiple parsers in one optlib file, but honestly? It's better to make a separate file for each parser definition. Cleaner. Main character energy. 👑

Oh and --_echo=<msg> and --_force-quit=<num> options are for debugging optlib parsers. 🔥

🌸 Overview for defining a parser 🌸

  1. Design the parser

You need to know both the target language and the ctags concepts — that's definition, reference, kind, role, field, extra. About those concepts, ctags(1) of Universal Ctags may help you. 💜

  1. Give a name to the parser

Use --langdef=<name>. That <name> gets referred to as <LANG> in all the later steps. Iykyk.

  1. Give a file pattern or file extension for activating the parser

Use --map-<LANG>=[+|-]<extension>|<pattern>.

  1. Define kinds

Use --kinddef-<LANG>=<letter>,<name>,<description>. Universal Ctags introduces this option — Exuberant Ctags doesn't have it. In Exuberant Ctags, a kind gets defined as a side effect of specifying the --regex-<LANG>= option, which means the user never gets a chance to recognize how important the definition of a kind actually is. 😭 Wild.

  1. Define patterns

Use --regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/[<flags>] for a single-line regular expression. You can also use --mline-regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/{mgroup=<N>}[<flags>] for a multi-line regular expression.

And as your <kind-spec>, you can use the one-letter flag defined with the --kinddef-<LANG>=<letter>,<name>,<description> option.

👑 OPTIONS 👑

💅 --langdef=<name>

Defines a new user-defined language, <name>, to be parsed with regular expressions. Once it's defined, <name> may be used in other options that take language names. Slay.

<name> must consist of alphanumeric characters, #, or + ([a-zA-Z0-9#+]+). The graph characters other than # and + are disallowed (or reserved). Some of them ([-=:{.]) are disallowed specifically because they can make the command line parser of ctags confused. The rest of them are just reserved for future extending of ctags. 🤷♀️

all is an exception. all as <name> is not acceptable. It's a reserved word. See the description of the --kinds-(<LANG>|all)=[+|-](<kinds>|*) option in ctags(1) about how that reserved word is used.

NONE is another exception. NONE as <name> is not acceptable. Nope.

The names of built-in parsers are capitalized. When ctags evaluates an option in a command line and chooses a parser, ctags uses the names of parsers in a case-insensitive way. So giving a name that starts with a lowercase character doesn't actually help you avoid a parser name conflict. However — in a tags file, ctags prints parser names in a case-sensitive way; it prints a parser name as specified in the --langdef=<name> option. So we recommend you give a name starting from a lowercase character to your private optlib parser. With that convention, people can tell whether a tag entry in a tag file came from a built-in parser or a private optlib parser. 💖

💅 --kinddef-<LANG>=<letter>,<name>,<description>

Define a kind for <LANG>. Do not get this confused with --kinds-<LANG>. Different things, bestie.

<letter> must be an alphabetical character ([a-zA-EG-Z]) other than "F". "F" has been reserved for representing a file since Exuberant Ctags. Periodt.

<name> must start with an alphabetic character, and the rest must be alphanumeric ([a-zA-Z][a-zA-Z0-9]*). Do not use "file" as <name>. It's been reserved for representing a file since Exuberant Ctags.

Heads up 🚨 — using a number character in a <name> violates the version 2 of tags file format, even though ctags accepts it. For more detail, see tags(5).

<description> comes from any printable ASCII characters. The exception is { and \. { is reserved for adding flags to this option in the future. So put \ before { to include { in a description. And to include \ itself in a description, put \ before \.

Both <letter>, <name>, and their combination must be unique in a <LANG>.

This option is newly introduced in Universal Ctags. It reduces the typing when defining a regex pattern with --regex-<LANG>=, and it keeps the consistency of kind definitions in a language. Iconic behavior, honestly. ✨

The <letter> can be used as an argument for the --kinds-<LANG> option to enable or disable the kind. Unless the K field is enabled, the <letter> is used as the value in the "kind" extension field in tags output.

The <name> surrounded by braces can be used as an argument for the --kind-<LANG> option. If the K field is enabled, the <name> is used as the value in the "kind" extension field in tags output.

The <description> and <letter> are listed in --list-kinds output. All three elements of the kind-spec are listed in --list-kinds-full output. And don't use braces in the <description> — they'll be used as meta characters in the future. 🫠

💅 --regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/[<flags>]

Define a single-line regular expression. Let's go.

The /<line_pattern>/<name_pattern>/ pair defines a regular expression replacement pattern, similar in style to sed substitution commands, s/regexp/replacement/, with which to generate tags from source files mapped to the named language, <LANG>, (case-insensitive; either a built-in or user-defined language).

The regular expression, <line_pattern>, defines an extended regular expression (roughly that used by egrep(1)), which is used to locate a single source line containing a tag, and it may specify tab characters using \t.

When a matching line is found, a tag will be generated for the name defined by <name_pattern>, which generally will contain the special back-references \1 through \9 to refer to matching sub-expression groups within <line_pattern>. 💫

The / separator characters shown in the parameter to the option can actually be replaced by any character. But note: whichever separator character you use will have to be escaped with a backslash (\) character wherever it's used in the parameter as something other than a separator. The regular expression defined by this option is added to the current list of regular expressions for the specified language unless the parameter is omitted — in which case the current list is cleared. 🙈

Unless modified by <flags>, <line_pattern> is interpreted as a POSIX extended regular expression. The <name_pattern> should expand for all matching lines to a non-empty string of characters, or a warning message will be reported unless the {placeholder} regex flag is specified.

A kind specifier (<kind-spec>) for tags matching regexp may follow <name_pattern>, which will determine what kind of tag is reported in the kind extension field (see tags(5)).

<kind-spec> has two forms: the one-letter form and the full form.

The one-letter form is in the form of <letter>. It just refers to a kind <letter> defined with --kinddef-<LANG>. This form is recommended in Universal Ctags. 👑

The full form of <kind-spec> is in the form of <letter>,<name>,<description>. Either the kind <name> and/or the <description> can be omitted. See the description of the --kinddef-<LANG>=<letter>,<name>,<description> option about the elements.

The full form is supported only for keeping compatibility with Exuberant Ctags, which does not have the --kinddef-<LANG> option. Supporting this form will be removed from Universal Ctags in the future. So maybe don't get too attached. 😭

...and that's only part 1, bestie. More coming. Stay tuned. 🎀

About <flags>, see "FLAGS FOR --regex-<LANG> OPTION". 👀

For more deets on the regular expressions ctags uses, bestie, go read either the regex(5,7) man page or the GNU info documentation for regex (like, info regex). 📖✨

  • --list-regex-flags — Lists all the flags you can use in the --regex-<LANG> option. 🎀
  • --list-mline-regex-flags — Lists all the flags you can use in the --mline-regex-<LANG> option. 🎀
  • --mline-regex-<LANG>=/<line_pattern>/<name_pattern>/<kind-spec>/{mgroup=<N>}[<flags>] — Defines a multi-line regular expression. It's basically the same vibe as the --regex-<LANG> option, except the pattern gets applied to the whole file's contents, not line by line. 🦋 See "FLAGS FOR --mline-regex-<LANG> OPTION" for the tea on {mgroup=<N>}. And heads up: {mgroup=<N>} flag is a must, no skips. 💅
  • --_echo=<message> — Prints <message> to the standard error stream. Super helpful for understanding (and debugging) the optlib loading feature of Universal Ctags. 🔍
  • --_force-quit[=<num>] — Exits immediately when this option is processed. If <num> is given, it's used as the exit status. The default is 0. Also helpful for debugging that optlib loading feature. 🚪💨

✨ FLAGS FOR --regex-<LANG> OPTION ✨

You can specify more than one flag, <letter>|{<name>}, at the end of --regex-<LANG> to control how Universal Ctags uses the pattern. 🎛️

Exuberant Ctags uses a <letter> to represent a flag. In Universal Ctags, a <name> surrounded by braces (the name form) can be used in addition to <letter>. The name form makes life easier for anyone reading an optlib file, fr fr. 🫶

Most of the flags newly added in Universal Ctags don't have a one-letter representation — they only have the name representation. --list-regex-flags lists all the flags. 📜

  • basic (one-letter form b) — The pattern is interpreted as a POSIX basic regular expression.
  • exclusive (one-letter form x) — Skip testing the other patterns if a line is matched to this pattern. Lowkey useful to avoid burning CPU on parsing line comments. 🔥
  • extend (one-letter form e) — The pattern is interpreted as a POSIX extended regular expression (this is the default).
  • pcre2 (one-letter form p, experimental) — The pattern is interpreted as a PCRE2 regular expression explained in pcre2syntax(3). This flag is available only if the ctags is built with the pcre2 library. Check the output of --list-features to know whether your ctags is built-with pcre2 or not. 🧪
  • icase (one-letter form i) — The regular expression is applied in a case-insensitive manner. 🙈
  • placeholder — Don't emit a tag captured with a regex pattern. The replacement can be an empty string. See the description of the scope=... flag below for how this is useful. 👻
  • postrun — Match the pattern at the end of all the parsing processes, including:
    • running the built-in code for <LANG>,
    • applying --mline-regex-<LANG> patterns,
    • applying --_mtable-regex-<LANG> patterns, and
    • applying non-postrun --regex-<LANG>= patterns.

This flag is helpful when combined with scope=intervaltab. 💖

The built-in code processes source files line-by-line delivered by the main part of ctags. The main part applies non-postrun --regex-<LANG>= patterns to a line just after delivering the line to the code of the built-in code. So basically, non-postrun --regex-<LANG>= patterns cannot refer to the tags information finally extracted by the built-in code. 😭

This is where the postrun comes into play. The main part never applies postrun --regex-<LANG>= patterns when delivering lines to the code of the built-in code. Instead, it applies the postrun patterns in batch after delivering all lines to the built-in code. That means the postrun patterns can refer to the tags information extracted by the built-in code. ✨

--mline-regex-<LANG> and --_mtable-regex-<LANG> have no {postrun} flag because the main part always applies the patterns specified with the options after running the built-in code for <LANG>.

See also the description of the scope=intervaltab flag. 👑

  • scope=intervaltab — Use the interval table maintained by the main part of ctags to fill in the scope: field. This flag is useful for extending a built-in parser with the --regex-<LANG>= option with the postrun flag. 🗂️

The interval table holds tag entries having both line: and end: fields. These tag entries are stored in a table keyed by their line: and end: field pairs. So the table can answer queries like, "Is there a tag entry that includes this line?" or "Which tag entry contains this line?" 🤔

The source line, where a postrun --regex-<LANG> pattern finds a language object, can be a key for such queries. The tag entry returned by the table gets set in the scope: field of the newly created tag entry for the language object. 💫

The postrun flag is needed for running the built-in parser that stores tag entries before applying patterns specified with --regex-<LANG>. 🌸

See also the example in "Using the interval table".

  • scope=(ref|push|pop|clear|set|replace) — Specify what to do with the internal scope stack. 🥞

A parser programmed with --regex-<LANG> has a stack (scope stack) internally. You can use it for tracking scope information. The scope=... flag is for manipulating and utilizing the scope stack.

If {scope=push} is specified, a tag captured with --regex-<LANG> is pushed to the stack. {scope=push} implies {scope=ref}.

You can fill the scope field (scope:) of a captured tag with {scope=ref}. If the {scope=ref} flag is given, ctags attaches the tag at the top to the tag captured with --regex-<LANG> as the value for the scope: field.

ctags pops the tag at the top of the stack when --regex-<LANG> with {scope=pop} is matched to the input line.

Specifying {scope=clear} removes all the tags in the scope. Specifying {scope=set} removes all the tags in the scope, and then pushes the captured tag as {scope=push} does.

{scope=replace} does the three things sequentially. First it does the same as {scope=pop}, then fills the scope: field of the tag captured with --regex-<LANG>, and pushes the tag to the scope stack as if {scope=push} was given finally. You cannot specify another scope action together with {scope=replace}. 🚫

You don't want to specify {scope=pop}{scope=push} as an alternative to {scope=replace}; {scope=pop}{scope=push} fills the scope: field of the tag captured with --regex-<LANG> first, then pops the tag at the top of the stack, and pushes the captured tag to the scope stack finally. The timing when filling the end field is different between {scope=replace} and {scope=pop}{scope=push}. ⏱️

In some cases, you may want to use --regex-<LANG> only for its side effects: using it only to manipulate the stack but not for capturing a tag. In such a case, make the <name_pattern> component of the --regex-<LANG> option empty while specifying {placeholder} as a regex flag. For example, a non-named tag can be put on the stack by giving a regex flag "{scope=push}{placeholder}". 🎭

You may wonder what happens if a regex pattern with the {scope=ref} flag matches an input line but the stack is empty, or a non-named tag is at the top. If the regex pattern contains a {scope=ref} flag and the stack is empty, the {scope=ref} flag is ignored and nothing is attached to the scope: field. 🙅

If the top of the stack contains an unnamed tag, ctags searches deeper into the stack to find the top-most named tag. If it reaches the bottom of the stack without finding a named tag, the {scope=ref} flag is ignored and nothing is attached to the scope: field.

When a named tag on the stack is popped or cleared as the side effect of a pattern matching, ctags attaches the line number of the match to the end: field of the named tag. 📍

ctags clears all of the tags on the stack when it reaches the end of the input source file. The line number of the end is attached to the end: field of the cleared tags. 🧹

  • warning=<message> — print the given <message> at WARNING level ⚠️
  • fatal=<message> — print the given <message> and exit 💀

🦋 FLAGS FOR --mline-regex-<LANG> OPTION 🦋

  • mgroup=<N> — decide the location of the tag extracted with the --mline-regex-<LANG> option. 🎯

<N> is the number of a capture group in the pattern, which is used to record the line number location of the tag. mgroup=<N> flag is not optional. You must add an mgroup=<N> flag, even if the <N> is 0 (which means the start position of the whole regex pattern). 💅

  • scope=intervaltab — See the description for this flag in "FLAGS FOR --regex-<LANG> OPTION". 🌈

Unlike the --regex-<LANG> option, you don't have to specify the postrun flag. 🫶

✨ EXAMPLES ✨

Bestie, this is the fun part — actual real-life glow-ups you can copy-paste 💅

🎀 Perl Pod

Okay so this is the definition (pod.ctags) that ctags itself uses for parsing Pod (<https://perldoc.perl.org/perlpod.html>) files. Iconic behavior tbh.

--langdef=pod
--map-pod=+.pod

--kinddef-pod=c,chapter,chapters
--kinddef-pod=s,section,sections
--kinddef-pod=S,subsection,subsections
--kinddef-pod=t,subsubsection,subsubsections

--regex-pod=/^=head1[ \t]+(.+)/\1/c/
--regex-pod=/^=head2[ \t]+(.+)/\1/s/
--regex-pod=/^=head3[ \t]+(.+)/\1/S/
--regex-pod=/^=head4[ \t]+(.+)/\1/t/

🦋 Using scope regex flags

okay so imagine we're writing a parser for a very small subset of the Ruby language. tiny. baby. adorable.

here's the input source file (input.srb):

class Example
  def methodA
        puts "in class_method"
  end
  def methodB
        puts "in class_method"
  end
end

so the parser for this input should capture Example with the class kind, plus methodA and methodB with the method kind. and methodA and methodB should have Example as their scope, no cap 🫶 and the end: fields of each tag should have proper values, periodt.

the optlib file (sub-ruby.ctags):

--langdef=subRuby
--map-subRuby=.srb
--kinddef-subRuby=c,class,classes
--kinddef-subRuby=m,method,methods
--regex-subRuby=/^class[ \t]+([a-zA-Z][a-zA-Z0-9]+)/\1/c/{scope=push}
--regex-subRuby=/^end///{scope=pop}{placeholder}
--regex-subRuby=/^[ \t]+def[ \t]+([a-zA-Z][a-zA-Z0-9_]+)/\1/m/{scope=push}
--regex-subRuby=/^[ \t]+end///{scope=pop}{placeholder}

and the command line plus its output:

$ ctags --quiet --fields=+eK \
--options=./sub-ruby.ctags -o - input.srb
Example input.srb       /^class Example$/;"     class   end:8
methodA input.srb       /^  def methodA$/;"     method  class:Example   end:4
methodB input.srb       /^  def methodB$/;"     method  class:Example   end:7

look at those scopes. look at those end: fields. ate and left no crumbs 💖

🌸 Using the interval table

now let's try to extract kernel threads defined in the Linux kernel. big girl stuff 🔥

kthread_run is the function for creating and starting a kernel thread. for example, the kswapd kernel thread is created with the function like this (quoted from linux/mm/vmscan.c):

void kswapd_run(int nid)
{
        /* ... */
        pgdat->kswapd = kthread_run(kswapd, pgdat, "kswapd%d", nid);
        /* ... */

kthread-v1.ctags illustrates the way to extract the name of kernel threads appearing at the third argument of kthread_run.

kthread-v1.ctags:

--kinddef-C=K,kernelThread,the name of kernel thread in Linux kernel
--regex-C=/kthread_run\([^"]+"([^"]+)"/\1/K/

with kthread-v1.ctags, ctags emits the following tags:

$ ctags --options=linux/kthread-v1.ctags  -o - vmscan.c
...
kswapd%d        vmscan.c        /^... = kthread_run(kswapd, pgdat, "kswapd%d"...$/;"    K
...

using the interval table, you can attach the name of the function where the kernel thread is defined. that's the glow-up right there ✨

kthread-v2.ctags:

--kinddef-C=K,kernelThread,the name of kernel thread in Linux kernel
--regex-C=/kthread_run\([^"]+"([^"]+)"/\1/K/{postrun}{scope=intervaltab}

with kthread-v2.ctags, ctags emits the following tags:

$ ctags --options=linux/kthread-v2.ctags  -o - vmscan.c
...
kswapd%d        vmscan.c        /^... = kthread_run(kswapd, pgdat, "kswapd%d"...$/;"    K       function:kswapd_run
...

with the new .ctags file, function:kswapd_run is attached to the tag entry as its scope field. she got a whole scope now, we love to see it 😭💖

👑 SEE ALSO 👑

the official Universal Ctags web site, at:

<https://ctags.io/>

also besties:

🌈 AUTHOR 🌈

Universal Ctags project <https://ctags.io/> (this man page is partially derived from ctags(1) of Executable-ctags)

Darren Hiebert <<dhiebert@users.sourceforge.net>> <http://DarrenHiebert.com/>

and that's a wrap on ctags-optlib(7), bestie 🫶 this whole page is from Universal Ctags 6.2.1 — go write some optlib files, you're gonna slay 💅✨

📦 where this page comes from