BTPARSE(1)

📦
version 0.91 · 7 versions
, in alpine 3.24

🎀 NAME 🎀

okay so bestie, welcome to btparse(1) — it's a C library for parsing and processing BibTeX data files 💖. that's the whole vibe. one job. iconic.

✨ SYNOPSIS ✨

like, here's the whole menu of what you can call, fr fr 👑 (copy-paste these exactly, i'm not kidding, character for character):

#include <btparse.h>

/* Basic library initialization / cleanup */
void bt_initialize (void);
void bt_free_ast (AST *ast);
void bt_cleanup (void);

/* Input / interface to parser */
void   bt_set_stringopts (bt_metatype_t metatype, btshort options);
AST * bt_parse_entry_s (char *    entry_text,
                        char *    filename,
                        int       line,
                        btshort    options,
                        boolean * status);
AST * bt_parse_entry   (FILE *    infile,
                        char *    filename,
                        btshort    options,
                        boolean * status);
AST * bt_parse_file    (char *    filename,
                        btshort    options,
                        boolean * overall_status);

/* AST traversal/query */
AST * bt_next_entry (AST * entry_list,
                     AST * prev_entry)
AST * bt_next_field (AST *entry, AST *prev, char **name);
AST * bt_next_value (AST *head,
                     AST *prev,
                     bt_nodetype_t *nodetype,
                     char **text);

bt_metatype_t bt_entry_metatype (AST *entry);
char *bt_entry_type (AST *entry);
char *bt_entry_key (AST *entry);
char *bt_get_text (AST *node);

/* Splitting names and lists of names */
bt_stringlist * bt_split_list (char *   string,
                               char *   delim,
                               char *   filename,
                               int      line,
                               char *   description);
void bt_free_list (bt_stringlist *list);
bt_name * bt_split_name (char *  name,
                         char *  filename,
                         int     line,
                         int     name_num);
void bt_free_name (bt_name * name);

/* Formatting names */
bt_name_format * bt_create_name_format (char * parts, boolean abbrev_first);
void bt_free_name_format (bt_name_format * format);
void bt_set_format_text (bt_name_format * format,
                         bt_namepart part,
                         char * pre_part,
                         char * post_part,
                         char * pre_token,
                         char * post_token);
void bt_set_format_options (bt_name_format * format,
                            bt_namepart part,
                            boolean abbrev,
                            bt_joinmethod join_tokens,
                            bt_joinmethod join_part);
char * bt_format_name (bt_name * name, bt_name_format * format);

/* Construct tree from TeX groups */
bt_tex_tree * bt_build_tex_tree (char * string);
void          bt_free_tex_tree (bt_tex_tree **top);
void          bt_dump_tex_tree (bt_tex_tree *node, int depth, FILE *stream);
char *        bt_flatten_tex_tree (bt_tex_tree *top);

/* Miscellaneous string utilities */
void bt_purify_string (char * string, btshort options);
void bt_change_case (char transform, char * string, btshort options);

🌸 DESCRIPTION 🌸

okay so the tea ☕: btparse is a C library for parsing and processing BibTeX files. that's the whole main character energy of this page. 💅

here's what it's actually packing, and honestly? it ate:

  • a lexical scanner and LR parser (constructed by PCCTS) — both of which are efficient and offer good error detection and recovery 🔥. like, it doesn't just fall apart the second something's weird, it recovers. we love a resilient icon.
  • a set of functions for traversing the AST (abstract syntax tree) that the parser generates 🦋
  • utility functions for manipulating strings according to BibTeX conventions ✨

and bestie, an important side note: nothing in the library assumes that you're using BibTeX files for their original purpose of bibliographic data for scholarly publications 🙈. you could use the file format for any conceivable purpose that fits it. go off, honestly. use it for whatever.

that said 👀 — there's some code in the library that's really only appropriate for use with strings meant to be processed in the same way that BibTeX itself does. but it's all entirely optional, so no pressure, babes. 🫶

one more thing ngl — the interface provided by btparse, while complete, is fairly low-level. so if you have more sophisticated needs, you might be interested in my Text::BibTeX module for Perl 5 (available on CPAN) 👑💖. iykyk.

🌸 CONCEPTS AND TERMINOLOGY 🌸

Okay bestie, before we go any further — to actually understand this doc and use btparse properly, you already need to be fluent in the BibTeX language. Like, specifically the BibTeX data description language. 💅

Real talk about "BibTeX" as a word: BibTeX is a whole complex beast, and honestly you could argue the term covers the program, the data language, the particular database structure described in the original BibTeX docs, the ".bst" formatting language, and the whole set of conventions baked into the standard styles that ship with the BibTeX distribution. In this document I'm only ever talking about the first two meanings — the data language, because that's literally what btparse deals with, and the program, because sometimes I gotta explain where my parser and BibTeX's parser disagree. 🙈

So you should be comfy with stuff like this:

@string{and = { and },
        joe = "Blow, Joe",
        john = "John Smith"}

@book(ourbook,
      author = joe # and # john,
      title = {Our Little Book})

If that looks like something you want to parse but you do NOT want to write your own parser for it — congrats, you're in exactly the right place. 🫶

🦋 the terminology deep-dive (yes, we're doing this) 🦋

Before we go much further you're gonna need my vocab for describing BibTeX data. Most of it matches what you'd find in any BibTeX documentation, but like — we have to be sure we're talking about the same things, okay? So. Definitions, bestie:

  • top-level — all the text in a BibTeX file from the start of the file up to the start of the first entry, and between entries after that.
  • name — a string of letters, digits, and these characters:
! $ & * + - . / : ; < > ? [ ] ^ _ ` |

"Name" is the catch-all term I use for entry types, entry keys, and field and macro names. For BibTeX compatibility there are slightly different rules for each of those four entities — and rn the only one of those rules actually implemented is that field and macro names may not begin with a digit. Some names from the example above: "string", "and".

  • entry — a chunk of text starting with an "at" sign (@) at top-level, then a name (the entry type), then an entry delimiter ({ or (), and then continuing to the matching closing delimiter. Also: the data structure that results from parsing that chunk of text. There are two entries in the example above. ✌️
  • entry type — the name that comes right after an @ at top-level. From the example above: "string", "book".
  • entry metatype — a classification of entry types that lets us group one or more entry types under the same heading. With the standard BibTeX database structure, "article", "book", "inbook", etc. all fall under the "regular entry" metatype. The other metatypes are "macro definition" (for "string" entries), "preamble" (for "preamble" entries), and "comment" ("comment" entries). Fr fr, any entry whose type isn't one of "string", "preamble", or "comment" is called a "regular" entry.
  • entry delimiters — { and }, or ( and ): the pair of characters that (almost) mark the boundaries of an entry. "Almost" because the start of an entry is marked by an @, not by the "entry open" delimiter. Sneaky, I know. 👀
  • entry key — (or just key, when it's obvious what we mean.) The name immediately following the entry open delimiter in a regular entry, which uniquely identifies that entry. From the example above: "ourbook". Only regular entries have keys, mmkay?
  • field — a name to the left of an equals sign in a regular or macro-definition entry. In that second context it might also be called a macro name. From the example above: "joe", "author".
  • field list — in a regular entry, everything between the entry delimiters except the entry key. In a macro definition entry, everything between the entry delimiters (and yeah, that one might also get called a macro list).
  • compound value — (usually just "value".) The text that follows an equals sign (=) in a regular or macro definition entry, up to a comma or the entry close delimiter; a list of one or more simple values joined by hash signs (#).
  • simple value — a string, macro, or number.
  • string — (or, sometimes, "quoted string.") A chunk of text between quotes (") or braces ({ and }). Braces must balance: {this is a {string} is not a BibTeX string, but {this is a {string}} is. (this is a {string is also illegal — mainly to dodge the possibility of generating bogus TeX code, which BibTeX will absolutely do in certain cases. 😭)
  • macro — a name that appears on the right-hand side of an equals sign (i.e. as one simple value in a compound value). This implies that the name was defined as a macro in an earlier macro definition entry, but that's only actually checked if btparse is being asked to expand macros to their full definitions.
  • number — an unquoted string of digits. That's it. That's the whole vibe. 🔢

So working with btparse generally goes like: you pass the library some BibTeX data (or a source for some BibTeX data — like a filename or a file pointer ✨), it lexically scans it, parses it, and builds an abstract syntax tree (AST) out of it. Then it hands that AST back to you, and you call other btparse functions to traverse and query the tree. Iconic behaviour, honestly.

Heads up: the contents of AST nodes are the private domain of the library — do not go poking around in there, bestie. This being C, though, nothing physically stops you except good manners and the very real possibility that I change the AST structure in a future release and break all your badly-behaved code. 💀 Also, you don't need to know the structural relationships between nodes in the AST at all — the query/traversal functions handle that for you. 🎀

That said, it is useful to know some of the stuff btparse deposits in the AST and hands back to you through those query/traversal functions. First off: every node has a "node type," which records the syntactic element that node corresponds to. For example, the entry

@book{mybook, author = "Joe Blow", title = "My Little Book"}

is rooted by an "entry" node; underneath that you'd find a "key" node (for the entry key), two "field" nodes (for the "author" and "title" fields), and associated with each field node there'd be a "string" node. The only time any of this matters to you is when you ask the library for a simple value — just looking at the text isn't enough to tell quoted strings, numbers, and macro names apart, so btparse returns the nodetype too. 😌

On top of the nodetype, btparse also records the metatype of each "entry" node. That's what lets you (and the library) tell regular entries apart from comment entries, for instance. Not only do those have wildly different structures and therefore have to be traversed differently by the library — some traversal functions genuinely make no sense on certain entry metatypes, so you need to be able to make that distinction yourself too. 🫡

That said! Everything you actually need to know to work with the AST is explained in bt_traversal. Go read that next, bestie. 🦋

💖 DATA TYPES AND MACROS 💖

Okay so btparse defines several types required for the external interface. First up, it trivially defines a boolean type (along with TRUE and FALSE macros). This might affect you when including the btparse.h header in your own code — since it's not possible for the code to detect if there is already a boolean type defined, you might have to define the HAVE_BOOLEAN pre-processor token to deactivate btparse.h's typedef of boolean. 💅

Next, two enumeration types are defined: bt_metatype and bt_nodetype. Both of these are used extensively in the library itself, and are made available to users of the library because they can be found in nodes of the btparse AST (abstract syntax tree). (I.e., querying the AST can give you bt_metatype and bt_nodetype values, so the typedefs must be available to your code.) iykyk ✨

🎀 Entry metatype enum 🎀

bt_metatype_t has the following values:

  • BTE_UNKNOWN
  • BTE_REGULAR
  • BTE_COMMENT
  • BTE_PREAMBLE
  • BTE_MACRODEF

which are determined by the entry type token. (@string entries have the BTE_MACRODEF metatype; @comment and @preamble correspond to BTE_COMMENT and BTE_PREAMBLE; and any other entry type has the BTE_REGULAR metatype.) periodt.

🦋 AST nodetype enum 🦋

bt_nodetype has the following values:

  • BTAST_UNKNOWN
  • BTAST_ENTRY
  • BTAST_KEY
  • BTAST_FIELD
  • BTAST_STRING
  • BTAST_NUMBER
  • BTAST_MACRO

Of these, you'll only ever deal with the last three. They are returned when you query the AST for a simple value — just seeing the text isn't enough to distinguish between a quoted string, a number, and a macro, so the AST nodetype is supplied along with the text. ✨

🌸 String processing option macros 🌸

Since BibTeX is essentially a system for glueing strings together in a wide variety of ways, the processing done to its strings is fairly important. Most of the string transformations are done outside of the lexer/parser; this reduces their complexity, and makes it easier to switch different transformations on and off. This switching is done with an options bitmap which can be specified on a per-entry-metatype basis. (That is, you can have one set of transformations done to the strings in all regular entries, another set done to the strings in all macro definition entries, and so on.) If you need finer control than that, it's currently unavailable outside of the library (but it's just a matter of making a couple functions available and documenting them — so bug me if you need this feature). 😭

Okay, so the page says there are three basic macros for constructing this bitmap, and then lists four — not me side-eyeing the math 💀. Here's every one:

  • BTO_CONVERT — Convert number values to strings. (The conversion is trivial, involving changing the type of the AST node representing the number from BTAST_NUMBER to BTAST_STRING. number values are stored as strings of digits, just as they are in the input data.)
  • BTO_EXPAND — Expand macro invocations to the full macro text.
  • BTO_PASTE — Paste simple values together.
  • BTO_COLLAPSE — Collapse whitespace according to the BibTeX rules.

For instance, supplying BTO_CONVERT | BTO_EXPAND as the string options bitmap for the BTE_REGULAR metatype means that all simple values in regular entries will be converted to strings: numbers will simply have their nodetype changed, and macros will be expanded. Nothing else will be done to the simple values, though — they will not be concatenated, nor will whitespace be collapsed. See the bt_set_stringopts() and bt_parse_*() functions in bt_input for more information on the various options for parsing; see bt_postprocess for details on the post-processing. 💖

✨ USING THE LIBRARY ✨

Here's the skeletal example of using the btparse library — copy it, love it, live it:

#include <btparse.h>

int main (void)
{
   bt_initialize ();

   /* process some data */

   bt_cleanup ();
   exit (0);
}

Please note the call to bt_initialize(); this is very important! Without it, the library may crash or fail mysteriously. You must call bt_initialize() before calling any other btparse functions. bt_cleanup() just frees the memory allocated by bt_initialize(); if you are careful to call it before exiting, and bt_free_ast() on any abstract syntax trees generated by btparse when you are done with them, then your program shouldn't have any memory leaks. (Unless they're due to your own code, of course!) 🫶

💀 BUGS AND LIMITATIONS 💀

btparse has several inherent limitations that are due to the lexical scanner and parser generated by PCCTS 1.x. In short, the scanner and parser are both heavily dependent on global variables, meaning that thread safety — or even the ability to have two files open and being parsed at the same time — is well-nigh impossible. This will not change until I get with the times and adopt ANTLR 2.0, the successor to PCCTS — presuming of course that it can generate more modular C scanners and parsers. 😭

Another limitation that is due to PCCTS: entries with a large number of fields (more than about 90, if each field value is just a single string) will cause the parser to crash. This is unavoidable due to the parser using statically-allocated stacks for attributes and abstract-syntax tree nodes. I could increase the static allocation, but that would just decrease the likelihood of encountering the problem, not make it go away. Again, the chances of this changing as long as I'm using PCCTS 1.x are nil. 💀

Apart from those inherent limitations, there are no known bugs in btparse. Any segmentation faults or bus errors from the library should be considered bugs. They probably result from using the library incorrectly (eg. attempting to interleave the parsing of two files), but I do make an attempt to catch all such mistakes, and if I've missed any I'd like to know about it.

Any memory leaks from the library are also a concern; as long as you are conscientious about calling the cleanup functions (bt_free_ast() and bt_cleanup()), then the library shouldn't leak. 🐛

🌈 SEE ALSO 🌈

  • To read and parse BibTeX data files, see bt_input.
  • To traverse the syntax tree that results, see bt_traversal.
  • To learn what is done to values in parsed entries, and how to customize that munging, see bt_postprocess.
  • To learn how btparse deals with strings, see bt_strings (oops, I haven't written this one yet!).
  • To manipulate and access the btparse macro table, see bt_macros.
  • For splitting author names and lists "the BibTeX way" using btparse, bt_split_names.
  • To put author names back together again, see bt_format_names.
  • Miscellaneous functions for processing strings "the BibTeX way": bt_misc.
  • A semi-formal language definition is in bt_language.

🌸 AUTHOR 🌸

Greg Ward <gward@python.net>

Copyright (c) 1996-97 by Gregory P. Ward.

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Library General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Library General Public License for more details.

You should have received a copy of the GNU Library General Public License along with this library; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

🦋 AVAILABILITY 🦋

Okay, where do you get it? The btOOL home page, where you can get up-to-date information about btparse (and download the latest version) is

http://starship.python.net/~gward/btOOL/

You will also find the latest version of Text::BibTeX, the Perl library that provides a high-level front-end to btparse, there. btparse is needed to build Text::BibTeX, and must be downloaded separately.

Both libraries are also available on CTAN (the Comprehensive TeX Archive Network, http://www.ctan.org/tex-archive/) and CPAN (the Comprehensive Perl Archive Network, http://www.cpan.org/). Look in biblio/bibtex/utils/btOOL/ on CTAN, and authors/Greg_Ward/ on CPAN. For example,

http://www.ctan.org/tex-archive/biblio/bibtex/utils/btOOL/
http://www.cpan.org/authors/Greg_Ward

will both get you to the latest version of Text::BibTeX and btparse — but of course, you should always access busy sites like CTAN and CPAN through a mirror. 🌈

✨ That's the whole page, bestie — btparse, version 0.91, dated 2025-09-04, BTPARSE(1). Stay iconic 💅✨

📦 where this page comes from