BIBCLEAN(1)

📦 version 3.07, in debian trixie, nixos 26.05, nixpkgs unstable, ubuntu questing, ubuntu resolute, ubuntu stonking

💖 bibclean(1) — General Commands Manual 💖

okay so bestie, welcome to the manual page for bibclean, and omg it's giving "I organize your BibTeX like a pro". no cap, it prettyprints and syntax-checks BibTeX and Scribe bibliography data base files. Let's get into it. 💅

✨ NAME ✨

bibclean — prettyprint and syntax check BibTeX and Scribe bibliography data base files. 🦋

🌸 SYNOPSIS 🌸

       bibclean [ -author ] [ -copyleft ] [ -copyright ]
                [ -error-log filename ] [ -help ] [ '-?'  ]
                [ -init-file filename ] [ -ISBN-file filename ]
                [ -keyword-file filename ] [ -max-width nnn ]
                [ -[no-]align-equals ] [ -[no-]brace-protect ]
                [ -[no-]check-values ] [ -[no-]debug-match-failures ]
                [ -[no-]delete-empty-values ] [ -[no-]file-position ]
                [ -[no-]fix-accents ] [ -[no-]fix-braces ]
                [ -[no-]fix-degrees ] [ -[no-]fix-font-changes ]
                [ -[no-]fix-initials ] [ -[no-]fix-math ] [ -[no-]fix-names ]
                [ -[no-]German-style ] [ -[no-]keep-linebreaks ]
                [ -[no-]keep-parbreaks ] [ -[no-]keep-preamble-spaces ]
                [ -[no-]keep-spaces ] [ -[no-]keep-string-spaces ]
                [ -[no-]parbreaks ] [ -[no-]prettyprint ]
                [ -[no-]print-ISBN-table ] [ -[no-]print-keyword-table ]
                [ -[no-]print-patterns ] [ -[no-]quiet ]
                [ -[no-]read-init-files ] [ -[no-]remove-OPT-prefixes ]
                [ -[no-]scribe ] [ -[no-]trace-file-opening ]
                [ -[no-]warnings ] [ -output-file filename ] [ -version ]
                <infile or  bibfile1 bibfile2 bibfile3 ...
                >outfile

All options can be abbreviated to a unique leading prefix. 💫

An explicit file name of ``-'' represents standard input; it is assumed if no input files are specified. 🫶

On VAX VMS and IBM PC DOS, the leading -'' on option names may be replaced by a slash, /''; however, the ``-'' option prefix is always recognized. 🌈

🦋 DESCRIPTION 🦋

bibclean prettyprints input BibTeX files to stdout, or to a user-specified file, and checks the brace balance and bibliography entry syntax as well. It can be used to detect problems in BibTeX files that sometimes confuse even BibTeX itself, and importantly, can be used to normalize the appearance of collections of BibTeX files. 💖

Here's a summary of the formatting actions, bestie:

  • BibTeX items are formatted into a consistent structure with one field = "value" pair per line, and the initial @ and trailing right brace in column 1. ✨
  • Tabs are expanded into blank strings; their use is discouraged because they inhibit portability, and can suffer corruption in electronic mail. 🚫
  • Long string values are split at a blank and continued onto the next line with leading indentation. 🌸
  • A single blank line separates adjacent bibliography entries. 🎀
  • Text outside BibTeX entries is passed through verbatim. 💜
  • Outer parentheses around entries are converted to braces. 🦋
  • Personal names in author and editor field values are normalized to the form P. D. Q. Bach'', from P.D.Q. Bach'' and ``Bach, P.D.Q.''. 👑
  • Hyphen sequences in page numbers are converted to en-dashes. ➖
  • Month values are converted to standard BibTeX string abbreviations. 📅
  • In titles, sequences of upper-case characters at brace level zero are braced to protect them from being converted to lower-case letters by some bibliography styles. 🛡️
  • CODEN, ISBN (International Standard Book Number) and ISSN (International Standard Serial Number) entry values are examined to verify the checksums of each listed number, and correct ISBN hyphenation is automatically supplied. 🔍

The standardized format of the output of bibclean facilitates the later application of simple filters, such as bibcheck(1), bibdup(1), bibextract(1), bibindex(1), bibjoin(1), biblabel(1), biblook(1), biborder(1), bibsort(1), citefind(1), and citetags(1), to process the text, and also is the one expected by the GNU Emacs BibTeX support functions. 💅

💅 OPTIONS 💅

Command-line switches may be abbreviated to a unique leading prefix, and letter case is not significant. All options are parsed before any input bibliography files are read, no matter what their order on the command line. Options that correspond to a yes/no setting of a flag have a form with a prefix "no-" to set the flag to no. For such options, the last setting determines the flag value used. That is significant when options are also specified in initialization files (see the INITIALIZATION FILES manual section). ✨

The leading hyphen that distinguishes an option from a filename may be doubled, for compatibility with GNU and POSIX conventions. Thus, -author and --author are equivalent. 💫

To avoid confusion with options, if a filename begins with a hyphen, it must be disguised by a leading absolute or relative directory path, e.g., /tmp/-foo.bib or ./-foo.bib. 🫶

  • -author — Display an author credit on the standard error unit, stderr, and then terminate with a success return code. Sometimes an executable program is separated from its documentation and source code; this option provides a way to recover from that. 👑
  • -copyleft — Display copyright information on the standard error unit, stderr, and then terminate with a success return code. 🎀
  • -copyright — Display copyright information on the standard error unit, stderr, and then terminate with a success return code. 💖
  • -error-log filename — Redirect stderr to the indicated file, which then contains all of the error and warning messages. This option is provided for those systems that have difficulty redirecting stderr. 📝
  • -help or -? — Display a help message on stderr, giving a usage description, similar to this section of the manual pages, and then terminate with a success return code. 🆘
  • -ISBN-file filename — Provide an explicit ISBN-range initialization file. It is processed after any system-wide and job-wide ISBN initialization files found on the PATH (for VAX VMS, SYS$SYSTEM) and BIBINPUTS search paths, respectively, and may override them. The ISBN initialization file name can be changed at compile time, or at run time through a setting of the environment variable BIBCLEANISBN, but defaults to .bibclean.isbn on UNIX, and bibclean.isb elsewhere. For further details, see the ISBN INITIALIZATION FILES manual section. 📚
  • -init-file filename — Provide an explicit value pattern initialization file. It is processed after any system-wide and job-wide initialization files found on the PATH (for VAX VMS, SYS$SYSTEM) and BIBINPUTS search paths, respectively, and may override them. It in turn may be overridden by a subsequent file-specific initialization file. The initialization file name can be changed at compile time, or at run time through a setting of the environment variable BIBCLEANINI, but defaults to .bibcleanrc on UNIX, and to bibclean.ini elsewhere. For further details, see the INITIALIZATION FILES manual section. ⚙️
  • -keyword-file filename — Provide an explicit keyword initialization file. It is processed after any system-wide and job-wide keyword initialization files found on the PATH (for VAX VMS, SYS$SYSTEM) and BIBINPUTS search paths, respectively, and may override them. The keyword initialization file name can be changed at compile time, or at run time through a setting of the environment variable BIBCLEANKEY, but defaults to .bibclean.key on UNIX, and bibclean.key elsewhere. For further details, see the KEYWORD INITIALIZATION FILES manual section. 🔑
  • -max-width nnn — bibclean normally limits output line widths to 72 characters, and in the interests of consistency, that value should not be changed. Occasionally, special-purpose applications may require different maximum line widths, so this option provides that capability. The number following the option name can be specified in decimal, octal (starting with 0), or hexadecimal (starting with 0x). A zero or negative value is interpreted to mean unlimited, so -max-width 0 can be used to ensure that each field/value pair appears on a single line.

When -no-prettyprint requests bibclean to act as a lexical analyzer, the default line width is unlimited, unless overridden by this option.

When bibclean is prettyprinting, line wrapping is done only at a space. Consequently, a long non-blank character sequence may result in the output exceeding the requested line width.

When bibclean is lexing, line wrapping is done by inserting a backslash-newline pair when the specified maximum is reached, so no line length ever exceeds the maximum. 📏

  • -[no-]align-equals — bestie, when you use the positive form, this lines up the = sign in key/value assignments in the same column, with just a single space between it and the value string 🎀. Otherwise? The equals sign just chills right after the key, one single space away. Default: no.
  • -[no-]brace-protect — positive form wraps uppercase and mixedcase words at brace-level zero in braces, so those BibTeX styles that love to downcase everything can't touch them 💅. Default: yes.
  • -[no-]check-values — positive form runs heuristic pattern matching on your field values to sniff out possible oopsies (think year = "192" when you meant year = "1992") and yells a warning whenever it spots a sus pattern 👀.

Honestly the checking is usually a whole win, but if it's throwing way too many bogus warnings at one particular bibliography file, you can just turn it off with the negative form. Default: yes.

  • -[no-]debug-match-failures — positive form prints a warning every time a value pattern fails to match a value string 🔍.

Super helpful when you're debugging new patterns, but the output can get so voluminous, so only use this one with tiny test files and init files that kill off every pattern except the ones you're actually testing. Default: no.

  • -[no-]delete-empty-values — positive form yeets every field/value pair whose value is an empty string 🗑️. That's a lifesaver for cleaning up bibliographies that got generated from text editor templates. Compare this one with -[no-]remove-OPT-prefixes down below. Default: no.
  • -[no-]file-position — positive form gives you detailed file position info in warning and error messages 📍. Default: no.
  • -[no-]fix-accents — positive form normalizes TeX accents in annotes, authors, booktitles, editors, notes, remarks, and titles ✨. Default: no.
  • -[no-]fix-braces — positive form normalizes bracing in annotes, authors, booktitles, editors, notes, remarks, and titles by stripping out unnecessary levels of braces 🎀. Default: no.
  • -[no-]fix-degrees — positive form removes spaces in author/editor fields inside braces after letter-ending periods 🎓. That's what takes reductions like J. J. {Thomson, M. A., F. R. S.}, Frederick {Soddy, B. A. (Oxon.)}, and John A. {Cable, M. A., M. Ed., Dipl. Deutsch (Marburg), A. L. C. M.} down to J. J. {Thomson, M.A., F.R.S.}, Frederick {Soddy, B.A. (Oxon.)}, and John A. {Cable, M.A., M.Ed., Dipl.Deutsch (Marburg), A.L.C.M.} respectively. Iconic behavior, no cap.

So in journals in the humanities and history of science, plus some scientific journals all the way into the 20th Century, academic, honorary, and professional titles and degrees are commonly attached to personal names 📚. Even though modern publishing practice avoids these decorations, bibliography entries should preferably keep them for accuracy's sake. Journal typographical practice generally follows the reductions described right here. Default: no.

  • -[no-]fix-font-changes — positive form supplies an additional brace level around font changes in titles, protecting against downcasing by some BibTeX styles 🛡️. Font changes that already have more than one level of braces? Not modified. Periodt.

For example, if a title contains the Latin phrase {\em Dictyostelium discoideum} or {\em {D}ictyostelium discoideum}, then downcasing incorrectly converts the phrase to lower-case letters 😭. Most BibTeX users are shocked that bracing the initial letters does not prevent the downcase action. The correct coding is {{\em Dictyostelium discoideum}}. BUT — there are also legit cases where an extra level of bracing wrongly protects from downcasing. So consequently, bibclean normally does not supply an extra level of braces, but if you've got a bibliography where the extra braces are routinely missing, you can use this option to supply them.

If you think you need this option, it is strongly recommended that you apply bibclean to your bibliography file with and without -fix-font-changes, then compare the two output files to make sure extra braces aren't being supplied in titles where they shouldn't be 💀. You must decide which of the two output files is the better choice, then repair the incorrect title bracing by hand.

Because font changes in titles are uncommon (except for the exact case this option is designed to fix), it should do more good than harm. Default: no.

  • -[no-]fix-initials — positive form inserts a space after a period that follows author initials 🌸. Default: yes.
  • -[no-]fix-math — positive form makes math mode in titles way more readable by inserting spaces around operators, deleting other unnecessary space, and removing braces around single-character subscripts and superscripts 🧮. Default: no.
  • -[no-]fix-names — positive form reorders author and editor name lists to remove commas at brace level zero, putting first names or initials before last names 👑. Default: yes.
  • -[no-]German-style — positive form interprets quote characters ["] inside braced value strings at brace level 1 according to the conventions of the TeX style file german.sty 🇩🇪, which overloads quote to simplify input and representation of German umlaut accents, sharp-s (es-zet), ligature separators, invisible hyphens, raised/lowered quotes, French guillemets, and discretionary hyphens. Recognized character combinations get braced so BibTeX can't interpret the quote as a string delimiter.

Quoted strings get no special handling from this option, and since German nouns in titles have to be protected from the downcasing operation of most BibTeX bibliography styles anyway, German value strings that use the overloaded quote character can always be entered in the form "{...}" — no need to specify this option at all 💅.

Default: no.

  • -[no-]keep-linebreaks — normally, line breaks inside value strings get collapsed into a single space, so that long value strings can later be broken to give you lines of reasonable length 📏.

Positive form preserves linebreaks in value strings. If -max-width is set to zero, this preserves the original line breaks. Spacing outside value strings stays under bibclean's control and is not affected by this option at all.

Default: no.

  • -[no-]keep-parbreaks — positive form preserves paragraph breaks (either formfeeds, or lines containing only spaces) in value strings 🦋. Normally, paragraph breaks get collapsed into a single space. Spacing outside value strings stays under bibclean's control and is not affected by this option either.

Default: no.

  • -[no-]keep-preamble-spaces — positive form preserves all whitespace in @Preamble{...} entries 🌈. Default: no.
  • -[no-]keep-spaces — with the positive form, every single space in value strings stays exactly where it is 💅 normally multiple spaces get squished down to one lonely space. bestie you can stack this with -keep-linebreaks, -keep-parbreaks, and -max-width 0 to keep value strings looking how they looked while still getting syntax and value checking done. the spacing outside value strings? that's still fully bibclean's call, this option does not touch it. default: no.
  • -[no-]keep-string-spaces — positive form = preserve ALL the whitespace inside @String{...} entries. every lil space, untouched. default: no.
  • -[no-]parbreaks — with the negative form, a paragraph break (that's either a formfeed, or a line that's nothing but spaces 🫠) is not allowed inside value strings, or between field/value pairs. why would you want that? to quickly catch runaway strings that happen when your delimiters don't match up — it's giving trap. default: yes.
  • -[no-]prettyprint — normally bibclean is out here being a prettyprinter ✨ but with the negative form it switches roles and becomes a lexical analyzer instead, spitting out a whole stream of lexical tokens. more on that in the LEXICAL ANALYSIS manual section. default: yes.
  • -[no-]print-ISBN-table — positive form prints the ISBN-range table to stderr, then bails with a success return code 🎀

that move happens after all command-line options are processed, and before any input files get read (except the ones that are values of command-line options, those don't count).

the format of the output ISBN-range table is totally acceptable as input for an ISBN initialization file (peep the ISBN INITIALIZATION FILES manual section). default: no.

  • -[no-]print-keyword-table — positive form prints the keyword initialization table to stderr, then terminates with a success return code 💖

same timing as above: after all command-line options are processed, and before any input files are read (again, other than files that are values of command-line options).

the format of the output table is acceptable as input for a keyword initialization file (see the KEYWORD INITIALIZATION FILES manual section). default: no.

  • -[no-]print-patterns — positive form prints the value patterns read from initialization files as they get added to internal tables. use this to check brand-new patterns, or just to see which patterns are in play 👀

when bibclean is compiled with native pattern-matching code (that's the default), those patterns are the exact ones used to check value strings for valid syntax, and all of them come from initialization files — they are not hard-coded into the program. details are in the INITIALIZATION FILES manual section. default: no.

  • -[no-]quiet — this one is literally just the opposite of -[no-]warning; it exists for user convenience and for compatibility with other programs that use -q for quiet operation, no warning messages. periodt 🤫
  • -[no-]read-init-files — negative form suppresses loading of system-, user-, and file-specific initialization files. so initializations then come only from files you explicitly hand over with -init-file filename options. default: yes.
  • -[no-]remove-OPT-prefixes — positive form removes the OPT prefix from each field name where the matching value is not an empty string. the prefix OPT has to be entirely upper-case to get recognized, no exceptions 🚨

this option is for bibliographies generated with the GNU Emacs BibTeX editing support, which makes templates with optional fields marked by that OPT prefix. the function M-x bibtex-remove-OPT, normally bound to C-c C-o, does the job — but users forget, and then BibTeX doesn't recognize the field name and just ignores the value string 😭 compare this with -[no-]delete-empty-values described above. default: no.

  • -[no-]scribe — positive form accepts input syntax that conforms to the Scribe document system. the output gets converted to conform to BibTeX syntax. more in the SCRIBE BIBLIOGRAPHY FORMAT manual section. default: no.
  • -[no-]trace-file-opening — positive form records the names of all files bibclean attempts to open in the error log file. use it to figure out where initialization files are hiding 🔍 default: no.
  • -[no-]warnings — positive form allows all warning messages. the negative form is not recommended, because it can mask problems that should be repaired. default: yes.
  • -output-file filename — supply an alternate output file to replace stdout. if that filename can't be opened for output, execution terminates immediately with a nonzero exit code 💀
  • -version — display the program version number on stderr, then terminate with a success return code. that includes an indication of who compiled the program, the host name it was compiled on, the time of compilation, and the type of string-value matching code that was selected — when that info is available to the compiler. iconic 🫶

😭 ERROR RECOVERY AND WARNINGS 😭

so when bibclean detects an error, it sends an error message to both stderr and stdout. that way you're clearly notified, and the output bibliography also carries the message right at the point of the error. no cap, you can't miss it.

error messages start with a very distinctive pair of question marks, ??, beginning in column 1, followed by the input file name and line number. if the -file-position option was specified, they also carry the input and output positions of the current file, entry, and value. each position includes the file byte number, the line number, and the column number. in the event of a runaway string argument, the entry and value positions should pinpoint the erroneous bibliography entry exactly, and the file positions show where it was detected — which might be a fair bit later in the files 👀

warning messages flag possible problems, so they go only to stderr and never to stdout, meaning they never show up in your output file. they're marked with a distinctive pair of percents, %%, beginning in column 1, and just like error messages they may be followed by file position messages if -file-position was specified.

for convenience, the first line of every error and warning message sent to stderr is formatted according to what the GNU Emacs next-error command expects. you can invoke bibclean with the Emacs command:

M-x compile<RET>bibclean filename.bib >filename.new

then use the next-error command, normally bound to C-x `` `` ` (that's a grave, or back, accent) to jump to the location of the error in the input file 🎯

if error messages get ignored and left in the output bibliography file, that precipitates an error when the bibliography is next processed with BibTeX. not the vibe.

after issuing an error message, bibclean resynchronizes its input by copying it verbatim to stdout until it recognizes a new bibliography entry on a line where the first non-blank character is an at-sign (@). that ensures nothing is lost from the input file(s), so you can make corrections in either the input or the output files 💖

however — and this is a big however — if bibclean detects an internal error in its data structures, it terminates abruptly with no further input or output processing. that kind of error should never happen, and if it does, report it immediately to the author of the program. errors in initialization files, and running out of dynamic memory, also immediately terminate bibclean 🧚

🔎 SEARCH PATHS 🔎

okay so gather round bestie, because this is the ✨lore✨ of where your files actually live and it matters.

Versions of bibclean before 3.00 found some of their initialization files in the same directory as the executable program. That design choice means that those files can be copied anywhere in the file system, and still be found at run time. Some software distributions, however, prefer to follow the model where initialization and other related files are instead stored in a directory whose name is related to that of the executable by a conventional difference in filepath. For example, a program might be installed in /opt/bin and its associated files in /opt/share/lib/PROGRAMNAME/ or /opt/share/lib/PROGRAMNAME/PROGRAMVERSION/.

The second form is preferable, because it permits multiple versions of the same program to be installed, as long as the executable program names carry a version suffix. Thus, a site might have installed programs named bibclean-1.00, bibclean-2.00, bibclean-2.15, and bibclean-3.00, with the versionless name bibclean being a symbolic link to whichever version is the desired local default. iconic behavior honestly 💅

With most software packages, the absolute path to the directory containing associated files is compiled into the program, making it impossible to change the installation locations after the program has been built from source code. not me being stuck forever 🙈

Some packages, however, instead use the location of the executable program to find files by relative path at runtime. In the above example, the program would determine its filesystem location at runtime, say /opt/bin, then find its associated files relative to that location in ../share/lib/PROGRAMNAME/PROGRAMVERSION/.

From version 3.00, bibclean uses that second approach, with an associated directory like ../share/lib/bibclean/3.00. That allows an installation directory tree to be distributed to other systems and unbundled anywhere in the file system, as long as the relative paths are not changed. so portable, so slay 🦋

bibclean tests whether its compiled-in library path is a directory on the local system, and if so, uses it. Otherwise, it replaces that path by a reconstructed one based on the location of the executable program. If the reconstructed path for the library directory does not exist, it uses a warning. In either case, it continues normally. no drama, just vibes 😌

With the old approach, initialization files on Unix systems were named with a leading period, making them hidden files for the ls command. With the new practice, initialization files are no longer named as hidden files. the glow-up is real ✨

🎀 INITIALIZATION FILES 🎀

bibclean can be compiled with one of three different types of pattern matching; the choice is made by the installer at compile time:

  • The original version uses explicit hand-coded tests of value-string syntax.
  • The second version uses regular-expression pattern-matching host library routines together with regular-expression patterns that come entirely from initialization files.
  • The third version uses special patterns that come entirely from initialization files.

The second and third versions are the ones of most interest here, because they allow the user to control what values are considered acceptable. However, command-line options can also be specified in initialization files, no matter which pattern matching choice was selected. iykyk 🫶

When bibclean starts, it searches for initialization files, finding the first one in the system executable program search path (on UNIX and IBM PC DOS, PATH) and the first one in the BIBINPUTS search path, and processes them in turn. Then, when command-line arguments are processed, any additional files specified by -init-file filename options are also processed. Finally, immediately before each named bibliography file is processed, an attempt is made to process an initialization file with the same name, but with the extension changed to .ini. The default extension can be changed by a setting of the environment variable BIBCLEANEXT. That scheme permits system-wide, user-wide, session-wide, and file-specific initialization files to be supported. layers on layers bestie 🧚

When input is taken from stdin, there is no file-specific initialization.

For precise control, the -no-read-init-files option suppresses all initialization files except those explicitly named by -init-file filename options, either on the command line, or in requested initialization files.

Recursive execution of initialization files with nested -init-file options is permitted; if the recursion is circular, bibclean finally gets a non-fatal initialization file open failure after opening too many files. That terminates further initialization file processing. As the recursion unwinds, the files are all closed, then execution proceeds normally. we love a graceful exit 😭

An initialization file may contain empty lines, comments from percent to end of line (just like TeX), option switches, and field/pattern or field/pattern/message assignments. Leading and trailing spaces are ignored. That is best illustrated by a short example:

% This is a small bibclean initialization file

-init-file /u/math/bib/.bibcleanrc  %% departmental patterns

chapter = "\"D\""                 %% 23

pages   = "\"D--D\""              %% 23--27

volume  = "\"D \\an\\d D\""       %% 11 and 12

year    = \
   "\"dddd, dddd, dddd\"" \
   "Multiple years specified."      %% 1989, 1990, 1991

-no-fix-names   %% do not modify author/editor lists

Long logical lines can be split into multiple physical lines by breaking at a backslash-newline pair; the backslash-newline pair is discarded. That processing happens while characters are being read, before any further interpretation of the input stream. sneaky but valid 💖

Each logical line must contain a complete option (and its value, if any), or a complete field/pattern pair, or a field/pattern/message triple.

Comments are stripped during the parsing of the field, pattern, and message values. The comment start symbol is not recognized inside quoted strings, so it can be freely used in such strings. go off, percent sign 💅

Comments on logical lines that were input as multiple physical lines via the backslash-newline convention must appear on the last physical line; otherwise, the remaining physical lines become part of the comment. careful with that one ⭐

Pattern strings must be enclosed in quotation marks; within such strings, a backslash starts an escape mechanism that is commonly used in UNIX software. The recognized escape sequences are:

  • \a — alarm bell (octal 007)
  • \b — backspace (octal 010)
  • \f — formfeed (octal 014)
  • \n — newline (octal 012)
  • \r — carriage return (octal 015)
  • \t — horizontal tab (octal 011)
  • \v — vertical tab (octal 013)
  • \ooo — character number octal ooo (e.g \012 is linefeed). Up to 3 octal digits may be used.
  • \0xhh — character number hexadecimal hh (e.g., \0x0a is linefeed). xhh may be in either letter case. Any number of hexadecimal digits may be used.

Backslash followed by any other character produces just that character. Thus, \% gets a literal percent into a string (preventing its interpretation as a comment), \" produces a quotation mark, and \\ produces a single backslash. escape game strong 🔥

An ASCII NUL (\0) in a string terminates it; that is a feature of the C programming language in which bibclean is implemented.

Field/pattern pairs can be separated by arbitrary space, and optionally, either an equals sign or colon functioning as an assignment operator. Thus, the following are equivalent:

pages="\"D--D\""
pages:"\"D--D\""
pages "\"D--D\""
  pages = "\"D--D\""
  pages : "\"D--D\""
pages   "\"D--D\""

Each field name can have an arbitrary number of patterns associated with it; however, they must be specified in separate field/pattern assignments.

An empty pattern string causes previously-loaded patterns for that field name to be forgotten. That feature permits an initialization file to completely discard patterns from earlier initialization files. reset era 🎀

Patterns for value strings are represented in a tiny special-purpose language that is both convenient and suitable for bibliography value-string syntax checking. While not as powerful as the language of regular-expression patterns, its parsing can be portably implemented in less than 3% of the code in a widely-used regular-expression parser (the GNU regexp package). efficiency queen 👑

The patterns are represented by the following special characters:

  • <space> — one or more spaces
  • a — exactly one letter
  • A — one or more letters
  • d — exactly one digit
  • D — one or more digits
  • r — exactly one Roman numeral
  • R — one or more Roman numerals (i.e. a Roman number)
  • w — exactly one word (one or more letters and digits)
  • W — one or more space-separated words, beginning and ending with a word
  • . — one special' character, one of the characters <space>!#()*+,-./:;?[]~`, a subset of punctuation characters that are typically used in string values
  • : — one or more `special' characters
  • X — one or more `special'-separated words, beginning and ending with a word
  • \x — exactly one x (x is any character), possibly with an escape sequence interpretation given earlier
  • x — exactly the character x (x is anything but one of these pattern characters: aAdDrRwW.:<space>\)

The X pattern character is very powerful, but generally inadvisable, because it matches almost anything likely to be found in a BibTeX value string. The reason for providing pattern matching on the value strings is to uncover possible errors, not mask them. 💀

There is no provision for specifying ranges or repetitions of characters, but that can usually be done with separate patterns. It is a good idea to accompany the pattern with a comment showing the kind of thing it is expected to match. Here is a portion of an initialization file giving a few of the patterns used to match number value strings:

number  =       "\"D\""         %% 23
number  =       "\"A AD\""      %% PN LPS5001
number  =       "\"A D(D)\""    %% RJ 34(49)
number  =       "\"A D\""       %% XNSS 288811
number  =       "\"A D\\.D\""   %% Version 3.20
number  =       "\"A-A-D-D\""   %% UMIAC-TR-89-11
number  =       "\"A-A-D\""     %% CS-TR-2189
number  =       "\"A-A-D\\.D\"" %% CS-TR-21.7

For a bibliography that's literally only article entries, bestie, you should probably trim that pattern list down to just the first pattern, so that anything that isn't a digit string just fails the pattern-match test. And that's lowkey easy — just keep your bibliography-specific patterns in a matching file with extension .ini, because that file gets read automatically. 💅

You should also be sure to use empty pattern strings in the pattern file to throw away patterns from earlier initialization files. Yes, discard. Gone. Bye. ✨

The value strings passed to the pattern matcher contain surrounding quotes, so your patterns should too. BUT — you could also use a pattern specification like "\"D" to match an initial digit string followed by anything else. How? Because omitting that final quotation mark \" in the pattern lets the match succeed without checking that the next character in the value string is a quotation mark. Sneaky. Iconic.

Because the value strings are meant to get processed by TeX, the pattern matching ignores braces, and TeX control sequences, together with any space following those control sequences. Spaces around braces are preserved though. That convention is what lets the pattern fragment A-AD-D match the value string TN-K\slash 27-70, because the value is implicitly collapsed to TN-K27-70 during the matching operation. Slay.

bibclean's normal action when a string value fails to match any of the corresponding patterns is to drop a warning message kinda like this: "Unexpected value in ``year = "192"''. In most cases that's enough to alert the user to a problem. But sometimes you want a different message tied to a particular pattern. You do that by supplying a message string right after the pattern string. The format items available for expanding current values in messages are %% (single percent), %e (entry name), %f (field name), %k (citation key), and %v (string value). Here's an example:

chapter = "\"D:D\"" "Colon found in ``%f = %v''" %% 23:2

To stay consistent with the other messages bibclean outputs, the message string should not end with punctuation.

If you want to make the message an error instead of just a warning, start it with a query (?), like this:

chapter = "\"D:D\"" "?Colon found in ``%f = %v''" %% 23:2

The query gets included in the output message. 🫶

Escape sequences are supported in message strings, just like they are in pattern strings. Use that to your advantage for fancy stuff, like terminal display mode control. If you rewrite the previous example as

chapter = "\"D:D\"" \
          "?\033[7mColon found in ``%f = %v''\033[0m" %% 23:2

then the error message shows up in inverse video on display screens that support ANSI terminal control sequences. Ngl though, that practice is not normally recommended, because it may have undesirable effects on some output devices. Still, you might find it useful for restricted applications.

For some types of bibliography fields, bibclean has special-purpose code to supplement or replace the pattern matching:

  • CODEN, ISBN and ISSN field values are handled that way because validating them requires evaluating checksums that can't be expressed by simple patterns; no patterns are even used in these three cases.
  • When bibclean is compiled with pattern-matching code support, chapter, number, pages, and volume values are checked only by pattern matching.
  • month values are first checked against the standard BibTeX month abbreviations, and only if no match is found are patterns then used.
  • year values are first checked against patterns, then if no match is found, the year numbers are found and converted to integer values for testing against reasonable bounds.

Values for other fields are checked only against patterns. You can provide patterns for any field you like, even ones bibclean doesn't already know about. New ones just get added to an internal table that's searched for each string to be validated. 🦋

The special field, key, represents the bibliographic citation key. It can be given patterns, like any other field. Here's an initialization file pattern assignment that matches an author name, a colon, a four-digit year, a colon, and an alphabetic string, in the BibNet Project style:

key = "A:dddd:A"                     %% Knuth:1986:TB

Notice that no quotation marks are included in the pattern, because the citation keys are not quoted. You can use patterns like this to help enforce uniform naming conventions for citation keys — which gets increasingly important as your bibliography data base grows. Periodt. 👑

✨ ISBN INITIALIZATION FILES ✨

bibclean contains a compiled-in table of ISBN ranges and country/language settings that's suitable for most applications.

However — plot twist — ISBN data change yearly, as new countries adopt ISBNs, and as publishers are granted new, or additional, ISBN prefixes. 😭

So, from version 2.12, bibclean supports reading of run-time ISBN initialization files found on the PATH (for VAX VMS, SYS$SYSTEM) and BIBINPUTS search paths, and then any specified by -ISBN-file filename options.

That feature makes it possible to incorporate new ISBN data without having to produce a new bibclean release and reinstall the software at end-user sites. Genius fr fr.

The format of an ISBN initialization file is similar to that of the bibclean initialization files described in the preceding section: comments begin with percent and continue to end of line, blank and empty lines are ignored, backslash-newline joins adjacent lines, and otherwise, lines are expected to contain a required pair of ISBN country/language-publisher prefixes forming a non-decreasing range, optionally followed by one or more words of text that are treated as the country/language group value. That latter value plays no part in ISBN validation, but its presence is strongly recommended, so the ISBN table stays understandable for humans. 🫶

Here's a short example:

%% The Faeroes got ISBN assignments between 1993 and 1998
99918-0         99918-3        Faeroes
99918-40        99918-61
99918-900       99918-938

It is not necessary to repeat the country names on succeeding entries with the same initial number (99918 in that example); that's handled internally. 💅

Data from ISBN files normally augment the compiled-in data. However, if the first prefix begins with a hyphen, then bibclean deletes the first entry in the table matching that first prefix (ignoring the leading hyphen):

%% Latvia got ISBN ranges between 1993 and 1998
%% so we remove the old placeholder, then add the
%% new ranges.
-9984-0         9984-9         This one is no longer valid

9984-00         9984-20        Latvia
9984-500        9984-770
9984-9000       9984-9984

✨ KEYWORD INITIALIZATION FILES ✨

bibclean contains a compiled-in table of keyword mappings that's suitable for most applications. The default settings merely adjust lettercase in certain keyword names, so that, for example, isbn is output as ISBN. Cute.

From version 2.12, bibclean supports reading of run-time keyword initialization files found on the PATH (for VAX VMS, SYS$SYSTEM) and BIBINPUTS search paths, and then any specified by -keyword-file filename options.

That feature makes it possible to incorporate special spellings of new keywords without having to produce a new bibclean release and reinstall the software at end-user sites. 🔥

The format of a keyword initialization file is similar to that of the other bibclean initialization files described in the preceding sections: comments begin with percent and continue to end of line, blank and empty lines are ignored, backslash-newline joins adjacent lines, and otherwise, lines are expected to contain a required pair of old and new keyword names.

Here's a short example:

%% We want special handling of MathReviews keywords
mrclass         MRclass
mrnumber        MRnumber
mrreviewer      MRreviewer

Data from keywords files normally augment the compiled-in data. However, if the first keyword begins with a hyphen, then bibclean deletes the first entry in the table matching that keyword (ignoring the leading hyphen):

%% Remove special handling of ISBN, ISSN, and LCCN values.
-issn           ISSN
-isbn           ISBN
-lccn           LCCN

Even though the second keyword in each deletion pair is not used, it still must be specified. No skipping, bestie. 🙈

Notice that this feature can be used to regularize keyword names, but use it with care, in order to avoid producing duplicate key names in output BibTeX entries:

%% Map variations of keywords into a common name:
keys            keywords
keywds          keywords
keyword         keywords
keywrd          keywords
keywrds         keywords
searchkey       keywords

🔍 LEXICAL ANALYSIS 🔍

okay bestie so when you pass -no-prettyprint, bibclean fully switches careers 💅 — it stops being a prettyprinter and becomes a lexical analyzer instead. its output comes in lines shaped exactly like this:

<token-number><tab><token-name><tab>"<token-value>"

so every single output line is one complete token, and that token has three parts: a small integer number (for computer programs to chew on), a token type name (for human eyes, hi 👋), and a string value wrapped in quotes. three-in-one, iconic ✨

Special characters inside that token value string get written out as ANSI/ISO Standard C escape sequences. Translation: every character except NUL is representable, and multi-line values can chill on a single line. no cap 🌸

here's the full token lineup — the numbers and the type names you might see in the output when -prettyprint is specified:

        0   UNKNOWN
        1   ABBREV
        2   AT
        3   COMMA
        4   COMMENT
        5   ENTRY
        6   EQUALS
        7   FIELD
        8   INCLUDE
        9   INLINE
       10   KEY
       11   LBRACE
       12   LITERAL
       13   NEWLINE
       14   PREAMBLE
       15   RBRACE
       16   SHARP
       17   SPACE
       18   STRING
       19   VALUE

ngl, if you're writing a program that parses this output, you also gotta be ready for lines that start with the warning prefix %% or the error prefix ?? 😭 — and for ANSI/ISO Standard C line-number directives shaped like this:

# line 273 "texbook1.bib"

those ones record the line number and file name of the current input file. very ✨metadata✨ of them.

and if you passed a -max-width nnn command-line option, long output lines get wrapped at a backslash-newline pair 🎀 which means any software processing the lexical token stream has to be prepared to collapse those wrapped lines back into single lines. like, un-wrap them. you got this.

as an example of using -no-prettyprint, this UNIX command pipeline:

bibclean -no-prettyprint mylib.bib | \
    awk '$2 == "KEY" {print $3}' | \
    sed -e 's/"//g' | \
    sort

extracts a sorted list of all citation keys in the file mylib.bib. obsessed with her 🔥

also, a certain amount of processing has already happened to the tokens. specifically: delimiters equivalent to braces have been replaced by braces, and braced strings have become quoted strings. the glow-up happens before you even see it 💅

the LITERAL token type is for arbitrary text that bibclean does not examine any further — think the contents of a @Preamble{...} or a @Comment{...}. left alone, living its truth 🦋

and the UNKNOWN token type should never show up in the output stream, fr fr. it's used internally to initialize token type variables. it's giving backstage crew, not main character energy 🎭

📜 SCRIBE BIBLIOGRAPHY FORMAT 📜

bibclean's support for the Scribe bibliography format is based on the syntax description in the Scribe Introductory User's Manual, 3rd Edition, May 1980. Scribe was originally developed by Brian Reid at Carnegie-Mellon University, and was marketed by Unilogic, Ltd., later renamed to Scribe Systems, and apparently now long defunct. rip a real one 🕯️

the BibTeX bibliography format was strongly influenced by Scribe — and honestly, with care, you can share bibliography files between the two systems. but there ARE differences, so here's the whole rundown of the Scribe bibliography file format:

  1. Letter case is not significant in field names and entry names, but case is preserved in value strings. selective memory, iconic
  2. In field/value pairs, the field and value may be separated by one of three characters: =, /, or space. Space may optionally surround these separators.
  3. Value delimiters are any of these seven pairs: { } [ ] ( ) < > ' ' " " ` `
  4. Value delimiters may not be nested — even though with the first four delimiter pairs, nested balanced delimiters would be unambiguous. rules are rules bestie
  5. Delimiters can be omitted around values that contain only letters, digits, sharp (#), ampersand (&), period (.), and percent (%).
  6. Outside of delimited values, a literal at-sign (@) is represented by doubled at-signs (@@).
  7. Bibliography entries begin with @name, same as for BibTeX, but any of the seven Scribe value delimiter pairs may be used to surround the values in field/value pairs. And as in (4): nested delimiters are forbidden 🚫
  8. Arbitrary space may separate entry names from the following delimiters.
  9. @Comment is a special command whose delimited value is discarded. As in (4), nested delimiters are forbidden.
  10. The special form
@Begin{comment}
 ...
@End{comment}

permits encapsulating arbitrary text containing any characters or delimiters, other than @End{comment}. Any of the seven delimiter pairs may be used around the word comment following the @Begin or @End; the delimiters in the two cases need not be the same, and consequently, @Begin{comment}/@End{comment} pairs may not be nested. no inception, sorry 🌀

  1. The key field is required in each bibliography entry. periodt. 👑
  2. A backslashed quote in a string is assumed to be a TeX accent, and braced appropriately. While such accents do not conform to Scribe syntax, Scribe-format bibliographies have been found that appear to be intended for TeX processing. mystery behavior, love that for them ✨

because of that loose syntax, bibclean's normal error detection heuristics are less effective, and consequently, Scribe mode input is not the default — you have to explicitly ask for it. no auto-detection, bestie, you gotta use your words 🗣️

🌈 ENVIRONMENT VARIABLES 🌈

  • BIBCLEANEXT — file extension of bibliography-specific initialization files. Default: .ini.
  • BIBCLEANINI — name of bibclean initialization files. Default: .bibcleanrc (UNIX), bibclean.ini (non-UNIX).
  • BIBCLEANISBN — name of bibclean ISBN initialization files. Default: .bibclean.isbn (UNIX), bibclean.isb (non-UNIX).
  • BIBCLEANKEY — name of bibclean keyword initialization files. Default: .bibclean.key (UNIX), bibclean.key (non-UNIX).
  • BIBINPUTS — search path for bibclean and BibTeX input files. On UNIX, it is a colon-separated list of directories that are searched in order from first to last. It is not an error for a specified directory to not exist. 🫶

On other operating systems, the directory names should be separated by whatever character is used in system search path specifications, such as a semicolon on IBM PC DOS.

  • PATH — on Atari TOS, IBM PC DOS, IBM PC OS/2, Microsoft NT, and UNIX, the search path for system executable files. The system-wide bibclean initialization file is searched for in that path.
  • SYS$SYSTEM — on VAX VMS, the search path for system executable files and the system-wide bibclean initialization file.

📂 FILES 📂

  • *.bib — BibTeX and Scribe bibliography data base files. the whole point honestly
  • *.ini — file-specific initialization files.
  • .bibclean.isbn — UNIX system-wide and user-specific ISBN initialization files.
  • .bibclean.key — UNIX system-wide and user-specific keyword initialization files.
  • .bibcleanrc — UNIX system-wide and user-specific initialization files.
  • bibclean.ini — non-UNIX system-wide and user-specific initialization files.
  • bibclean.isb — non-UNIX system-wide and user-specific ISBN initialization files.
  • bibclean.key — non-UNIX system-wide and user-specific keyword initialization files.

👀 SEE ALSO 👀

the whole friend group, go say hi:

  • bibcheck(1), bibdup(1), bibextract(1), bibindex(1), bibjoin(1), biblabel(1), biblex(1), biblook(1), biborder(1), bibparse(1), bibsearch(1), bibsort(1), bibtex(1), bibunlex(1), citefind(1), citesub(1), citetags(1), latex(1), scribe(1), tex(1).

💖 AUTHOR 💖

Nelson H. F. Beebe
University of Utah
Department of Mathematics, 110 LCB
155 S 1400 E RM 233
Salt Lake City, UT 84112-0090
USA
Tel: +1 801 581 5254
FAX: +1 801 581 4148
Email: beebe@math.utah.edu, beebe@acm.org, beebe@computer.org (Internet)
URL: http://www.math.utah.edu/~beebe
########################################################################
########################################################################
########################################################################
###                                                                  ###
###     bibclean: prettyprint and syntax check BibTeX and Scribe     ###
###                   bibliography data base files                   ###
###                                                                  ###
###           Copyright (C) 1990--2016 Nelson H. F. Beebe            ###
###                                                                  ###
### This program is covered by the GNU General Public License (GPL), ###
### version 2 or later, available as the file COPYING in the program ###
### source distribution, and on the Internet at                      ###
###                                                                  ###
###               ftp://ftp.gnu.org/gnu/GPL                          ###
###                                                                  ###
###               http://www.gnu.org/copyleft/gpl.html               ###
###                                                                  ###
### This program is free software; you can redistribute it and/or    ###
### modify it under the terms of the GNU General Public License as   ###
### published by the Free Software Foundation; either version 2 of   ###
### the License, or (at your option) any later version.              ###
###                                                                  ###
### This program is distributed in the hope that it will be useful,  ###
### but WITHOUT ANY WARRANTY; without even the implied warranty of   ###
### MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the    ###
### GNU General Public License for more details.                     ###
###                                                                  ###
### You should have received a copy of the GNU General Public        ###
### License along with this program; if not, write to the Free       ###
### Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,   ###
### MA 02111-1307 USA                                                ###
########################################################################
########################################################################
########################################################################

and that's the whole page bestie, we made it to the end 🎉 this is Version 3.05 from 18 May 2020, and you just read the BIBCLEAN(1) manual page. go forth and prettyprint responsibly. slay 💅✨

📦 where this page comes from