CSV(3)

✨ NAME ✨

csv — a whole CSV parser and writer library, bestie. it does both. slay.

---

🌸 SYNOPSIS 🌸

#include <csv.h>

int csv_init(struct csv_parser *p, unsigned char options);
size_t csv_parse(struct csv_parser *p,
        const void *s,
        size_t len,
        void (*cb1)(void *, size_t, void *),
        void (*cb2)(int, void *),
        void *data);
int csv_fini(struct csv_parser *p,
        void (*cb1)(void *, size_t, void *),
        void (*cb2)(int, void *),
        void *data);
void csv_free(struct csv_parser *p);

unsigned char csv_get_delim(struct csv_parser *p);
unsigned char csv_get_quote(struct csv_parser *p);
void csv_set_space_func(struct csv_parser *p, int (*f)(unsigned char));
void csv_set_term_func(struct csv_parser *p, int (*f)(unsigned char));

int csv_get_opts(struct csv_parser *p);
int csv_set_opts(struct csv_parser *p, unsigned char options);
int csv_error(struct csv_parser *p);
char * csv_strerror(int error);

size_t csv_write(void *dest, size_t dest_size, const void *src,
        size_t src_size);
int csv_fwrite(FILE *fp, const void *src, size_t src_size);

size_t csv_write2(void *dest, size_t dest_size, const void *src,
        size_t src_size, unsigned char quote);
int csv_fwrite2(FILE *fp, const void *src, size_t src_size, unsigned char quote);

void csv_set_realloc_func(struct csv_parser *p, void *(*func)(void *, size_t));
void csv_set_free_func(struct csv_parser *p, void (*func)(void *));
void csv_set_blk_size(struct csv_parser *p, size_t size);
size_t csv_get_blk_size(struct csv_parser *p);
size_t csv_get_buffer_size(struct csv_parser *p);

(this is the CSV(3) page from the Library Functions Manual, btw 🫶)

---

💜 DESCRIPTION 💜

The CSV library gives you a flexible, intuitive interface for parsing and writing csv data. that's the whole pitch. it just gets you. 🧚

---

🌈 OVERVIEW 🌈

okay so the idea behind parsing with libcsv is genuinely so straight-forward it's giving main character:

1. you initialize a parser object with csv_init()
2. you feed data to the parser over one or more calls to csv_parse(), handing it callback functions that handle end-of-field and end-of-row events. csv_parse() parses the data you gave it, calling your user-defined callbacks as it reads fields and rows.
3. when you're done, you call csv_fini() to finish processing the current field and make a final call to the callback functions if necessary.
4. then csv_free() is called to free the parser object.

csv_error() and csv_strerror() give you info about errors the functions ran into. and csv_write() and csv_fwrite() give you a simple little interface for converting raw data into CSV data and storing the result into a buffer or a file respectively. ✨

CSV is a binary format allowing the storage of arbitrary binary data. files opened for reading or writing CSV data should be opened in binary mode. not me stressing about this — please do it. 🙈

libcsv ships with a default mode where the parser will happily process any data as CSV without complaint. this is lowkey iconic for parsing files that don't adhere to all the traditional rules. a strict mode is also supported, which will cause any violation of the imposed rules to cause a parsing failure. no vibes allowed in strict mode. 💅

---

👑 ROUTINES 👑

🎀 PARSING DATA 🎀

csv_init() initializes a pointer to a csv_parser structure. this structure holds all the housekeeping info — the current state of the parser, the buffer, current size and position, etc. it's the whole backstage crew fr.

csv_init() returns 0 on success and a non-zero value upon failure. it will fail if the pointer passed to it is a null pointer. the options argument specifies the parser options, and these may be changed later with the csv_set_opts() function. so you're not locked in, bestie. 🦋

✨ OPTIONS ✨

• CSV_STRICT — enables strict mode. rules are rules.
• CSV_REPALL_NL — causes each instance of a carriage return or linefeed outside of a record to be reported.
• CSV_STRICT_FINI — causes unterminated quoted fields encountered in csv_fini() to cause a parsing error (see below).
• CSV_APPEND_NULL — will cause all fields to be nul-terminated when provided to cb1. introduced in 3.0.0.
• CSV_EMPTY_IS_NULL — will cause NULL to be passed as the first argument to cb1 for empty, unquoted, fields. "empty" means consisting only of either spaces and tabs, or the values defined by a custom function registered via csv_set_space_func(). added in 3.0.3.

Multiple options can be specified by OR-ing them together. go wild, combine them. 🌈

---

csv_parse() is the function that does the actual parsing. it takes 6 arguments:

• p — a pointer to an initialized struct csv_parser.
• s — a pointer to the data to read in, like a dynamically allocated region of memory containing data you read in from a call to fread().
• len — the number of bytes of data to process.
• cb1 — a pointer to the callback function that will be called from csv_parse() after an entire field has been read. cb1 gets called with a pointer to the parsed data (which is NOT nul-terminated unless the CSV_APPEND_NULL option is set), the number of bytes in the data, and the pointer that was passed to csv_parse().
• cb2 — a pointer to the callback function that will be called when the end of a record is encountered. it gets called with the character that caused the record to end, cast to an unsigned char, or -1 if called from csv_fini, plus the pointer that was passed to csv_parse().
• data — a pointer to user-defined data that will be passed to the callback functions when they're invoked.

cb1 and/or cb2 may be NULL, in which case no function will be called for the associated actions. data may also be NULL, but then your callback functions must be prepared to handle receiving a null pointer. be nice to them. 🫶

---

By default, cb2 is not called when rows that do not contain any fields are encountered. this behavior is meant to accommodate files using only either a linefeed or a carriage return as a record separator to be parsed properly, while at the same time being able to parse files with rows terminated by multiple characters from resulting in blank rows after each actual row of data — for example, processing a text CSV file that was created on a Windows machine on a Unix machine. 😭 the classic.

The CSV_REPALL_NL option will cause cb2 to be called once for every carriage return or linefeed encountered outside of a field. cb2 is called with the character that prompted the call to the function, cast to an unsigned char — either CSV_CR for carriage return, CSV_LF for linefeed, or -1 for record termination from a call to csv_fini() (see below).

A carriage return or linefeed within a non-quoted field always marks both the end of the field and the row. Other characters can be used as row terminators and thus be provided as an argument to cb2 using csv_set_space_func().

Note: the first parameter of the cb1 function is void *, not const void *. the pointer passed to the callback function is actually a pointer to the entry buffer inside the csv_parser struct. this data may safely be modified from the callback function (or any function that the callback function calls), but you must not attempt to access more than len bytes, and you should not access the data after the callback function returns, as the buffer is dynamically allocated and its location and size may change during calls to csv_parse(). respect the buffer's boundaries, bestie. 💀

Note: different callback functions may safely be specified during each call to csv_parse(), but keep in mind that the callback functions may be called many times during a single call to csv_parse() depending on the amount of data being processed in a given call. it's giving popular.

---

csv_parse() returns the number of bytes processed. on a successful call this will be len; if it is less than len, an error has occurred. an error can occur, for example, if there is insufficient memory to store the contents of the current field in the entry buffer. an error can also occur if malformed data is encountered while running in strict mode. 🚨

The csv_error() function can be used to determine what the error is, and the csv_strerror() function can be used to provide a textual description of the error. csv_error() takes a single argument, a pointer to a struct csv_parser, and returns one of the following values defined in csv.h:

• CSV_EPARSE — a parse error has occurred while in strict mode
• CSV_ENOMEM — there was not enough memory while attempting to increase the entry buffer for the current field
• CSV_ETOOBIG — continuing to process the current field would require a buffer of more than SIZE_MAX bytes

The value passed to csv_strerror() should be one returned from csv_error(). the return value of csv_strerror() is a pointer to a static string. the pointer may be used for the entire lifetime of the program and the contents will not change during execution, but you must not attempt to modify the string it points to. hands off. 🙅

---

When you have finished submitting data to csv_parse(), you need to call the csv_fini() function. this function will call the cb1 function with any remaining data in the entry buffer (if there is any) and call the cb2 function unless we are already at the end of a row (the last byte processed was a newline character, for example).

It is necessary to call this function because the file being processed might not end with a carriage return or newline, but the data that has been read in to this point still needs to be submitted to the callback routines. ✨ if cb2 is called from within csv_fini(), it will be because the row was not terminated with a newline sequence — in this case cb2 will be called with an argument of -1.

Note: a call to csv_fini implicitly ends the current field and row. if the last field processed is a quoted field that ends before a closing quote is encountered, no error will be reported by default, even if CSV_STRICT is specified. to cause csv_fini() to report an error in such a case, set the CSV_STRICT_FINI option (new in version 1.0.1) in addition to the CSV_STRICT option. two options, one very specific outcome. iykyk.

csv_fini() also reinitializes the parser state so that it is ready to be used on the next file or set of data. it does not alter the current buffer size. if the last set of data that was being parsed contained a very large field that increased the size of the buffer, and you need to free that memory before continuing, you must call csv_free() — you do not need to call csv_init() again after csv_free(). 🎀

Like csv_parse, the callback functions provided to csv_fini() may be NULL. csv_fini() returns 0 on success and a non-zero value if you pass it a null pointer.

Okay so real talk about csv_fini(): after you call it, you can keep on using that same struct csv_parser pointer, no re-init required. 💖 But bestie — you must NOT call csv_init() on an already-initialized csv_parser object, because the memory that was allocated for the original structure is just gone. Poof. Lost. 😭

Then, when you're truly done with your csv_parser object, you can free all the dynamically allocated memory hanging off it by calling csv_free(). ✨ And the vibe is very flexible here: you may call csv_free() at any time — it does not need to be preceded by a csv_fini() call. 🫶 The one hard rule: you must only call csv_free() on a csv_parser object that's been initialized by a successful call to csv_init(). Periodt. 💅

✍️ WRITING DATA ✍️

libcsv hands you two functions for turning raw data into CSV-formatted data, and they're giving very different energy:

• csv_write() — writes the result into a buffer you provide.
• csv_fwrite() — writes the result to a file.

Both are super straightforward: they write out a single field, quotes included (opening and closing), and they escape every quote they run into by doubling it up with another quote. 🎀

Now let's talk csv_write(), because the details matter. It takes a pointer to a source buffer (src) and processes at most src_size characters from src. It'll write at most dest_size characters to dest, and it returns the number of characters that would have been written if dest were big enough. 👑

That return value is genuinely useful: you can use it to check whether everything got written, and if not, to figure out exactly how big dest needs to be to hold all the data. And here's a cute trick — you can call csv_write() with a null pointer for the dest argument, and in that case nothing gets written, but you still get back the size required to write out the data. 🌈

So the space you need to write out your data is:

• the size of the data,
• plus the number of quotes appearing in the data (each one gets escaped),
• plus 2 (that's the leading quote and the terminating quote). 💫

csv_write() and csv_fwrite() always surround the output data with quotes. No exceptions, that's the look. 💅

Heads up though: if src_size is very large — like SIZE_MAX/2 or greater — it's possible that the number of bytes needed to represent the data, once you've inserted all the escaping quotes, ends up greater than SIZE_MAX. In that specific case, csv_write returns SIZE_MAX, and you should read that as "this data is too large to write to a single field." 😭 Good news: csv_fwrite() is not similarly limited.

Now csv_fwrite(): it takes a FILE pointer (which should've been opened in binary mode, bestie, don't skip that) and it converts and writes the data pointed to by src of size src_size. It returns 0 on success and EOF if there was an error writing to the file. 📝

One thing it does not give you: the number of characters processed or written. Nope. If you need that info, use csv_write() combined with fwrite(). 🦋

And then we've got the glow-up versions: csv_write2() and csv_fwrite2() work similarly, but they take an additional argument — the quote character to use when composing the field. ✨

🎀 CUSTOMIZING THE PARSER 🎀

Want to change what counts as a delimiter or a quote? Say less:

• csv_set_delim() — changes the character the parser treats as the delimiter, cast to unsigned char.
• csv_set_quote() — changes the character the parser treats as the quote, cast to unsigned char.
• csv_get_delim() — returns the current delimiter character.
• csv_get_quote() — returns the current quote character.

When csv_init() gets called, the delimiter is set to CSV_COMMA and the quote is set to CSV_QUOTE. 💖

Note that the rest of the CSV conventions still apply even when you use these functions to change the delimiter and/or quote characters. Fields containing the new quote character or the new delimiter must be quoted, and quote characters must be escaped with an immediately preceding instance of the same character. No cap. 🫶

You can also bring your own functions to the party:

• csv_set_space_func() — lets you supply a user-defined function that determines what constitutes a space character.
• csv_set_term_func() — lets you supply a user-defined function that determines what constitutes a record terminator character.

Space characters decide which characters get removed from the beginning and end of non-quoted fields, and terminator characters govern when a record ends. 🌸

When csv_init() is called, the effect is exactly as if these two functions were each called with a NULL argument — in which case no function gets called, CSV_SPACE and CSV_TAB are used for space characters, and CSV_CR and CSV_LF are used for terminator characters. 👑

More customization, because we love options:

• csv_set_realloc_func() — sets the function called when the internal buffer needs to be resized. Only realloc is used internally, not malloc; the default is to use the standard realloc function.
• csv_set_free_func() — sets the function called to free the internal buffer. The default is the standard free function.
• csv_get_blk_size() / csv_set_blk_size() — get and set the block size of the parser respectively. The block size is the amount of extra memory allocated every time the internal buffer needs to be increased, and the default is 128. 💫
• csv_get_buffer_size() — returns the current number of bytes allocated for the internal buffer.

---

📜 THE CSV FORMAT 📜

So here's the tea: even though the CSV format is everywhere, there is genuinely no standard for it. There are, however, a set of traditional conventions that lots of applications use. libcsv follows the conventions described at:

http://www.creativyst.com/Doc/Articles/CSV/CSV01.htm

which seem to reflect the most common usage of the format. Namely: 🌈

• Fields are separated with commas.
• Rows are delimited by newline sequences (see below).
• Fields may be surrounded with quotes.
• Fields that contain comma, quote, or newline characters MUST be quoted. 🚨
• Each instance of a quote character must be escaped with an immediately preceding quote character.
• Leading and trailing spaces and tabs are removed from non-quoted fields.
• The final line need not contain a newline sequence.

And in strict mode, any detectable violation of these rules results in an error. 💀

🦋 RFC 4180 vs libcsv 🦋

RFC 4180 is an informational memo that tries to document the CSV format, especially with regards to its use as a MIME type. But there are several parts of the description in that memo which either don't accurately reflect widely used conventions, or artificially limit how useful the format can be. Here are the differences between the RFC and libcsv:

• "Each line should contain the same number of fields throughout the file"
  • libcsv doesn't care if every record contains a different number of fields. Such a restriction could easily be enforced by the application itself, if it wants that. 💅
• "Spaces are considered part of a field and should not be ignored"
  • Leading and trailing spaces that are part of non-quoted fields are ignored, because this is by far the most common behavior and what many applications expect.

abc ,  def

is considered equivalent to:

"abc", "def"

• "The last field in the record must not be followed by a comma"
  • The meaning of this statement isn't clear, but if the last character of a record is a comma, libcsv will interpret that as a final empty field, i.e.:

"abc", "def",

will be interpreted as 3 fields, equivalent to:

"abc", "def", ""

• RFC 4180 limits the allowable characters in a CSV field. libcsv allows any character to be present in a field, provided it adheres to the conventions mentioned above. This makes it possible to store binary data in CSV format — an attribute that many applications rely on. 🔥
• RFC 4180 states that a Carriage Return plus Linefeed combination is used to delimit records. libcsv allows any combination of Carriage Returns and Linefeeds to signify the end of a record. This is to increase portability among systems that use different combinations to denote a newline sequence. 🌍

---

🕵️ PARSING MALFORMED DATA 🕵️

libcsv should correctly parse any CSV data that conforms to the rules we just talked about. But by default, it'll also attempt to parse malformed CSV data — stuff like unescaped quotes, or quotes sitting inside non-quoted fields. For example:

a"c, "d"f"

would be parsed equivalently to the correct form:

"a""c", "d""f"

This is often desirable, because some applications out there just do not adhere to the specifications we discussed. 😭

But — and this is the important part — there are instances where malformed CSV data is genuinely ambiguous. Specifically when a comma or a newline is the next non-space character following a quote. Like:

"Sally said "Hello", Wally said "Goodbye""

This could be parsed as a single field containing the data:

Sally said "Hello", Wally said "Goodbye"

or as 2 separate fields:

Sally said "Hello and Wally said "Goodbye""

Since the data is malformed, there is no way to know whether the quote before the comma is meant to be a literal quote, or whether it signifies the end of the field. 💀 This is, of course, not an issue for properly formed data, because all quotes must be escaped. libcsv will parse this example as 2 separate fields.

Good news though: libcsv provides a strict mode that will return with a parse error if a quote is seen inside a non-quoted field, or if a non-escaped quote is seen whose next non-space character isn't a comma or newline sequence. 👑

🔍 PARSER DETAILS 🔍

okay bestie, this is the part where we get into the actual brain of the parser — like, how does it even decide what a field is? Here's the tea 💅

A field is considered quoted if the first non-space character for a new field is a quote.

Now, when a quote shows up inside a quoted field, the parser checks what the next non-space character is:

• If it's a comma — the field ends at the closed quote, and the field data gets submitted the moment that comma is hit.
• If it's a newline character — the row has ended, the field data is submitted, and the end of row is signalled (via the appropriate callback function).
• If two quotes are immediately adjacent — the first one is interpreted as escaping the second one, and one quote gets written to the field buffer. 🫶
• If it's anything else — the quote is interpreted as a non-escaped literal quote, and it plus whatever follows get written to the field buffer. Heads up: this would cause a parse error in strict mode.

---

✨ Example 1 ✨

"abc"""

Parses as: abc"

The first quote marks the field as quoted, the second quote escapes the following quote, and the last quote ends the field. This is valid in both strict and non-strict modes. Slay 💖

✨ Example 2 ✨

"ab"c

Parses as: ab"c

The first quote marks the field as quoted, the second quote is taken as a literal quote since the next non-space character is not a comma, or newline and the quote is not escaped. The last quote ends the field (assuming there is a newline character following). A parse error would result upon seeing the character c in strict mode.

✨ Example 3 ✨

"abc" "

Parses as: abc"

In this case, since the next non-space character following the second quote is not a comma or newline character, a literal quote is written, the space character after is part of the field, and the last quote terminated the field. This demonstrates the fact that a quote must immediately precede another quote to escape it. This would be a strict-mode violation as all quotes are required to be escaped.

---

and now, the non-quoted girlies:

If the field is not quoted, any quote character is taken as part of the field data, any comma terminates the field, and any newline character terminates the field and the record.

✨ Example 4 ✨

ab""c

Parses as: ab""c

Quotes are not considered special in non-quoted fields. This would be a strict mode violation since quotes may not exist in non-quoted fields in strict mode.

---

🧪 EXAMPLES 🧪

The following example prints the number of fields and rows in a file. It's a simplified version of the csvinfo program provided in the examples directory. Error checking not related to libcsv has been removed for clarity, the csvinfo program also provides an option for enabling strict mode and handles multiple files.

#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <stdlib.h>
#include <csv.h>

struct counts {
  long unsigned fields;
  long unsigned rows;
};

void cb1 (void *s, size_t len, void *data) {
  ((struct counts *)data)->fields++; }
void cb2 (int c, void *data) {
  ((struct counts *)data)->rows++; }

int main (int argc, char *argv[]) {
  FILE *fp;
  struct csv_parser p;
  char buf[1024];
  size_t bytes_read;
  struct counts c = {0, 0};

  if (csv_init(&p, 0) != 0) exit(EXIT_FAILURE);
  fp = fopen(argv[1], "rb");
  if (!fp) exit(EXIT_FAILURE);

  while ((bytes_read=fread(buf, 1, 1024, fp)) > 0)
    if (csv_parse(&p, buf, bytes_read, cb1, cb2, &c) != bytes_read) {
      fprintf(stderr, "Error while parsing file: %s\n",
      csv_strerror(csv_error(&p)) );
      exit(EXIT_FAILURE);
    }

  csv_fini(&p, cb1, cb2, &c);

  fclose(fp);
  printf("%lu fields, %lu rows\n", c.fields, c.rows);

  csv_free(&p);
  exit(EXIT_SUCCESS);
}

See the examples directory for several complete example programs. 👑

---

👑 AUTHOR 👑

Written by Robert Gamble.

---

💀 BUGS 💀

Please send questions, comments, bugs, etc. to:

rgamble99@gmail.com

---

okay that's the whole page bestie, you're officially a libcsv girlie now 🎀 This is the CSV(3) page from the libcsv source, dated 20 August 2021. Go forth and parse responsibly, mwah 💋✨🦋