VNL-FILTER(1)

📦
version 1.42 · 5 versions
, in ubuntu resolute, ubuntu stonking

💅 vnl-filter 💅

yep bestie, this is a page from the vnlog manual, section 1

🎀 NAME 🎀

vnl-filter — filters vnlogs to select particular rows, fields

✨ SYNOPSIS ✨

okay so picture this: you've got a little file called run.vnl and it's just sitting there, living its truth:

$ cat run.vnl

# time x   y   z   temperature
3      1   2.3 4.8 30
4      1.1 2.2 4.7 31
6      1   2.0 4.0 35
7      1   1.6 3.1 42

$ <run.vnl vnl-filter -p x,y,z | vnl-align

# x  y   z
1   2.3 4.8
1.1 2.2 4.7
1   2.0 4.0
1   1.6 3.1

$ <run.vnl vnl-filter -p i=NR,time,'dist=sqrt(x*x + y*y + z*z)' | vnl-align

# i time   dist
1   3    5.41572
2   4    5.30471
3   6    4.58258
4   7    3.62905

$ <run.vnl vnl-filter 'temperature >= 35' | vnl-align

# time x  y   z  temperature
6      1 2.0 4.0 35
7      1 1.6 3.1 42

$ <run.vnl vnl-filter --eval '{s += temperature} END { print "mean temp: " s/NR}'

mean temp: 34.5

$ <run.vnl vnl-filter -p x,y | feedgnuplot --terminal 'dumb 80,30' --unset grid --domain --lines --exit

  2.3 +---------------------------------------------------------------------+
      |           +          +          ***************         +           |
      |                                                **************       |
      |                                                              *******|
  2.2 |-+                                                       ************|
      |                                                 ********            |
      |                                         ********                    |
  2.1 |-+                              *********                          +-|
      |                        ********                                     |
      |                ********                                             |
      |            ****                                                     |
    2 |-+         *                                                       +-|
      |           *                                                         |
      |           *                                                         |
      |           *                                                         |
  1.9 |-+         *                                                       +-|
      |           *                                                         |
      |           *                                                         |
      |           *                                                         |
  1.8 |-+         *                                                       +-|
      |           *                                                         |
      |           *                                                         |
  1.7 |-+         *                                                       +-|
      |           *                                                         |
      |           *                                                         |
      |           *          +           +           +          +           |
  1.6 +---------------------------------------------------------------------+
     0.98         1         1.02        1.04        1.06       1.08        1.1

look at her go 😭 she's doing things.

🌸 DESCRIPTION 🌸

okay so the tea: this tool is largely a frontend for awk that operates on vnlog files. Vnlog is both an input and an output here. And the whole point is that this tool makes it very simple to select specific rows and columns for output, and to manipulate the data in various ways. no cap.

This is a UNIX-style tool, so the input/output of this tool is strictly STDIN/STDOUT. Furthermore, in its usual form this tool is a filter, so the format of the output is exactly the same as the format of the input. The exception to this is when using --eval, in which case the output is dependent on whatever expression we're evaluating. (so like, it does whatever it wants based on your expression, which is kind of the whole vibe)

This tool is convenient to process both stored data or live data; in the latter case, it's very useful to pipe the streaming output to feedgnuplot --stream to get a realtime visualization of the incoming data. obsessed honestly.

This tool reads enough of the input file to get a legend, at which point it constructs an awk program to do the main work, and execs to awk (it's possible to use perl as well, but this isn't as fast).

💖 Input/output data format 💖

The input/output data is vnlog: a plain-text table of values. Any lines beginning with # are treated as comments, and are passed through. The first line that begins with # but not ## or #! is a legend line. After the #, follow whitespace-separated field names. Each subsequent line is whitespace-separated values matching this legend. For instance, this is a valid vnlog file:

#!/usr/bin/something
## more comments
# x y z
-0.016107 0.004362 0.005369
-0.017449 0.006711 0.006711
-0.018456 0.014093 0.006711
-0.017449 0.018791 0.006376

vnl-filter uses this format for both the input and the output. The comments are preserved, but the legend is updated to reflect the fields in the output file. periodt.

A string - is used to indicate an undefined value, so this is also a valid vnlog file:

# x y z
1 2 3
4 - 6
- - 7

🦋 Filtering 🦋

To select specific columns, pass their names to the -p option (short for --print or --pick, which are synonyms — pick your fave, they're the same thing). In its simplest form, to grab only columns x and y, do

vnl-filter -p x,y

See the detailed description of -p below for more detail. 👀

To select specific rows, we use matches expressions. Anything on the vnl-filter commandline and not attached to any --xxx option is such an expression. For instance

vnl-filter 'size > 10'

would select only those rows whose size column contains a value > 10. See the detailed description of matches expressions below for more detail.

🧚 Context lines 🧚

vnl-filter supports the context output options (-A, -B and -C) exactly like the grep tool. I.e. to print out all rows whose size column contains a value > 10 but also include the 3 rows immediately before and after such matching rows, do this:

vnl-filter -C3 'size > 10'

-B reports the rows before matching ones and -A the rows after matching ones. -C reports both. Note that this applies only to matches expressions: records skipped because they fail --has or --skipempty are not included in contextual output.

👑 Backend choice 👑

By default, the parsing of arguments and the legend happens in perl, which then constructs a simple awk script, and invokes mawk to actually read the data and to process it. This is done because awk is lighter weight and runs faster, which is important because our data sets could be quite large. We default to mawk specifically, since this is a simpler implementation than gawk, and runs much faster. If for whatever reason we want to do everything with perl, this can be requested with the --perl option.

🍓 Special functions 🍓

For convenience we support several special functions in any expression passed on to awk or perl (named expressions, matches expressions, --eval strings). These generally maintain some internal state, and vnl-filter makes sure that this state is consistent. Note that these are evaluated after --skipcomments and --has. So any record skipped because of a --has expression, for instance, will not be considered in prev(), diff() and so on. ikyk. 🫶

  • rel(x) — returns value of x relative to the first value of x. For instance we might want to see the time or position relative to the start, not relative to some absolute beginning. Example:
$ cat tst.vnl

# time x
100    200
101    212
102    209

$ <tst.vnl vnl-filter -p 't=rel(time),x=rel(x)

# t x
0 0
1 12
2 9
  • diff(x) — returns the difference between the current value of x and the previous value of x. The first row will always be -. Example:
$ <tst.vnl vnl-filter -p x,'d1=diff(x),d2=diff(diff(x))' | vnl-align

# x d1 d2
  1  -  -
  8  7  7
 27 19 12
 64 37 18
125 61 24
  • sum(x) — returns the cumulative sum of x. As diff(x) can be thought of as a derivative, sum(x) can be thought of as an integral. So diff(sum(x)) would return the same value as x (except for the first row; diff() always returns - for the first row).

Example:

$ <tst.vnl vnl-filter -p 'x,s=sum(x),ds=diff(sum(x))' | vnl-align

# x  s   ds
  1   1   -
  8   9   8
 27  36  27
 64 100  64
125 225 125
  • prev(x) — returns the previous value of x. One could construct sum() and rel() using this, if they weren't already available. iconic honestly.
  • latestdefined(x) — returns the most recent value of x that isn't -. If x isn't -, this simply returns x.
part 2 is coming bestie, don't go anywhere 💖

💬 ARGUMENTS 💬

Matches expressions 🔍

Okay so bestie, here's the tea: literally anything on the commandline that isn't glued to some --xxx option is a matches expression. What are they for? They pick which records (aka rows) in your data file get to stay. 🎯

For every single row, we evaluate all the expressions. If all of them come out true, that row gets output. And the way this works is that the expression is passed directly to the awk (or perl) backend. No middleman, no filter, no chill.

Example time, slay: to select all rows that have valid data in column "a" or column "b" or column "c" you can do

vnl-filter 'a != "-" || b != "-" || c != "-"'

or, if you're feeling perl-y 🐪

vnl-filter --perl 'defined a || defined b || defined c'

Same energy as the named expressions you give to -p (that's the section above, iykyk): these get handed straight to awk, so literally anything awk can do is fair game here. periodt ✨

-p|--print|--pick expr 🎀

These options are the whole mechanism for picking specific columns to output. Like, say you wanna pull out columns called "lat", "lon", and any column whose name contains the string "feature_", you'd do

vnl-filter -p lat,lon,'feature_.*'

or, equivalently, one at a time like a little gremlin 🧚

vnl-filter --print lat --print lon --print 'feature_.*'

Here's the order of operations: we look for exact column name matches first, and if none are found, then we try a regex. So if there was no column called exactly "feature_", then the above would be equivalent to

vnl-filter -p lat,lon,feature_

And honestly? This mechanism is way more powerful than just selecting columns. First off, we can rename chosen fields:

vnl-filter -p w=feature_width

That would pick the "feature_width" field, but the resulting column in the output would be named "w". Cute, right? 💅

⚠️ But heads up: when you're renaming a column like this, regexen are not supported, and exact field names must be given. No cap.

The flip side though — the string to the right of the "=" is passed on directly to awk (after replacing field names with column indices), so literally any awk expression can live there. For instance, to compute the length of a vector in separate columns "x", "y", and "z" you can do:

vnl-filter -p 'l=sqrt(x*x + y*y + z*z)'

And a single column called "l" would be produced. Math ate that. 🧮✨

We can also exclude columns by putting a "!" in front of their name. This works exactly like you expect. The rules, bestie, are:

  • The pick/exclude directives are processed in order given to produce the output picked-column list
  • If the first -p item is an exclusion, we implicitly pick all the columns prior to processing the -p.
  • The exclusion expressions match the output column names, not the input names.
  • We match the exact column names first. If that fails, we match as a regex

Example. To grab all the columns except the temperature(s) do this:

vnl-filter -p !temperature

To grab all the columns that describe something about a robot (columns whose names have the string "robot_" in them), but not its temperature (i.e. not "robot_temperature"), do this:

vnl-filter -p robot_,!temperature

Robot columns minus the temp. We love to see it. 🤖🌡️

--has a,b,c,... ✅

This one selects records (rows) that have a non-empty value in a particular field (column). A null value in a column is designated with a single "-".

So if we want to select only records that have a value in the "x" column, we pass --has x. To select records that have data for all of a given set of columns, the --has option can be repeated, or those multiple columns can be given in a whitespace-less comma-separated list. For instance if we want only records that have data in both columns "x" and "y" we can pass in --has x,y or --has x --has y.

If you want to combine multiple columns in an or (select rows that have data in any of a given set of columns), use a matches expression, as documented below.

And here's a cute little shorthand 🫶 — if we want to select a column and pick only rows that have a value in this column:

vnl-filter --has col -p col

is equivalent to

vnl-filter -p +col
📌 Note that just like the column specifications in -p, the columns given to --has must match exactly or as a regex. In either case, a unique matching column must be found.

-l|list-columns 📋

Instead of doing any processing, parse the input to get the available columns, print those out, and exit. Just a little peek at what's inside. 👀

-A N|--after-context N ⬇️

Output N lines following each matches expression, even those lines that do not themselves match. This works just like the grep options of the same name. See "Context lines"

-B N|--before-context N ⬆️

Output N lines preceding each matches expression, even those lines that do not themselves match. This works just like the grep options of the same name. See "Context lines"

-C N|--context N ↕️

Output N lines preceding and following each matches expression, even those lines that do not themselves match. This works just like the grep options of the same name. See "Context lines"

--eval expr 🧪

Instead of printing out all matching records and picked columns, just run the given chunk of awk (or perl). In this mode of operation, vnl-filter acts just like a glorified awk, that allows fields to be accessed by name instead of by number, as it would be in raw awk. Glow up. 💅

Since the expression may print anything or nothing at all, the output in this mode is not necessarily itself a valid vnlog stream. And no column-selecting arguments should be given, since they make no sense in this mode.

In awk the expr is a full set of pattern/action statements. So to print the sum of columns "a" and "b" in each row, and at the end, print the sum of all values in the "a" column

vnl-filter --eval '{print a+b; suma += a} END {print suma}'

In perl the arbitrary expression fits in like this:

sub evalexpr
{
  eval expression;     # evaluate the arbitrary expression
}
while(<>) # read each line
{
  chomp;
  next unless matches; # skip non-matching lines
  evalexpr();
}

--function|--sub 🛠️

Evaluates the given expression as a function that can be used in other expressions. This is most useful when you want to print something that can't trivially be written as a simple expression. For instance:

$ cat tst.vnl
# s
1-2
3-4
5-6

$ < tst.vnl
  vnl-filter --function 'before(x) { sub("-.*","",x); return x }' \
             --function 'after(x)  { sub(".*-","",x); return x }' \
             -p 'b=before(s),a=after(s)'
# b a
1 2
3 4
5 6

See the CAVEATS section below if you're doing something sufficiently-complicated where you need this.

--function-abs|--sub-abs 📐

Convenience option to add an absolute-value abs() function. This is only useful for awk programs (the default, no --perl given) since perl already provides abs() by default. Thanks perl, so generous. 🙏

--begin|--BEGIN 🎬

Evaluates the given expression in the BEGIN {} block of the generated awk (or perl) program.

--end|--END 🎞️

Evaluates the given expression in the END {} block of the generated awk (or perl) program.

--[no]skipempty 🫧

Do [not] skip records where all fields are blank. By default we do skip all empty records; to include them, pass --noskipempty.

--skipcomments 🤫

Don't output non-legend comments.

--perl 🐪

By default all processing is performed by mawk, but if for whatever reason we want perl instead, pass --perl. Both modes work, but mawk is noticeably faster. --perl could be useful because it is more powerful, which could be important since a number of things pass commandline strings directly to the underlying language (named expressions, matches expressions, --eval strings).

📌 Note that while variables in perl use sigils, column references should not use sigils.

To print the sum of all values in column "a" you'd do this in awk

vnl-filter --eval '{suma += a} END {print suma}'

and this in perl

vnl-filter --perl --eval '{$suma += a} END {say $suma}'

The perl strings are evaluated without use strict or use warnings so I didn't have to declare $suma in the example. Living dangerously, love that for us. 😌

With --perl, empty strings ("-" in the vnlog file) are converted to "undef".

--dumpexprs 🔬

Used for debugging. This spits out all the final awk (or perl) program we run for the given commandline options and given input. This is the final program, with the column references resolved to numeric indices, so one can figure out what went wrong. Detective mode activated. 🕵️‍♀️

--unbuffered ⚡

Flushes each line after each print. This makes sure each line is output as soon as it is available, which is crucial for realtime output and streaming plots. No lag, no waiting, just vibes in real time. 🌈

--stream 🌊

Synonym for --unbuffered.

💀 CAVEATS 💀

okay bestie, sit down for this one, because this is the part where I'm brutally honest with you. This tool is very lax in its input validation — and yes, that's on purpose. The whole reason is that columns with names like %CPU and "TIME+" actually work, which means you can more or less feed in output straight from top -b. 🎀

The downside? Shooting yourself in the foot is absolutely possible. 😭 This tradeoff is currently tuned to be very permissive, which works great for my use cases, and I'd genuinely love to hear other people's experiences. So here's the list of potential pitfalls and unexpected behaviors:

  • All column names get replaced in all eval strings, with zero regard for context. Remember that earlier example that sums up the values in a column: vnl-filter --eval '{suma += a} END {print suma}' — this works totally fine if we have a column named "a" and we do not have a column named "suma". But it will not do the right thing if either of those gets violated. Like, say the column "a" doesn't exist — then "awk" would literally see suma += a instead of something like suma += $5. And "a" would be an uninitialized variable, which evaluates to 0, so the whole vnl-filter command wouldn't even fail… it would just print 0 instead. 🙈 It's your responsibility to make sure we're talking about the right columns. The focus here was one-liners, so hopefully nobody has so many columns that they can't keep track of them all in their head. I don't see any way to fix this without seriously wrecking the scope of the tool, so I'm leaving it alone.
  • It is so natural to use vnlog as a database. You can run queries like
vnl-filter 'key == 5'

and it works! But unlike a real database, this is clearly a linear lookup. With large data files, this would be significantly slower than the logarithmic searches a real database gives you. What counts as "large" and "significant" varies, so test it yourself, bestie. In my experience vnlog "databases" scale surprisingly well 🦋 — but at some point, importing your data into something like sqlite is well worth it.

  • When substituting column names I match either a word-nonword transition ("\b") or a whitespace-nonword transition. The word boundary is what you'd use 99% of the time. But the keys may have special characters in them, which don't play nice with "\b". Which means whitespace becomes important: 1+%CPU will not be parsed as expected — which is correct, since +%CPU is also a valid field name. But 1+ %CPU will be parsed correctly. So if you have weird field names, put the whitespace into your expressions. It'll make them more readable anyway. 💅
  • Strings passed to "-p" are split on "," except if the "," is inside balanced "()". That's what makes it possible to say things like vnl-filter --function 'f(a,b) { ... }' -p 'c=f(a,b)'. This is probably the right behavior, although some questionable-looking field names become potentially impossible: "f(a" and "b)" could otherwise be legal field names, but honestly you're probably asking for trouble if you do that. 😅
  • Currently there're two modes: a pick/print mode and an "--eval" mode. Then there's also "--function", which adds bits of "--eval" to the pick/print mode, but it feels maybe insufficient. I don't yet have strong feelings about what this should become. Comments welcome ✨

🌈 REPOSITORY 🌈

https://github.com/dkogan/vnlog/

👑 AUTHOR 👑

Dima Kogan "<dima@secretsauce.net>"

Copyright 2016-2017 California Institute of Technology

Copyright 2017-2019 Dima Kogan "<dima@secretsauce.net>"

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

and that's a wrap, bestie 🫶 this page is dated 2026-01-26 and it lives in the VNL-FILTER(1) section of your manual, so go forth and filter some columns. slay, periodt. 💅✨

📦 where this page comes from