STILTS-TCAT(1)

📦
version 3.5.4 · 5 versions
, in ubuntu stonking

💖 STILTS-TCAT — stilts-tcat 💖

welcome to the Stilts commands manual, bestie 🦋 this lil page is all about stilts-tcat, one of the Stilts commands, and honestly? it's giving table-glue. let's get into it 💅

✨ NAME ✨

stilts-tcat — Concatenates multiple similar tables 🧚 like, it stacks them. one after the other. iconic behaviour fr fr.

🌸 SYNOPSIS 🌸

stilts tcat [in=<table> [<table> ...]] [ifmt=<in-format>]
            [multi=true|false] [istream=true|false] [icmd=<cmds>]
            [ocmd=<cmds>]
            [omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui]
            [out=<out-table>] [ofmt=<out-format>] [seqcol=<colname>]
            [loccol=<colname>] [uloccol=<colname>] [lazy=true|false]
            [countrows=true|false]

look at all those options 😭 we love a flexible queen. every single one of them gets its own moment later, don't worry.

👑 DESCRIPTION 👑

okay so tcat is the tool for concatenating any number of similar tables one after the other 🫶. "Similar" is doing real work in that sentence, so let's be clear: the tables have to be of similar form to each other, meaning the same number and types of columns. no cap, that part is non-negotiable 💀.

Preprocessing of the tables may be done using the icmd parameter, and the way that works is that it operates in the same way on all the input tables ✨ — so it's one shared glow-up for the whole squad, not a bespoke makeover per table.

Table parameters of the output table will be taken from the first of the input tables 👑. first table = main character, and the rest are just... supporting cast, i guess 💅.

Subject to some constraints on the details of the input and output formats and processing, tcat is capable of joining an unlimited number of tables together to produce an output table of unlimited length, without large memory requirements 🌈. like, it really said "memory? never heard of her." that's the slay.

but — and this is the important bit — if there are very many input files, it may be necessary to set the lazy parameter so that they are not all kept open at once 🙈. so many open files at once is not the vibe, and lazy=true is how you fix it.

💜 quick heads up: if you have heterogeneous tables — ones in different formats from each other, or ones that need different preprocessing steps from each other before they can be concatenated — then tcat is not your girl. use tcatn instead 🎀.

more coming in part 2, bestie, don't go anywhere 🦋

✨ OPTIONS ✨

  • in=<table> [<table> ...] — where your input tables live, bestie. You can either pass this parameter multiple times, or just drop all the input tables in as a space-separated list in one go. Either way slays.

These are the table location forms you're allowed to use:

  • A filename. Classic.
  • A URL.
  • The special value -, which means standard input. Lowkey important: if you do this, you have to state the input format explicitly with the ifmt parameter. Also heads up, not every format can be streamed this way.
  • A scheme specification shaped like :<scheme-name>:<scheme-args>.
  • A system command line with either a < character at the start, or a | character at the end (<syscmd or syscmd|). This runs the pipeline you gave and reads from its standard output. Ngl this will probably only work on unix-like systems.

Compression: anything in one of the supported compression formats (Unix compress, gzip or bzip2) gets expanded automatically. No effort required from you 💅

You can also list your input table locations in an external file using the indirection character @. So in=@filename makes the file filename get read for a list of input table locations. The locations inside that file should each be on their own separate line.

  • ifmt=<in-format> — tells the program what format the input table (the one from in) is in. All the known formats are listed in SUN/256. Use this flag when you already know what format your table is. If it's set to the special value (auto) (which is the default), then it'll try to detect the format for you automatically. Not always done correctly though — when that happens the program exits with an error explaining which formats it attempted. This parameter is ignored for scheme-specified tables.

The same format parameter applies to all the tables you specified with in.

  • multi=true|false — decides whether we use all the tables, or just the first one, from the input table files. If false, only the first table from each file named by in gets used. If true, every table present in those input files gets used. This only does anything for file formats capable of holding more than one table — which effectively means FITS and VOTable and their variants. 🫶
  • istream=true|false — if true, the input table given by the in parameter gets read as a stream. You must give the ifmt parameter in this case. Depending on the operations you need and the processing mode, this can make the read fail (sometimes the table has to be read more than once, oops). Normally you don't need to set this flag — in most cases the data streams automatically if that's the best thing to do. But sometimes it can mean less resource usage when you're processing huge files in certain formats (like VOTable). This parameter is ignored for scheme-specified tables.

The same streaming flag applies to all the tables you specified with in.

  • icmd=<cmds> — processing to be performed on each input table specified by in, before anything else has happened. The value is one or more of the filter commands described in SUN/256. If you give more than one, separate them with semicolon characters (;). You can repeat this parameter multiple times on the same command line to stack up a list of processing steps. The sequence of commands you give this way defines the processing pipeline performed on the table. ✨

You can alternatively supply commands in an external file using the indirection character @. So a value of @filename reads the file filename for a list of filter commands to execute. Commands in that file can be separated by newline characters and/or semicolons, and lines that are blank or start with a # character are ignored. A backslash character (\) at the end of a line joins it with the following line.

  • ocmd=<cmds> — processing to be performed on the output table, after all other processing has taken place. Same deal as above: value is one or more of the filter commands described in SUN/256, more than one must be separated by semicolon characters (;), and you can repeat this parameter multiple times on the same command line to build up a list of processing steps. The sequence of commands you give defines the processing pipeline performed on the table. 💖

You can alternatively supply commands in an external file using the indirection character @. So a value of @filename reads the file filename for a list of filter commands to execute. Commands in that file can be separated by newline characters and/or semicolons, and lines that are blank or start with a # character are ignored. A backslash character (\) at the end of a line joins it with the following line.

  • omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui — the mode your result table gets output in. Default mode is out, meaning the result gets written as a new table to disk or wherever, as determined by the out and ofmt parameters. But there are other possibilities, matching all the other things you might want to do with a table instead of outputting it — like displaying metadata, calculating statistics, or populating a table in an SQL database. For some values of this parameter you'll need extra parameters (<mode-args>) to pin down the exact behaviour.

Possible values are:

  • out
  • meta
  • stats
  • count
  • checksum
  • cgi
  • discard
  • topcat
  • samp
  • tosql
  • gui

Use the help=omode flag or see SUN/256 for more info, iykyk.

  • out=<out-table> — where the output table goes. Usually a filename to write to. If it equals the special value - (the default), the output table gets written to standard output.

You must only give this parameter if omode has its default value of out.

  • ofmt=<out-format> — the format the output table will be written in (one of the ones in SUN/256 — matching is case-insensitive and you can get away with just the first few letters). If it has the special value (auto) (the default), then the output filename gets examined to try to guess what sort of file you want, usually by looking at the extension. If it's not obvious from the filename what output format is intended, you get an error. 😭

You must only give this parameter if omode has its default value of out.

  • seqcol=<colname> — name of a column to add to the output table, containing the sequence number of the input table each row came from. That column holds 1 for rows from the first concatenated table, 2 for the second, and so on. 👑
  • loccol=<colname> — name of a column to add to the output table, containing the location (as specified in the input parameter(s)) of the input table each row came from.
  • uloccol=<colname> — name of a column to add to the output table, containing the unique part of the location (as specified in the input parameter(s)) of the input table each row came from. If it's not null, parameters also get added to the output table giving the pre- and post-fix string common to all the locations. For example, if the input tables are "/data/cat_a1.fits" and "/data/cat_b2.fits" then the output table will contain a new column <colname> which takes the value "a1" for rows from the first table and "b2" for rows from the second, plus new parameters "<colname>_prefix" and "<colname>_postfix" with the values "/data/cat_" and ".fits" respectively. Obsessed with how clean that is.
  • lazy=true|false — whether to do table resolution lazily. If true, each table is only accessed when the time comes to add its rows to the output; if false, all the tables are accessed up front. This is mostly a tuning parameter and honestly it doesn't matter much how you set it — but if you're joining an enormous number of tables, setting it true may keep you from running out of resources. 💀
  • countrows=true|false — whether to count the rows in the table before starting the output. This is essentially a tuning parameter: if you're writing to an output format that needs the number of rows up front (like normal FITS) it may mean skipping the number of passes through the input files needed for processing. Unless you really understand the internals of this software, your best bet for working out whether to set this true or false is to try it both ways. Fr fr.

👀 SEE ALSO 👀

If the package stilts-doc is installed, the full documentation SUN/256 is available in HTML format: file:///usr/share/doc/stilts/sun256/index.html

🌸 VERSION 🌸

STILTS version 3.5.4-debian

This is the Debian version of Stilts, which lack the support of some file formats and network protocols. For differences see file:///usr/share/doc/stilts/README.Debian

💖 AUTHOR 💖

Mark Taylor (Bristol University)

Okay bestie that's the whole thing 🎀 massive shoutout to Mark Taylor (Bristol University) for writing it, and this is STILTS version 3.5.4-debian, the Debian build from way back in Mar 2017 — I know, ancient, but she ate and left no crumbs. Go read a table or something. Periodt. 💅✨🦋

📦 where this page comes from