STILTS-ARRAYJOIN(1)

📦
version 3.5.4 · 4 versions
, in ubuntu stonking

💖 STILTS-ARRAYJOIN(1) 💖

okay so first things first bestie: this page lives in the Stilts commands part of the manual (section 1, like all the other stilts crew). Go grab a snack 🍓 because she's a long one, but we're gonna get through it together, no cap.

✨ NAME ✨

stilts-arrayjoin — Adds table-per-row data as array-valued columns 💅

🌸 SYNOPSIS 🌸

       stilts arrayjoin [ifmt=<in-format>] [istream=true|false] [in=<table>]
                        [icmd=<cmds>] [ocmd=<cmds>]
                        [omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui]
                        [out=<out-table>] [ofmt=<out-format>] [atable=<loc-
                        expr>] [afmt=<in-format>] [astream=true|false]
                        [acmd=<cmds>] [keepall=true|false] [aparams=<name-
                        list>] [cache=true|false] [fixcols=none|dups|all]
                        [suffixarray=<label>]

🦋 DESCRIPTION 🦋

So here's the vibe: arrayjoin takes an input table, and for every single row it goes and grabs the contents of a whole separate "array" table and staple-guns it on. ✨

The columns it adds are literally the columns from the array table. And the value of each cell? It's the value of the whole column from the array table, squished down into a single array value. That's the whole trick, that's the magic. 🪄

The assumption running under all of this: every one of those array tables has the same form — same columns — though not necessarily the same row counts. Row counts can absolutely differ, she doesn't care about that.

When is this a slay? When you're building one single table that has array-valued columns, where the data is actually scattered across multiple external files, like via the DataLink protocol. There's a whole Examples subsection below illustrating it. 💖

⚠️ but hold on, important: this command does not understand DataLink directly. She can't figure out where the external array tables live all by herself. You have to hand her an expression that gives their per-row location (filename or URL). Periodt.

👑 OPTIONS 👑

Yes bestie, there are a lot. We're doing all of them. Sit down, hydrate. 🫶

  • ifmt=<in-format> — Tells her the format of the input table, the one named by the in parameter. All the known formats are listed in SUN/256. Use this when you already know what format your table is in. If you give it the special value (auto) — which is the default — she'll try to detect the format automatically. Heads up though: automatic detection is not always gonna get it right, and when it doesn't, the program exits with an error explaining which formats it tried. 👀 Also, this parameter gets ignored for scheme-specified tables.
  • istream=true|false — Set it true and the input table from the in parameter is read as a stream. You must give ifmt if you do this, no exceptions. Depending on what operations you're asking for and what processing mode you're in, this might make the read straight-up fail — sometimes the table needs to be read more than once. 😭 It's not normally necessary to set this flag; most of the time the data gets streamed automatically when that's the best call. But sometimes it can use less resources when you're processing big files in certain formats (VOTable, for instance). Ignored for scheme-specified tables.
  • in=<table> — Where the input table is at. She'll accept any of these, and honestly? range:
    • A filename. 📄
    • A URL. 🌐
    • The special value -, meaning standard input. If you use this, you must give the input format explicitly with the ifmt parameter. Also note: not every format can be streamed this way.
    • A scheme specification of the form :<scheme-name>:<scheme-args>.
    • A system command line with either a < character at the start, or a | character at the end ("<syscmd" or "syscmd|"). This runs that pipeline and reads from its standard output. Realistically this will probably only work on unix-like systems.

In any of these cases, compressed data in one of the supported compression formats — gzip, Unix compress or bzip2 — gets decompressed transparently. You don't have to do anything. 💅

  • icmd=<cmds> — Processing to be done on the input table specified by in, before any other processing happens. The value is one or more of the filter commands described in SUN/256. If you give more than one, separate them with semicolon characters (";"). You can repeat this parameter multiple times on the same command line to build up a list of processing steps — the sequence of commands you give that way defines the processing pipeline performed on the table.

Alternatively, you can supply commands in an external file using the indirection character '@'. So a value of "@filename" makes the file filename get read for a list of filter commands to execute. Commands inside the file can be separated by newline characters and/or semicolons, and lines that are blank or that start with a '#' character are ignored. A backslash character '\fR' at the end of a line joins it with the following line. 🧚

  • ocmd=<cmds> — Same deal as icmd, except it's processing on the output table, and it happens after all other processing has taken place. Value is one or more filter commands from SUN/256, more than one means semicolons (";") between them, and yes you can repeat the parameter multiple times on one command line to build up the step list. The sequence defines the processing pipeline performed on the table.

External file time again: use the indirection character '@', so "@filename" reads the file filename for a list of filter commands to execute. Commands may be separated by newline characters and/or semicolons, blank lines and lines starting with '#' are ignored, and a backslash character '\fR' at the end of a line joins it with the following line. 🎀

  • omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui — The mode your result table gets output in. Default is out, meaning the result is written as a new table to disk or wherever, as decided by the out and ofmt parameters. But that's not the only option — there are other possibilities that correspond to other uses for a table besides printing it out, like displaying metadata, calculating statistics, or populating a table in an SQL database. 🗄️ For some values of this parameter you'll also need extra parameters (<mode-args>) to pin down the exact behaviour.

The possible values are:

  • out
  • meta
  • stats
  • count
  • checksum
  • cgi
  • discard
  • topcat
  • samp
  • tosql
  • gui

Use the help=omode flag or see SUN/256 for more on this one, iykyk. 🔥

  • out=<out-table> — Where the output table goes. Usually a filename to write to. If it's the special value - (which, btw, is the default), the output table gets written to standard output.
⚠️ You may only give this parameter if omode has its default value of "out". Don't try it otherwise, she will not be happy.
  • ofmt=<out-format> — The format the output table gets written in. Pick one of the ones in SUN/256 — matching is case-insensitive and you can get away with just the first few letters, we love a shortcut. ✨ If it has the special value (auto) (the default), the output filename gets examined to guess what sort of file is required, usually by looking at the extension. If it's not obvious from the filename what output format you meant, you get an error.
⚠️ Same rule as out: only give this if omode is sitting at its default value of "out".
  • atable=<loc-expr> — Where's the table whose rows are gonna get turned into an array-valued column? Right here. This is generally an expression giving a URL or filename that's different for each row of the input table. If loading the table fails for the given location — because the file isn't found, or an HTTP 404 comes back, whatever 😭 — then the array cells in the corresponding row will just be blank.

The first non-blank table that loads is the one that defines which array columns get added. If later tables have a different structure (meaning they don't contain similar columns in a similar sequence), you may get an error. If your external array tables aren't all homogenous in this way, you can use the acmd parameter to filter them until they are. 💅

  • afmt=<in-format> — Specifies the format of the array tables, the ones named by the atable parameter. Known formats live in SUN/256. Use this if you already know what format your table is in. If it has the special value (auto) — the default — she'll try to detect the format automatically. That can't always be done correctly, and when it can't, the program exits with an error explaining which formats were attempted. Ignored for scheme-specified tables. 🦋
  • astream=true|false — Set it true and the array tables named by atable get read as a stream. You must give the afmt parameter in that case. Depending on the operations required and the processing mode, this might make the read fail (sometimes the table has to be read more than once, it's a whole thing). You don't normally need to set this flag — most of the time the data streams automatically when that's the best move. But sometimes it can mean less resource usage when processing large files in certain formats (such as VOTable). Ignored for scheme-specified tables.
  • acmd=<cmds> — Processing to be done on the array tables named by atable, before any other processing has taken place. Value is one or more filter commands described in SUN/256. More than one? Semicolon characters (";") between them. Repeat the parameter multiple times on the same command line and you build up a list of processing steps — that sequence defines the processing pipeline performed on the table. 🌈
🫶 okay bestie, that's the end of this part — more is coming, don't go anywhere 💖

okay bestie, we're picking up right where we left off, mid-OPTIONS, no time to breathe 💅

Oh and also? Commands don't have to live inside the parameter at all — you can yeet them into an external file and just point at it with the indirection character '@'. So like, a value of "@filename" means "go read the file called filename, it's got a list of filter commands for you to execute." Inside that file, commands can be separated by newlines and/or semicolons, and any line that's blank or that starts with a '#' is simply ignored. And a backslash character '\fR' at the end of a line joins it with the following line. Continuation lines, we love to see it 🫶

  • keepall=true|false — this one decides what happens when the atable parameter points at a table that straight up can't be loaded. If it's false, the input table row still gets output, just with blank values in the columns that the array tables were supposed to fill — so the output table ends up with the same number of rows as the input table. If it's true, only the rows whose tables actually loaded successfully make it into the output. No failed table, no row, periodt.
  • aparams=<name-list> — lists the table parameters (that's the per-table metadata, iykyk) that get read from the loaded tables and turned into scalar-valued columns in the output. By default these parameters get discarded 💀, but you can drag them into the output by naming them here.

You supply the parameters as a space- or comma-separated list. Matching against table names is case-insensitive, and the asterisk character "*" can be used as a wildcard to match any sequence of characters. The list is interpreted relative to the first external table that gets loaded — so if you pass "*", you get a column for every single parameter in that first loaded table. Maximalism, iconic behavior ✨

  • cache=true|false — decides whether the array data gets cached the first time an array table is read (true) or whether it's re-read from the array table every single time the row is accessed (false). Row construction can be a whole expensive step, especially if the tables are being downloaded, so honestly it usually makes sense to set this to true — which is, no surprise, the default. Setting it true also enables the metadata to be adjusted so it reports constant array length where applicable; that's something that literally cannot be done until all the rows have been scanned, and it may enable more efficient file output. But if you're trying to stream the data? Set it false, bestie.
  • fixcols=none|dups|all — determines how input columns get renamed before they're used in the output table. Choices are:
    • none: columns are not renamed
    • dups: columns which would otherwise have duplicate names in the output will be renamed to indicate which table they came from
    • all: all columns will be renamed to indicate which table they came from

If columns are renamed, the new ones are determined by suffix* parameters.

  • suffixarray=<label> — if fixcols is set so that input columns get renamed for insertion into the output table, this parameter decides how the renaming is done. It gives a suffix which is appended to all renamed columns from the array tables. name construction but make it fashion 💅

✨ SEE ALSO ✨

If the package stilts-doc is installed, the full documentation SUN/256 is available in HTML format at: file:///usr/share/doc/stilts/sun256/index.html

✨ VERSION ✨

STILTS version 3.5.4-debian

This is the Debian version of Stilts, which lack the support of some file formats and network protocols. For differences see file:///usr/share/doc/stilts/README.Debian

✨ AUTHOR ✨

Mark Taylor (Bristol University)

and that's the whole page, bestie, you ate that 💖 the version on this one was STILTS version 3.5.4-debian, it was dated Mar 2017, and it came to you from the STILTS-ARRAYJOIN(1) manual — go forth and join those arrays fr fr 🦋👑

📦 where this page comes from