STILTS-TMATCH1(1)

📦
version 3.5.4 · 5 versions
, in ubuntu stonking

💖 stilts-tmatch1 💖 — the Stilts command that does an internal crossmatch

✨ NAME ✨

stilts-tmatch1 — performs a crossmatch internal to a single table 🫶 like, one table, matching itself, no second table needed, we love a self-sufficient queen 👑

🌸 SYNOPSIS 🌸

       stilts tmatch1 [matcher=<matcher-name>] [params=<match-params>]
                      [tuning=<tuning-params>] [values=<expr-list>]
                      [action=identify|keep0|keep1|wide2|wideN]
                      [progress=none|log|time|profile]
                      [runner=parallel|parallel<n>|parallel-
                      all|sequential|classic|partest] [ifmt=<in-format>]
                      [istream=true|false] [in=<table>] [icmd=<cmds>]
                      [ocmd=<cmds>]
                      [omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui]
                      [out=<out-table>] [ofmt=<out-format>]

💅 DESCRIPTION 💅

okay so tmatch1 does efficient and flexible crossmatching between the rows of a single table. one table. rows matching each other. that's the whole vibe ✨

and bestie, it's flexible-flexible. it can match rows based on their relative position in the sky, or honestly a whole bunch of other criteria — separation in some isotropic or anisotropic Cartesian space, identity of a key value, or literally any combination of those. the full range of match criteria is discussed in SUN/256 (iykyk, go read it 📖).

the basic task the intra-table matcher does is identify groups of rows within the table which match each other. if you want the exact definition of what actually constitutes a match group, that's in SUN/256 too — no cap. and then the result of finding those groups gets expressed as an output table in one of a bunch of different ways, which you pick with the action parameter 🎀

those options include marking group membership in added columns and eliminating some or all rows which form part of a match group. choose your fighter 💅

👑 OPTIONS 👑

  • matcher=<matcher-name> — this defines the nature of the matching that's gonna happen 🦋 depending on the name you hand over, it might be positional matching using celestial or Cartesian coordinates, exact matching on the value of a string column, or other things entirely. a list and explanation of the available matching algorithms lives in SUN/256. the value you give for this parameter is what determines the meanings of the values required by params, values* and the tuning parameter(s) 👑
  • params=<match-params> — sets the parameters of this match. typically it's one or more tolerances like error radii. it can hold zero or more values, and which values are required depends on the match type selected by the matcher parameter. if it contains multiple values they must be separated by spaces; values which contain a space can be 'quoted' or "quoted" 💅
  • tuning=<tuning-params> — tuning values for the matching process, if appropriate 🎚️ it may contain zero or more values, and which values are permitted depends on the match type selected by the matcher parameter. multiple values must be separated by spaces; values which contain a space can be 'quoted' or "quoted". and if you just... don't supply this optional parameter at all? sensible defaults will be chosen 🫶 so it's fine, breathe.
  • values=<expr-list> — defines the values from the input table which are used to figure out whether a match happened 💖 these will typically be coordinate values like RA and Dec, and maybe some per-row error values as well — though exactly what values are required is determined by the kind of match, which comes from matcher. depending on the kind of match, the number and type of the values required will be different. multiple values should be separated by whitespace; if whitespace occurs within a single value it must be 'quoted' or "quoted". elements of the expression list are commonly just column names, but they may also be algebraic expressions calculated from zero or more columns, as explained in SUN/256 🧮
  • action=identify|keep0|keep1|wide2|wideN — decides what form of table comes out as a result of the internal match 🎀
    • identify: the output table is the same as the input table, except it contains two additional columns, GroupID and GroupSize, following the input columns. each group of rows which matched gets a unique integer assigned to it, recorded in the GroupID column, and the size of each group is recorded in the GroupSize column. rows which don't match any others (the singles 💔) have null values in both these columns.
    • keep0: the result is a new table containing only "single" rows — that is, ones which don't match any other rows in the table. any other rows are thrown out 🗑️ bye.
    • keep1: the result is a new table in which only one row (the first in the input table order) from each group of matching ones is retained. so a subsequent intra-table match with the same criteria would therefore show no matches. slay, deduplicated.
    • wideN: the result is a new "wide" table consisting of matched rows in the input table stacked next to each other 🫂 only groups of exactly N rows in the input table are used to form the output table; each row of the output table consists of the columns of the first group member, followed by the columns of the second group member and so on. the output table therefore has N times as many columns as the input table. the column names in the new table have _1, _2, ... appended to them to avoid duplication 👑
  • progress=none|log|time|profile — decides whether info about progress of the match gets written to the standard error stream while it's running. for lengthy matches this is a useful reassurance and can give you guidance about how much longer it'll take ⏳ it can also be useful as a performance diagnostic.

the options are:

  • none: no progress is shown 🙈
  • log: progress information is shown 📋
  • time: progress information plus some time profiling information is shown ⏱️
  • profile: progress information plus limited time/memory profiling information is shown 📊
  • runner=parallel|parallel<n>|parallel-all|sequential|classic|partest — selects the threading implementation 🧵 the options are currently:
    • parallel: uses multithreaded implementation for large tables, with default parallelism, which is the smaller of 6 and the number of available processors
    • parallel<n>: uses multithreaded implementation for large tables, with parallelism given by the supplied value n
    • parallel-all: uses multithreaded implementation for large tables, with a parallelism given by the number of available processors
    • sequential: uses multithreaded implementation but with only a single thread
    • classic: uses the legacy sequential implementation
    • partest: uses multithreaded implementation even when tables are small

the parallel* options should normally run faster than sequential or classic (which are provided mainly for testing purposes), at least for large matches and where multiple processing cores are available 🔥

the default value "parallel" is currently limited to a parallelism of 6 since larger values yield diminishing returns given that some parts of the matching algorithms run sequentially (Amdahl's Law, bestie, it's real), and using too many threads can sometimes end up doing more work or impacting on other operations on the same machine 😭 but you can experiment with other concurrencies, e.g. "parallel16" to run on 16 cores (if available) or "parallel-all" to run on all available cores 💅

the value of this parameter should make no difference to the matching results. if you notice any discrepancies please report them 🫶
  • ifmt=<in-format> — specifies the format of the input table as specified by parameter in. the known formats are listed in SUN/256 📚 this flag can be used if you know what format your table is in. if it has the special value (auto) (which is the default), then an attempt will be made to detect the format of the table automatically ✨ this cannot always be done correctly however, in which case the program will exit with an error explaining which formats were attempted 💀 this parameter is ignored for scheme-specified tables.
  • istream=true|false — if set true, the input table specified by the in parameter will be read as a stream 🌊 it is necessary to give the ifmt parameter in this case. depending on the required operations and processing mode, this may cause the read to fail (sometimes it is necessary to read the table more than once) 😭 it's not normally necessary to set this flag; in most cases the data will be streamed automatically if that's the best thing to do. however it can sometimes result in less resource usage when processing large files in certain formats (such as VOTable) 🫶 this parameter is ignored for scheme-specified tables.
  • in=<table> — the location of the input table 📍 this may take one of the following forms:
    • a filename 📄
    • a URL 🔗
    • the special value "-", meaning standard input. in this case the input format must be given explicitly using the ifmt parameter. note that not all formats can be streamed in this way ⚠️
    • a scheme specification of the form :<scheme-name>:<scheme-args> 🧚
    • a system command line with either a "<" character at the start, or a "|" character at the end ("<syscmd" or "syscmd|") 💻 this executes the given pipeline and reads from its standard output. this will probably only work on unix-like systems.

in any case, compressed data in one of the supported compression formats (gzip, Unix compress or bzip2) will be decompressed transparently ✨

  • icmd=<cmds> — okay so before literally anything else happens to your input table (the one from the in parameter), this is where you get to boss it around. 💅 The value is one or more of the filter commands described in SUN/256. Got more than one? Separate them with semicolons (;). You can repeat this parameter as many times as you want on the same command line to stack up processing steps — and that sequence is literally the processing pipeline that gets performed on the table. ✨

You can also put the commands in an external file using the indirection character @. So a value of "@filename" makes it read filename for a list of filter commands to execute. In that file, commands may be separated by newline characters and/or semicolons, and any lines that are blank or start with a # character are ignored. Also a backslash character (\fR) at the end of a line joins it with the following line. 🌸

  • ocmd=<cmds> — this is the after-party for your output table 🎉 — it runs after all other processing has taken place. The value is one or more filter commands described in SUN/256. More than one? Semicolon-separate them (;). Repeat this parameter multiple times on the same command line to build a list of processing steps — that sequence defines the processing pipeline for the table. 💖

Commands can also be supplied in an external file via the indirection character @. So "@filename" reads filename for a list of filter commands to execute. In the file, commands may be separated by newlines and/or semicolons; blank lines or lines starting with # are ignored. A backslash character (\fR) at the end of a line joins it with the following line. 🌈

  • omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui — the mode in which the result table will be output. Default is out, meaning the result gets written as a new table to disk or elsewhere, as determined by the out and ofmt parameters. But bestie, there are other possibilities that correspond to uses for a table other than outputting it — like displaying metadata, calculating statistics, or populating a table in an SQL database. For some values of this parameter, extra parameters (<mode-args>) are required to determine exact behaviour. 🧐

Possible values are:

  • out
  • meta
  • stats
  • count
  • checksum
  • cgi
  • discard
  • topcat
  • samp
  • tosql
  • gui

Use the help=omode flag or see SUN/256 for more info. 📚

  • out=<out-table> — where the output table goes. Usually a filename to write to. If it's the special value "-" (the default), the output table is written to standard output. 📝
Heads up: this parameter must only be given if omode has its default value of "out". No exceptions, bestie. 🚫
  • ofmt=<out-format> — the format the output table gets written in (one of the ones in SUN/256 — matching is case-insensitive and you can use just the first few letters). If it's the special value "(auto)" (the default), then the output filename is examined to try to guess what sort of file is required, usually by looking at the extension. If it's not obvious from the filename what output format is intended, an error will result. 😭
Another heads up: this parameter must only be given if omode has its default value of "out". 💯

👀 SEE ALSO 👀

If the package stilts-doc is installed, the full documentation SUN/256 is available in HTML format:

file:///usr/share/doc/stilts/sun256/index.html

🌟 VERSION 🌟

STILTS version 3.5.4-debian — slay. This is the Debian version of Stilts, which lacks support for some file formats and network protocols. For differences, see:

file:///usr/share/doc/stilts/README.Debian

✍️ AUTHOR ✍️

Mark Taylor (Bristol University) — iconic. 👑

And that's the whole page, bestie! This manual page is STILTS-TMATCH1(1) from STILTS version 3.5.4-debian, dated Mar 2017. Go forth and match those tables. 💖✨🦋

📦 where this page comes from