STILTS-TAPSKYMATCH(1)

📦
version 3.5.4 · 5 versions
, in ubuntu stonking

💫 STILTS-TAPSKYMATCH(1) 💫

okay bestie, so this is stilts-tapskymatch — a Stilts command, manual section 1. it's giving "let me crossmatch your little local table against a whole TAP service and make it look effortless" and honestly? iconic. 🛰️✨

🎀 NAME 🎀

stilts-tapskymatch — crossmatches a table on sky position against a TAP table 🌌💖

📜 SYNOPSIS 📜

stilts tapskymatch [ifmt=<in-format>] [istream=true|false] [in=<table>]
                   [icmd=<cmds>] [ocmd=<cmds>]
                   [omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui]
                   [out=<out-table>] [ofmt=<out-format>]
                   [inlon=<expr/deg>] [inlat=<expr/deg>] [tapurl=<url-
                   value>] [interface=tap1.0|tap1.1|cap]
                   [auth=true|false] [taptable=<name>] [taplon=<column>]
                   [taplat=<column>] [tapcols=<colname,...>]
                   [sr=<expr/deg>] [find=all|best|each|each-dist]
                   [blocksize=<int-value>] [maxrec=<int-value>]
                   [sync=true|false] [blockmaxrec=<nrow>]
                   [compress=true|false] [fixcols=none|dups|all]
                   [suffixin=<label>] [suffixremote=<label>]

💅 DESCRIPTION 💅

so tapskymatch lets you do a positional crossmatch between a table you've got locally and one that lives in a remote TAP service — as long as that TAP supports upload queries. no cap, that's the whole entry fee. 🎟️

this task does three main jobs, and it's lowkey serving in all of them:

  1. it prepares the ADQL queries and the TAP negotiations for you, so you literally never have to remember the syntax for doing positional crossmatches against a TAP service. the syntax stays in its lane, you stay in yours. 🫶
  2. it organises the data transfer so that only the columns that are actually required get sent to and from the service — basically the positional ones — which saves you bandwidth. frugal queen behaviour. 💸
  3. it divides the job up into chunks, so the TAP service only ever has to run a query of a manageable size at a time. if the job is large, this chunking is genuinely useful for monitoring the progress of the job, and it also lets you pull off a match that would otherwise hit the upload or output limits imposed by the service. we love a workaround that isn't even sketchy. 🔥

heads up bestie, the positional match can be done in any spherical coordinate system — it is up to you, the user, to make sure that the same coordinates are provided for the local and the remote tables. the task is not gonna do the vibe check on your coordinate frames for you. 🙈

also, quick note: cdsskymatch provides similar functionality by accessing a different external service, and that one is usually much faster. so if the table you want to match is part of the VizieR database, you may wish to use that command instead. just saying. 🧚

✨ OPTIONS ✨

  • ifmt=<in-format> — tells everyone what the format of the input table is, the one you set with the in parameter. all the known formats are listed in SUN/256. use this flag when you already know what format your table is in. if it's set to the special value (auto) — which is the default — then the program will have a go at detecting the format of the table automatically. and here's the tea: it cannot always be done correctly, and in that case the program will exit with an error explaining which formats it tried. also, this parameter is ignored for scheme-specified tables. 💀
  • istream=true|false — if you set this true, the input table you named with the in parameter gets read as a stream. in that case you must give the ifmt parameter too, no skipping. depending on the required operations and the processing mode, this may cause the read to fail — sometimes it's necessary to read the table more than once, and streaming doesn't love that. it is not normally necessary to set this flag; in most cases the data will be streamed automatically if that's the best thing to do. <emphasis>but</emphasis> it can sometimes result in less resource usage when you're processing large files in certain formats (like VOTable). this parameter is ignored for scheme-specified tables. 🦋
  • in=<table> — the location of the input table. bestie, it can take one of these forms:
    • a filename. 📄
    • a URL. 🌐
    • the special value "-", which means standard input. in this case the input format must be given explicitly using the ifmt parameter. and note that not all formats can be streamed in this way. 🙈
    • a scheme specification of the form :<scheme-name>:<scheme-args>.
    • a system command line with either a "<" character at the start, or a "|" character at the end ("<syscmd" or "syscmd|"). this executes the given pipeline and reads from its standard output. realistically this will probably only work on unix-like systems. 💅

in any case, compressed data in one of the supported compression formats (gzip, Unix compress or bzip2) will be decompressed transparently. you don't even have to ask. ✨

  • icmd=<cmds> — processing to be performed on the input table named by the in parameter, before any other processing has taken place. the value is one or more of the filter commands described in SUN/256. if you give more than one, they must be separated by semicolon characters (";"). you can repeat this parameter multiple times on the same command line to build up a list of processing steps — and the sequence of commands you give this way defines the processing pipeline performed on the table. stacking, but make it science. 🧚

commands may alternatively be supplied in an external file, by using the indirection character '@'. so a value of "@filename" causes the file filename to be read for a list of filter commands to execute. the commands in that file may be separated by newline characters and/or semicolons, and lines which are blank or which start with a '#' character are ignored. a backslash character '\fR' at the end of a line joins it with the following line. 🌸

  • ocmd=<cmds> — processing to be performed on the output table, after all other processing has taken place. the value is one or more of the filter commands described in SUN/256. if more than one is given, they must be separated by semicolon characters (";"). you can repeat this parameter multiple times on the same command line to build up a list of processing steps, and the sequence of commands given this way defines the processing pipeline performed on the table. same energy as icmd, different end of the pipe. 💜

commands may alternatively be supplied in an external file using the indirection character '@', so "@filename" makes the file filename get read for a list of filter commands to execute. commands in the file may be separated by newline characters and/or semicolons, blank lines or lines starting with '#' are ignored, and a backslash character '\fR' at the end of a line joins it with the following line. 🎀

  • omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui — the mode in which the result table gets output. default mode is out, which means the result is written as a new table to disk or elsewhere, as determined by the out and ofmt parameters. but there are other possibilities, and they correspond to other uses a table can be put to besides just outputting it — like displaying metadata, calculating statistics, or populating a table in an SQL database. for some values of this parameter, additional parameters (&lt;mode-args&gt;) are required to determine the exact behaviour. 👑

possible values are:

  • out
  • meta
  • stats
  • count
  • checksum
  • cgi
  • discard
  • topcat
  • samp
  • tosql
  • gui

use the help=omode flag, or go read SUN/256, for more info. iykyk. 📚

  • out=<out-table> — the location of the output table. usually this is a filename to write to. if it's equal to the special value "-" (the default), the output table gets written to standard output. 🖨️
this parameter must only be given if omode has its default value of "out". otherwise, no thank you. 🚫
  • ofmt=<out-format> — the format the output table will be written in (one of the ones in SUN/256 — matching is case-insensitive and you can use just the first few letters, cute right?). if it has the special value (auto) (the default), then the output filename gets examined to try to guess what sort of file is required, usually by looking at the extension. if it's not obvious from the filename what output format is intended, an error will result. no guessing games, the program will simply refuse. 😭
this parameter must only be given if omode has its default value of "out". same rule as out, fr fr. 💅
  • inlon=<expr/deg> — longitude in degrees for the position of each row in the input table. this may simply be a column name, or it may be an algebraic expression as explained in SUN/256. the coordinate system must match the one used for the coordinates in the remote table. 🧭
  • inlat=<expr/deg> — longitude in degrees for the position of each row in the input table. yes bestie, the page really does say "Longitude" here too — that's not me being chaotic, that's the original, i'm just the messenger 🙈. this may simply be a column name, or it may be an algebraic expression as explained in SUN/256. the coordinate system must match the one used for the coordinates in the remote table. 🌈
  • tapurl=<url-value> — the base URL of a Table Access Protocol service. this is the bare URL, without a trailing "/[a]sync". keep it clean. ✨

in the usual case, the default values of the various endpoints (sync and async query submission, tables metadata, service-provided examples, etc.) use this URL as a parent and append standard sub-paths. 🦋

in some cases though, determination of the endpoints is more complicated, as determined by the interface parameter — which may cause endpoints to be read from the capabilities document at tapurl/capabilities — and by other endpoint-specific parameters (syncurl, asyncurl, tablesurl, capabilitiesurl, availabilityurl, examplesurl) for fine tuning. yes, that's a lot of URLs. we're fine. we're totally fine. 💀

  • interface=tap1.0|tap1.1|cap — defines how the service endpoints and the version of the TAP protocol used for queries get determined. it can take one of the following (case-insensitive) values:
    • TAP1.0: the standard TAP endpoints are used, without examining the service's capabilities document. the service is queried using version 1.0 of the TAP protocol.
    • TAP1.1: the standard TAP endpoints are used, without examining the service's capabilities document. the service is queried using version 1.1 of the TAP protocol.
    • cap: the service's capabilities document is examined, and the endpoints listed there are used. 👑

Like, the capabilities doc — if it even gets used — gets read from tapurl/capabilities, unless the capabilitiesurl parameter is defined, in which case that one wins. Obviously. 💅

And the baseline for all your TAP service endpoints (sync, async, tables, capabilities, examples) comes from this parameter. But every single one of those can be overridden individually by its own endpoint-specific param: syncurl, asyncurl, tablesurl, capabilitiesurl, availabilityurl, examplesurl. Mix and match, bestie, it's your world. 🌈

For default (unauthenticated) access, the default value is usually suitable. ✨

  • auth=true|false — if true, we make an attempt to authenticate with the TAP service even if anonymous operation is permitted. If the service offers authentication, you will be asked for credentials. 👑

To use this option in non-interactive contexts, you may want to use the auth.username and auth.password system properties. (iykyk — no typing your password into a prompt that can't hear you 😭)

  • taptable=<name> — name of the table in the given TAP service against which the matching will be performed.
  • taplon=<column> — longitude in degrees for the position of each row in the remote table. This is an ADQL expression interpreted within the TAP service, typically just a column name. The coordinate system must match that used for the input table. 🦋
  • taplat=<column> — latitude in degrees for the position of each row in the remote table. Also an ADQL expression interpreted within the TAP service, typically just a column name. Coordinate system must match the input table. Same energy, different axis. ✨
  • tapcols=<colname,...> — comma-separated list of column names to retrieve from the remote table. If no value is supplied (that's the default), all columns from the remote table come back. Not me asking for everything because I can 😌
  • sr=<expr/deg> — maximum distance in degrees from the local table (lat,lon) position at which counterparts from the remote table will be identified. This is an ADQL expression interpreted within the TAP service, so it may be a constant value or may involve columns in the remote table. Slay, flexible queen. 🔥
  • find=all|best|each|each-dist — determines which pair matches land in the result. The choices:
    • all: all matches ✅
    • best: matched rows, best remote row for each input row ✨
    • each: one row per input row, contains best remote match or blank
    • each-dist: one row per input row, column giving distance only for best match
Note: only the all mode is symmetric between the two tables. The other three are giving main character energy for the input table specifically. 💀
  • blocksize=<int-value> — the number of rows uploaded in each TAP query. TAP services may have limits on the number of rows in a table uploaded for matching, so this command can break up input tables into blocks and make a number of individual TAP queries to generate the result. This parameter controls the maximum number of rows uploaded in each individual request. For an input table with fewer rows than this value, the whole thing is done as a single query. One query, one vibe. 🎀
  • maxrec=<int-value> — limit to the number of rows resulting from this operation. If the value is negative (the default), no limit is imposed. But heads up: there can be truncation of the result if the number of records returned from a single chunk exceeds limits imposed by the service. 🙈
  • sync=true|false — determines whether the TAP queries are submitted in synchronous or asynchronous mode. Since this command uses chunking to keep requests to a reasonable size, hopefully requests will not take too long to execute, therefore the default is synchronous (true). ✨
  • blockmaxrec=<nrow> — sets the MAXREC parameter passed to the TAP service for each uploaded block. This allows you to request that the service overrides its default limit for the number of rows output from a single query. The service may still impose some "hard" limit beyond which the output row count cannot be increased. 🚧
Note this differs from the maxrec parameter, which gives the maximum total number of rows to be returned from this command. blockmaxrec is per-block, maxrec is the whole thing. Do not confuse them, bestie, I'm begging. 😭
  • compress=true|false — if true, the service is requested to provide HTTP-level compression for the response stream (the Accept-Encoding header is set to "gzip", see RFC 2616). This does not guarantee that compression will happen, but if the service honours this request it may result in a smaller amount of network traffic at the expense of more processing on the server and client. Trade-offs, iconic behaviour honestly. 🌸
  • fixcols=none|dups|all — determines how input columns are renamed before use in the output table. The choices are:
    • none: columns are not renamed
    • dups: columns which would otherwise have duplicate names in the output will be renamed to indicate which table they came from
    • all: all columns will be renamed to indicate which table they came from

If columns are renamed, the new ones are determined by suffix* parameters. ✨

  • suffixin=<label> — if the fixcols parameter is set so that input columns are renamed for insertion into the output table, this parameter determines how the renaming is done. It gives a suffix which is appended to all renamed columns from the input table. 🎀
  • suffixremote=<label> — if the fixcols parameter is set so that input columns are renamed for insertion into the output table, this parameter determines how the renaming is done. It gives a suffix which is appended to all renamed columns from the TAP result table. 🦋

👀 SEE ALSO

  • stilts(1)
  • If the package stilts-doc is installed, the full documentation SUN/256 is available in HTML format: file:///usr/share/doc/stilts/sun256/index.html
  • (go read it, it's giving ✨scholarly✨)

🌟 VERSION

STILTS version 3.5.4-debian 👑

This is the Debian version of Stilts, which lack the support of some file formats and network protocols. For differences see file:///usr/share/doc/stilts/README.Debian. (yes it says "lack", we're quoting it verbatim, do not come for me 💅)

💖 AUTHOR

Mark Taylor (Bristol University)

and that's the whole thing, bestie 🫶 STILTS-TAPSKYMATCH(1), dated Mar 2017, from the stilts package — you absolutely ate that read, now go match some sky objects. periodt. 💅✨

📦 where this page comes from