STILTS-CONESKYMATCH(1)

📦
version 3.5.4 · 5 versions
, in ubuntu stonking

🦋 stilts-coneskymatch 🦋 — from the STILTS commands manual

ok so bestie. welcome to the girliepop guide to stilts-coneskymatch, the crossmatch-er that goes out to the internet on your behalf. she's a network girlie. she's a remote girlie. let's get into it 💖

✨ NAME ✨

stilts-coneskymatch — crossmatches table on sky position against remote cone service 🫶

✨ SYNOPSIS ✨

stilts coneskymatch [ifmt=<in-format>] [istream=true|false] [in=<table>]
                    [icmd=<cmds>] [ocmd=<cmds>]
                    [omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui]
                    [out=<out-table>] [ofmt=<out-format>] [ra=<expr>]
                    [dec=<expr>] [sr=<expr/deg>] [find=best|all|each]
                    [usefoot=true|false] [footnside=<int-value>]
                    [copycols=<colid-list>] [scorecol=<col-name>]
                    [parallel=<n>] [erract=abort|ignore|retry|retry<n>]
                    [ostream=true|false] [fixcols=none|dups|all]
                    [suffix0=<label>] [suffix1=<label>]
                    [servicetype=cone|ssa|sia1|sia2|sia]
                    [serviceurl=<url-value>] [verb=1|2|3]
                    [dataformat=<value>] [emptyok=true|false]
                    [compress=true|false]

✨ DESCRIPTION ✨

⚠️ read this first, bestie: this command is very inefficient for large tables, and in most cases cdsskymatch or tapskymatch are better alternatives. no cap. she said what she said.

coneskymatch is a utility that performs a cone search-like query to a remote server, once for each row of an input table. Each of those queries comes back with a table that has one row for every item the server is holding in the region of sky that the input row represents. Then all those query results get concatenated into one big output table, and that big table is the output of this command. literally just vibes and concatenation ✨

The type of virtual observatory service being queried is decided by the servicetype parameter. Usually it'll be a Cone Search service, which queries a remote catalogue for astronomical objects or sources sitting in a particular region. But plot twist 🎀 — you can also query Simple Image Access and Simple Spectral Access services in exactly the same way, to get back tables of the available image and spectral resources in the relevant regions.

The identity of the server you're querying is given by the serviceurl parameter. If you need advice on how to hunt down URLs for suitable services, that's covered in SUN/256.

The effect of this command is like doing a positional crossmatch where one of the catalogues is local and the other one is remote and exposes its data via a cone search / SIA / SSA service. However — because of the network communication involved, and because of the necessarily naive crossmatching algorithm (which scales linearly with the size of the local catalogue), she's only suitable if your local catalogue has a reasonably small number of rows… unless you're genuinely prepared to wait a very long time. 💀

The parallel parameter lets you run multiple cone searches concurrently. So instead of finishing the first cone search, then the second, then the third, the program can be executing a bunch of them at once. This can speed things up considerably, especially when network latency is in the picture 🌐 — but beware: submitting a very large number of queries simultaneously to the same server may overload it, and you'll end up with some combo of failed queries, ultimately slower runtimes, and unpopularity with server admins. 😭 bestie behavior is: start with a low parallelism and cautiously increase it to see whether there are actually any gains in performance.

Also, heads up: when running, coneskymatch can generate a lot of WARNING messages. Most of these are her complaining about badly formed VOTables being returned from the cone search services. STILTS does its best to work out what the service responses mean in this case, and usually makes a good enough job of it. 🙈

📝 Note: this task was known as multicone in its experimental form in STILTS v1.2 and v1.3. glow-up arc 👑

✨ OPTIONS ✨

  • ifmt=<in-format> — says what format the input table (the one from the in parameter) is in. All the known formats are listed in SUN/256. Use this flag when you already know what format your table is. If it has the special value (auto) — which is the default — then she'll attempt to detect the table's format automatically. That can't always be done correctly though, and when it can't, the program exits with an error explaining which formats were attempted. This parameter is ignored for scheme-specified tables.
  • istream=true|false — if you set it true, the input table given by the in parameter gets read as a stream. In that case you must give the ifmt parameter. Depending on what operations and processing mode you need, this may make the read fail (sometimes it's necessary to read the table more than once). Normally you don't need to set this flag — in most cases the data gets streamed automatically if that's the best thing to do. That said, it can sometimes result in less resource usage when processing large files in certain formats (such as VOTable). This parameter is ignored for scheme-specified tables.
  • in=<table> — where the input table lives. She accepts one of these forms:
    • A filename.
    • A URL.
    • The special value -, meaning standard input. If you do this, the input format must be given explicitly using the ifmt parameter. Note that not all formats can be streamed in this way.
    • A scheme specification of the form :<scheme-name>:<scheme-args>.
    • A system command line with either a < character at the start, or a | character at the end (<syscmd or syscmd|). This executes the given pipeline and reads from its standard output. This will probably only work on unix-like systems.

In any case, compressed data in one of the supported compression formats (gzip, Unix compress or bzip2) will be decompressed transparently. 🎁

  • icmd=<cmds> — specifies processing to be performed on the input table (the one from in) before any other processing has taken place. The value is one or more of the filter commands described in SUN/256. If you give more than one, they must be separated by semicolon characters (;). You can repeat this parameter multiple times on the same command line to build up a list of processing steps — and the sequence of commands you give this way defines the processing pipeline that gets performed on the table. 🧚

Commands can alternatively be supplied in an external file, using the indirection character @. So a value of "@filename" causes the file filename to be read for a list of filter commands to execute. The commands in the file may be separated by newline characters and/or semicolons, and lines which are blank or which start with a # character are ignored. A backslash character '\fR' at the end of a line joins it with the following line.

  • ocmd=<cmds> — specifies processing to be performed on the output table, after all other processing has taken place. The value is one or more of the filter commands described in SUN/256. If you give more than one, they must be separated by semicolon characters (;). You can repeat this parameter multiple times on the same command line to build up a list of processing steps — and again, the sequence of commands given this way defines the processing pipeline that gets performed on the table. 💅

Commands can alternatively be supplied in an external file, using the indirection character @. So a value of "@filename" causes the file filename to be read for a list of filter commands to execute. The commands in the file may be separated by newline characters and/or semicolons, and lines which are blank or which start with a # character are ignored. A backslash character '\fR' at the end of a line joins it with the following line.

  • omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui — the mode the result table gets output in. The default mode is out, which means the result gets written as a new table to disk or wherever, as determined by the out and ofmt parameters. But there are other possibilities too — they correspond to other uses you can put a table to besides outputting it, like displaying metadata, calculating statistics, or populating a table in an SQL database. For some values of this parameter you'll need extra parameters (<mode-args>) to determine the exact behaviour. ✨

The possible values are:

  • out
  • meta
  • stats
  • count
  • checksum
  • cgi
  • discard
  • topcat
  • samp
  • tosql
  • gui

Use the help=omode flag or see SUN/256 for more information. 📚

  • out=<out-table> — where the output table goes. Usually a filename to write to. If it's equal to the special value - (the default), the output table gets written to standard output.

This parameter must only be given if omode has its default value of "out".

  • ofmt=<out-format> — the format the output table gets written in (one of the ones in SUN/256 — matching is case-insensitive and you can use just the first few letters). If it has the special value (auto) — the default — then the output filename gets examined to try to guess what sort of file is required, usually by looking at the extension. If it's not obvious from the filename what output format is intended, an error will result. 😬

This parameter must only be given if omode has its default value of "out".

  • ra=<expr> — right ascension in degrees, in the ICRS coordinate system, for the position of each row of the input table. This may simply be a column name, or it may be an algebraic expression calculated from columns, as explained in SUN/256. If left blank, an attempt is made to guess from UCDs, column names and unit annotations what expression to use. 🦋
  • dec=<expr> — the declination, in degrees, in the ICRS coordinate system, for the position of every row of the input table. 💫 This can literally just be a column name, or it can be an algebraic expression worked out from columns (the whole vibe of how that works lives in SUN/256, bestie). If you leave it blank, it'll have a little guess based on UCDs, column names and unit annotations to figure out which expression to use. 🔭
  • sr=<expr/deg> — an expression that evaluates to the search radius in degrees for the request at each row of the input table. Usually it's just a constant number, but ngl it can also be the name or the ID of a column in the input table, or a function that involves one. ✨
  • find=best|all|each — this one decides which matches actually get kept. 👑
    • best: only the matching query table row that's closest to the input table row gets output. Input table rows with no matches just get omitted. (heads up: this is the same as the best1 option in the pair matching commands, and best1 is a permitted alias.)
    • all: all query table rows that match the input table row get output. Input table rows with no matches are omitted.
    • each: there'll be one output table row for every input table row. If matches are found, the closest one from the query table gets output, and in the no-matches case, the query table columns come out blank. 💀
  • usefoot=true|false — controls whether it'll try to restrict searches according to whatever footprint info is available. Set it true and then before any of the per-row queries run, it may try to acquire footprint information about the service. If that info is obtainable, queries that fall outside the footprint — and are therefore known to give zero results — get skipped. This can speed the search up considerably, fr fr. 🚀

Currently the only footprints available are the ones from the CDS MOC (Multi-Order Coverage map) service, which covers VizieR and a few other cone search services.

  • footnside=<int-value> — sets the HEALPix Nside parameter used with the MOC footprint service. It's a tuning parameter, and it decides the resolution of the footprint (if there is one). Bigger values = better resolution = a better chance of dodging unnecessary queries, but processing them takes longer and retrieving and storing them costs more. 🎀

The value must be a power of 2, and at the time of writing the MOC service won't hand over footprints at resolutions greater than nside=512, so it should be <=512.

Only used if usefoot=true.

  • copycols=<colid-list> — the list of columns from the input table that get copied to the output table. Every column named here gets prepended to the columns of the combined output table, and its value for each row is taken from the input table row that supplied the parameters of the query that produced it. List syntax is in SUN/256. The default is "*", which means all the input table columns are included in the output. 🌈
  • scorecol=<col-name> — names the column in the output table that holds the distance between the requested central position and the actual position of the returned row. That distance is an angular distance in degrees. Pick a null value and no distance column shows up in the output table at all. 📏
  • parallel=<n> — lets multiple cone searches run concurrently. Leave it at the default 1 and the cone query for the first row of the input table goes out, and when that one's done, the query for the second row goes out, and so on. Set it to <n> and queries get overlapped so that up to approximately <n> may be running at any one time. ⚡

Whether bumping <n> up is even a good idea, and what a sensible max would be, depends on the characteristics of the service you're querying. Setting it too high can overload the service, giving you some combo of failed queries, ultimately slower runtimes, and unpopularity with server admins. 😭

The maximum value this parameter permits by default is 5. That limit can be raised with the service.maxparallel system property — but use that option with great care, because you may overload services and make yourself unpopular with data centre admins. As a rule, you should only increase this value if you've obtained permission from the data centres whose services you'll be using the increased parallelism on. 🙏

  • erract=abort|ignore|retry|retry<n> — what happens if any of the individual cone search requests fails. By default the task aborts. That might genuinely be the best move, but for unreliable or poorly implemented services you may find some searches fail while others succeed, so it can be best to keep going in the face of a few failures. The options are:
    • abort: failure of any query terminates the task. 💀
    • ignore: failure of a query is treated exactly the same as a query that returns no rows.
    • retry: failed queries are retried until they succeed, with an increasing delay introduced for each failure. Use with care — if the failure is for some good, or at least reproducible, reason, this could stop the task from ever completing. 😵
    • retry<n>: failed queries are retried at most a fixed number <n> of times, with an increasing delay introduced for each failure. If failures persist, the task terminates.
  • ostream=true|false — set it true and the operation streams on output, so the output table gets built up as results come in from the cone search service. The downside? Some output modes and formats need multiple passes through the data to work, so depending on your output destination, the operation may fail if this is set. Use with care (or just be prepared for the operation to fail, bestie). 🫠
  • fixcols=none|dups|all — decides how input columns get renamed before they're used in the output table. The choices:
    • none: columns are not renamed.
    • dups: columns that would otherwise have duplicate names in the output get renamed to show which table they came from.
    • all: all columns get renamed to show which table they came from.

If columns do get renamed, the new names are determined by the suffix* parameters.

  • suffix0=<label> — if fixcols is set so input columns get renamed for insertion into the output table, this parameter decides how that renaming goes. It's the suffix appended to all renamed columns from the input table. 🎀
  • suffix1=<label> — same deal, same rules, but this one's the suffix appended to all renamed columns from the cone result table. ✨
  • servicetype=cone|ssa|sia1|sia2|sia — selects the type of data access service to contact. Most of the time it's the Cone Search service itself, but there are one or two other possibilities:
    • cone: Cone Search protocol — returns a table of objects found near each location. See Cone Search standard. 🔭
    • ssa: Simple Spectral Access protocol — returns a table of spectra near each location. See SSA standard. 🌈
    • sia1: Simple Image Access protocol version 1 — returns a table of images near each location. See SIA 1.0 standard. 🖼️
    • sia2: Simple Image Access protocol version 2 — returns a table of images near each location. See SIA 2.0 standard. 🖼️
    • sia: alias for sia
  • serviceurl=<url-value> — the base part of a URL that defines the queries to be made. Extra parameters get appended to it using CGI syntax ("name=value", separated by & characters). And if this value doesn't end in either a ? or a &, one gets added as appropriate. 🔗

See SUN/256 for the discussion of how to locate service URLs corresponding to given datasets.

  • verb=1|2|3 — verbosity level of the tables returned by the query service. 1 means the bare minimum, 3 means all available information. 💅
  • dataformat=<value> — indicates the format of the data objects described in the returned table. What it actually means depends on the value of the servicetype parameter:
    • servicetype=cone: not used. 🙈
    • servicetype=ssa: gives the MIME type of the spectra referenced in the output table, plus special values "votable", "fits", "compliant", "graphic", "all", and others (it's the value of the SSA FORMAT parameter). 📊
    • servicetype=sia1: gives the MIME type required for the images/resources referenced in the output table, corresponding to the SIA FORMAT parameter. The special values "GRAPHIC" (all graphics formats) and "ALL" (no restriction) as defined by SIAv1 are also permissible. For SIA version 1 only, this defaults to "image/fits". 🖼️
    • servicetype=sia2: gives the MIME type required for the images/resources referenced in the output table, corresponding to the SIA FORMAT parameter. The special values "GRAPHIC" (all graphics formats) and "ALL" (no restriction) as defined by SIAv1 are also permissible. 🖼️
    • servicetype=sia: gives the MIME type required for the images/resources referenced in the output table, corresponding to the SIA FORMAT parameter. The special values "GRAPHIC" (all graphics formats) and "ALL" (no restriction) as defined by SIAv1 are also permissible. For SIA version 1 only, this defaults to "image/fits". 🖼️
  • emptyok=true|false — do we believe the table metadata that comes back from a search result with zero rows in it? 👀 So like, according to the spirit (not the letter, bestie, the spirit) of the cone search standard, a cone search service that returns no data should still hand back the correct column headings. And unfortunately that is just not always what happens. 😭 If you set this true, we assume the service behaves properly on that front — but if it doesn't, an error may result. Which, if that's your situation, set this parameter false. ⚠️ And here's the tradeoff you need to know: a consequence of setting it false is that when no results come back, the task will return no table at all, rather than an empty one. So it's giving "nothing vs a whole empty vibe" and you gotta pick. 💅
  • compress=true|false — if true, we ask the service to give us HTTP-level compression on the response stream (the Accept-Encoding header gets set to "gzip", see RFC 2616). 🗜️✨ Now, this does not guarantee compression will actually happen — it's a request, not a demand, bestie — but if the service honours it, you may end up with a smaller amount of network traffic, at the expense of more processing on the server and the client. So, tiny packets, slightly sweatier CPU. Balance accordingly. 🌸

👀 SEE ALSO 👀

And okay so if the package stilts-doc is installed, the full documentation SUN/256 is available in HTML format:

file:///usr/share/doc/stilts/sun256/index.html

go read it, it's literally right there 💖

🎀 VERSION 🎀

STILTS version 3.5.4-debian ✨

This is the Debian version of Stilts, which lack the support of some file formats and network protocols. 😭 For differences see:

file:///usr/share/doc/stilts/README.Debian

so if something's missing, that's why — it's not you, it's the packaging 🫶

👑 AUTHOR 👑

Mark Taylor (Bristol University) — the icon, the legend, the main character who wrote all this 💅🌟

okay bestie that's the whole page, we ate and left no crumbs 🍓✨ this is the Debian build of STILTS, dropped as version 3.5.4-debian and dated Mar 2017 — STILTS-CONESKYMATCH(1), signed sealed and slayed, periodt 💖🦋🌸

📦 where this page comes from