STILTS-TGRIDMAP(1)

📦
version 3.5.4 · 5 versions
, in ubuntu stonking

💅 STILTS-TGRIDMAP(1) 💅

okay so bestie, you're on the stilts-tgridmap page from the Stilts commands manual, section 1. It's giving… density maps. No cap. Let's go 🫶

✨ NAME ✨

stilts-tgridmap — Calculates N-dimensional density maps

🎀 SYNOPSIS 🎀

stilts tgridmap [ifmt=<in-format>] [istream=true|false] [in=<table>]
                [icmd=<cmds>] [ocmd=<cmds>]
                [omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui]
                [out=<out-table>] [ofmt=<out-format>] [coords=<expr>
                ...] [logs=true|false ...] [bounds=[<lo>]:[<hi>] ...]
                [binsizes=<size> ...] [nbins=<num> ...]
                [cols=<expr>[;<combiner>[;<name>]] ...]
                [combine=sum|sum-per-unit|count|count-per-
                unit|mean|median|Q1|Q3|min|max|stdev|stdev_pop|hit]
                [sparse=true|false]
                [runner=sequential|parallel|parallel<n>|partest]

🦋 DESCRIPTION 🦋

okay so picture this: tgridmap scans your input table and builds one or more N-dimensional density maps — which is literally just a fancy way of saying N-dimensional histograms — of the values sitting in that table. Then it drops the whole thing out as a table (optionally sparse) with one row per grid cell. Obsessed. 💖

and listen, the maps/histograms can optionally be weighted by some quantity from the input table, and you've got a whole menu of options for aggregating inputs into the output bins — summing, averaging, counting, the works. it's giving flexible.

so here's how the pieces snap together: the coords parameter you supply tells it which N numeric columns of the input table are the coordinates of the bin grid, and the cols parameter tells it which quantities get aggregated into each bin. and bestie, you must supply either the binsizes or the nbins parameter to define the extents of the bins on each axis — that's not optional, that's the vibe check. 🌈

the output table has a row for every bin, with columns giving the central (and upper/lower bound) values of each grid coordinate, plus a column for each aggregated value. rows come out in first-coordinate-slowest sequence (iykyk), and the sparse parameter decides whether you get a row for every cell in the hypercube defined by the grid dimensions, or only for the cells that actually have non-blank data. 💅

ngl, the tabular form of the output may not be the most appropriate or compact way to write a density map, especially for multi-dimensional grids — but it means the output can be manipulated later by other STILTS commands or by TOPCAT. so it's a tradeoff, and it's a good one.

if you want to do a similar job with more compact output, go peep tcube. Also see tskymap, which does the same thing for sky geometry — and honestly it's probably a better choice if you find yourself accumulating onto a longitude-latitude grid. main character energy, that one. 👑

🔥 OPTIONS 🔥

  • ifmt=<in-format> — says what format the input table (the one from parameter in) is in. all the known formats are listed in SUN/256. use this flag when you already know what format your table is, no guessing needed. but if you set it to the special value (auto) — which is the default — then it'll try to detect the format for you automatically. lowkey that doesn't always work correctly though, and when it fails the program just exits with an error explaining which formats it attempted. 💀 this parameter is ignored for scheme-specified tables.
  • istream=true|false — set it true and the input table from the in parameter gets read as a stream. heads up: you have to give the ifmt parameter in that case. depending on what operations and processing mode you need, this can make the read fail (sometimes a table genuinely needs to be read more than once, sorry bestie). it's not normally necessary to set this flag — in most cases the data streams automatically when that's the best thing to do. but sometimes it does mean less resource usage when you're processing large files in certain formats (like VOTable), so ✨situational✨. this parameter is also ignored for scheme-specified tables.
  • in=<table> — where your input table lives. it can be one of these shapes:
    • A filename.
    • A URL.
    • The special value "-", meaning standard input. if you do this, the input format must be given explicitly using the ifmt parameter. and ngl, not every format can be streamed this way. 🙈
    • A scheme specification of the form :<scheme-name>:<scheme-args>.
    • A system command line with either a "<" character at the start, or a "|" character at the end ("<syscmd" or "syscmd|"). this runs the pipeline and reads from its standard output. it will probably only work on unix-like systems, just so you know. 🧚

in any case, compressed data in one of the supported compression formats (gzip, Unix compress or bzip2) gets decompressed transparently. magic. 💫

  • icmd=<cmds> — processing to do on the input table (the one from in) before any other processing has taken place. the value is one or more of the filter commands described in SUN/256. give more than one and they must be separated by semicolon characters (";"). and you can repeat this parameter multiple times on the same command line to build up a whole list of processing steps — the sequence you give defines the processing pipeline performed on the table. 🌸

alternatively, commands can come from an external file using the indirection character '@'. so a value of "@filename" makes the file filename get read for a list of filter commands to execute. commands in that file may be separated by newline characters and/or semicolons, and lines that are blank or that start with a '#' character are ignored. a backslash character \ at the end of a line joins it with the following line. periodt. 💅

  • ocmd=<cmds> — processing to do on the output table, after all other processing has taken place. same deal: value is one or more of the filter commands described in SUN/256, multiple ones must be separated by semicolon characters (";"), and you can repeat the parameter multiple times on the same command line to build up a list of processing steps. the sequence you give defines the processing pipeline performed on the table. ✨

and again, commands may alternatively be supplied in an external file using the indirection character '@'. so "@filename" causes the file filename to be read for a list of filter commands to execute. the commands in the file may be separated by newline characters and/or semicolons, and lines which are blank or which start with a '#' character are ignored. a backslash character \ at the end of a line joins it with the following line. 💖

  • omode=out|meta|stats|count|checksum|cgi|discard|topcat|samp|tosql|gui — the mode your result table gets output in. default is out, which means the result is written as a new table to disk or wherever, as determined by the out and ofmt parameters. but bestie there are so many other possibilities, for all the other things you might want to do with a table besides outputting it — displaying metadata, calculating statistics, populating a table in an SQL database, the whole roster. for some values of this parameter you'll need extra parameters (<mode-args>) to pin down the exact behaviour. 👑

possible values are:

  • out
  • meta
  • stats
  • count
  • checksum
  • cgi
  • discard
  • topcat
  • samp
  • tosql
  • gui

use the help=omode flag or see SUN/256 for more info. 📚

  • out=<out-table> — where the output table goes. usually a filename to write to. if it equals the special value "-" (which is the default), the output table gets written to standard output. 🫶
⚠️ this parameter must only be given if omode has its default value of "out".
  • ofmt=<out-format> — the format the output table gets written in, one of the ones in SUN/256. matching is case-insensitive and you can use just the first few letters, so you can absolutely be lazy about it. 💅 if it has the special value "(auto)" (the default), the output filename gets examined to try to guess what sort of file is required — usually by looking at the extension. if it's not obvious from the filename what output format is meant, you get an error. 😭
⚠️ this parameter must only be given if omode has its default value of "out".
  • coords=<expr> ... — defines the dimensions of the grid that accumulation happens over. the form of this value is a space-separated list of words, each one giving a column name or algebraic expression defining one of the dimensions of the output grid. for a 1-dimensional histogram you only need one value. cute and simple. 🌸
  • logs=true|false ... — decides whether each coordinate axis is linear or logarithmic. by default the grid axes are linear, but if you supply this parameter with one or more true values, the bins on the corresponding axes get assigned logarithmically instead. 📈

if you supply this, it must have the same number of words as the coords parameter. no mismatches allowed, bestie.

  • bounds=[<lo>]:[<hi>] ... — gives the bounds for each dimension of the cube, in data coordinates. the form of the value is a space-separated list of words, each giving an optional lower bound, then a colon, then an optional upper bound. for instance "1:100 0:20" represents a range for two-dimensional output between 1 and 100 of the first coordinate (table column) and between 0 and 20 for the second. either or both numbers may be omitted to say that the bounds should be determined automatically by assessing the range of the data in the table. a null value for the parameter means all bounds should be determined automatically for all dimensions. ✨
⚠️ if any of the bounds need to be determined automatically in this way, two passes through the data will be required — the first to determine bounds, and the second to calculate the map. it's giving double the work, so just know that going in.

if supplied, this parameter must have the same number of words as the coords parameter. 💅

  • binsizes=<size> ... — tells you how big each data bin is in every dimension, measured in data coordinates 🎀. The value is a space-separated list of numbers: first one is the extent for the first dimension, second for the second dimension, and so on forever. You gotta give either this one or nbins — one of them, bestie, not neither 🙈.

And if you do pass it, it's gotta have the exact same number of words as your coords parameter. No matching words, no vibe ✨.

  • nbins=<num> ... — the approximate number of bins per dimension 💅. Again a space-separated list of integers: how many bins the output histogram wants in the first, second, ... dimension. It tries to land on round numbers for the bin sizes, so lowkey the counts you get back may not be exactly what you asked for 😭. Either this or binsizes must be supplied.

Same rule as before: if you supply it, it must have the same number of words as the coords parameter. Word count match or it's over 🫶.

  • cols=<expr>[;<combiner>[;<name>]] ... — this is where you define the quantities that actually get calculated 👑. Value is a space-separated list of items, one item per aggregated column in the output table.

Each item is made of one, two, or three tokens, all glued together with semicolon (;) characters:

  • <expr>: (required, no skipping) a column name or an expression using the expression language for the quantity you want aggregated.
  • <combiner>: (optional) the combination method, taking the same options as the combine parameter does. Leave it out and whatever you set for that parameter gets used instead.
  • <name>: (optional) the name of the output column. Skip it and the <expr> value gets used — maybe slightly sanitised, so heads up 💀.

Honestly most of the time you can just hand it a space-separated list of input table column names and it's giving exactly what you wanted 🌸. But the extra syntax earns its keep when you need something like both a sum and a mean of the same input column — one column, two outputs, iconic.

The default value is "1;count;COUNT", which is just an unweighted histogram — a count of the rows in each bin. That's aggregation of the value "1" using the combination method "count", giving you an output column named "COUNT". Simple, clean, slay ✨.

  • combine=sum|sum-per-unit|count|count-per-unit|mean|median|Q1|Q3|min|max|stdev|stdev_pop|hit — sets the default way values landing in the same density map bin get combined into that bin's final value 🧚. Possible values, all of them, no gatekeeping:
    • sum: the sum of all the combined values per bin
    • sum-per-unit: the sum of all the combined values per unit of bin size
    • count: the number of non-blank values per bin (weight is ignored, she doesn't care)
    • count-per-unit: the number of non-blank values per unit of bin size (weight is ignored too)
    • mean: the mean of the combined values
    • median: the median
    • Q1: first quartile
    • Q3: third quartile
    • min: the minimum of all the combined values
    • max: the maximum of all the combined values
    • stdev: the sample standard deviation of the combined values
    • stdev_pop: the population standard deviation of the combined values
    • hit: 1 if any values are present, NaN otherwise (weight is ignored)
    • Q.nnn: quantile nnn — e.g. Q.05 is the fifth percentile 💜
Heads up bestie: this value can be overridden on a per-column basis by the cols parameter. So cols always gets the final word, periodt.
  • sparse=true|false — decides whether a row gets written for every cell in the defined grid, or only for the cells where data actually shows up in the input 🙈. The result will usually be more compact if you set this to false. But if you wanna compare results from different runs, setting it true may be way more convenient for you. Your call, main character 🔥.
  • runner=sequential|parallel|parallel<n>|partest — picks the threading implementation, aka how much of this is happening in parallel ⭐. Current options:
    • sequential: runs using only a single thread
    • parallel: runs using multiple threads for large tables, with parallelism given by the number of available processors
    • parallel<n>: runs using multiple threads for large tables, with parallelism given by the supplied value <n>
    • partest: runs using multiple threads even when tables are small (only intended for testing purposes, don't get comfy)

Parallel processing can speed things up considerably — but ngl, depending on the I/O operations required, it can also slow it down by disrupting patterns of disk access 😭. If the content of a file lives on a solid state disk, or is already in cache (like because a similar command got run recently), then parallel will probably be faster. But if the data's being read directly from a spinning disk, like because the file is too large to fit in RAM, then sequential or parallel<n> with a small <n> may be faster. Read the room, and read the disk 💅.

The value of this parameter should make only very tiny differences to the output table. If you notice significant discrepancies, please report them 🙏.

🌈 SEE ALSO 🌈

If the package stilts-doc is installed, the full documentation SUN/256 is available in HTML format at file:///usr/share/doc/stilts/sun256/index.html 📚.

💖 VERSION 💖

STILTS version 3.5.4-debian

This is the Debian version of Stilts, which lack the support of some file formats and network protocols 😔. For differences see file:///usr/share/doc/stilts/README.Debian.

👑 AUTHOR 👑

Mark Taylor (Bristol University)

And that's the whole page, bestie — this one shipped in Mar 2017 under the name STILTS-TGRIDMAP(1), running version 3.5.4-debian 🦋. Go map some grids, you've got this. 💖✨

📦 where this page comes from