mcxassemble(1)

📦 version 14-137, in ubuntu jammy

💖 mcxassemble — the mcxassemble(1) glow-up 💖

okay bestie, sit DOWN, we're learning mcxassemble and it's actually so good 🫶 it lives in the USER COMMANDS section of the manual, and it does one iconic thing: turns your messy raw cooccurrence data into a real, valid mcl matrix. no cap.

✨ NAME ✨

mcxassemble — transform raw cooccurrence data to mcl matrix format. literally that's the whole job and it ate 💅

✨ SYNOPSIS ✨

mcxassemble -b base (base name) [-o fname (write to file fname)]
[--write-binary (write output in binary format)] [--map (apply base.map)]
[-raw-tf (apply transform spec to input)] [-rv MODE (repeated vectors)]
[-re MODE (repeated entries)] [-ri MODE (adding mirror image)] [-r MODE
(repeated entries/vectors/images)] [-prm-tf (apply transform spec to
primary matrix)] [-sym-tf (apply transform spec to symmetrified matrix)]
[-q (quiet mode)]

so those options above? that's the default setup when you use mcxassemble. like, the vanilla vibe. but there are so many more options — mostly subtly different ways of doing input/output, setting warning levels, or regulating how repeated entries and vectors get treated. the full list is down below in OPTIONS, bestie, don't worry. and if you wanna actually understand what mcxassemble eats and spits out, read DESCRIPTION. 🌸

⚠️ NOTE ⚠️

As of release 05-314, mcl(1) is able to cluster label-type input on the fly. In most cases, this will be sufficient. Alternatively, mcxload(1) can be used to map label-type input onto mcl matrices. Consequently, there are likely fewer scenarios nowadays where mcxassemble is the best solution. Consider first whether mcl in label mode or mcxload can do the job as well.

fr fr, check your other options first before you commit to mcxassemble — it's giving "we love a specialist but know your alternatives" energy 😭

mcxassemble [-b base (base name)] [-hdr fname (read header file)] [-raw
fname (read raw file)] [--map (apply base.map)] [--cmap (apply
base.cmap)] [--rmap (apply base.rmap)] [-map fname (apply fname)] [-rmap
fname (apply fname)] [-cmap fname (apply fname)] [-tag tag (apply
base.tag)] [-rtag tag (apply base.tag)] [-ctag tag (apply base.tag)]
[-skw fname (write skew matrix)] [-prm fname (write primary result
matrix)] [--skw (write base.skw)] [--prm (write base.prm)] [-xo suf
(write base.suf)] [-o fname (write to file fname)] [-n (do not write
default symmetrized result)] [-i (read from single data file)] [-digits
int (digits width)] [-s (check for symmetry)] [-raw-tf (apply transform
spec to input)] [-rv <mode> (action for repeated vectors)] [-re <mode>
(action for repeated entries)] [-ri <mode> (adding mirror image)] [-r
<mode> (same for entries and vectors)] [-prm-tf (apply transform spec to
primary matrix)] [-sym-tf (apply transform spec to symmetrified matrix)]
[--quiet-re (quiet for repeated entries)] [--quiet-rv (quiet for repeated
vectors)] [-q (the two above combined)] [-h (print synopsis, exit)]
[--apropos (print synopsis, exit)] [--version (print version, exit)]

✨ DESCRIPTION ✨

mcxassemble lets you make matrices easily from an intermediate raw matrix format — a format you can build super fast from a one-pass-parse of cooccurrence data. here's the basic setup, bestie: 🦋

  • Parse cooccurrence data from some external format.
  • Transform cooccurrence data to raw mcl data as you parse.
  • When done, write out required header and domain information to a separate file. The domain information can be built during the parsing stage.
  • Use mcxassemble to construct a valid matrix from the raw data and the header information.
  • Nodes can optionally be relabeled by writing a separate map file to be read by mcxassemble, which takes the form of a very thin matrix file.

the easiest thing to do is group all your input/output files under the same base name — say, base. the standard glow-up that leads to a nice concise mcxassemble command line is: create the input files base.raw and base.hdr, and optionally the file base.map. then the default behaviour of mcxassemble is to create base.sym as the resulting matrix file, containing the symmetrized matrix built from the raw input. periodt. 👑

💫 Example 💫

say blastresult is a file containing blast results. these two commands construct an mcl matrix file from it:

mcxdeblast --score=e --sort=a blastresult
mcxassemble -b blastresult -r max --map

mcxdeblast will generate the files blastresult.hdr, blastresult.raw, and blastresult.map. the --sort=a option creates a map file corresponding with alphabetic ordering. these files get processed by mcxassemble and it generates the file blastresult.sym. the -r option tells mcxassemble that repeated entries should be maxed — each time, the largest entry seen thus far will be taken. iconic behaviour, honestly 💅

📄 Header file 📄

this file contains a header, as usually found in generic mcl matrix files — i.e. the required header part, and optionally the domain part(s) if not all domains are canonical. refer to mcxio(5) for more info, bestie. the domain information in the header file gets used to pre-construct a skeleton matrix and to validate the entries in the raw data file as they fill that skeleton matrix. 🧚

📥 Raw input format 📥

the file from which raw input is read should have the raw format described in mcxio(5). simply put: no header specification, no domain specification, and no matrix introduction syntax. the file just contains a listing of vectors. an example fragment:

2  4:0.34 1:2.8838 4:2.328 1:4.238 1:12 $
1  2:7.8 $
2  1:0.01 4:20.3 3:2 $

the listing of vectors need not be sorted, and neither does a vector itself need to be sorted — the mcl generic matrix format is actually not different in this respect. also, duplicate entries and duplicate vectors are allowed! which, again, is allowed in the generic format too — except that where applications expect generic format, warnings get issued and duplicate entries get disregarded. mcxassemble allows customizable behaviour dictating how to merge repeated entries. refer to the -re, -rv, -r options below. 💀 (yes it's a lot, we'll get there)

the vectors read by mcxassemble do have to match the domains specified in the header file. the leading index that specifies the column index has to be present in the column domain; all subsequent indices that specify column entries have to be present in the row domain.

if you concatenate the contents of the header file and the data file, the result is almost but not quite a file containing a matrix in syntactically correct mcl generic matrix format. the parts missing are: the (mclmatrix introduction token, (followed by) the begin token, and the closing ) token. so close, yet so far 😭

🗺️ Map file 🗺️

this file must contain a map matrix — a matrix with these properties:

  • The column domain and row domain are of the same cardinality.
  • Each column has exactly one entry.
  • Each row domain index occurs in exactly one column.

such a matrix is used to relabel the nodes found in the raw data. a situation that might occur when parsing some external format (and producing raw matrix format) is that ID's (indices) get handed out on the fly during the parse. afterwards, you might wanna relabel those IDs so they correspond with an alphabetic listing of the quantity represented by the node domain, or by some other sort criterion. a map file is then typically generated by the parser, since that's the utility in charge of the IDs. a small example of a map file for a graph containing five nodes:

(mclheader
mcltype matrix
dimensions 5x5
)
(mclmatrix
begin
0  4  $  #  mno
1  2  $  #  ghi
2  1  $  #  def
3  3  $  #  jkl
4  0  $  #  abc
)

this corresponds to a relabeling such that the associated strings get ordered alphabetically. note that comments can be used to link string identifiers with indices 🎀. this map file says e.g. that the string identifier "mno" is represented by index 0 in the raw data, and by index 4 in the matrix output by mcxassemble.

✨ OPTIONS ✨

  • -b base (base name) — base name of files to be processed and output. refer to DESCRIPTION above and the entries of other options below. this is the main character of your command line, ngl. 👑
  • -hdr fname (read header file)
  • -raw fname (read raw file) — explicitly specify the header file and the data file (rather than constructing the file names from a base name and suffixes). for when you wanna be ✨specific✨.
  • --map (apply base.map)
  • --cmap (apply base.cmap)
  • --rmap (apply base.rmap)
  • -map fname (apply fname)
  • -rmap fname (apply fname)
  • -cmap fname (apply fname)
  • -tag tag (apply base.tag)
  • -rtag tag (apply base.tag)
  • -ctag tag (apply base.tag) — map options, bestie. --cmap combines with the -b option, and says that the map file in base.cmap (where base was specified with -b base) should be applied to the column domain only. --rmap works the same for the row domain, and --map can be used to apply the same map to both the column and row domains. -cmap and its siblings are used to explicitly specify the map file to be used, rather than combining a base name with a fixed suffix. -tag and its siblings work in conjunction with the -b option, and require that a tag be specified from which to construct the map file (by appending it to the base name). 🦋
  • -skw fname (write skew matrix)
  • -prm fname (write primary result matrix)
  • --prm (write base.prm)
  • --skw (write base.skw)
  • -n (do not write default symmetrized result) — options for writing matrices other than the default symmetrized result. the primary result matrix is the matrix constructed from reading in the raw data and adding entries to the skeleton matrix as specified with the -r, -re, and -rv options. this matrix can be written using one of the prm options. calling the primary matrix A, the skew matrix (as defined here) is the matrix A - A^T, i.e. A minus its transposed matrix. it can be written using one of the skw options. and if for some reason the symmetrized result is not needed, its output can be prevented using the -n option. 💅
  • -xo suf (write base.suf)
  • -o fname (write to file fname)
  • -i (read from single data file)
  • -digits int (digits width)
  • --write-binary (write output in binary format) — the -xo option is used in conjunction with the -b option in order to change the suffix for the file in which the symmetrized result matrix is written. use e.g. -xo mci to change the suffix from the default value sym to mci. use -o to explicitly specify the filename in full. use -digits to set the number of digits written for matrix entries (c.q. edge weights). the -i option is special — it causes mcxassemble to read both the header information and the raw data from the same file, where the syntax should be fully conforming to generic mcl matrix format. 🤯
  • -s (check for symmetry) — this will check whether the primary result matrix was symmetric. it reports the number of failing (or skew) edges. a little vibe check for your matrix 🔍
  • -raw-tf <tf-spec> (apply transform spec to input)
  • -prm-tf (apply transform spec to primary matrix)
  • -sym-tf (apply transform spec to symmetrified matrix) — the first applies its transformation spec to the values as found in the raw data. the second applies its transformation spec to the primary matrix. the third applies its transformation step to the symmetrified matrix. refer to mcxio(5) for documentation on the transformation spec syntax. ✨

okay that's the options section done and we're still going — there's more coming in the next part, don't go anywhere bestie 🫶

  • -rv add|max|min|mul|left|right — what to do when whole vectors get repeated (action for repeated vectors)
  • -re add|max|min|mul|left|right — what to do when entries inside a column get repeated (action for repeated entries)
  • -ri add|max|min|mul — how to fold in the mirror image (adding mirror image)
  • -r add|max|min|mul|left|right — the lazy-girl special: sets the same behaviour for entries and vectors at once

✨ THE MERGE OPTIONS ✨ — okay bestie, this is the part where mcxassemble decides what happens when the same thing shows up more than once, and it's lowkey the whole personality of the tool.

First, the vibes check: there are two flavours of repeat, and you have to keep them straight or you will be confused, and I refuse to let that happen to you.

  • Repeats inside the same column listing — same entry, same column, more than once.
  • Repeats across different column listings — the same entry appearing in two separate columns.

And plot twist 👑: one entry can be both kinds at the same time. Not a bug, just the situation.

On top of that, the final result is by default symmetrized — meaning it gets combined with its mirror image (in matrix-speak: the transposed matrix). And yes, that symmetrization can be done in exactly the same whole menu of ways. So the same add|max|min|mul|left|right energy applies there too. 💅

Order of operations, because it matters:

  1. The -re option — repeats within the same column — runs first. Heads up though: it's applied after the column's entries have been sorted, so left and right are not guaranteed to follow the order found in the raw input. The sort already ate your original ordering, sorry babe 😭
  2. The -rv option — repeats across different columns — runs second.
Fun fact you can actually use: -ri min can help you implement a (top-list) best reciprocal hit criterion. Slay.

🌸 Examples 🌸

Say this column shows up in the input — it's listing entries for the vector labeled with index 0:

0 1:30 1:50 2:60 4:70 3:20 1:40 2:40 $

If you're running -re add or -r add, it transforms into this vector:

0 1:120 2:60  3:20 4:70 $

But if it's -re max or -r add instead, it becomes this instead:

0 1:40 2:60 3:20 4:70 $

Now imagine add mode is on, and later another vector specification for index 0 shows up, giving you this transformed vector:

0 1:60 2:80 4:40 $

If -rv max was specified, this new vector gets combined with the previous one by taking the entry wise maximum:

0 1:120 2:60 3:20 4:70 $      # first (transformed) vector
0 1:60 2:80 4:40 $            # second vector

0 1:120 2:80 3:20 4:70 $      # entry wise maximum

And finally — last one, promise 🫶 — suppose somewhere along the way one or more vector listings were specified for index 3, and they eventually led to an entry 0:50. The final symmetrization step takes the [0,3] entry of weight 20 and combines it with the [3,0] entry of weight 50. The resulting matrix will then have the [0,3] and the [3,0] entry both equal to either the maximum, the sum, or the product of the two quantities 50 and 20. Depends on your mode, bestie. Choose wisely 💖

And now, the warning squad 🚨

  • --quiet-re — quiet for repeated entries
  • --quiet-rv — quiet for repeated vectors
  • -q — the two above combined, one flag to silence them both
These are warning options. Turn them on if you expect the raw data to be free of repeats. That's the whole point — if you're confident there are no dupes, hush the noise. If you're wrong... well, you told it to be quiet 🙈

💖 AUTHOR 💖

Stijn van Dongen. (The icon. The legend. The moment. 👑)

🌈 SEE ALSO 🌈

mcxio(5), mcl(1), mcxload(1) and mclfamily(7) — the last one for an overview of all the documentation and the utilities in the mcl family. Go read it, it's giving library.

And that's the whole page, bestie! 🎀 This was mcxassemble version 14-137, dated 16 May 2014, and it lives in section mcxassemble(1). You read the entire thing, you understood the merge options, you are literally the main character of the mcl family now. Periodt. 💅✨

📦 where this page comes from