mcxarray(1)

mcxarray(1) 💖

okay so bestie, this is a USER COMMANDS page from the mcl family, and it's literally the glow-up station where your boring array data becomes an MCL matrix. mcxarray said "let me make that data cute and comparable" and honestly? she ate. 🍓

---

✨ NAME ✨

mcxarray — transform array data to MCL matrices 🦋

---

✨ SYNOPSIS ✨

mcxarray [options]

mcxarray [-data fname (input data file)]
[-imx fname (input matrix file)]
[-co num ((absolute) cutoff for output values (required))]
[-skipr <num> (skip <num> data rows)]
[-skipc <num> (skip <num> data columns)]
[-o fname (output file fname)]
[--text-table (write output in full text table format)]
[-write-tab <fname> (write row labels to file)]
[-l <num> (take labels from column <num>)]

[--pearson (use Pearson correlation (default))]
[--spearman (use Spearman rank correlation)]
[--dot (use dot product)]
[--cosine (use cosine (similarity))]
[--slow-cosine (use cosine(0.5 alpha) (similarity))]

[--angle (use angle between vectors (note: a metric distance))]
[--acute-angle (use acute angle between vectors)]
[--angle-norm (use normalised angle between vectors (by pi))]
[--acute-angle-norm (use normalised acute angle between
vectors (by pi/2))]
[--sine (use sine (note: a metric distance))]
[--slow-sine (use sine(0.5 alpha) (note: a metric distance))]
[--euclid (use Euclidean distance between vectors)]
[--max (use L-oo, aka Chebyshev distance)]
[--taxi (use L-1, aka taxi, aka city-block distance)]
[-minkowski <num> (use Minkowski distance with power <num>)]
[-fp <mode> (use fingerprint measure)]

[-digits <num> (output precision)]
[--write-binary (write output in binary format)]
[-t <int> (use <int> threads)]
[-J <intJ> (a total of <intJ> jobs are used)]
[-j <intj> (this job has index <intj>)]
[-start <int> (start at column <int> inclusive)]
[-end <int> (end at column <int> EXclusive)]
[--transpose-data (work with the transposed data matrix)]
[--rank-transform (rank transform the data first)]
[-tf spec (transform result network)]
[-table-tf spec (transform input table before processing)]
[-n mode (normalize input)]
[--zero-as-na (treat zeroes as missing data)]
[--sparse (do not store zero values)]
[-write-data <fname> (write data to file)]
[-write-na <fname> (write NA matrix to file)]
[--job-info (print index ranges for this job)]
[--help (print this help)]
[-h (print this help)]
[--version (print version information)]

---

✨ DESCRIPTION ✨

so mcxarray can go two ways, bestie. it can read a flat file full of array data with -data, or it can read a matrix file that satisfies the mcl input format with -imx. 💅

in the flat-file case, she'll work with the rows as the data vectors by default. in the matrix-file case, she'll work with the columns as the data vectors by default — and yeah, that's on purpose, because mcl matrices are presented as a listing of columns (iykyk). 🫶

both of those defaults can be flipped with --transpose-data. main character energy.

the input data is allowed to have missing data, in the form of empty columns, NA values (not available/applicable), or NaN values (not a number). the program keeps track of all of these, and when it computes the correlation between two rows or columns it ignores every position where either one of the two has missing data. no cap, she's careful like that. 😌

---

✨ OPTIONS ✨

🎀 input & output basics

• -data fname — (input data file) point her at the data file holding the expression values. it should be tab-separated. 🍓
• -imx fname — (input matrix file) the expression values get read from a file in mcl matrix format.
• -o fname — (output file fname) the output file name. that's it, that's the whole vibe. 💖
• --text-table — (write output in full text table format) the output gets written in tabular format instead of native mcl-edge format.
• -digits num — (output precision) specify the precision to use in native interchange format.
• --write-binary — (write output in binary format) write output matrices in native binary format.
• -co num — ((absolute) cutoff for output values) output values of magnitude smaller than num are removed (set to zero). so heads up: negative values are removed only if their positive counterpart is smaller than num. 🫶

---

💫 similarity & correlation measures

• --pearson — (use Pearson correlation (default)) the default, bestie ✨
• --spearman — (use Spearman rank correlation)
• --cosine — (use cosine)
• --slow-cosine — (use cosine(0.5 alpha) (similarity))
• --dot — (use the dot product)

okay so all of these express the level of similarity or correlation between two vectors. but listen: the dot product is not normalised, and you should only use it with a very good reason. 🚨 a few more similarity measures are provided by the fingerprint option -fp, described right below.

---

🧚 the fingerprint zone (-fp)

• -fp mode — (specify fingerprint measure)

fingerprints are how you define an entity in terms of it having or not having certain traits. which means a fingerprint can be represented by a boolean vector, and a set of fingerprints can be represented by an array of such vectors. and when you have a lot of traits and a lot of entities, the dimensions of that matrix can get big. 💀

the sparse storage employed by MCL-edge is ideally suited to this, and mcxarray is ideally suited to computing all pairwise comparisons between such fingerprints. currently mcxarray supports five different types of fingerprint, described below.

given two fingerprints: the number of traits unique to the first is denoted by a, the number unique to the second is denoted by b, and the number they have in common is denoted by c. 🌸

• hamming — the Hamming distance, defined as a+b.
• tanimoto — the Tanimoto similarity measure, c/(a+b+c).
• cosine — the cosine similarity measure, c/sqrt((a+c)*(b+c)).
• meet — simply the number of shared traits, identical to c.
• cover — a normalised and non-symmetric similarity measure, representing the fraction of traits shared relative to the number of traits by a single entity. this gives the value c/(a+c) in one direction, and the value c/(b+c) in the other. 👑

---

🦋 distance measures

• --sine — (use sine (note: a metric distance))
• --slow-sine — (use sine(0.5 alpha) (note: a metric distance))
• --angle — (use angle between vectors (note: a metric distance))
• --acute-angle — (use acute angle between vectors)
• --angle-norm — (use normalised angle between vectors (by pi))
• --acute-angle-norm — (use normalised acute angle between vectors (by pi/2))
• --euclid — (use Euclidean distance between vectors)
• --max — (use L-oo, aka Chebyshev distance)
• --taxi — (use L-1, aka taxi, aka city-block, aka Manhattan distance)
• -minkowski num — (use Minkowski distance with power num)

all of these express the level of dissimilarity or distance between two vectors. opposite energy from the similarity gang above. 🌈

---

🎀 skipping & labels

• -skipr num — (skip num data rows) skip the first num data rows.
• -skipc num — (skip num data columns) ignore the first num data columns.
• -l num — (take labels from column num) specifies to construct a tab of labels from this data column. the tab can be written to file using -write-tab fname.
• -write-tab fname — (write row labels to file) write a tab file. in the simple case where the labels are in the first data column it is sufficient to issue -skipc 1. if more data columns need to be skipped, you must explicitly specify the data column to take labels from with -l l. 📝

---

⚡ threads, jobs & splitting the work

• -t int — (use int threads)
• -J intJ — (a total of intJ jobs are used)
• -j intj — (this job has index intj)

computing all pairwise correlations is time-intensive for large input, bestie. if you have multiple CPUs available, consider using as many threads. additionally it's possible to spread the computation over multiple jobs/machines. these three options are described in the clmprotocols manual page.

the following set of options, if given to as many commands, defines three jobs, each running four threads. 🔥

-t 4 -J 3 -j 0 -o out.0
-t 4 -J 3 -j 1 -o out.1
-t 4 -J 3 -j 2 -o out.2

the output can then be collected with

mcx collect --add-matrix -o out.all out.[0-2]

---

🔍 job info & manual ranges

• --job-info — (print index ranges for this job)
• -start int — (start at column int inclusive)
• -end int — (end at column int EXclusive)

--job-info can be used to list the set of column ranges to be processed by the job as a result of the command line options -t, -J, and -j. if a job has failed, this option can be used to manually split those ranges into finer chunks, each to be processed as a new sub-job specified with -start and -end. with the latter two options, it is impossible to use parallelization of any kind (i.e. any of the -t, -J, and -j options). 😭

---

🌸 data wrangling & transforms

• --transpose-data — (work with the transpose) work with the transpose of the input data matrix.
• --rank-transform — (rank transform the data first) the data is rank-transformed prior to the computation of pairwise measures.
• -write-data fname — (write data to file) this writes the data that was read in to file. if --spearman is specified, the data will be rank-transformed. ✨
• -write-na fname — (write NA matrix to file) this writes all positions for which no data was found to file, in native mcl matrix format.
• -n mode — (normalization mode) if mode is set to z, the data will be normalized based on z-score. no other modes are currently supported. 🫶
• -tf spec — (transform result network)
• -table-tf spec — (transform input table before processing)

the transformation syntax is described in mcxio(5).

---

💀 missing data, sparsity & the rest

• --zero-as-na — (treat zeroes as missing data)

this option can be useful when reading data with the -imx option, for example after it has been loaded from label input by mcxload. an example case is the processing of a large number of probe rankings, where not all rankings contain all probe names. the rankings can be loaded using mcxload with a tab file containing all probe names. probes that are present in the ranking are given a positive ordinal number reflecting the ranking, and probes that are absent are implicitly given the value zero. with the present option mcxarray will handle the correlation computation in a reasonable way. 🧠

• --sparse — (do not store zero data value) with this option, internal calculations are performed on compressed data where zeroes are not stored. this can be useful when the input data is very large.

---

🎀 help & version

• --help — (print help)
• -h — (print help)
• --version — (print version information)

---

✨ AUTHOR ✨

Stijn van Dongen. 👑

---

✨ SEE ALSO ✨

mcl(1), mclfaq(7), and mclfamily(7) for an overview of all the documentation and the utilities in the mcl family. 🌈

---

and that's the whole page, bestie 💖 this was mcxarray 22-282, dated 9 Oct 2022 — go forth and matrix-ify your arrays, you absolute icon 🎀✨🦋