SIDER(1)

📦
version 1.1.6 · 3 versions
, in debian trixie, ubuntu questing

✨ SIDER(1) ✨ — sideRETRO 💅

💖 NAME 💖

okay so bestie, this is the manual for sider, aka sideRETRO Documentation. That's it. That's the name section. iconic.

🌈 GENERAL SYNTAX 🌈

sideRETRO has a very straightforward syntax, fr fr. Basically, there are three main commands, and each one has a plethora of available options:

  • process-sample
  • merge-call
  • make-vcf

So, to test the installation process and run a first example, the user can call it without any argument from the command line, like this:

$ sider
Usage: sider [-hv]
       sider <command> [options]

A pipeline for detecting
Somatic Insertion of DE novo RETROcopies

Options:
   -h, --help            Show help options
   -v, --version         Show current version
   -c, --cite            Show citation in BibTeX

Commands:
   ps,  process-sample   Extract alignments related
                         an event of retrocopy
   mc,  merge-call       Discover and annotate
                         retrocopies
   vcf, make-vcf         Generate VCF file with all
                         annotate retrocopies

In the above situation, if sideRETRO was correctly installed, it will give that default usage help. 💖

Another classical example is to print sideRETRO's installed version using the -v option:

$ sider --version
sideRETRO 1.0.0

And, if the user needs further help, he can find it both at the sideRETRO's readthedocs page or in the already installed software documentation, from command line:

$ sider --help

Please, see A Practical Workflow and Running with Docker sections for more examples and tips for using with Docker.

Now, to get more familiar with sideRETRO main commands and results, let's try some basic examples for each command. 🫶

🧬 COMMAND PROCESS-SAMPLE 🧬

The first one is process-sample, or ps for short, and it was intended to act as the "evidence's grounding faith" for sideRETRO. Here, we're saying "first" because of an order in which the user must run the commands. The file resultant from this command will become the input to the next one, merge-call.

As explained in the Introduction section, the command process-sample creates a database of abnormal reads from a SAM/BAM file set. To do this, there are some mandatory options the user must supply to do a correct search. Calling the command process-sample without any argument will give a specific help where the user can know all the mandatory options for this command:

$ sider process-sample

📥 Arguments

  • One or more alignment file in SAM/BAM format

🔒 Mandatory Options

  • -a, --annotation-file — Gene annotation on the reference genome in GTF/GFF3 format. sider will look for 'exon' with the attribute 'transcript_type=protein_coding'. The attributes 'gene_name', 'gene_id' and 'exon_id' are also required.
  • -i, --input-file — File containing a newline separated list of alignment files in SAM/BAM/CRAM format. This option is not mandatory if one or more SAM/BAM/CRAM files are passed as argument. If 'input-file' and arguments are set concomitantly, then the union of all alignment files is used.

📁 Input/Output Options

  • -h, --help — Show help options.
  • -q, --quiet — Decrease verbosity to error messages only or suppress terminal outputs at all if 'log-file' is passed.
  • --silent — Same as '--quiet'.
  • -d, --debug — Increase verbosity to debug level.
  • -l, --log-file — Print log messages to a file.
  • -o, --output-dir — Output directory. Create the directory if it does not exist [default:"."].
  • -p, --prefix — Prefix output files [default:"out"].

🗄️ SQLite3 Options

  • -c, --cache-size — Set SQLite3 cache size in KiB [default:"200000"].

🎯 Read Quality Options

  • -Q, --phred-quality — Minimum mapping quality of the reads required [default:"8"].
  • -M, --max-base-freq — Maximum base frequency fraction allowed [default:"0.75"].
  • -D, --deduplicate — Remove duplicated reads. Reads are considered duplicates when they share the 5 prime positions of both reads and read-pairs.

⚙️ Processing Options

  • -s, --sorted — Assume all reads are grouped by queryname, even if there is no SAM/BAM/CRAM header tag 'SO:queryname'.
  • -t, --threads — Number of threads [default:"1"].
  • -m, --max-distance — Maximum distance allowed between paired-end reads [default:"10000"].
  • -f, --exon-frac — Minimum overlap required as a fraction of exon [default:"1e-09"; 1 base].
  • -F, --alignment-frac — Minimum overlap required as a fraction of alignment [default:"1e-09"; 1 base].
  • -e, --either — The minimum fraction must be satisfied for at least exon OR alignment. Without '-e', both fractions would have to be satisfied.
  • -r, --reciprocal — The fraction overlap must be reciprocal for exon and alignment. If '-f' is 0.5, then '-F' will be set to 0.5 as well.

So, supposing that the user has three files: f1.bam, f2.bam, f3.sam, he can type:

$ sider process-sample f2.bam f2.bam f3.sam \
    -a annotation_file.gtf

Note the mandatory -a option specifying the annotation file. And, in this unique exception, we suppressed the -i mandatory option cause all the files were explicitly called.

Let's see another example that shows the convenient use of the -i option to call a list of input files (e.g. my_files_list.txt) instead of them directly:

$ sider process-sample \
    -i my_files_list.txt \
    -a annotation_file.gtf

Both commands above will produce only one output database file out.db containing all relevant reads for non-fixed retrocopies search, whose prefix out can be easily changed with the -p option. The abnormal reads from all input files will be merged in just one table. To produce one database for each input file separately, user must run one distinct instance of sideRETRO per file.

Some options' values can affect drastically the output. Let's play a little bit with some of them while using the short version of the command ps:

$ sider ps \
    -i my_files_list.txt \
    -a annotation_file.gtf \
    -o output_dir \
    -p my_reads_database \
    -l my_log_file.log \
    -c 2000000 \
    -Q 20 \
    -F 0.9 \
    -t 3

Wow! The number of options can be overwhelming. 😭

Here used -o option to specify the directory output_dir to write our database as my_reads_database.db (-p option). Also, we chose to save the log messages in my_log_file.log file (-l option), a cache size of 2Gb (-c option), a minimum phred score cutoff of 20 for alignments (-Q option), a minimum overlap ratio of 0.9 for read alignments over exonic regions (-F option) and 3 threads to process those files in parallel (-t option).

To see another example of the process-sample command chained in a real workflow, please refer to the A Practical Workflow section.

🎀 COMMAND MERGE-CALL 🎀

okay bestie, step two of sideRETRO's "journey for the truth of retrocopies" is here and it's called merge-call, or mc if you're feeling lazy (and honestly, same). The whole vibe of this command? It takes the database that process-sample built for you as input, and then it populates more tables in it — all that new info comes straight from a clustering process running over the abnormal reads regions. It's giving glow-up, but for a database. 💅

And like process-sample, merge-call has some mandatory options, which you can literally just discover by calling it with no arguments at all:

$ sider merge-call

Arguments: One or more SQLite3 databases generated in the process-sample step

Mandatory Options:

  • -i, --input-file — a file containing a newline separated list of SQLite3 databases to be processed. Not mandatory if one or more SQLite3 databases are passed as argument. And if 'input-file' and arguments are set concomitantly? Then the union of all files is used. ✨

Input/Output Options:

  • -h, --help — show help options
  • -q, --quiet — decrease verbosity to error messages only, or suppress terminal outputs at all if 'log-file' is passed
  • --silent — same as '--quiet'
  • -d, --debug — increase verbosity to debug level
  • -l, --log-file — print log messages to a file
  • -o, --output-dir — output directory. Creates the directory if it does not exist [default:"."]
  • -p, --prefix — prefix output files [default:"out"]
  • -I, --in-place — merge all databases with the first one of the list, instead of creating a new file

SQLite3 Options:

  • -c, --cache-size — set SQLite3 cache size in KiB [default:"200000"]

Clustering Options:

  • -e, --epsilon — DBSCAN: maximum distance between two alignments inside a cluster [default:"300"]
  • -m, --min-pts — DBSCAN: minimum number of points required to form a dense region [default:"10"]

Filter & Annotation Options:

  • -b, --blacklist-chr — avoid clustering from and to this chromosome. This option can be passed multiple times [default:"chrM"]
  • -B, --blacklist-region — GTF/GFF3/BED blacklisted regions. If the file is in GTF/GFF3 format, the user may indicate the 'feature' (third column), the 'attribute' (ninth column) and its values
  • -P, --blacklist-padding — increase the blacklisted regions ranges (left and right) by N bases [default:"0"]
  • -T, --gff-feature — the value of 'feature' (third column) for GTF/GFF3 file [default:"gene"]
  • -H, --gff-hard-attribute — the 'attribute' (ninth column) for GTF/GFF3 file. It may be passed in the format key=value (e.g. gene_type=pseudogene). Each value will match as regex, so 'pseudogene' can capture IG_C_pseudogene, IG_V_pseudogene etc. This option can be passed multiple times and must be true in all of them
  • -S, --gff-soft-attribute — works as 'gff-hard-attribute'. The difference is that if this option is passed multiple times, it needs to be true only once [default:"gene_type=processed_pseudogene tag=retrogene"]
  • -x, --parental-distance — minimum distance allowed between a cluster and its putative parental gene [default:"1000000"]
  • -g, --genotype-support — minimum number of reads coming from a given source (SAM/BAM/CRAM) within a cluster [default:"3"]
  • -n, --near-gene-rank — minimum ranked distance between genes in order to consider them close [default:"3"]

Genotyping Options:

  • -t, --threads — number of threads [default:"1"]
  • -Q, --phred-quality — minimum mapping quality used to define reference allele reads [default:"8"]

And same energy as before — you can call a set of database files directly, or hand it a list of files:

$ sider merge-call database1.db database2.db -I

or

$ sider merge-call -i my_databases_list.txt -I
NOTE: again bestie, notice that -I option that is not mandatory but would lead to the creation of duplicated output databases if absent. This option does the clustering "in place" over the input files, overwriting them (so be careful 🫣). And if the user does not use the -p or -I options, the output files will be named out.db.

Now for a ✨more sophisticated✨ example, we're gonna use the short version of the command, mc, with a whole bunch of other options:

$ sider mc \
    -i my_databases_list.txt \
    -o output_dir \
    -p my_database \
    -l my_log_file.log \
    -I \
    -c 2000000 \
    -B my_black_list.bed \
    -x 1000000 \
    -g 5 \
    -Q 20 \
    -C 15 \
    -t 3

So here, the options -i, -o, -p, -l, -I, -c, -Q and -t keep the same meaning they have in the process-sample command. The others need some explanation, so let's get into it 🧚:

  • All we've done here was ask for a minimum number of 5 reads of contribution from each input SAM/BAM file to consider a clustering region as a retrocopy candidate (that's the -g option);
  • a minimum distance of 1000000 bp from the parental gene to resolve some doubtful overlaps (that's -x);
  • a minimum number of 15 crossing reads over the putative insertion point to consider heterozygosis evidence (-C — iykyk);
  • and, importantly, a BED file with a list of regions to be ignored at the clustering process called my_black_list.txt (the -B option). This last option's file can describe entire chromosomes (like chrM) and many chromosomal regions with poor insertion evidences taken from literature, like centromers. All specified regions won't be targets for clustering. 🚫

To see another example of the merge-call command chained in a real workflow, please refer to the A Practical Workflow section.

🌈 COMMAND MAKE-VCF 🌈

And now, the third and last step of sideRETRO's "crusade to retrocopies" — the make-vcf command, or vcf for short. This one takes the already clustered tables in the database files that got populated back at the merge-call step, and creates one VCF file with all statistically significant retrocopy insertions annotated in a convenient format. Chef's kiss. 👑

This command has no mandatory options, but it's genuinely worth trying to discover the others:

$ sider make-vcf

Arguments: SQLite3 database generated at process-sample and merge-call steps

Input/Output Options:

  • -h, --help — show help options
  • -q, --quiet — decrease verbosity to error messages only, or suppress terminal outputs at all if 'log-file' is passed
  • --silent — same as '--quiet'
  • -d, --debug — increase verbosity to debug level
  • -l, --log-file — print log messages to a file
  • -o, --output-dir — output directory. Creates the directory if it does not exist [default:"."]
  • -p, --prefix — prefix output files [default:"out"]

Filter & Annotation Options:

  • -n, --near-gene-dist — minimum distance between genes in order to consider them close [default:"10000"]
  • -e, --orientation-error — maximum error allowed for orientation rho [default:"0.05"]
  • -r, --reference-file — FASTA file for the reference genome

So, in order to produce a VCF file from a database input file like my_database.db, you literally just type:

$ sider make-vcf my_database.db

This will produce a out.vcf output file. Slay. 💅

Let's add more options to customize it to our needs (using the short version of the command, purely for symmetry bestie):

$ sider vcf my_database.db \
    -o output_dir \
    -p my_retrocopies \
    -l my_log_file.log \
    -r my_reference_genome.fa \
    -n 50000

Command make-vcf is very simple and doesn't allow the user to use threads. The only new options are -r, which must specify the reference genome in FASTA format (like gencode's Hg38.fa), and -n, where the user can establish a distance threshold for genes surrounding insertion points for additional information in the output VCF file. 🦋

DEALING WITH CRAM FORMAT 💾✨

okay so CRAM files? lowkey a little tricky, especially if you grabbed your data off a public repository. no cap. let's look at the two possible situations you might be in:

  • Local alignment
  • External alignment

Local alignment 🧬

so to make an alignment file in CRAM format, bestie, first we gotta index the reference genome:

# Inde for BWA: .fa.amb, .fa.ann, .fa.bwt, .fa.pac, .fa.sa files
bwa index hg38.fa

# Index reference genome for CRAM: .fa.fai file
samtools faidx hg38.fa

then we can align with bwa:

# Align with BWA and generate a CRAM
bwa mem hg38.fa file_R1.fastq file_R2.fastq | \
   samtools view -T hg38.fa -C -o file.cram -

that alignment, file.cram, can be processed with sider — as long as we don't change the reference genome and its index (.fa.fai) path. but if you do change it? then you gotta set the environment variables REF_PATH and REF_CACHE, see External alignment below. 💅

External alignment 🌐

when we download public data that's already aligned in CRAM format, we may be stressed about the reference genome index. probably we won't have the required genome index to read the .cram, and the htslib library — the one used by sider and samtools — is able to download that index from the CRAM Reference Registry. iconic of it tbh.

BUT. and this is the important part. for htslib to pull that off, we need to compile the library with the required flags and we also need the required dependencies (like libcurl). so, to be able to read these files without depending on those details, we need to generate a new local index and set the environment variables — REF_PATH and REF_CACHE — to the correct path:

# Create cache dir
mkdir -p /my/cache

# Construct the index
perl seq_cache_populate.pl -root /my/cache hg38.fa

# Now before running samtools or sider, we need to
# set the environment variables REF_PATH and REF_CACHE
export REF_PATH=/my/cache/%2s/%2s/%s:http://www.ebi.ac.uk/ena/cram
export REF_CACHE=/my/cache/%2s/%2s/%s

# So ...
sider ps -a annot.gff3.gz -o result file.cram

the script seq_cache_populate.pl can be found in the samtools, or at ‐ seq_cache_populate.pl. 🫶

for more info, see Samtools Worflow.

A PRACTICAL WORKFLOW 🎀🔥

okay NOW let's do something actually fun — a real exercise with real experimental data from the 1000 Genomes Project.

Warning: This example requires 16GB of RAM. 💀 for real, check your machine first bestie.

to run siderRETRO searching for retrocopies, we're gonna download 2 whole-genome sequenced CRAM files, both aligned on the gencode's hg38 genome: NA12878 and NA12778. slay.

at the beginning of a run, the files listed below must be in the same directory where the user is running sideRETRO, or their correct paths must be supplied at the corresponding option. the files are:

  1. A GTF gene annotation file from gencode project (here gencode.v32.annotation.gtf).
  2. A FASTA file with the gencode's Human reference genome, version 38 (here GRCh38_full_analysis_set_plus_decoy_hla.fa).
  3. A custom perl script, seq_cache_populate.pl, to construct a new local index. the seq_cache_populate.pl script can be found in ‐ seq_cache_populate.pl.
  4. A custom perl script, analyser.pl, to do the final analysis over the VCF file and produce the TSV file in a tabular format. the analyser.pl script can be downloaded here.

also, we're gonna set the environment variables REF_PATH and REF_CACHE, as a requirement to work with CRAM files — more info at Dealing with CRAM format. ✨

see the complete command sequence below for the whole analysis.

Tip: Copy and paste line by line in your terminal.

Tip 2: If you are running line by line in your terminal don't paste the "$" character. It is already in your terminal.

# Do things inside a clean directory.
# Average time: irrelevant
$ mkdir -p sider_test
$ cd sider_test

# Download annotation from gencode
wget ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/gencode.v32.annotation.gtf.gz

# Download the reference genome from 1000 genomes
wget ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/GRCh38_reference_genome/GRCh38_full_analysis_set_plus_decoy_hla.fa

# Make the CRAM index
# Create cache dir
mkdir -p cache

# create index
perl seq_cache_populate.pl -root cache GRCh38_full_analysis_set_plus_decoy_hla.fa

# Set environment variables
export REF_PATH=$PWD/cache/%2s/%2s/%s:http://www.ebi.ac.uk/ena/cram
export REF_CACHE=$PWD/cache/%2s/%2s/%s

# Create a download list (WGS.list) containing all files of interest.
# Average time: irrelevant
$ echo "ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR323/ERR3239334/NA12878.final.cram" > WGS_download.list
$ echo "ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR323/ERR3239484/NA12778.final.cram" >> WGS_download.list

# Download all files: NA12878 and NA12778.
# Average time: network dependent
$ wget -c -i WGS_download.list

# Create the list of BAM files.
# Average time: irrelevant
$ ls *.cram > WGS_genomes.list

# First sideRETRO step: process-sample
# Input file: WGS_genomes.list
# Output file: 1000_genomes.db
# Average time: 62m34.541
$ sider process-sample \
    -i WGS_genomes.list \
    -a gencode.v32.annotation.gtf.gz \
    -p 1000_genomes \
    -c 2000000 \
    -Q 20 \
    -F 0.9 \
    -t 2

# Second sideRETRO step: merge-call
# Input file: 1000_genomes.db
# Output file: 1000_genomes.db (same file)
# Average time: 62m34.541
$ sider merge-call 1000_genomes.db \
    -c 2000000 \
    -x 1000000 \
    -g 5 \
    -I \
    -t 2

# Second sideRETRO step: merge-call
# Input file: 1000_genomes.db
# Output file: 1000_genomes.vcf
# Average time: 62m34.541
$ sider make-vcf 1000_genomes.db \
    -p 1000_genomes \
    -r GRCh38_full_analysis_set_plus_decoy_hla.fa

# Some analysis over the final VCF file.
# Input file: 1000_genomes.vcf
# Output file: 1000_genomes.tsv
# Average time: 62m34.541
$ perl analyser.pl 1000_genomes.vcf > 1000_genomes.tsv

and that was a simple but complete pipeline to obtain a final TSV file with all the relevant results in a tabular format, ready to import in a R or Python script and plot some graphics. ate. no crumbs. 📊💖

RUNNING WITH DOCKER 🐳✨

notwithstanding sideRETRO's native run, the user can happily run it from a Docker image just by prepending Docker's directives to any example shown. so, supposing the user has Docker installed and has pulled the image galantelab/sider:latest from DockerHub, they can just prepend docker --rm -ti -v $(pwd):/home/sider -w /home/sider galantelab/sider to the ordinary sider command, like:

$ docker --rm -ti -v $(pwd):/home/sider -w /home/sider galantelab/sider \
  sider ps \
      -i my_files_list.txt \
      -a annotation_file.gtf \
      -o output_dir \
      -p my_reads_database \
      -l my_log_file.log \
      -c 2000000 \
      -Q 20 \
      -F 0.9 \
      -t 3

containerized and cute. the way that just works 🫶

AUTHOR 👑

Thiago L. A. Miller

2020, The sideRETRO Team

okay that's the whole page bestie, we did it 💅✨ this one's from sideRETRO (source/version SIDER(1)), dated Mar 17, 2025 — go forth and find those retrocopies, mwah 🎀🦋💖

📦 where this page comes from