Blastall(3pm)

📦 version 0.66, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu resolute, ubuntu stonking

💖 Blastall(3pm) 💖

okay so this whole thing lives in the User Contributed Perl Documentation corner of the manual-verse, and the actual star of the show is NHGRI::Blastall — the Perl extension that runs and parses NCBI's BLAST 2.x. iykyk ✨

🌸 NAME 🌸

NHGRI::Blastall — it's a Perl extension for running and parsing NCBI's BLAST 2.x. That's it, that's the name, no notes. 💅

🎀 SYNOPSIS 🎀

bestie there is literally nothing here. The synopsis is empty — the real lore lives down in DESCRIPTION and all the sections after it. Consider this a vibe check that you passed by scrolling past. 🦋

✨ DESCRIPTION ✨

okay so here's the situation. If you've got NCBI's BLAST2 or WU-BLAST installed locally and your environment is already set up, you can use Perl's object-oriented capabilities to run your BLASTs. Iconic. Also, if you have a blastcl3 binary from the toolkit (or binaries from our FTP site), you can run BLAST over the network. network blast, main character energy.

There are also methods to blast single sequences against each other using the bl2seq binaries (also in the toolkit and binaries). And you can blast one sequence against a library of sequences using the blast_one_to_many method. You can format databases with the formatdb method. You can also have NHGRI::Blastall read existing BLAST reports.

And if you have a database of repetitive DNA or other DNA you'd like to mask out, you can use the mask method to mask the data against these databases. Then you can use either the filter or result methods to parse the report and access the various elements of the data. slay, everything is accessible. 🌈

🏃 RUNNING NEW BLASTS 🏃

use NHGRI::Blastall;
my $b = new NHGRI::Blastall;
# If you are running NCBI's Local BLAST
$b->blastall( p => 'blastn',
              d => 'nr',
              i => 'infile',
              o => 'outfile'
            );
# If you are running NCBI's blastcl3 network client
$b->blastcl3( p => 'blastn',
              d => 'nr',
              i => 'infile',
              o => 'outfile'
            );
# If you are running WU-BLAST locally
$b->wu_blastall( p      => 'blastn',
                 d      => 'nr',
                 nogap  => '!',     #use ! for arguments w/o parameter
                 i      => 'infile',
                 o      => 'outfile'
               );

See BLASTALL for more info 💖

🧬 BLASTING 2 SEQUENCES 🧬

use NHGRI::Blastall;
my $b = new NHGRI::Blastall;
$b->bl2seq(i => 'seq1',
           j => 'seq2',
           p => 'tblastx'
          );

See BL2SEQ for more info 💖

🦋 BLASTING 1 SEQUENCE AGAINST A FASTA LIBRARY OF SEQUENCES 🦋

# a library is a FASTA file with multiple FASTA formatted sequences
use NHGRI::Blastall;
my $b = new NHGRI::Blastall;
$b->blast_one_to_many(i => 'seq1',
                      d => 'seq2.lib',
                      p => 'tblastx',
                     );

See BLAST_ONE_TO_MANY for more info 💖

📂 INITIALIZING EXISTING BLAST REPORTS 📂

use NHGRI::Blastall;
my $b = new NHGRI::Blastall;
$b->read_report('/path/to/report');

🎭 MASKING SEQUENCES 🎭

use NHGRI::Blastall;
my $b = new NHGRI::Blastall;
$masked_seq = $b->mask( type => 'wu_blastall',
                        p    => 'blastn',
                        d    => 'alu',
                        i    => 'infile'
                      );

See MASKING for more info 💖

🗂️ CREATING BLAST INDEXES 🗂️

use NHGRI::Blastall;
my $b = new NHGRI::Blastall;
$b->formatdb( i => 'est',
              p => 'F',
              o => 'T',
            );

See FORMATDB for more info 💖

🖨️ PRINTING REPORTS 🖨️

$b->print_report();
# this method only opens the report and prints.  It does not print
# summary reports

🔍 FILTERING BLAST RESULTS 🔍

@hits = $b->filter( scores     => '38.2',
                    identities => '.98'
                  );

# returns an array of hash references.

See HASHREF for more info on manipulating the results. See FILTERING for more info on using the filter method 💖

💎 GETTING AT ELEMENTS 💎

@ids = $b->result('id');
@scores = $b->result('scores',$ids[0]);  # second param must be an id

See RESULT for more info on using the result method. See ELEMENTS for element names 💖

🌈 GETTING AT ALL THE DATA 🌈

@results = $b->result();  # returns an array of hashes

See HASHREF for information on the array of hashes that is returned. See DUMP RESULTS to see how to work with the array of hashes 💖

🎯 ADJUSTING THE DEFLINE REGEX 🎯

$b = new NHGRI::Blastall (-DB_ID_REGEX => '[^ ]+');

See DB_ID_REGEX for more info 💖

👑 BLASTALL 👑

okay so this method is the moment. It gives you a simple object-oriented frontend to BLAST. This module works with either NCBI's blastall binary distributed with BLAST 2.x, WU-BLAST, or over the web through NCBI's Web Site.

The blastall function accepts as a parameter an anonymous hash with keys that are the command line options (see BLASTALL OPTIONS) and values which are the corresponding values to those options.

pro tip energy: you may want to set the BLASTALL variable in Blastall.pm to the full path of your blastall binary — especially if you'll be running scripts as cron jobs, or if blastall is not in the system path. Trust. 🫶

✨ BLASTALL OPTIONS ✨

For wu_blastall you need to use NCBI type switches for the following:

-i for infile
-o for outfile
-p for program
-d for database

the rest of the parameters MUST be the parameters available through WU-BLAST (e.g. -sump, -nogap -compat1.4, etc.). Use a ! to specify that an argument has no parameters. See the example at the top of the manpage. 👀

These are the options that NCBI's blastall and binary accepts, and these are the same options that are accepted by the blastall and blastcl3 methods.

NOTE: You must set the proper environmental variables for the blastall method to work (BLASTDB, BLASTMAT).

p — Program Name
d — Database — default=nr
i — QueryFile
e — Expectation value (E) — default=10.0
m — alignment view — default=0
- 0 = pairwise,
- 1 = master-slave showing identities,
- 2 = master-slave no identities,
- 3 = flat master-slave, show identities,
- 4 = flat master-slave, no identities,
- 5 = master-slave no identities and blunt ends,
- 6 = flat master-slave, no identities and blunt ends
o — BLAST report Output File — default=stdout
F — Filter query sequence — default=T (DUST with blastn, SEG with others)
G — Cost to open a gap — default=0 (zero invokes default behavior)
E — Cost to extend a gap — default=0 (zero invokes default behavior)
X — X dropoff value for gapped alignment (in bits) — default=0 (zero invokes default behavior)
I — Show GI's in deflines — default=F
q — Penalty for a nucleotide mismatch (blastn only) — default=-3
r — Reward for a nucleotide match (blastn only) — default=1
v — Number of one-line descriptions (V) — default=500
b — Number of alignments to show (B) — default=250
f — Threshold for extending hits, default if zero — default=0
g — Perform gapped alignment (NA with tblastx) — default=T
Q — Query Genetic code to use — default=1
D — DB Genetic code (for tblast[nx] only) — default=1
a — Number of processors to use — default=1
O — SeqAlign file — Optional
J — Believe the query defline — default=F
M — Matrix — default=BLOSUM62
W — Word size, default if zero — default=0
z — Effective length of the database — default=0 (use zero for the real size)
K — Number of best hits from a region to keep — default=100
L — Length of region used to judge hits — default=20
Y — Effective length of the search space — default=0 (use zero for the real size)
S — Query strands to search against database — default=3 (for blast[nx], and tblastx). 3 is both, 1 is top, 2 is bottom
T — Produce HTML output [T/F] — default=F
l — Restrict search of database to list of GI's [String]

⚠️ NOTE: If you do not supply an o option (outfile), the following environment variables are checked in order: TMPDIR, TEMP, and TMP. If one of them is set, outfiles are created relative to the directory it specifies. If none of them are set, the first possible one of the following directories is used: /var/tmp, /usr/tmp, /temp, /tmp. This file is deleted after the NHGRI::Blastall object is destroyed. It is recommended that you create a tmp directory in your home directory and set one of the above environmental vars to point to this directory and then set the permissions on this directory to 0700. Writing to a "public" tmp directory can have security ramifications. bestie please, don't do that one. 😭

🧚 BL2SEQ 🧚

This method uses the bl2seq binary (distributed with BLAST executables and source) to BLAST one sequence against another sequence. Like the blastall method, the bl2seq method accepts the same options that the bl2seq binary accepts. Run bl2seq without options from the command line to get a full list of options.

important note about the options: when running blastx the 1st sequence should be nucleotide; when running tblastn the 2nd sequence should be nucleotide. the way that matters so much. 💅

use NHGRI::Blastall;
my $b = new NHGRI::Blastall;
$b->bl2seq(i => 'seq1.nt',
           j => 'seq2.aa',
           p => 'blastx'
          );

🌟 BLAST_ONE_TO_MANY 🌟

This method allows for blasting one sequence against a FASTA library of sequences. Behind the scenes, BLAST indexes are created (in the same directory as the FASTA library) using the provided FASTA library and the one sequence is used to search against this database. If the program completes successfully, the databases are removed.

To compare two sequences, use the bl2seq method which is faster and less messy (no tmp indexes). This method accepts the same options as the blastall binary with the d option corresponding to the FASTA library.

use NHGRI::Blastall;
my $b = new NHGRI::Blastall;
$b->blast_one_to_many(i => 'seq.aa',
                      d => 'seq.nt.lib',
                      e => '0.001',
                      p => 'tblastn',
                     );

and that's just the setup arc — the deep lore keeps going 👇 💜

🧬 MASKING 💅

okay bestie, this one screens your DNA sequences (fasta format, obviously) against whatever database you told the blastall d option about. the mask method takes the same parameters as the blastall method — no new rules, same vibe. any matches to the masking database get swapped out for "N"s. then the mask method hands you back the masked sequence. it does basically the same job as xblast, which is this old NCBI program written in C. iconic ancestor behavior, honestly.

set the type parameter to wu_blastall, blastcl3 or blastall depending on your configuration. iykyk.

$masked_seq = $b->mask( type => 'blastcl3', # defaults to blastall
                        p    => 'blastn',
                        d    => 'alu',
                        i    => 'infile'
                      );

and if you want the mask coordinates back?? call the mask method in an array context. so simple. so slay.

@mask = $b->mask(p    => 'blastn',
                 d    => 'alu',
                 i    => 'infile'
                );
$masked_seq = $mask[0];        # same as above masked seq
$ra_masked_coords = $mask[1];  # reference to array of mask coordinates

📚 FORMATDB 🎀

this method creates BLAST indexes using the formatdb binary which is distributed with BLAST. it accepts the same parameters as formatdb. and the remove_formatdb_indexes method will remove databases created using the formatdb method — but only if it's called by the same object. okay so important thing: formatdb leaves a file called formatdb.log by default in the current working directory (if it has permission, like, if it can). to change this behavior, use the l option to direct the sequence to /dev/null or somewhere else. bye bye log file 💀

use NHGRI::Blastall;
my $b = new NHGRI::Blastall;
$b->formatdb( i => 'swissprot',
              p => 'T',
              l => '/dev/null',
              o => 'T',
            );

🌸 DB_ID_REGEX 🌸

by default Blastall.pm expects FASTA deflines of BLAST databases to be formatted like a Genbank database:

gi|GINUMBER|DATABASE|ACCESSION|SUMMARY

the default regular expression is [^\|]+(?:\|[^\|,\s]*){1,10}

when you're using non-genbankformatted deflines, it may become necessary to adjust the regular expression that identifies the unique identifier in a defline. you can do this with the -DB_ID_REGEX parameter to the new method. for example:

$b = new NHGRI::Blastall( -DB_ID_REGEX => '[^ ]+' );

✨ FILTERING ✨

the filter method accepts an anonymous hash where the keys are elements of the blast report and the values are limits that are put on the result set.

the following are the Filter elements and their default operations:

id                  => regular expression match
defline             => regular expression match
subject_length      => greater than
scores              => greater than
expects             => less than
identities          => greater than
match_length        => greater than
subject_strand      => equals
query_frames        => equals
subject_frames      => equals

so if you would like to limit your results to entries that have scores greater than 38.2 and identities greater than 98% you would say...

@hits = $b->filter( scores      => '38.2',
                    identities  => '.98'
                  );

you can also override the defaults. if you would like only scores that are less than 38.2 you could say...

@hits = $b->filter( scores => '<38.2' );

or if you wanted only identities that were equal to 1 and you didn't care about the hits array you could say...

$b->filter( identities => '=1' );

regular expression matches are case insensitive. so if you wanted only records with the word "human" in the defline you could say...

@hits = $b->filter( defline => 'HuMaN' );

🫶 heads up bestie: after you run the filter method on an object, the object only contains those results which passed the filter. this will effect additional calls to the filter method as well as calls to other methods (e.g. result). to reset the NHGRI::Blastall object you can use the unfilter method.

$b->unfilter();

see DUMP RESULTS for info on how to manipulate the array of hash refs.

🦋 RESULT 🦋

the result method has 3 possible invocations. the first invocation is when it is called without parameters.

@results = $b->result();

this invocation returns an array of hash references. see HASHREF for further explanation of this structure.

to get a list of all the ids do...

@ids = $b->result('id');

these ids can be used to get at specific elements. if 2 parameters are present and the first one is an element (see ELEMENTS for a list of ELEMENTS) and the second one is an id then the routine will return a list of elements corresponding to the id.

@scores = $b->result('scores',$ids[0]);  # second param must be an id

if more than 2 elements are passed the function will return undef. that's it, that's the rule, no cap.

👑 ACCESSOR METHODS 👑

get_report

returns the filename of the BLAST report.

get_database_description

returns description given to the database during formatting of db. e.g. All non-redundant GenBank CDStranslations+PDB+SwissProt+PIR

get_database_sequence_count

returns the number of sequences in the database.

get_database_letter_count

returns the number of total letters in the database.

get_blast_program

returns the BLAST program name that appears at the top of the report. either BLASTN, BLASTP, BLASTX, TBLASTN or TBLASTX

get_blast_version

returns the version of the BLAST program that was used.

💖 ELEMENTS 💖

id

an example of an id is >gb|U19386|MMU19386 the initial > is just a flag. the next characters up until the first pipe is the database the subject was taken from. the next characters up to the next pipe is the Genbank accession number. the last characters are the locus. this element is used as a unique identifier by the NHGRI::Blastall module. (SCALAR)

defline

The definition line taken from the subject (SCALAR)

subject_length

This is the length of the full subject, not the length of the match. (SCALAR)

scores

This is score (in bits) of the match. (ARRAY)

expects

This is the statistical significance (E value) for the match. (ARRAY)

identities

This is the number of identities divided by the match length in decimal format. (Listed as a fraction and a percentage in a BLAST report.) (ARRAY)

match_lengths

this is the number of base pairs that match up. (ARRAY)

query_starts

This is the number of the first base which matched with the subject. (ARRAY)

query_ends

This is the number of the last base which matched with the subject. (ARRAY)

subject_starts

This is the number of the first base which matched with the query. (ARRAY)

subject_ends

This is the number of the last base which matched with the query. (ARRAY)

subject_strands

This is either plus or minus depending on the orientation of the subject sequence in the match. (ARRAY)

query_strands

This is either plus or minus depending on the orientation of the query sequence in the match. (ARRAY)

query_frames

If you are running a blastx or tblastx search in which the query_sequence is translated this is the frame the query sequence matched. (ARRAY)

subject_frames

If you are running a tblastn or tblastx search in which the subject sequence is translated, this is the frame where the subject sequence matched. (ARRAY)

🌈 HASHREF 🌈

each hash ref contains an id, defline and subject Length. because there can be multiple scores, expect values, Identities, match_lengths, query_starts, query_strands and subject_starts, these are stored as array references. the following is an array containing two hash refs. look at her. she's beautiful. 💅

@hits = (
    {'id'                 => '>gb|U79716|HSU79716',
     'defline'            => 'Human reelin (RELN) mRNA, complete cds',
     'subject_length'     => '11580',
     'scores'             => [ 684, 123               ],
     'expects'            => [ 0.0, 3e-26             ],
     'identities'         => [ .99430199, .89256198   ],
     'match_lengths'      => [ 351, 121               ],
     'query_starts'       => [ 3, 404                 ],
     'query_ends'         => [ 303, 704                 ],
     'subject_starts'     => [ 5858, 6259             ],
     'subject_ends'       => [ 6158, 6559             ],
     'subject_strands'    => [ 'plus', 'minus'        ],
     'query_strands'      => [ 'plus', 'plus'         ],
     'query_frames'       => [ '+1', '-3'             ],
     'subject_frames'     => [ '+2', '-1'             ],
    },
    {'id'                 => '>gb|U24703|MMU24703',
     'defline'            => 'Mus musculus reelin mRNA, complete cds',
     'subject_length'     => '11673',
     'scores'             => [ 319, 38.2              ],
     'expects'            => [ 2e-85, 1.2             ],
     'identities'         => [ .86455331, 1           ],
     'match_lengths'      => [ 347, 19                ],
     'query_starts'       => [ 3, 493                 ],
     'query_ends'         => [ 303, 793                 ],
     'subject_starts'     => [ 5968, 6457             ]
     'subject_ends'       => [ 6268, 6757             ],
     'subject_strands'    => [ 'plus', 'minus'        ],
     'query_strands'      => [ 'plus', 'plus'         ],
     'query_frames'       => [ '+3', '-3'             ],
     'subject_frames'     => [ '+1', '-2'             ],
    }
);

see ELEMENTS for explanation of each element.

see DUMP RESULTS and/or the perlref(1) manpage for clues on working with this structure.

DUMP RESULTS 💅✨

Okay bestie, listen. When you call the result function — or call it with no parameters at all — or when you call the filter function, what comes back? An array of references to hashes. Every single element of that array? A reference to a hash that is holding exactly 1 record. Need the full structure? Go see HASHREF. Need more on references in general? perlref(1) has you covered, fr fr.

The routine below is gonna loop through each element of that array of hashes and then print out the element and its corresponding value or values. Iconic behavior, no cap.

sub dump_results {
    foreach $rh_r (@results) {
        while (($key,$value) = each %$rh_r) {
            if (ref($value) eq "ARRAY") {
                print "$key: ";
                foreach $v (@$value) {
                    print "$v ";
                }
                print "\n";
            } else {
                print "$key: $value\n";
            }
        }
    }
}

AUTHOR 👑✨

Joseph Ryan — jfryan@nhgri.nih.gov

CONTACT ADDRESS 🫶

If you have problems, questions, comments send to webblaster@nhgri.nih.gov. That's the whole contact address, bestie. No gatekeeping, just vibes.

COPYRIGHT INFORMATION 📜💖

Okay legal moment, but make it cute: this software/database is "United States Government Work" under the terms of the United States Copyright Act. It was written as part of the authors' official duties for the United States Government and thus cannot be copyrighted. This software/database is freely available to the public for use without a copyright notice. Restrictions cannot be placed on its present or future use. Periodt, that's the legal tea.

Although all reasonable efforts have been taken to ensure the accuracy and reliability of the software and data, the National Human Genome Research Institute (NHGRI) and the U.S. Government does not and cannot warrant the performance or results that may be obtained by using this software or data. NHGRI and the U.S. Government disclaims all warranties as to performance, merchantability or fitness for any particular purpose.

In any work or product derived from this material, proper attribution of the authors as the source of the software or data should be made, using http://genome.nhgri.nih.gov/blastall as the citation.

ENVIRONMENT VARIABLES 🌈🦋

BLASTDB — location of BLAST formatted databases.
BLASTMAT — location of BLAST matrices.
TMPDIR TEMP TMP — if the o option is not passed to the blastall method than NHGRI::Blastall looks for one of these vars (in order) to store the BLAST report. This report is destroyed after the NHGRI::Blastall.pm object is destroyed.

Heads up, bestie: that report gets destroyed after the NHGRI::Blastall.pm object is destroyed. So don't get too attached unless you saved it. 💀