Bio::D...bi::pg(3pm)

📦
version 1.7.4 · 2 versions
, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble

💖 Bio::DB::GFF::Adaptor::dbi::pg — the pg(3pm) manual, but make it cute 💖

okay bestie, gather round, because today we're talking about Bio::DB::GFF::Adaptor::dbi::pg — aka the database adaptor for a specific postgres schema. yes it's Perl, yes it's a manpage, yes we're doing this. 🎀

✨ NAME ✨

Bio::DB::GFF::Adaptor::dbi::pg — a database adaptor for a specific postgres schema. that's it, that's the name, she's straightforward and we love her for it. 💅

🌸 NOTES 🌸

okay so before this adaptor will even breathe, you gotta run some SQL. like, it needs these commands executed first:

CREATE DATABASE <dbname>;

that's the vibe check. no database, no adaptor, periodt.

Also!! if you're using this adaptor with the Generic Genome Browser (gbrowse), you need to grant select permission for every single table in the database to the owner of the httpd process. who's that, you ask? usually nobody, but on some RedHat systems it's apache. iykyk.

CREATE USER nobody;
GRANT SELECT ON TABLE fmeta                 TO nobody;
GRANT SELECT ON TABLE fgroup                TO nobody;
GRANT SELECT ON TABLE fdata                 TO nobody;
GRANT SELECT ON TABLE fattribute_to_feature TO nobody;
GRANT SELECT ON TABLE fdna                  TO nobody;
GRANT SELECT ON TABLE fattribute            TO nobody;
GRANT SELECT ON TABLE ftype                 TO nobody;

look at her go. seven whole tables getting their permissions. iconic. 👑

🦋 Optimizing the database 🦋

so here's the tea: PostgreSQL generally requires some tuning before you get very good performance for large databases. that's just the way it is, no cap. for general info on tuning a PostgreSQL server, peep this link: http://www.varlena.com/GeneralBits/Tidbits/perf.html

and of particular importance — like, the main character of tuning moves here — is executing VACUUM FULL ANALYZE whenever you change the database. every time. don't skip it. she's not optional. 😤

Additionally (yes there's more, we're not done), for a GFF database there are a few extra items you can tune. so for each automatic class in your GBrowse conf file, there will be one or two searches done when searching for a feature. and if there are lots of features? those searches can take several seconds. like, several. 😭 to speed these searches up, do two things:

  1. Set 'enable_seqscan = false' in your postgresql.conf file (and restart your server, bestie, don't forget that part).
  2. Create 'partial' indexes for each automatic class — here's how you'd do it for the example class 'Allele':
CREATE INDEX partial_allele_gclass ON
  fgroup (lower('gname')) WHERE gclass='Allele';

and be sure to run VACUUM FULL ANALYZE after creating the indexes. (see? i told you she'd come back. she always comes back.) ✨

💜 DESCRIPTION 💜

so this adaptor implements a specific postgres database schema that's compatible with Bio::DB::GFF. she inherits from Bio::DB::GFF::Adaptor::dbi, which itself inherits from Bio::DB::GFF. it's giving generational talent. 🧬

The schema uses several tables, and we're gonna meet every single one of them because they all matter:

🍓 fdata

This is the feature data table. Its columns are:

fid            feature ID (integer)
fref           reference sequence name (string)
fstart         start position relative to reference (integer)
fstop          stop position relative to reference (integer)
ftypeid        feature type ID (integer)
fscore         feature score (float); may be null
fstrand        strand; one of "+" or "-"; may be null
fphase         phase; one of 0, 1 or 2; may be null
gid            group ID (integer)
ftarget_start  for similarity features, the target start position (integer)
ftarget_stop   for similarity features, the target stop position (integer)

note that it would be desirable to normalize the reference sequence name, since there are usually many features that share the same reference feature. BUT — and this is a big but — in the current schema, query performance suffers dramatically when this additional join is added. so we don't. tragic? kinda. reality? yes. 💀

🌈 fgroup

This is the group table. There is one row for each group. Columns:

gid       the group ID (integer)
gclass    the class of the group (string)
gname     the name of the group (string)

The group table serves multiple purposes. As you might expect, it is used to cluster features that logically belong together, such as the multiple exons of the same transcript. She's also used to assign a name and class to a singleton feature. And finally — plot twist — the group table is used to identify the target of a similarity hit. This is consistent with the way in which the group field is used in the GFF version 2 format. multitasking queen. 👑

The fgroup.gid field joins with the fdata.gid field. they're linked, your honor.

Examples:

sql> select * from fgroup where gname='sjj_2L52.1';
+-------+-------------+------------+
| gid   | gclass      | gname      |
+-------+-------------+------------+
| 69736 | PCR_product | sjj_2L52.1 |
+-------+-------------+------------+
1 row in set (0.70 sec)
sql> select fref,fstart,fstop from fdata,fgroup
          where gclass='PCR_product' and gname = 'sjj_2L52.1'
                and fdata.gid=fgroup.gid;
+---------------+--------+-------+
| fref          | fstart | fstop |
+---------------+--------+-------+
| CHROMOSOME_II |   1586 |  2355 |
+---------------+--------+-------+
1 row in set (0.03 sec)

see how fast that second one was?? 0.03 seconds. we love a join. 🫶

⭐ ftype

This table contains the feature types, one per row. Columns are:

ftypeid      the feature type ID (integer)
fmethod      the feature type method name (string)
fsource      the feature type source name (string)

The ftype.ftypeid field joins with the fdata.ftypeid field. Example:

sql> select fref,fstart,fstop,fmethod,fsource from fdata,fgroup,ftype
       where gclass='PCR_product'
             and gname = 'sjj_2L52.1'
             and fdata.gid=fgroup.gid
             and fdata.ftypeid=ftype.ftypeid;
+---------------+--------+-------+-------------+-----------+
| fref          | fstart | fstop | fmethod     | fsource   |
+---------------+--------+-------+-------------+-----------+
| CHROMOSOME_II |   1586 |  2355 | PCR_product | GenePairs |
+---------------+--------+-------+-------------+-----------+
1 row in set (0.08 sec)

three tables in one query and she's still serving. main character energy fr fr. 🔥

🧚 fdna

This table holds the raw DNA of the reference sequences. It has three columns:

fref          reference sequence name (string)
foffset       offset of this sequence
fdna          the DNA sequence (longblob)

To overcome problems loading large blobs, DNA is automatically fragmented into multiple segments when loading, and the position of each segment is stored in foffset. The fragment size is controlled by the -clump_size argument during initialization. little DNA confetti, chopped up for convenience. 🎀

💅 fattribute_to_feature

This table holds "attributes", which are tag/value pairs stuffed into the GFF line. The first tag/value pair is treated as the group, and anything else is treated as an attribute (weird, huh? — their words, not mine, but honestly, yeah).

CHR_I assembly_tag Finished     2032 2036 . + . Note "Right: cTel33B"
CHR_I assembly_tag Polymorphism 668  668  . + . Note "A->C in cTel33B"

The columns of this table are:

fid                 feature ID (integer)
fattribute_id       ID of the attribute (integer)
fattribute_value    text of the attribute (text)

The fdata.fid column joins with fattribute_to_feature.fid. another iconic pairing. 💖

🎀 fattribute

This table holds the normalized names of the attributes. Fields are:

fattribute_id      ID of the attribute (integer)
fattribute_name    Name of the attribute (varchar)

🌟 Data Loading Methods 🌟

In addition to implementing the abstract SQL-generating methods of Bio::DB::GFF::Adaptor::dbi, this module also implements the data loading functionality of Bio::DB::GFF. she does it all, bestie. no notes.

🦋 new

Title   : new
Usage   : $db = Bio::DB::GFF->new(@args)
Function: create a new adaptor
Returns : a Bio::DB::GFF object
Args    : see below
Status  : Public

The new constructor is identical to the "dbi" adaptor's new() method, EXCEPT that the prefix "dbi:pg" is added to the database DSN identifier automatically if it is not there already. she's helpful like that. 💅

Argument       Description
--------       -----------

-dsn           the DBI data source, e.g. 'dbi:Pg:dbname=:ens0040' or "ens0040"

-user          username for authentication

-pass          the password for authentication

🌸 schema

Title   : schema
Usage   : $schema = $db->schema
Function: return the CREATE script for the schema
Returns : a list of CREATE statemetns
Args    : none
Status  : protected

(and yes, the original really does say "statemetns" — we keep the typo, we respect the typo, the typo is canon now. 😭)

This method returns a list containing the various CREATE statements needed to initialize the database tables. she's the architect. the blueprint. the moment. ✨

⭐ setup_load

Title   : setup_load
Usage   : $db->setup_load
Function: called before load_gff_line()
Returns : void
Args    : none
Status  : protected

This method performs schema-specific initialization prior to loading a set of GFF records. It prepares a set of DBI statement handlers to be used in loading the data. setting the stage, prepping the outfit, getting ready for the main event. 👑

🧚 load_gff_line

Title   : load_gff_line
Usage   : $db->load_gff_line($fields)
Function: called to load one parsed line of GFF
Returns : true if successfully inserted
Args    : hashref containing GFF fields
Status  : protected

This method is called once per line of the GFF and passed a series of parsed data items that are stored into the hashref $fields. The keys are:

ref          reference sequence
source       annotation source
method       annotation method
start        annotation start
stop         annotation stop
score        annotation score (may be undef)
strand       annotation strand (may be undef)
phase        annotation phase (may be undef)
group_class  class of annotation's group (may be undef)
group_name   ID of annotation's group (may be undef)
target_start start of target of a similarity hit
target_stop  stop of target of a similarity hit
attributes   array reference of attributes, each of which is a [tag=>value] array ref

look at that list. every single key accounted for. no crumbs left behind. 🍓

💜 get_table_id

Title   : get_table_id
Usage   : $integer = $db->get_table_id($table,@ids)
Function: get the ID of a group or type
Returns : an integer ID or undef
Args    : none
Status  : private

This internal method is called by load_gff_line to look up the integer ID of an existing feature type or group. The arguments are the name of the table, and two string identifiers. For feature types, the identifiers are the method and source. For groups, the identifiers are group name and class.

This method requires that a statement handler named lookup_$table, have been created previously by setup_load(). It is here to overcome deficiencies in mysql's INSERT syntax. (yes, mysql, in the postgres file. we don't ask questions, we just respect the lore. 🙈)

🎀 range_query

Title   : range_query
Usage   : $db->range_query($range_type,$refseq,$refclass,$start,$stop,$types,$order_by_group,$attributes,$binsize)
Function: create statement handle for range/overlap queries
Returns : a DBI statement handle
Args    : see below
Status  : Protected

and that's where part one leaves us, bestie — the range_query args are still coming, so don't go anywhere. 💖

okay so this method? it's the one that builds the statement handle for this module's whole main-character query 🎀 — you hand it a range and/or a list of feature types, and it fetches their GFF records. That's the gig. It overrides a method in dbi.pm so the overlaps query can write SQL that's been optimized for Postgres. And the way it does that? Instead of writing the bin-related section as a whole pile of ORs, each bin piece gets parked in its own separate select, and then they all get UNIONed together. 💅

This subroutine needs several replacements for other subroutines in dbi.pm. In this module, those replacements are named exactly the same as the ones in dbi.pm, just with "pg_" slapped on the front. iykyk.

The positional arguments go like this:

  • $isrange — a flag saying "bestie, this is a range query". If you don't pass it, an overlap query is what's assumed.
  • $refseq — the reference sequence name (undef if there's no range).
  • $refclass — the reference sequence class (undef if there's no range).
  • $start — the start of the range (undef if none).
  • $stop — the stop of the range (undef if none).
  • $types — array ref containing zero or feature types, in the format [method,source].
  • $order_by_group — a flag saying the statement handler should group the features by group id (super handy for iterative fetches, fr).
  • $attributes — a hash containing select attributes.
  • $binsize — a bin size for generating tables of feature density.

🌸 search_notes 🌸

ngl, this PostgreSQL adaptor does NOT implement the search notes method, and it's not being shady about it — it's just that it can be very slow. (The code for the method is literally right there in this method, but it's commented out. 💀 So close yet so far.)

There is, however, a PostgreSQL adaptor that implements it in a way more efficient way: Bio::DB::GFF::Adaptor::dbi::pg_fts, which inherits from this adaptor and uses the optional PostgreSQL module TSearch2 for full text indexing. Go read that adaptor's docs for the tea. 🫖

See also Bio::DB::GFF.

 Title   : search_notes
 Usage   : @search_results = $db->search_notes("full text search string",$limit)
 Function: Search the notes for a text string, using mysql full-text search
 Returns : array of results
 Args    : full text search string, and an optional row limit
 Status  : public

This one is a replacement for the mysql-specific method. Give it a search string and it performs an ILIKE search of the notes table, then hands you back an array of results. Every row in that returned array is an arrayref holding these fields:

  • column 1 — a Bio::DB::GFF::Featname object, suitable for passing to segment()
  • column 2 — the text of the note
  • column 3 — a relevance score.
⚠️ heads up bestie: on large databases this can be very slow and may result in a time out or 500-cgi errors. If this is happening on the regular, you should look into using Bio::DB::GFF::Adaptor::dbi::pg_fts, which implements the TSearch2 full text indexing scheme. 😭

✨ make_meta_set_query ✨

 Title   : make_meta_set_query
 Usage   : $sql = $db->make_meta_set_query
 Function: return SQL fragment for setting a meta parameter
 Returns : SQL fragment
 Args    : none
 Status  : public

By default this does absolutely nothing — meta parameters are not stored or retrieved. that's it, that's the whole vibe. 🙈

🦋 make_features_by_name_where_part 🦋

 Title   : make_features_by_name_where_part
 Usage   : $db->make_features_by_name_where_part
 Function: Overrides a function in Bio::DB::GFF::Adaptor::dbi to insure
           that searches will be case insensitive. It creates the SQL
           fragment needed to select a feature by its group name & class
 Returns : a SQL fragment and bind arguments
 Args    : see below
 Status  : Protected

and that's the whole page, periodt. 💖 this one's from perl v5.30.0, dated 2020-01-13, and it lives its life as Bio::D...bi::pg(3pm) — shoutout to the version and the date, iconic of them, no notes. ✨🎀🦋🌸💅

📦 where this page comes from