ref-cache(1)

📦
version 1.24 · 3 versions
, in epel 10, nixpkgs unstable

🎀 ref-cache — CRAM reference caching proxy 🎀

okay bestie, so this is ref-cache(1) from the bioinformatics tools girlies (that's the manual page name, keepin' it official), and honestly? she's a whole vibe. let's get into it 💅

✨ NAME ✨

ref-cache — CRAM reference caching proxy. that's it, that's the name, no cap.

✨ SYNOPSIS ✨

ref-cache [-bLUv] [-l LOG_DIR] [-u URL] -d CACHE_DIR -p PORT

like, all of the short flags up front, and then the ones you actually need at the end. iconic formatting honestly.

✨ DESCRIPTION ✨

so ref-cache is a caching proxy for reference sequences — the thing you want in your life when you're encoding and decoding CRAM format sequence alignment files. 🧬

here's the tea: CRAM can use reference based compression where individual bases in aligned records get compared against a known reference sequence, and it only stores the bases that differ. better compression, sure, but it means the reference sequence has to be supplied from some external source. one way to get these sequences is by querying a server implementing the GA4GH refget standard (yes, links work in manpage source: <https://ga4gh.github.io/refget/>), but that can lead to excessive network traffic and server load if — as is often the case — the same reference is needed more than once. and it always is. 😭 ref-cache makes reference handling easier by keeping copies of downloaded files so they can be reused when they're needed again. obsessed.

because it's been specifically designed to serve reference sequences for CRAM encoders and decoders, ref-cache behaves rather differently to general-purpose caching web proxies. like, the way that it's just built different:

  • it only makes requests to a single upstream server.
  • sequences are requested using MD5 checksum identifiers, as stored in the M5 tag in CRAM @SQ header lines.
  • the requested sequence comes back as a single string in uppercase ASCII text with no line terminators or other formatting characters. clean. ✨
  • downloaded sequences are checked to make sure they have the correct MD5 checksum before being stored in the cache. verification queen.
  • cached sequences are never removed. they live there forever, main character energy.
  • cached sequences are stored in a way that allows them to be accessed directly on the filesystem. this can allow the web server to be bypassed in some set-ups (for example where the cache is on a shared drive), letting already-downloaded files be accessed more efficiently. 🔥
  • when started, ref-cache will test to see if it's already running on the specified port, and exits silently if it finds that it is. this gives you a simple way of ensuring the server is up — just try to restart it every few minutes. lowkey genius.

🌈 QUICK-START GUIDE 🌈

first step bestie: create directories for the cache and (optionally) log files. then start up the server in the background, listening on port 8080, with the EBI's CRAM reference server as the upstream source. 🫶

mkdir cached_refs
mkdir logs
ref-cache -b -d cached_refs -l logs -p 8080 -u https://www.ebi.ac.uk/ena/cram/md5/

to make SAMtools and HTSlib use the server, set its URL in the REF_PATH environment variable — and heads up, colons should be doubled up in the URL, plus you should substitute the hostname of your actual server:

REF_PATH='http:://myserver.example.com::8080/%s'
export REF_PATH

if the cache directory can be made visible to SAMtools/HTSlib processes, you can also add it directly to REF_PATH by putting it before the web server URL. it's necessary to use the full path to the directory, followed by /%2s/%2s/%s for the file location, due to the way they're stored inside the cache:

REF_PATH='/path/to/cache/%2s/%2s/%s:http:://myserver.example.com::8080/%s'
export REF_PATH

this is useful because accessing the files directly is more efficient than using http. files are downloaded to a temporary name and then renamed after validation, so processes directly using the cache will never try to use a partly downloaded file. slay. and by putting the URL at the end, the web server will pick up any requests for references not already in the cache, download them, provide them to the requester, and store them in the cache. the whole cycle. 💖

💅 OPTIONS 💅

  • -b — run in the background as a System V-style daemon. ⚠️ this option must not be used with -s.
  • -d <dir> — directory where cached files will be stored.
  • -h — show help. (hi, hello, here's how this works.)
  • -l <dir> — directory for log files. if not set and running in the foreground, logs will be sent to stdout.
  • -L — don't log. silence, but make it fashion. 🤫
  • -m all|default|localhost|<network-list> — reply to connections from the listed network(s). this option can be given more than once, with the final allow list being the union of all listed networks along with localhost (which is always enabled). see CLIENT ADDRESS CHECKING below. 👀
  • -n <1-4> — number of server processes to run.
  • -p <port> — port number to listen on.
  • -r <num> — number of request log files to keep.
  • -R <num> — maximum size of a request log file (MiB).
  • -s — run as a systemd-style socket service. since the service manager handles socket allocation, the -p option is ignored when running in this mode. ⚠️ this option must not be used with -b.
  • -u <url> — URL of the upstream server. if not set or overridden using -U, the EBI's server (https://www.ebi.ac.uk/ena/cram/md5/) will be used.
  • -U — do not attempt to get files from an upstream server. only files already in the local cache will be served. no outside world, just us. 🙈
  • -v — turn on debugging output.

🦋 CLIENT ADDRESS CHECKING 🦋

ref-cache is designed to serve references to local networks. to ensure it only responds to the desired clients, it has an allow list of address ranges that it will talk to. if a connection attempt comes from an IP address not in the allowed set, it will be closed immediately. 🚫

N.B.: rejected clients will see a connection open and immediately close, because it's necessary for connections to be opened for the server to discover the peer address. if you want to drop or reject unwanted requests without opening them, you'll need to use your operating system's firewall. fr fr, that's a firewall job.

the address ranges can be set using the -m option, which may be used more than once. networks can be specified either as a comma-separated list of CIDR-format blocks (e.g. 192.0.2.0/24, 2001:db8::/32) or using one of these synonyms:

  • all — any address (not recommended). 💀
  • default — 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 (the private ranges listed in RFC 1918); fc00::/7 (the local IPv6 Unicast address range in RFC 4193); and fe80::/10 (IPv6 link-local addresses).
  • localhost — 127.0.0.0/8 and ::1/128 (loop-back addresses).

if no -m option is given, the default list will be used, since most organisations will be using one or more of these internally. this will be overridden if any -m option appears, in which case -m default will need to be specified explicitly if you also want to reply to addresses in the IPv4 and IPv6 private ranges. for example:

ref-cache -m 192.0.2.0/24 -m default ...

ref-cache will always listen to the loop-back address, even if this was not specified. using -m localhost will limit it to only respond to loop-back requests. 🎀

👑 AUTHOR 👑

written by Rob Davies from the Wellcome Sanger Institute. legend behaviour, honestly.

⭐ SEE ALSO ⭐

and that's the page bestie!! 💜 this little beauty is htslib-1.24, dated 9 July 2026 — so go forth, cache those references, and remember: rejected clients get the immediate close, not the explanation. byeeee 🦋✨🎀

📦 where this page comes from