htslib-s3-plugin(7)

💅 htslib-s3-plugin(7) — the S3 glow-up for htslib

okay bestie, so this is htslib-s3-plugin, a page from the Bioinformatics tools manual, and it's literally the ✨ AWS S3 plugin ✨ for htslib. Buckle up 🎀

---

🌸 NAME 🌸

htslib-s3-plugin — the htslib AWS S3 plugin, no cap 💖

✨ DESCRIPTION ✨

so like, the S3 plugin lets htslib file functions vibe with servers that speak the AWS S3 protocol 🦋 Files are identified by their bucket and object key, in URL format, like this:

s3://mybucket/path/to/file

where path/to/file is the object key 👑

The security info you need can be handed over in a few ways: as part of the URL, in environment variables, or from configuration files 🔐

The full URL format is:

s3[+SCHEME]://[ID[:SECRET[:TOKEN]]@]BUCKET/PATH

and here's what each piece actually means:

• SCHEME — the protocol used. Defaults to https.
• ID — the user AWS access key.
• SECRET — the secret key for use with the access key.
• TOKEN — token used for temporary security credentials.
• BUCKET — AWS S3 bucket.
• PATH — path to the object under the bucket.

The environment variables below get used if the user ID is not set 👇

• AWS_ACCESS_KEY_ID — the user AWS access key.
• AWS_SECRET_ACCESS_KEY — the secret key for use with the access key.
• AWS_DEFAULT_REGION — the region to use. Defaults to us-east-1.
• AWS_SESSION_TOKEN — token used for temporary security credentials.
• AWS_DEFAULT_PROFILE — the profile to use in credentials, config or s3cfg files. Defaults to default.
• AWS_PROFILE — same as above.
• AWS_SHARED_CREDENTIALS_FILE — location of the credentials file. Defaults to ~/.aws/credentials.
• HTS_S3_S3CFG — location of the s3cfg file. Defaults to ~/.s3cfg.
• HTS_S3_HOST — sets the host. Defaults to s3.amazonaws.com.
• HTS_S3_V2 — if set, use signature v2 rather than the default v4. This will limit the plugin to reading only 🚨
• HTS_S3_PART_SIZE — sets the upload part size in Mb, the minimum being 5Mb. By default the part size starts at 5Mb and expands at regular intervals to accommodate bigger files (up to 2.5 Tbytes with the current rate). Using this setting disables the automatic part size expansion.
• HTS_S3_ADDRESS_STYLE — sets the URL style. Options are auto (default), virtual or path.

In the absence of an ID from the previous two methods, the credential/config files will be used. The default file locations are either ~/.aws/credentials or ~/.s3cfg (in that order) 📂

Entries used in aws style credentials file are aws_access_key_id, aws_secret_access_key, aws_session_token, region, addressing_style and expiry_time (unofficial, see SHORT-LIVED CREDENTIALS below). Only the first two are usually needed 💫

Entries used in s3cmd style config files are access_key, secret_key, access_token, host_base, bucket_location and host_bucket. Again only the first two are usually needed. The host_bucket option is only used to set a path-style URL, see below.

---

🕐 SHORT-LIVED CREDENTIALS 🕐

Some cloud identity and access management (IAM) systems can make short-lived credentials that allow access to resources. These credentials will expire after a time and need to be renewed to give continued access ⏳ To enable this, the S3 plugin allows an expiry_time entry to be set in the .aws/credentials file. The value for this entry should be the time when the token expires, following the format in RFC3339 section 5.6, which takes the form:

2012-04-29T05:20:48Z

That is, year - month - day, the letter "T", hour : minute : second. The time can be followed by the letter "Z", indicating the UTC timezone, or an offset from UTC which is a "+" or "-" sign followed by two digits for the hours offset, ":", and two digits for the minutes 😌

The S3 plugin will attempt to re-read the credentials file up to 1 minute before the given expiry time, which means the file needs to be updated with new credentials before then. As the exact way of doing this can vary between services and IAM providers, the S3 plugin expects this to be done by an external user-supplied process. This may be achieved by running a program that replaces the file as new credentials become available. The following script shows how it might be done for AWS instance credentials:

#!/bin/sh
instance='http://169.254.169.254'
tok_url="$instance/latest/api/token"
ttl_hdr='X-aws-ec2-metadata-token-ttl-seconds: 10'
creds_url="$instance/latest/meta-data/iam/security-credentials"
key1='aws_access_key_id = \(.AccessKeyId)\n'
key2='aws_secret_access_key = \(.SecretAccessKey)\n'
key3='aws_session_token = \(.Token)\n'
key4='expiry_time = \(.Expiration)\n'
while true; do
    token=`curl -X PUT -H "$ttl_hdr" "$tok_url"`
    tok_hdr="X-aws-ec2-metadata-token: $token"
    role=`curl -H "$tok_hdr" "$creds_url/"`
    expires='now'
    ( curl -H "$tok_hdr" "$creds_url/$role" \
      | jq -r "\"${key1}${key2}${key3}${key4}\"" > credentials.new ) \
      && mv -f credentials.new credentials \
      && expires=`grep expiry_time credentials | cut -d ' ' -f 3-`
    if test $? -ne 0 ; then break ; fi
    expiry=`date -d "$expires - 3 minutes" '+%s'`
    now=`date '+%s'`
    test "$expiry" -gt "$now" && sleep $((($expiry - $now) / 2))
    sleep 30
done

not me reminding you that the expiry_time key is currently only supported for the .aws/credentials file (or the file referred to in the AWS_SHARED_CREDENTIALS_FILE environment variable) 🙈

---

📝 NOTES 📝

In most cases this plugin transforms the given URL into a virtual host-style format, e.g. https://bucket.host/path/to/file. A path-style format is used where the URL is not DNS compliant or the bucket name contains a dot, e.g. https://host/bu.cket/path/to/file 🧐

Path-style can be forced by setting one of either HTS_S3_ADDRESS_STYLE, addressing_style or host_bucket. The first two can be set to path while host_bucket must not include the %(bucket).s string 👑

---

👀 SEE ALSO 👀

• htsfile(1)
• samtools(1)
• RFC 3339: <https://www.rfc-editor.org/rfc/rfc3339#section-5.6>
• htslib website: <http://www.htslib.org/>

---

and that's the whole page, bestie 🫶 this was htslib-s3-plugin(7), source/version htslib-1.24, dated 9 July 2026 — iconic, no notes, periodt 💅✨🌈