git-annex-addurl(1)

📦
version 10.20261026 · 5 versions
, in alpine 3.24, arch rolling, nixos 26.05, nixpkgs unstable

💅 git-annex-addurl — add urls to annex

okay so first things first, this whole page is from the git-annex manual (the General Commands Manual section, iykyk) and it's literally about yeeting urls into your annex. bestie, let's get into it 🎀

✨ SYNOPSIS ✨

git annex addurl [url ...]

🦋 DESCRIPTION 🦋

It downloads each url to its own file, and that file gets added to the annex. simple as that, no cap 💖

When yt-dlp is installed, it can be used to check for a video embedded in a web page at the url, and that video is what gets added to the annex instead. But — big but — this is disabled by default because it can be a security risk 🙈. See the documentation of annex.security.allowed-ip-addresses in git-annex(1) for the deets.

Special remotes can add their own special handling of particular urls. For example, the bittorrent special remotes make urls to torrent files (including magnet links) download the content of the torrent, using aria2c. iconic behavior honestly 👑

Normally the filename is based on the full url, so it'll look like "www.example.com_dir_subdir_bigfile". In some cases, addurl is able to come up with a better filename based on other information, and options can also be used to get better filenames. slay 💅

🌸 OPTIONS 🌸

  • --fast — Avoid immediately downloading the url. The url is still checked (via HEAD) to verify that it exists, and to get its size if possible. vibe check only, no actual download 🫶
  • --relaxed — Don't immediately download the url, and avoid storing the size of the url's content. This makes git-annex accept whatever content is there at a future point.

This is the fastest option, but it still has to access the network to check if the url contains embedded media. When adding large numbers of urls, using --relaxed --raw is much faster. ✨

  • --verifiable, -V — This option is now enabled by default when using --fast or --relaxed, but was not the default in older versions of git-annex.

When a file is added without first downloading its content from the web, the checksum of the file is not yet known.

To allow later learning and verifying the checksum, the VURL backend is used. With --fast, the checksum is learned on initial download of the file from the web, and all subsequent downloads from the web must have the same checksum. With --relaxed, additional checksums are added each time different content is downloaded from the web.

This improves the safety of the resulting annexed file, by letting its content be verified with a checksum when it is transferred between git-annex repositories, as well as by things like git-annex fsck. 🔥

Files that were added with old versions of addurl (or with git-annex registerurl or git-annex fromkey) are not verifiable. They can be converted to verifiable by migrating them from the URL backend to the VURL backend. glow-up time 💖

For example: git-annex migrate foo --backend=VURL

  • --raw — Prevent special handling of urls by yt-dlp, and by bittorrent and other special remotes. This will for example, make addurl download the .torrent file and not the contents it points to. no middlemen, just the raw file 💀
  • --no-raw — Require content pointed to by the url to be downloaded using yt-dlp or a special remote, rather than the raw content of the url. If that cannot be done, the add will fail. all or nothing, periodt 🙈
  • --raw-except=remote — Prevent special handling of urls by all special remotes except for the specified one. To allow special handling only by yt-dlp, use --raw-except=web. 🎀
  • --file=name — Use with a filename that does not yet exist to add a new file with the specified name and the content downloaded from the url.

If the file already exists, addurl will record that it can be downloaded from the specified url(s). 🌈

  • --preserve-filename — When the web server (or torrent, etc) provides a filename, use it as-is, avoiding sanitizing unusual characters, or truncating it to length, or any other modifications.

git-annex will still check the filename for safety, and if the filename has a security problem such as path traversal or a control character, it will refuse to add it. bestie it will NOT let that slide 🚨

  • --pathdepth=N — Rather than basing the filename on the whole url, this causes a path to be constructed, starting at the specified depth within the path of the url.

For example, adding the url http://www.example.com/dir/subdir/bigfile with --pathdepth=1 will use "dir/subdir/bigfile", while --pathdepth=3 will use "bigfile".

It can also be negative; --pathdepth=-2 will use the last two parts of the url. math but make it cute ✨

  • --prefix=foo, --suffix=bar — Use to adjust the filenames that are created by addurl. For example, --suffix=.mp3 can be used to add an extension to the file. little accessory for your filename 💅
  • --no-check-gitignore — By default, gitignores are honored and it will refuse to download an url to a file that would be ignored. This makes such files be added despite any ignores. breaking the rules (respectfully) 🦋
  • --jobs=N, -JN — Enables parallel downloads when multiple urls are being added. For example: -J4

Setting this to "cpus" will run one job per CPU core. multiprocessing era 🔥

  • --batch — Enables batch mode, in which lines containing urls to add are read from stdin. feed her lines, she eats 🍓
  • -z — Makes the --batch input be delimited by nulls instead of the usual newlines.
  • --with-files — When batch mode is enabled, makes it parse lines of the form: "$url $file"

That adds the specified url to the specified file, downloading its content if the file does not yet exist; the same as git annex addurl $url --file $file

  • --json — Enable JSON output. This is intended to be parsed by programs that use git-annex. Each line of output is a JSON object. for the machine besties 🤖💖
  • --json-progress — Include progress objects in JSON output.
  • --json-error-messages — Messages that would normally be output to standard error are included in the JSON instead.
  • --backend — Specifies which key-value backend to use.

Also the git-annex-common-options(1) can be used. the usual suspects, love them 🫶

💀 CAVEATS 💀

If annex.largefiles is configured, and does not match a file, git annex addurl will add the non-large file directly to the git repository, instead of to the annex. However, this is not done when --fast or --relaxed is used. read that twice, bestie, it matters 😭

👑 SEE ALSO 👑

🧚 AUTHOR 🧚

Joey Hess <id@joeyh.name>

and that's the whole page, bestie — brought to you by git-annex-addurl(1) from the General Commands Manual, the one and only. mwah 💜🎀✨

📦 where this page comes from