git-annex-...red-content(1)

📦
version 10.20261026 · 7 versions
, in alpine 3.24, arch rolling

Hey bestie 💖 welcome to the git-annex-...red-content(1) page from the General Commands Manual. This is the git-annex-preferred-content manual, and it's literally about which files are wanted in a repository. It's giving main character energy for your repos, no cap. ✨

🎀 NAME 🎀

git-annex-preferred-content — which files are wanted in a repository

🌸 DESCRIPTION 🌸

Okay so every repository has a preferred content setting. It specifies content that the repository wants to have present. You can configure these settings using git annex vicfg or git annex wanted. They're used by the --wanted and --auto options, by git annex sync --content, by clusters, and by the git-annex assistant. The assistant is doing the most, honestly.

Preferred content expresses a preference, but it can be overridden by simply using git annex drop. On the other hand, required content settings are enforced: git annex drop will refuse to drop a file if doing so would violate its required content settings. A repository's required content can be configured using git annex vicfg or git annex required.

🦋 SYNTAX 🦋

Preferred content expressions use a similar syntax to git-annex-matching-options(1), without the dashes. For example:

exclude=archive/* and (include=*.mp3 or smallerthan=1mb)

The vibe is: you write an expression that files are matched against. If a file matches, the repository wants to store its content. If it doesn't match, the repository wants to drop its content (if there are enough copies elsewhere to allow removing it).

An empty preferred content expression is treated the same as preferred content not being configured. Periodt.

👑 EXPRESSIONS 👑

These are the building blocks, bestie. Each one is iconic in its own way. 💅

🍓 include=glob / exclude=glob

Matches files to include, or exclude. Simple as that. 🫶

While the command-line options --include=glob and --exclude=glob match files relative to the current directory, preferred content expressions match files relative to the top of the git repository. Important little difference, iykyk.

A glob is something like foo.* or b?r. Globs can also contain character classes, like foo[Bb]ar, as well as additional POSIX character classes like [[:space:]]. Which is useful, since a glob in a preferred content expression cannot contain spaces. See the glob(7) man page for more about globs. Glob lore unlocked.

For example, suppose you put files into archive directories when you're done with them. Then you could configure your laptop to prefer to not retain those files, like this: exclude=*/archive/*

When a subdirectory is being exported or imported to a special remote (see git-annex-export(1)) and git-annex-import(1), these match relative to the top of the subdirectory.

Note that, when a command is run with the --all option, or in a bare repository, there is no filename associated with an annexed object, and so include= and exclude= will not match. So don't be surprised when they ghost you there.

🎀 copies=number

Matches only files that git-annex believes to have the specified number of copies, or more. Note that it does not check remotes to verify that the copies still exist. Trust but verify? Not here, bestie.

To decide if content should be dropped, git-annex evaluates the preferred content expression under the assumption that the content has already been dropped. If the content would not be wanted then, the drop can be done. So, for example, copies=2 in a preferred content expression lets content be dropped only when there are currently 3 copies of it, including the repo it's being dropped from. This is different than running git annex drop --copies=2, which will drop files that currently have 2 copies. The math is mathing.

💜 copies=trustlevel:number

Matches only files that git-annex believes have the specified number of copies, or more, on repositories with the specified trust level. For example, copies=trusted:2

To match any trust level at or higher than a given level, use trustlevel+. For example, copies=semitrusted+:2

🌸 copies=groupname:number

Matches only files that git-annex believes have the specified number of copies, or more, on repositories in the specified group. For example, copies=archive:2

Preferred content expressions have no equivalent to the --in option, but groups can accomplish similar things. You can add repositories to groups, and match against the groups in a preferred content expression. So rather than --in=usbdrive, put all the USB drives into a "transfer" group, and use copies=transfer:1. Group chat, but for repos.

🦋 copies=grouplimit=number

Matches only files that git-annex believes have the specified number of copies, or more, on repositories in groups allowed by the group limit.

In a group limit, use + before a group to include that group, and - before a group to exclude it. (The first group to be included does not need to be prefixed with a + though it may be.)

A group limit matches repositories that are in any of the included groups and are in none of the excluded groups.

For example, copies=backup+archive-offsite=3 matches files that are stored in 3 backup or archive repositories, excluding offsite repositories. And copies=-offsite=3 matches files that are stored in 3 repositories excluding offsite repositories. Math with group chat rules, honestly iconic.

🌈 lackingcopies=number

Matches only files that git-annex believes need the specified number or more additional copies to be made in order to satisfy their numcopies settings.

🍓 lackingcopies=groupname:number

Providing the name of a group limits lackingcopies to only considering repositories in that group toward the numcopies count.

For example, lackingcopies=backup:1 matches files that are not stored in enough backup repositories to satisfy numcopies, even if they are stored in enough other repositories.

🧚 lackingcopies=grouplimit=number

Providing a group limit makes lackingcopies only consider repositories in groups that match the group limit toward the numcopies count.

In a group limit, use + before a group to include that group, and - before a group to exclude it. (The first group to be included does not need to be prefixed with a + though it may be.)

A group limit matches repositories that are in any of the included groups and are in none of the excluded groups.

For example, lackingcopies=backup+archive-offsite=1 matches files that are not stored in enough backup or archive repositories, excluding offsite repositories. And lackingcopies=-offsite=1 matches files that are not stored in enough repositories, excluding offsite repositories.

⭐ approxlackingcopies=value

Like lackingcopies, but it does not look at .gitattributes annex.numcopies settings. This makes it significantly faster. Speedrun strats, no cap.

🎀 inbackend=backendname

Matches only files whose content is stored using the specified key-value backend.

See git-annex-backends(1) for information about available backends.

🔥 securehash

Matches only files whose content is hashed using a cryptographically secure function. Security slay.

💖 inallgroup=groupname

Matches only files that git-annex believes are present in all repositories in the specified group.

🦋 onlyingroup=groupname

Matches files that git-annex believes are present in at least one repository that is in the specified group, and are not present in any repositories that are not in the specified group.

🌸 smallerthan=size / largerthan=size

Matches only files whose content is smaller than, or larger than, the specified size.

The size can be specified with any commonly used units, for example, "0.5 gb" or "100 KiloBytes"

💜 metadata=field=glob

Matches only files that have a metadata field attached with a value that matches the glob. The values of metadata fields are matched case insensitively.

A glob is something like foo.* or b?r. Globs can also contain character classes, like foo[Bb]ar, as well as additional POSIX character classes like [[:space:]]. Which is useful, since a glob in a preferred content expression cannot contain spaces. See the glob(7) man page for more about globs.

To match a tag done, use metadata=tag=done

To match author metadata, use metadata=author=*Smith

🎀 metadata=field<number / metadata=field>number & metadata=field<=number / metadata=field>=number

Matches only files that have a metadata field attached with a value that is a number and is less than or greater than the specified number.

To match PDFs with between 100 and 200 pages (assuming something has set that metadata), use metadata=pagecount>=100 and metadata=pagecount<=200

🌈 url=glob

Matches when the content is recorded as being present in an url that matches the glob.

🫶 present

Makes content be wanted if it's present, but not otherwise.

This leaves it up to you to use git-annex manually to move content around. You can use this to avoid preferred content settings from affecting a subdirectory. For example: auto/* or (include=ad-hoc/* and present)

Note that not present is not a reasonable thing to put in a preferred content expression. It says to get content that's not present, but then drop it! If that somehow gets into a preferred content expression, git-annex will recognize that the preferred content expression is not stable, and make it never match. So maybe don't do that, bestie.

🧚 inpreferreddir

Makes content be preferred if it's in a directory (located anywhere in the tree) with a particular name.

The name of the directory can be configured using git annex enableremote $remote preferreddir=$dirname

(If no directory name is configured, it uses public by default.)

Note that, when a command is run with the --all option, or in a bare repository, there is no filename associated with an annexed object, and so inpreferreddir will not match.

👑 standard

git-annex comes with some built-in preferred content expressions, that can be used with repositories that are in some standard groups such as client and transfer.

When a repository is in exactly one such group, you can use the standard keyword in its preferred content expression, to match whatever content the group's expression matches.

Most often, the whole preferred content expression is simply standard. But, you can do more complicated things, for example: standard or include=otherdir/*

💖 groupwanted

Okay so the groupwanted keyword is basically a shortcut to a preferred content expression that’s tied to a group — but only if there’s exactly one such expression among all the groups your repo is in. It’s like the standard keyword, except you get to configure the preferred content expressions yourself with git annex groupwanted. Iconic.

When you’re writing a groupwanted preferred content expression, you can use all the keywords documented here, including standard. (But not groupwanted itself — no recursive vibe checks, bestie.)

For example, say you want a variant of the standard client preferred content expression that doesn’t want files in the out directory. You’d run:

git annex groupwanted client "standard and exclude=out/*"

Then repositories that are in the client group and have their preferred content expression set to groupwanted will use that. Meanwhile, other client repositories that have their preferred content expression set to standard will use the standard expression. Slay, everyone gets their own lane.

Or you could make a whole new group with your own custom preferred content expression tuned exactly to your needs. Every repository you put in that group and set its preferred content to groupwanted will use it. Main character group energy.

For example, the archive group only wants to archive 1 copy of each file, spread among every repository in the group. Here’s how to configure a group named redundantarchive that instead wants to contain 3 copies of each file:

git annex groupwanted redundantarchive "not (copies=redundantarchive:3)"
for repo in foo bar baz; do
     git annex group $repo redundantarchive
     git annex wanted $repo groupwanted
done

🌀 unused

unused matches only keys that git annex unused has determined to be unused. It’s related to the --unused option, but here’s the catch: putting unused in a preferred content expression doesn’t make git-annex consider those unused keys. So when git-annex is only checking preferred content expressions against files in the repository — which are obviously used — unused in a preferred content expression won’t match anything. It’s giving “I’m in the group chat but I’m not actually reading the messages.”

So when is unused useful in a preferred content expression?

Using git annex sync --content --all will operate on all files, including unused ones, and it takes unused in preferred content expressions into account. That’s the moment.

The git-annex assistant periodically scans for unused files and moves them to some repository whose preferred content expression says it wants them. (Or, if annex.expireunused is set, it may just delete them. 😭)

🧚 balanced=groupname[:number]

This makes content be evenly balanced among repositories in the group. The number is the number of repositories in the group that will want each file. When it’s not specified, the default is 1.

For example, balanced=backup:2, when there are 3 members of the backup group, will make each backup repository want 2/3rds of the files. Math bestie behavior.

For balancing to work, each repository in the group should have its preferred content set to the same expression. Using groupwanted is a good way to do that.

The sizes of files are not taken into account, so it’s possible for one repository to get larger-than-usual files and fill up before the other repositories. But files are only wanted by repositories that have enough free space to hold them. So once a repository is full, the remaining repositories will have any additional files balanced among them. For git-annex to know when a repository is full, you must use git-annex-maxsize(1) to specify the size of each repository in the group.

This usually avoids moving files between repositories, even if that means things are not optimally balanced. Some of the ways it can get out of balance include adding a new repository to the group, or a file getting copied into more repositories in the group than the specified number, or some of the repositories filling up.

Running git-annex commands with the --rebalance option will make this expression instead behave like the fullybalanced expression, which will make repositories want to move files around as necessary in order to get fully balanced.

Using this in a preferred content expression makes git-annex need to do some additional work to keep track of how full repositories are. Usually that won’t affect performance much. However, the first time git-annex processes this expression in a given git repository, it will need to calculate the sizes of all repositories, which can be slow when there are a lot of files. When this causes git-annex to do a lot of work, it will display "(calculating repository sizes)".

⚠️ Note that not balanced is not a reasonable thing to use in a preferred content expression for the same reasons as not present.

🍓 balanced=groupname:lackingcopies

This makes each file be balanced among as many repositories in the group as are needed to satisfy the numcopies configuration.

For example, balanced=backup:lackingcopies, when numcopies is 3 and there is 1 other copy (in a repository not in the backup group) will behave the same as balanced=backup:2.

🌈 balanced=groupname:lackingcopies=grouplimit

Specifying a group limit makes only repositories in groups that match the group limit, as well as repositories in the balanced group, be considered for the numcopies count.

In a group limit, use + before a group to include that group, and - before a group to exclude it. (The first group to be included does not need to be prefixed with a + though it may be.)

A group limit matches repositories that are in any of the included groups and are in none of the excluded groups.

For example, balanced=backup:lackingcopies=archive-offsite makes each file be balanced among as many repositories in the backup group as are needed to satisfy its numcopies configuration, only counting other copies in repositories in the archive group and not in the offsite group. Selective queen behavior.

🔥 fullybalanced=groupname[:number|:lackingcopies[=grouplimit]]

This is like balanced, but it allows moving content between repositories in the group at any time to keep it fully balanced. No “we’ll just leave it like that” energy.

Normally balanced=groupname:number is the same as (fullybalanced=groupname:number and not copies=groupname:number) or present.

When the --rebalance option is used, balanced is the same as fullybalanced.

When the specified number is greater than 1, and too many repositories in the group are more than 90% full (as configured by annex.fullybalancedthreshhold), this behaves like fullysizebalanced.

For example, fullybalanced=foo:3, when group foo has 5 repositories, two 50% full and three 99% full, will make some content move from the full repositories to the others. Moving content like that is expensive, but it allows new files to continue to be stored on the specified number of repositories. Expensive but functional, the ultimate combo.

🦋 sizebalanced=groupname[:number|:lackingcopies[=grouplimit]]

This distributes content among repositories in the group, keeping repositories proportionally full.

The number is the number of repositories in the group that will want each file. When not specified, the default is 1.

For this to work, you must use git-annex-maxsize(1) to specify the size of each repository in the group. When a repository’s maxsize has not been specified, it will not want any files.

For example, if one repository in the group has a maximum size of 100 gb with 60 gb used, and another has a maximum size of 50 gb with 25 gb used, the smaller one will want files (that fit in it), and the larger one won’t want any files (that would fit in the smaller one) until the smaller one gets equally full.

Note that, once a repository contains a file, it will continue to want it, even if it’s more full than other repositories. This is to avoid churn in moving files around.

This is more likely to get out of balance than the balanced= expression is, because git-annex does not always have a consistent knowledge of how full repositories are. Consider for example if a laptop and a desktop are each sending a new file to the group. They will both pick whichever repository was least full, but that means both files go to the same repository, when a better solution might have been to send the smaller file to a different repository. When using balanced= in the same situation, it’s less likely that a repository will want both files.

Running git-annex commands with the --rebalance option will make this expression instead behave like the fullysizebalanced expression, which will make repositories want to move files around as necessary in order to get fully balanced.

💀 Note that not sizebalanced is not a reasonable thing to use in a preferred content expression for the same reasons as not present.

🌸 fullysizebalanced=groupname[:number|:lackingcopies[=grouplimit]]

This is like sizebalanced, but it allows moving content between repositories in the group at any time to keep it fully balanced. Fully balanced, fully iconic.

Normally sizebalanced=groupname:number is the same as (fullysizebalanced=groupname:number and not copies=groupname:number) or present.

When the --rebalance option is used, sizebalanced is the same as fullysizebalanced.

✨ anything

Always matches. Periodt.

💀 nothing

Never matches. (Same as "not anything".)

🎀 not expression

Inverts what the expression matches. For example, not include=archive/* is the same as exclude=archive/*.

🌀 and / or / ( expression )

These can be used to build up more complicated expressions. Group-chat nesting, but for preferred content. 🫶

🧪 TESTING 🧪

ok so you've written your little preferred content expression and now you're like "did I cook or did I burn the house down" 💀 — bestie, there's a way to check, no cap.

To check at the command line which files are matched by a repository's preferred content settings, you can use the --want-get and --want-drop options. That's literally what they're for 🫶

For example, this one:

git annex find --want-get --not --in .

will find all the files that git annex get --wanted will want to get. and then this one:

git annex find --want-drop --in .

will find all the files that git annex drop --wanted will want to drop. see how that works?? matching files, revealed ✨

and when your expression is doing the absolute most and you're sitting there like "why is this matching, I have no idea" 😭 — the --explain option can be used to understand why a complex preferred content expression matches or fails to match. iconic behaviour honestly 👑

The expression will be displayed, with each term followed by [TRUE] or [FALSE] to indicate the value. Irrelevant terms will be omitted from the explanation, so like, for example exclude=* and copies=1 will be displayed as exclude=*[FALSE] — the parts that don't matter get ghosted 🙈

fr fr this is the vibe check for your expression. use it before you spend an afternoon confused.

👀 SEE ALSO 👀

like, go read these next, they're the supporting cast and they ate 💅

💌 AUTHOR 💌

Joey Hess <id@joeyh.name>

<http://git-annex.branchable.com/>

and that's a wrap on this page, bestie 🎀 the footer down there is just the manpage name git-annex-...red-content(1) chilling with its source/version and date — the little signature at the bottom of the page, not a whole extra section. go forth and match some files 💖✨

📦 where this page comes from