DATALAD-AGGREGATE-METADATA(1)

🎀 datalad aggregate-metadata 🎀

okay bestie, welcome to the General Commands Manual page for datalad aggregate-metadata, and lemme tell you, this one is a whole vibe. it's the command that goes around hoovering up all the metadata in your datasets so you can find stuff later without downloading the entire universe first. it's giving librarian energy. it's giving ✨organized✨. let's get into it 🫶

---

✨ NAME ✨

datalad aggregate-metadata — aggregate metadata of one or more datasets for later query. no cap, that's literally the whole job 💅

---

🌸 SYNOPSIS 🌸

datalad aggregate-metadata [-h] [-d DATASET] [-r] [-R LEVELS] [--update-
       mode {all|target}] [--incremental] [--force-extraction]
       [--nosave] [--version] [PATH ...]

---

🦋 DESCRIPTION 🦋

so metadata aggregation is a whole procedure, right? it extracts the metadata that's already chillin' in a dataset and turns it into a portable representation that's stored in a single standardized format. and it doesn't stop there — metadata aggregation can also extract metadata in this format from one dataset and store it in another (super)dataset. based on such collections of aggregated metadata, you can discover particular datasets and specific parts of their content without having to obtain the target datasets first. iykyk — see the DataLad search command for that 🔍

to enable aggregation of metadata that are contained in files of a dataset, you have to enable one or more metadata extractor for a dataset. DataLad supports a number of common metadata standards, such as the Exchangeable Image File Format (EXIF), Adobe's Extensible Metadata Platform (XMP), and various audio file metadata systems like ID3. DataLad extension packages can provide metadata data extractors for additional metadata sources. for example, the neuroimaging extension provides extractors for scientific (meta)data standards like BIDS, DICOM, and NIfTI1. some metadata extractors depend on particular 3rd-party software. the list of metadata extractors available to a particular DataLad installation is reported by the wtf command (datalad wtf) 👑

enabling a metadata extractor for a dataset is done by adding its name to the dataset's configuration file (.datalad/config), like this:

[datalad "metadata"]
  nativetype = exif
  nativetype = xmp

if an enabled metadata extractor is not available in a particular DataLad installation, metadata extraction will not succeed, in order to avoid inconsistent aggregation results. so like, don't expect it to just shrug and move on — it will not 💀

enabling multiple extractors is supported. in this case, metadata are extracted by each extractor individually, and stored alongside each other. metadata aggregation will also extract DataLad's own metadata (extractors

metadata aggregation can be performed recursively, in order to aggregate all metadata across all subdatasets — for example, to be able to search across any content in any dataset of a collection. aggregation can also be performed for subdatasets that are not available locally. in this case, pre-aggregated metadata from the closest available superdataset will be considered instead. iconic behavior honestly 🧚

depending on the versatility of the present metadata and the number of dataset or files, aggregated metadata can grow prohibitively large. a number of configuration switches are provided to mitigate such issues. so here's the config squad, one by one:

• datalad.metadata.aggregate-content-<extractor-name> — if set to false, content metadata aggregation will not be performed for the named metadata extractor (a potential underscore _ in the extractor name must be replaced by a dash -). this can substantially reduce the runtime for metadata extraction, and also reduce the size of the generated metadata aggregate. note, however, that some extractors may not produce any metadata when this is disabled, because their metadata might come from individual file headers only. datalad.metadata.store-aggregate-content might be a more appropriate setting in such cases.
• datalad.metadata.aggregate-ignore-fields — any metadata key matching any regular expression in this configuration setting is removed prior to generating the dataset-level metadata summary (keys and their unique values across all dataset content), and from the dataset metadata itself. this switch can also be used to filter out sensitive information prior aggregation 🙈
• datalad.metadata.generate-unique-<extractor-name> — if set to false, DataLad will not auto-generate a summary of unique content metadata values for a particular extractor as part of the dataset-global metadata (a potential underscore _ in the extractor name must be replaced by a dash -). this can be useful if such a summary is bloated due to minor uninformative (e.g. numerical) differences, or when a particular extractor already provides a carefully designed content metadata summary.
• datalad.metadata.maxfieldsize — any metadata value that exceeds the size threshold given by this configuration setting (in bytes/characters) is removed.
• datalad.metadata.store-aggregate-content — if set, extracted content metadata are still used to generate a dataset-level summary of present metadata (all keys and their unique values across all files in a dataset are determined and stored as part of the dataset-level metadata aggregate, see datalad.metadata.generate-unique-<extractor-name>), but metadata on individual files are not stored. this switch can be used to avoid prohibitively large metadata files. discovery of datasets containing content matching particular metadata properties will still be possible, but such datasets would have to be obtained first in order to discover which particular files in them match these properties. so like, you can find the dataset, but not the exact file, until you download it. bestie that's the trade-off 💅

---

💖 OPTIONS 💖

• PATH — path to datasets that shall be aggregated. when a given path is pointing into a dataset, the metadata of the containing dataset will be aggregated. if no paths given, current dataset metadata is aggregated. Constraints: value must be a string
• -h, --help, --help-np — show this help message. --help-np forcefully disables the use of a pager for displaying the help message
• -d DATASET, --dataset DATASET — topmost dataset metadata will be aggregated into. all dataset between this dataset and any given path will receive updated aggregated metadata from all given paths. Constraints: Value must be a Dataset or a valid identifier of a Dataset (e.g. a path)
• -r, --recursive — if set, recurse into potential subdataset.
• -R LEVELS, --recursion-limit LEVELS — limit recursion into subdataset to the given number of levels. Constraints: value must be convertible to type 'int'
• --update-mode {all|target} — which datasets to update with newly aggregated metadata: all datasets from any leaf dataset to the top-level target dataset including all intermediate datasets (all), or just the top-level target dataset (target). Constraints: value must be one of ('all', 'target') [Default: 'target']
• --incremental — If set, all information on metadata records of subdatasets that have not been (re-)aggregated in this run will be kept unchanged. this is useful when (re-)aggregation only a subset of a dataset hierarchy, for example, because not all subdatasets are locally available.
• --force-extraction — If set, all enabled extractors will be engaged regardless of whether change detection indicates that metadata has already been extracted for a given dataset state.
• --nosave — by default all modifications to a dataset are immediately saved. giving this option will disable this behavior.
• --version — show the module and its version which provides the command

heads up bestie: --nosave means your changes are not written out. if you forget it exists and wonder why nothing stuck, that's on you, not on DataLad 😭

---

🌈 AUTHORS 🌈

datalad is developed by The DataLad Team and Contributors <team@datalad.org>.

---

and that's the whole page, fr fr 🫶 this was datalad aggregate-metadata 0.15.5, dated 2022-02-10, straight out of the General Commands Manual. go aggregate something iconic, bestie 💜✨