DATALAD-SEARCH(1)

📦 version 0.15.5, in ubuntu jammy

✨ NAME ✨

datalad search — bestie, it searches dataset metadata 💖

🌸 SYNOPSIS 🌸

datalad search [-h] [-d DATASET] [--reindex] [--max-
       nresults MAX_NRESULTS] [--mode {egrep,textblob,autofield}]
       [--full-record] [--show-keys {name,short,full}] [--show-query]
       [--version] [QUERY ...]

💅 DESCRIPTION 💅

Okay so picture this: DataLad can search metadata that's been extracted from a dataset and/or aggregated up into a superdataset (peep the AGGREGATE-METADATA command for that part, iykyk). Which means you can literally discover datasets, or even individual files inside a dataset, even when they're not available locally. Obsessed. 🦋

Lowkey though, here's the tea: underneath it all, DataLad metadata is a graph of linked data structures. But this command does not (yet, bestie) support queries that can exploit all the info stored in the metadata. So right now we've got the following search modes implemented, and each one is a different trade-off between how expressive your query can be and how much compute + storage it takes to actually run it. ✨

  • egrep (the default, no cap)
  • egrepcs [case-sensitive egrep]
  • textblob
  • autofield

Wanna change the default mode? Totally valid, just tune the config variable 'datalad.search.default-mode':

[datalad "search"]
  default-mode = egrepcs

And here's the thing — every search mode has its own default config for what kind of documents it queries. You can switch that up too via config variables:

[datalad "search"]
  index-<mode_name>-documenttype = (all|datasets|files)

🍓 Mode: egrep/egrepcs

These two modes are largely ignorant of the metadata structure (not in a mean way, they're just built different 💀) — they simply match a search pattern against a flat string-representation of the metadata. Which is honestly iconic when your query is simple and the metadata structure is irrelevant, or precisely known. Bonus: it doesn't need a search index at all, so results can be reported with zero initial latency for building an index when the underlying metadata has changed (say, after a dataset update). By default these modes only consider datasets and don't dig into records for individual files, purely for speed reasons. Results come back in the order they were discovered. 🌈

Queries can use Python regular expression syntax (https://docs.python.org/3/library/re.html). In EGREP mode, matching is case-insensitive when the query has no upper case characters — but the second it does, it's case-sensitive. In EGREPCS mode, matching is always case-sensitive. And expressions will match anywhere in a metadata string, not just at the start. Periodt.

When you give multiple queries, all of them have to match for a search hit (that's AND behavior, bestie).

You can also search individual metadata key/value items by prefixing your query with a metadata key name, separated by a colon (':'). And the key name can itself be a regular expression, so it can match multiple keys. A query match happens when any value of an item with a matching key name matches the query (that's OR behavior). Check the examples below for the vibe. 🫶

Examples:

Query for (what happens to be) an author:

% datalad search haxby

Queries are case-INsensitive when the query contains no upper case characters, and can be regular expressions. Use EGREPCS mode when it is desired to perform a case-sensitive lowercase match:

% datalad search --mode egrepcs halchenko.*haxby
⚠️ Heads up bestie — this search mode performs NO analysis of the metadata content. So queries can easily fail to match. For example, the query above implicitly assumes that authors are listed in alphabetical order. If that's the case (which may or may not be true 👀), the following query would yield NO hits:
% datalad search Haxby.*Halchenko

The TEXTBLOB search mode represents an alternative that is more robust in such cases. 💖

For more complex queries, multiple query expressions can be provided that all have to match to be considered a hit (AND behavior). This query discovers all files (non-default behavior, so watch out) that match 'bids.type=T1w' AND 'nifti1.qform_code=scanner':

% datalad -c datalad.search.index-egrep-documenttype=all search
bids.type:T1w nifti1.qform_code:scanner

Key name selectors can be expressions too, which is how you select multiple keys or construct "fuzzy" queries. In those cases a query matches when any item with a matching key matches the query (OR behavior). But — and this is important — multiple queries are always evaluated using an AND conjunction. The following query extends the example above to match any files that have either 'nifti1.qform_code=scanner' or 'nifti1.sform_code=scanner':

% datalad -c datalad.search.index-egrep-documenttype=all search
bids.type:T1w nifti1.(q|s)form_code:scanner

🧚 Mode: textblob

This mode is very similar to EGREP mode, but with a few key differences. A search index gets built from the string-representation of metadata records. By default only datasets are included in this index, so the indexing usually wraps up within a few seconds — even for hundreds of datasets, slay. 💅 This mode uses its own query language (not regular expressions!) that's similar to other search engines. It supports logical conjunctions and fuzzy search terms. More deets on that are available from the Whoosh project (that's the search engine implementation):

And importantly: search hits are scored and reported in order of descending relevance. Which means limiting the number of search results is way more meaningful here than in 'egrep' mode, and it can also cut down your query duration. 🔥

Examples:

Search for (what happens to be) two authors, regardless of the order in which those names appear in the metadata:

% datalad search --mode textblob halchenko haxby

Fuzzy search when you only have an approximate idea what you're looking for or how it's spelled:

% datalad search --mode textblob haxbi~

Very fuzzy search, when you are basically only confident about the first two characters and how it sounds approximately (or more precisely: allow for three edits and require matching of the first two characters):

% datalad search --mode textblob haksbi~3/2

Combine fuzzy search with logical constructs:

% datalad search --mode textblob 'haxbi~ AND (hanke OR halchenko)'

👑 Mode: autofield

This mode is similar to 'textblob' mode, but it builds a vastly more detailed search index that represents individual metadata variables as individual fields. By default this search index includes records for datasets and individual fields — so yeah, it can grow very quickly into a huge structure that can easily take an hour or more to build and require more than a GB of storage 😭. That said, limiting it to documents on datasets (see above) keeps all the enhanced expressiveness of queries while dramatically reducing the resource demands. Bestie math. ✨

Examples:

List names of search index fields (auto-discovered from the set of indexed datasets) which either have a field starting with "age" or "gender":

% datalad search --mode autofield --show-keys name '.age' '.gender'

Fuzzy search for datasets with an author that is specified in a particular metadata field:

% datalad search --mode autofield bids.author:haxbi~ type:dataset

Search for individual files that carry a particular description prefix in their 'nifti1' metadata:

% datalad search --mode autofield nifti1.description:FSL* type:file

🎀 Reporting

Search hits get returned as standard DataLad results. On the command line you can use the '--output-format' (or '-f') option to tweak results for further processing. 🫶

Examples:

Format search hits as a JSON stream (one hit per line):

% datalad -f json search haxby

Custom formatting: which terms matched the query of particular results. Useful for investigating fuzzy search results:

$ datalad -f '{path}: {query_matched}' search --mode autofield
bids.author:haxbi~

…and bestie, there's literally more coming, don't go anywhere 💜

✨ OPTIONS ✨

okay bestie, here's the full option menu — every flag, every argument, no skips 💖

  • QUERY — the query string. supported syntax and features depends on the selected search mode (see documentation). 👑
  • -h, --help, --help-np — show this help message. --help-np forcefully disables the use of a pager for displaying the help message. 🙈
  • -d DATASET, --dataset DATASET — specify the dataset to perform the query operation on. If no dataset is given, an attempt is made to identify the dataset based on the current working directory and/or the PATH given. Constraints: Value must be a Dataset or a valid identifier of a Dataset (e.g. a path). 🫶
  • --reindex — force rebuilding the search index, even if no change in the dataset's state has been detected, for example, when the index documenttype configuration has changed. 🔥
  • --max-nresults MAX_NRESULTS — maxmimum number of search results to report. Setting this to 0 will report all search matches. Depending on the mode this can search substantially slower. If not specified, a mode-specific default setting will be used. Constraints: value must be convertible to type 'int'. 😭
  • --mode {egrep, textblob, autofield} — Mode of search index structure and content. See section SEARCH MODES for details. 💅
  • --full-record, -f — If set, return the full metadata record for each search hit. Depending on the search mode this might require additional queries. By default, only data that is available to the respective search modes is returned. This always includes essential information, such as the path and the type. 🌸
  • --show-keys {name, short, full} — if given, a list of known search keys is shown. Here's the breakdown:
    • If name — only the name is printed, one per line.
    • If short or full — statistics (in how many datasets, and how many unique values) are printed. short truncates the listing of unique values.
    • QUERY, if provided, is regular expressions any of which keys should contain.
    • No other action is performed (except for reindexing), even if other arguments are given.
    • Each key is accompanied by a term definition in parenthesis (TODO). In most cases a definition is given in the form of a URL. If an ontology definition for a term is known, this URL can resolve to a webpage that provides a comprehensive definition of the term. However, for speed reasons term resolution is solely done on information contained in a local dataset's metadata, and definition URLs might be outdated or point to no longer existing resources. 🦋
  • --show-query — if given, the formal query that was generated from the given query string is shown, but not actually executed. This is mostly useful for debugging purposes. 💜
  • --version — show the module and its version which provides the command. ⭐

👑 AUTHORS 👑

datalad is developed by The DataLad Team and Contributors <team@datalad.org>. 🫶

this iconic page is from datalad search version 0.15.5, dated 2022-02-10 — periodt 💅✨

📦 where this page comes from