DATALAD-CLONE(1)

📦
version 1.1.5 · 8 versions
, in debian trixie, ubuntu resolute, ubuntu stonking

💖 NAME 💖

datalad clone — bestie, this is how you obtain a dataset (a copy!) from a URL or a local directory ✨

✨ SYNOPSIS ✨

datalad clone [-h] [-d DATASET] [-D DESCRIPTION] [--reckless
       [auto|ephemeral|shared-...]] [--version] SOURCE [PATH] ...

🦋 DESCRIPTION 🦋

okay so the whole point of this command: you get a brand-new clone (copy) of a dataset and it lands in a directory that either doesn't exist yet or is empty. That's it, that's the vibe 🫶

Real talk though — CLONE is a strict subset of what install can do. Lowkey by design. You can only grab a single dataset, and it will not immediately recursively install subdatasets for you. BUT (and this is the iconic part) once a (super)dataset is installed via CLONE, you can get any content — including subdatasets — with a follow-up get command. iykyk 💅

So what makes this different from just raw-dogging a plain git clone call? Four things, and I'm gonna list them because they ate:

  1. it automatically initializes a dataset annex (pure Git repositories are equally supported, no cap)
  2. it automatically registers the newly obtained dataset as a subdataset (submodule) — if you specified a parent dataset
  3. it supports extra resource identifiers: DataLad resource identifiers like the ones used on datasets.datalad.org, and RIA store URLs like the ones used for store.datalad.org — optionally in specific versions identified by a branch or a tag (peep the examples 👀)
  4. it automatically (and configurably!) generates alternative access URLs for common cases — like appending '.git' to the URL when hitting the base URL didn't work. so clutch.

In case the clone gets registered as a subdataset, the original URL you passed to CLONE is recorded in .gitmodules of the parent dataset in addition to the resolved URL used internally for git-clone. Why do we care? Because that lets DataLad-specific URLs like ria+ssh://... be preserved for later GET calls — you know, for when the subdataset got locally removed at some point later on. forward-thinking queen behavior 👑

🌈 URL mapping configuration 🌈

This is about substitution specifications. A substitution specification is defined as a configuration setting datalad.clone.url-substition.<seriesID> whose value is a string containing a match expression and a substitution expression, each following Python's regular expression syntax. The two expressions get concatenated into a single string with an arbitrary delimiter character — and the delimiter itself is defined by prefixing the string with it. Prefix and delimiter get stripped from the expressions.

Example, so it's giving clarity: ",^http://(.*)$,https://1" 🎀

You can define this setting multiple times using the same '<seriesID>'. Substitutions in a series get applied incrementally, in the order they were defined. The first substitution in such a series must match — otherwise no further substitutions in that series will be considered. However! Following the first match, all further substitutions in a series are processed, regardless of whether intermediate expressions match or not. 😤

Substitution series themselves have no particular order — each matching series will result in a candidate clone URL. Consequently, that initial match specification in a series should be as precise as possible, to prevent inflation of candidate URLs. be specific, bestie 💀

👑 SEEALSO 👑

🎀 Examples 🎀

Install a dataset from GitHub into the current directory:

% datalad clone https://github.com/datalad-datasets/longnow-podcasts.git

Install a dataset into a specific directory:

% datalad clone https://github.com/datalad-datasets/longnow-podcasts.git    myfavpodcasts

Install a dataset as a subdataset into the current dataset:

% datalad clone -d . https://github.com/datalad-datasets/longnow-podcasts.git

Install the main superdataset from datasets.datalad.org:

% datalad clone ///

Install a dataset identified by a literal alias from store.datalad.org:

% datalad clone ria+http://store.datalad.org#~hcp-openaccess

Install a dataset in a specific version as identified by a branch or tag name from store.datalad.org:

% datalad clone ria+http://store.datalad.org#76b6ca66-36b1-11ea-a2e6-f0d5bf7b5561@myidentifier

Install a dataset with group-write access permissions:

% datalad clone http://example.com/dataset --reckless shared-group

💅 OPTIONS 💅

  • SOURCE — URL, DataLad resource identifier, local path, or an instance of a dataset to be cloned. Constraints: value must be a string.
  • PATH — path to clone into. If you don't give a PATH, a destination path will be derived from the source URL — similar to how git clone does it. 🎀

🧰 GIT CLONE OPTIONS 🧰

Options to pass along to git clone. Literally any argument you specify after SOURCE and the optional PATH gets passed to git-clone. 💫

⚠️ heads up bestie: not all of these will lead to viable results. For example, '--single-branch' will not result in a functional annex repository, because both a regular branch and the git-annex branch are required. Also note that a version in a RIA URL takes precedence over '--branch'.

the actual flags ✨

  • -h, --help, --help-np — show this help message. --help-np forcefully disables the use of a pager for displaying the help message (for when the pager is doing too much 😭).
  • -d DATASET, --dataset DATASET — (parent) dataset to clone into. If you give this, the newly cloned dataset gets registered as a subdataset of the parent. Also, if given, relative paths are interpreted as being relative to the parent dataset, not relative to the working directory. Constraints: value must be a Dataset or a valid identifier of a Dataset (e.g. a path), or value must be NONE.
  • -D DESCRIPTION, --description DESCRIPTION — short description to use for a dataset location. Its primary purpose is helping humans identify a dataset copy (like, "mike's dataset on lab server"). Note that when a dataset is published, this information becomes available on the remote side. Constraints: value must be a string or value must be NONE.
  • --reckless [auto|ephemeral|shared-...] — obtain a dataset or subdataset and set it up in a potentially unsafe way, for performance or access reasons. Use with care — any dataset set up this way is marked as 'untrusted'. 😬 The reckless mode is stored in a dataset's local configuration under 'datalad.clone.reckless', and it will be inherited by any of its subdatasets. Supported modes:
    • ['auto']: hard-link files between local clones. In-place modification in any clone will alter original annex content.
    • ['ephemeral']: symlink annex to origin's annex and discard local availability info via git-annex-dead 'here', and declares this annex private. Shares an annex between origin and clone w/o git-annex being aware of it. In case of a change in origin you need to update the clone before you're able to save new content on your end. Alternative to 'auto' when hardlinks are not an option, or the number of consumed inodes needs to be minimized. Note that this mode can only be used with clones from non-bare repositories or a RIA store! Otherwise two different annex object tree structures (dirhashmixed vs dirhashlower) will be used simultaneously, and annex keys using the respective other structure will be inaccessible. 💀
    • ['shared-<mode>']: set up repository and annex permission to enable multi-user access. This disables the standard write protection of annex'ed files. <mode> can be any value supported by 'git init --shared=', such as 'group', or 'all'.
    • Constraints: value must be one of (True, False, 'auto', 'ephemeral') or value must start with 'shared-'.
  • --version — show the module and its version which provides the command. 👑

🌟 AUTHORS 🌟

datalad is developed by The DataLad Team and Contributors <team@datalad.org>. 🫶

and that's the whole page, bestie!! 🦋 this was datalad clone version 1.1.5, dated 2025-06-15 — go forth and clone responsibly, mwah 💖🌈⭐

📦 where this page comes from