SPACK(1)
๐ SPACK(1) โ the spack manual, bestie edition
okay so this is spack(1), like from the Spack section, and honestly? it's a lot. but we're gonna get through it together ๐
โจ NAME โจ
spack โ Spack Documentation.
so like. these are the docs for the Spack package manager. if you were looking for sphere packing?? wrong page bestie, that's pyspack. different vibe entirely ๐ซถ
Spack is a package management tool, and it's designed to support multiple versions and configurations of software across a huge variety of platforms and environments. it was literally built for big supercomputing centers, where tons of users and application teams share common software installations on clusters with, and I quote, "exotic architectures," using libraries that do not have a standard ABI. and the best part? Spack is non-destructive: installing a new version does not break your existing installations, so a million different configs can coexist on the same system. no cap. โจ
and most importantly โ Spack is simple. it's giving effortless. it offers a simple spec syntax so you can specify versions and configuration options concisely. it's also simple for package authors: package files are written in pure Python, and specs let package authors keep a single file for many different builds of the same package. iconic behavior, honestly ๐
see the Feature Overview for examples and highlights (it's right below, keep scrolling).
get spack from the github repository and install your first package:
$ git clone -c feature.manyFiles=true https://github.com/spack/spack.git
$ cd spack/bin
$ ./spack install libelfif you're new to spack and wanna start using it, see Getting Started, or just refer to the full manual below. we gotchu ๐ธ
๐ฆ FEATURE OVERVIEW ๐ฆ
this is the high-level overview of the features that make Spack different from other package managers and port systems. the main character energy section, basically.
๐ Simple package installation
installing the default version of a package is simple โ like, simple simple. this installs the latest version of the mpileaks package plus all of its dependencies:
$ spack install mpileaksthat's it. that's the whole thing. slay.
๐ Custom versions & configurations
Spack lets you customize your installation. you can specify the version, build compiler, compile-time options, and cross-compile platform โ all on the command line. obsessed.
# Install a particular version by appending @
$ spack install mpileaks@1.1.2
# Specify a compiler (and its version), with %
$ spack install mpileaks@1.1.2 %gcc@4.7.3
# Add special compile-time options by name
$ spack install mpileaks@1.1.2 %gcc@4.7.3 debug=True
# Add special boolean compile-time options with +
$ spack install mpileaks@1.1.2 %gcc@4.7.3 +debug
# Add compiler flags using the conventional names
$ spack install mpileaks@1.1.2 %gcc@4.7.3 cppflags="-O3 -floop-block"
# Cross-compile for a different micro-architecture with target=
$ spack install mpileaks@1.1.2 target=icelakeyou can specify as many or as few options as you care about โ Spack fills in the unspecified values with sensible defaults. and fun fact: those two listed syntaxes for variants are identical when the value is boolean. iykyk ๐
๐ Customize dependencies
Spack lets you customize the dependencies of a particular installation extensively. say hdf5 depends on openmpi and indirectly on hwloc. using ^, you can add custom configurations for the dependencies:
# Install hdf5 and link it with specific versions of openmpi and hwloc
$ spack install hdf5@1.10.1 %gcc@4.7.3 +debug ^openmpi+cuda fabrics=auto ^hwloc+glthe way that ^ just does that?? chef's kiss ๐ง
๐ฅ Non-destructive installs
Spack installs every unique package/dependency configuration into its own prefix, so new installs will not break existing ones. periodt. nothing gets ruined, nothing gets overwritten. ๐
โญ Packages can peacefully coexist
Spack avoids library misconfiguration by using RPATH to link dependencies. when a user links a library or runs a program, it is tied to the dependencies it was built with, so there's no need to manipulate LD_LIBRARY_PATH at runtime. literally never have to touch it. healing. ๐ฆ
๐ Creating packages is easy
to create a new package, all Spack needs is a URL for the source archive. the spack create command creates a boilerplate package file, and package authors can fill in specific build steps in pure Python.
for example, this command:
$ spack create https://ftp.osuosl.org/pub/blfs/conglomeration/libelf/libelf-0.8.13.tar.gzcreates a simple python file:
from spack.package import *
class Libelf(AutotoolsPackage):
"""FIXME: Put a proper description of your package here."""
# FIXME: Add a proper url for your package's homepage here.
homepage = "https://www.example.com"
url = "https://ftp.osuosl.org/pub/blfs/conglomeration/libelf/libelf-0.8.13.tar.gz"
# FIXME: Add a list of GitHub accounts to
# notify when the package is updated.
# maintainers("github_user1", "github_user2")
version("0.8.13", sha256="591a9b4ec81c1f2042a97aa60564e0cb79d041c52faa7416acb38bc95bd2c76d")
# FIXME: Add dependencies if required.
# depends_on("foo")
def configure_args(self):
# FIXME: Add arguments other than --prefix
# FIXME: If not needed delete this function
args = []
return argsand it doesn't take much python coding to get from there to a working package:
from spack.package import *
class Libelf(AutotoolsPackage):
"""libelf lets you read, modify or create ELF object files in an
architecture-independent way. The library takes care of size
and endian issues, e.g. you can process a file for SPARC
processors on an Intel-based system. Note: libelf is no longer
maintained and packages that depend on libelf should migrate to
elfutils."""
# The original homepage no longer exists, but the tar file is
# archived at fossies.org.
# homepage = "http://www.mr511.de/software/english.html"
homepage = "https://directory.fsf.org/wiki/Libelf"
urls = [
"https://fossies.org/linux/misc/old/libelf-0.8.13.tar.gz",
"https://ftp.osuosl.org/pub/blfs/conglomeration/libelf/libelf-0.8.13.tar.gz",
]
license("LGPL-2.0-only")
version("0.8.13", sha256="591a9b4ec81c1f2042a97aa60564e0cb79d041c52faa7416acb38bc95bd2c76d")
provides("elf@0")
# configure: error: neither int nor long is 32-bit
depends_on("automake", when="platform=darwin", type="build")
depends_on("autoconf", when="platform=darwin", type="build")
depends_on("libtool", when="platform=darwin", type="build")
depends_on("m4", when="platform=darwin", type="build")
@property
def force_autoreconf(self):
return self.spec.satisfies("platform=darwin")
def configure_args(self):
args = ["--enable-shared", "--disable-debug"]
# config.sub: invalid option -apple-darwin21.6.0
if self.spec.satisfies("platform=darwin target=aarch64:"):
args.append("--build=aarch64-apple-darwin")
return args
def install(self, spec, prefix):
make("install", parallel=False)
def flag_handler(self, name, flags):
if name == "cflags":
if self.spec.satisfies("%clang@16:"):
flags.append("-Wno-error=implicit-int")
flags.append("-Wno-error=implicit-function-declaration")
return (flags, None, None)see how that escalated? that's the glow-up arc ๐
Spack also provides wrapper functions around common commands like configure, make, and cmake to make writing packages simple. ur welcome.
๐ธ GETTING STARTED ๐ธ
๐ง System Prerequisites
Spack has the following minimum system requirements, which are assumed to be present on the machine where Spack is run:
๐ธ System prerequisites for Spack ๐ธ
Okay bestie, hereโs the vibe check on what your machine lowkey needs before Spack can slay. The original table had Name, Supported Versions, Notes, and Requirement Reason columns โ so hereโs every single row, same details, just with more sparkle:
Pythonโ Supported versions:3.6--3.12. Notes: โ. Requirement reason: Interpreter for Spack.C/C++ Compilersโ Supported versions: โ. Notes: โ. Requirement reason: Building software.patchโ Supported versions: โ. Notes: โ. Requirement reason: Build software.tarโ Supported versions: โ. Notes: โ. Requirement reason: Extract/create archives.gzipโ Supported versions: โ. Notes: โ. Requirement reason: Compress/Decompress archives.unzipโ Supported versions: โ. Notes: โ. Requirement reason: Compress/Decompress archives.bzip2โ Supported versions: โ. Notes: โ. Requirement reason: Compress/Decompress archives.xzโ Supported versions: โ. Notes: โ. Requirement reason: Compress/Decompress archives.zstdโ Supported versions: โ. Notes:Optional. Requirement reason: Compress/Decompress archives.fileโ Supported versions: โ. Notes: โ. Requirement reason: Create/Use Buildcaches.lsb-releaseโ Supported versions: โ. Notes: โ. Requirement reason: Linux: identify operating system version.gnupg2โ Supported versions: โ. Notes: โ. Requirement reason: Sign/Verify Buildcaches.gitโ Supported versions: โ. Notes: โ. Requirement reason: Manage Software Repositories.svnโ Supported versions: โ. Notes:Optional. Requirement reason: Manage Software Repositories.hgโ Supported versions: โ. Notes:Optional. Requirement reason: Manage Software Repositories.Python header filesโ Supported versions: โ. Notes:Optional(e.g.python3-devon Debian). Requirement reason: Bootstrapping from sources.
These requirements can be easily installed on most modern Linux systems; on macOS, the Command Line Tools package is required, and a full XCode suite may be necessary for some packages such as Qt and apple-gl. Spack is designed to run on HPC platforms like Cray. Not all packages should be expected to work on all platforms.
A build matrix showing which packages are working on which systems is shown below.
โจ Installation โจ
Getting Spack is easy, bestie. You can clone it from the github repository using this command:
$ git clone -c feature.manyFiles=true https://github.com/spack/spack.gitThis will create a directory called spack. Slay.
๐ Shell support ๐
Once you have cloned Spack, we recommend sourcing the appropriate script for your shell:
# For bash/zsh/sh
$ . spack/share/spack/setup-env.sh
# For tcsh/csh
$ source spack/share/spack/setup-env.csh
# For fish
$ . spack/share/spack/setup-env.fishThatโs it! Youโre ready to use Spack. Periodt. ๐
Sourcing these files will put the spack command in your PATH, set up your MODULEPATH to use Spackโs packages, and add other useful shell integration for certain commands, environments, and modules. For bash and zsh, it also sets up tab completion.
In order to know which directory to add to your MODULEPATH, these scripts query the spack command. On shared filesystems, this can be a bit slow, especially if you log in frequently. If you donโt use modules, or want to set MODULEPATH manually instead, you can set the SPACK_SKIP_MODULES environment variable to skip this step and speed up sourcing the file.
If you do not want to use Spackโs shell support, you can always just run the spack command directly from spack/bin/spack.
When the spack command is executed it searches for an appropriate Python interpreter to use, which can be explicitly overridden by setting the SPACK_PYTHON environment variable. When sourcing the appropriate shell setup script, SPACK_PYTHON will be set to the interpreter found at sourcing time, ensuring future invocations of the spack command will continue to use the same consistent python version regardless of changes in the environment.
๐ฅ Bootstrapping clingo ๐ฅ
Spack uses clingo under the hood to resolve optimal versions and variants of dependencies when installing a package. Since clingo itself is a binary, Spack has to install it on initial use, which is called bootstrapping.
Spack provides two ways of bootstrapping clingo: from pre-built binaries (default), or from sources. The fastest way to get started is to bootstrap from pre-built binaries.
The first time you concretize a spec, Spack will bootstrap automatically:
$ spack spec zlib
==> Bootstrapping clingo from pre-built binaries
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-ba5ijauisd3uuixtmactc36vps7yfsrl.spec.json
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64/gcc-10.2.1/clingo-bootstrap-spack/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-ba5ijauisd3uuixtmactc36vps7yfsrl.spack
==> Installing "clingo-bootstrap@spack%gcc@10.2.1~docs~ipo+python+static_libstdcpp build_type=Release arch=linux-centos7-x86_64" from a buildcache
==> Bootstrapping patchelf from pre-built binaries
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.16.1-p72zyan5wrzuabtmzq7isa5mzyh6ahdp.spec.json
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64/gcc-10.2.1/patchelf-0.16.1/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.16.1-p72zyan5wrzuabtmzq7isa5mzyh6ahdp.spack
==> Installing "patchelf@0.16.1%gcc@10.2.1 ldflags="-static-libstdc++ -static-libgcc" build_system=autotools arch=linux-centos7-x86_64" from a buildcache
Input spec
--------------------------------
zlib
Concretized
--------------------------------
zlib@1.2.13%gcc@9.4.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu20.04-icelakeIf for security concerns you cannot bootstrap clingo from pre-built binaries, you have to disable fetching the binaries we generated with Github Actions.
so like, the whole point is you turn off the old ones and keep the shiny new one ๐
$ spack bootstrap disable github-actions-v0.4
==> "github-actions-v0.4" is now disabled and will not be used for bootstrapping
$ spack bootstrap disable github-actions-v0.3
==> "github-actions-v0.3" is now disabled and will not be used for bootstrappingand then you can verify the new settings actually took, bestie, with:
$ spack bootstrap list
Name: github-actions-v0.5 ENABLED
Type: buildcache
Info:
url: https://mirror.spack.io/bootstrap/github-actions/v0.5
homepage: https://github.com/spack/spack-bootstrap-mirrors
releases: https://github.com/spack/spack-bootstrap-mirrors/releases
Description:
Buildcache generated from a public workflow using Github Actions.
The sha256 checksum of binaries is checked before installation.
Name: github-actions-v0.4 ENABLED
Type: buildcache
Info:
url: https://mirror.spack.io/bootstrap/github-actions/v0.4
homepage: https://github.com/spack/spack-bootstrap-mirrors
releases: https://github.com/spack/spack-bootstrap-mirrors/releases
Description:
Buildcache generated from a public workflow using Github Actions.
The sha256 checksum of binaries is checked before installation.
Name: spack-install ENABLED
Type: install
Info:
url: https://mirror.spack.io
Description:
Specs built from sources downloaded from the Spack public mirror.โ ๏ธ NOTE: when you're bootstrapping from sources, Spack needs a full install of Python including header files (think python3-dev on Debian), plus a compiler with support for C++14 (GCC on Linux, Apple Clang on macOS) and static C++ standard libraries on Linux. like, no shortcuts here, she's needy ๐ญ
And Spack will build all the required software the very first time you ask it to concretize a spec:
$ spack spec zlib
[+] /usr (external bison-3.0.4-wu5pgjchxzemk5ya2l3ddqug2d7jv6eb)
[+] /usr (external cmake-3.19.4-a4kmcfzxxy45mzku4ipmj5kdiiz5a57b)
[+] /usr (external python-3.6.9-x4fou4iqqlh5ydwddx3pvfcwznfrqztv)
==> Installing re2c-1.2.1-e3x6nxtk3ahgd63ykgy44mpuva6jhtdt
[ ... ]
zlib@1.2.11%gcc@10.1.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell๐๏ธ The Bootstrap Store โจ
okay so every tool Spack needs for its own functioning gets installed in a separate store, and that store lives under the ${HOME}/.spack directory. lowkey iconic that it keeps its own stuff separate ๐ซถ you can peek at what's in there with:
$ spack -b find
-- linux-ubuntu18.04-x86_64 / gcc@10.1.0 ------------------------
clingo-bootstrap@spack python@3.6.9 re2c@1.2.1and if you ever need to, you can clean the bootstrap store out too:
$ spack clean -b
==> Removing bootstrapped software and configuration in "/home/spack/.spack/bootstrap"โ Check Installation ๐
once Spack is installed you should be able to run some basic Spack commands, fr fr. for example:
$ spack spec netcdf-c target=x86_64 os=SUSEin theory Spack doesn't need any additional installation โ just download and run, main character energy ๐ but in real life? yeah, there are usually extra steps required before Spack works in a practical sense. read on, bestie...
๐งผ Clean Environment ๐
many packages' installs can get broken by changing environment variables ๐ like, a package might pick up the wrong build-time dependencies (most of them aren't even specified) depending on what PATH is set to. and GCC seems to be particularly vulnerable to these issues, ngl.
so the recommendation is: run Spack with a clean environment, especially for PATH. only software that comes with the system, or that you know you want to use with Spack, should be included. this procedure avoids sooo many strange build errors, trust ๐ซถ
๐ Optional: Alternate Prefix ๐ธ
you may want to run Spack out of a prefix other than the git repository you cloned. the spack clone command provides this functionality! to install spack in a new directory, simply type:
$ spack clone /my/favorite/prefixthis installs a new spack script in /my/favorite/prefix/bin, which you can use just like you would the regular spack script. and each copy of spack installs packages into its own $PREFIX/opt directory โจ so they don't get all up in each other's business.
๐ ๏ธ Compiler configuration ๐
Spack has the ability to build packages with multiple compilers and compiler versions. compilers can be made available to Spack by specifying them manually in compilers.yaml or packages.yaml, or automatically by running spack compiler find โ but for convenience, Spack will automatically detect compilers the first time it needs them ๐โจ
spack compilers ๐
you can see which compilers are available to Spack by running spack compilers or spack compiler list:
$ spack compilers
==> Available compilers
-- gcc ---------------------------------------------------------
gcc@4.9.0 gcc@4.8.0 gcc@4.7.0 gcc@4.6.2 gcc@4.4.7
gcc@4.8.2 gcc@4.7.1 gcc@4.6.3 gcc@4.6.1 gcc@4.1.2
-- intel -------------------------------------------------------
intel@15.0.0 intel@14.0.0 intel@13.0.0 intel@12.1.0 intel@10.0
intel@14.0.3 intel@13.1.1 intel@12.1.5 intel@12.0.4 intel@9.1
intel@14.0.2 intel@13.1.0 intel@12.1.3 intel@11.1
intel@14.0.1 intel@13.0.1 intel@12.1.2 intel@10.1
-- clang -------------------------------------------------------
clang@3.4 clang@3.3 clang@3.2 clang@3.1
-- pgi ---------------------------------------------------------
pgi@14.3-0 pgi@13.2-0 pgi@12.1-0 pgi@10.9-0 pgi@8.0-1
pgi@13.10-0 pgi@13.1-1 pgi@11.10-0 pgi@10.2-0 pgi@7.1-3
pgi@13.6-0 pgi@12.8-0 pgi@11.1-0 pgi@9.0-4 pgi@7.0-6any of these compilers can be used to build Spack packages ๐ more on how that's done is in Specs & dependencies.
spack compiler add โญ
this one's just an alias for spack compiler find. periodt.
spack compiler find ๐
lists the compilers currently available to Spack. if you don't see a compiler in this list but you want to use it with Spack, you can simply run spack compiler find with the path to where the compiler is installed. like:
$ spack compiler find /usr/local/tools/ic-13.0.079
==> Added 1 new compiler to ~/.spack/linux/compilers.yaml
intel@13.0.079or you can run spack compiler find with no arguments to force auto-detection ๐ซถ this is useful if you don't know where compilers are installed, but you do know that new compilers have been added to your PATH. for example, you might load a module like this:
$ module load gcc/4.9.0
$ spack compiler find
==> Added 1 new compiler to ~/.spack/linux/compilers.yaml
gcc@4.9.0this loads the environment module for gcc-4.9.0 to add it to PATH, and then it adds the compiler to Spack. slay ๐
โ ๏ธ NOTE: by default, spack does not fill in themodules:field in thecompilers.yamlfile. if you're using a compiler from a module, you should add this field manually. see the section on Compilers Requiring Modules. ๐
spack compiler info ๐ง
if you want the specifics on a particular compiler, you can run spack compiler info on it:
$ spack compiler info intel@15
intel@15.0.0:
paths:
cc = /usr/local/bin/icc-15.0.090
cxx = /usr/local/bin/icpc-15.0.090
f77 = /usr/local/bin/ifort-15.0.090
fc = /usr/local/bin/ifort-15.0.090
modules = []
operating_system = centos6
...this shows which C, C++, and Fortran compilers were detected by Spack ๐ also notice we didn't have to be too specific about the version โ we just said intel@15, and info about the only matching Intel compiler was displayed. the way that just works ๐ญโจ
๐ Manual compiler configuration ๐
if auto-detection fails, you can manually configure a compiler by editing your ~/.spack/<platform>/compilers.yaml file. you can do this by running spack config edit compilers, which opens the file in your favorite editor ๐ซถ
each compiler configuration in the file looks like this:
compilers:
- compiler:
modules: []
operating_system: centos6
paths:
cc: /usr/local/bin/icc-15.0.024-beta
cxx: /usr/local/bin/icpc-15.0.024-beta
f77: /usr/local/bin/ifort-15.0.024-beta
fc: /usr/local/bin/ifort-15.0.024-beta
spec: intel@15.0.0for compilers that don't support Fortran (like clang), put None for f77 and fc:
compilers:
- compiler:
modules: []
operating_system: centos6
paths:
cc: /usr/bin/clang
cxx: /usr/bin/clang++
f77: None
fc: None
spec: clang@3.3svnonce you save the file, the configured compilers will show up in the list displayed by spack compilers. iconic ๐
you can also add compiler flags to manually configured compilers. these flags should be specified in the flags section of the compiler specification. the valid flags are cflags, cxxflags, fflags, cppflags, ldflags, and ldlibs. for example:
compilers:
- compiler:
modules: []
operating_system: centos6
paths:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
f77: /usr/bin/gfortran
fc: /usr/bin/gfortran
flags:
cflags: -O3 -fPIC
cxxflags: -O3 -fPIC
cppflags: -O3 -fPIC
spec: gcc@4.7.2these flags will be treated by spack as if they were entered from the command line every single time this compiler is used. the compiler wrappers then inject those flags into the compiler command ๐ compiler flags entered from the command line will be discussed in more detail in the following section, so stay tuned bestie.
some compilers also require additional environment configuration ๐ examples include Intel's oneAPI and AMD's AOCC compiler suites, which have custom scripts for loading environment variables and setting paths. these variables should be specified in the environment section of the compiler specification. the operations available to modify the environment are set, unset, prepend_path, append_path, and remove_path. for example:
compilers:
- compiler:
modules: []
operating_system: centos6
paths:
cc: /opt/intel/oneapi/compiler/latest/linux/bin/icx
cxx: /opt/intel/oneapi/compiler/latest/linux/bin/icpx
f77: /opt/intel/oneapi/compiler/latest/linux/bin/ifx
fc: /opt/intel/oneapi/compiler/latest/linux/bin/ifx
spec: oneapi@latest
environment:
set:
MKL_ROOT: "/path/to/mkl/root"
unset: # A list of environment variables to unset
- CC
prepend_path: # Similar for append|remove_path
LD_LIBRARY_PATH: /ld/paths/added/by/setvars/shNOTE: Spack is in the process of moving compilers from a separate attribute to be handled like all other packages. As part of this process, thecompilers.yamlsection will eventually be replaced by configuration in thepackages.yamlsection. This new configuration is now available, although it is not yet the default behavior. ๐
Okay bestie, compilers can also be configured as external packages in the packages.yaml config file. Any external package for a compiler (e.g. gcc or llvm) will be treated as a configured compiler assuming the paths to the compiler executables are determinable from the prefix. โจ
If the paths to the compiler executable are not determinable from the prefix, you can add them to the extra_attributes field. Similarly, all other fields from the compilers config can be added to the extra_attributes field for an external representing a compiler. ๐ซถ
Note that the format for the paths field in the extra_attributes section is different than in the compilers config. For compilers configured as external packages, the section is named compilers and the dictionary maps language names (c, cxx, fortran) to paths, rather than using the names cc, fc, and f77. iykyk ๐
packages:
gcc:
external:
- spec: gcc@12.2.0 arch=linux-rhel8-skylake
prefix: /usr
extra_attributes:
environment:
set:
GCC_ROOT: /usr
external:
- spec: llvm+clang@15.0.0 arch=linux-rhel8-skylake
prefix: /usr
extra_attributes:
compilers:
c: /usr/bin/clang-with-suffix
cxx: /usr/bin/clang++-with-extra-info
fortran: /usr/bin/gfortran
extra_rpaths:
- /usr/lib/llvm/๐ ๏ธ Build Your Own Compiler ๐ ๏ธ
If you are particular about which compiler/version you use, you might wish to have Spack build it for you. For example:
$ spack install gcc@4.9.3Once that has finished, you will need to add it to your compilers.yaml file. You can then set Spack to use it by default by adding the following to your packages.yaml file:
packages:
all:
compiler: [gcc@4.9.3]๐งฉ Compilers Requiring Modules ๐งฉ
Many installed compilers will work regardless of the environment they are called with. However, some installed compilers require $LD_LIBRARY_PATH or other environment variables to be set in order to run; this is typical for Intel and other proprietary compilers. ๐ญ
In such a case, you should tell Spack which module(s) to load in order to run the chosen compiler (If the compiler does not come with a module file, you might consider making one by hand). Spack will load this module into the environment ONLY when the compiler is run, and NOT in general for a package's install() method. See, for example, this compilers.yaml file:
compilers:
- compiler:
modules: [other/comp/gcc-5.3-sp3]
operating_system: SuSE11
paths:
cc: /usr/local/other/SLES11.3/gcc/5.3.0/bin/gcc
cxx: /usr/local/other/SLES11.3/gcc/5.3.0/bin/g++
f77: /usr/local/other/SLES11.3/gcc/5.3.0/bin/gfortran
fc: /usr/local/other/SLES11.3/gcc/5.3.0/bin/gfortran
spec: gcc@5.3.0Some compilers require special environment settings to be loaded not just to run, but also to execute the code they build, breaking packages that need to execute code they just compiled. If it's not possible or practical to use a better compiler, you'll need to ensure that environment settings are preserved for compilers like this (i.e., you'll need to load the module or source the compiler's shell script). ๐ฅ
By default, Spack tries to ensure that builds are reproducible by cleaning the environment before building. If this interferes with your compiler settings, you CAN use spack install --dirty as a workaround. Note that this MAY interfere with package builds. ๐
๐ Licensed Compilers ๐
Some proprietary compilers require licensing to use. If you need to use a licensed compiler (eg, PGI), the process is similar to a mix of build your own, plus modules:
- Create a Spack package (if it doesn't exist already) to install your compiler. Follow instructions on installing Licensed software. ๐
- Once the compiler is installed, you should be able to test it by using Spack to load the module it just created, and running simple builds (eg:
cc helloWorld.c && ./a.out) - Add the newly-installed compiler to
compilers.yamlas shown above. ๐
๐งฌ Mixed Toolchains ๐งฌ
Modern compilers typically come with related compilers for C, C++ and Fortran bundled together. When possible, results are best if the same compiler is used for all languages. ๐
In some cases, this is not possible. For example, starting with macOS El Capitan (10.11), many packages no longer build with GCC, but XCode provides no Fortran compilers. The user is therefore forced to use a mixed toolchain: XCode-provided Clang for C/C++ and GNU gfortran for Fortran. ๐ฆ
- You need to make sure that Xcode is installed. Run the following command:
$ xcode-select --installIf the Xcode command-line tools are already installed, you will see an error message:
xcode-select: error: command line tools are already installed, use "Software Update" to install updates- For most packages, the Xcode command-line tools are sufficient. However, some packages like
qtrequire the full Xcode suite. You can check to see which you have installed by running:
$ xcode-select -pIf the output is:
/Applications/Xcode.app/Contents/Developeryou already have the full Xcode suite installed. If the output is:
/Library/Developer/CommandLineToolsyou only have the command-line tools installed. The full Xcode suite can be installed through the App Store. Make sure you launch the Xcode application and accept the license agreement before using Spack. It may ask you to install additional components. Alternatively, the license can be accepted through the command line:
$ sudo xcodebuild -license acceptNote: the flag is -license, not --license. ๐
- Run
spack compiler findto locate Clang. โญ - There are different ways to get gfortran on macOS. For example, you can install GCC with Spack (
spack install gcc), with Homebrew (brew install gcc), or from a DMG installer. ๐ - The only thing left to do is to edit
~/.spack/darwin/compilers.yamlto provide the path to gfortran:
compilers:
- compiler:
# ...
paths:
cc: /usr/bin/clang
cxx: /usr/bin/clang++
f77: /path/to/bin/gfortran
fc: /path/to/bin/gfortran
spec: apple-clang@11.0.0If you used Spack to install GCC, you can get the installation prefix by spack location -i gcc (this will only work if you have a single version of GCC installed). Whereas for Homebrew, GCC is installed in /usr/local/Cellar/gcc/x.y.z. With the DMG installer, the correct path will be /usr/local/gfortran. ๐ง
โ Compiler Verification โ
You can verify that your compilers are configured properly by installing a simple package. For example:
$ spack install zlib%gcc@5.3.0๐ท๏ธ Vendor-Specific Compiler Configuration ๐ท๏ธ
With Spack, things usually "just work" with GCC. Not so for other compilers. This section provides details on how to get specific compilers working. ๐
๐ฅ Intel Compilers ๐ฅ
Intel compilers are unusual because a single Intel compiler version can emulate multiple GCC versions. In order to provide this functionality, the Intel compiler needs GCC to be installed. Therefore, the following steps are necessary to successfully use Intel compilers:
- Install a version of GCC that implements the desired language features (
spack install gcc). ๐ธ - Tell the Intel compiler how to find that desired GCC. This may be done in one of two ways:
"By default, the compiler determines which version of gcc or g++ you have installed from thePATHenvironment variable. If you want use a version of gcc or g++ other than the default version on your system, you need to use either the-gcc-nameor-gxx-namecompiler option to specify the path to the version of gcc or g++ that you want to use." โ Intel Reference Guide
Intel compilers may therefore be configured in one of two ways with Spack: using modules, or using compiler flags. ๐
๐งฑ Configuration with Modules ๐งฑ
One can control which GCC is seen by the Intel compiler with modules. A module must be loaded both for the Intel Compiler (so it will run) and GCC (so the compiler can find the intended GCC). The following configuration in compilers.yaml illustrates this technique:
compilers:
- compiler:
modules: [gcc-4.9.3, intel-15.0.24]
operating_system: centos7
paths:
cc: /opt/intel-15.0.24/bin/icc-15.0.24-beta
cxx: /opt/intel-15.0.24/bin/icpc-15.0.24-beta
f77: /opt/intel-15.0.24/bin/ifort-15.0.24-beta
fc: /opt/intel-15.0.24/bin/ifort-15.0.24-beta
spec: intel@15.0.24.4.9.3NOTE: The version number on the Intel compiler is a combination of the "native" Intel version number and the GNU compiler it is targeting. ๐
๐ฅ๏ธ Command Line Configuration ๐ฅ๏ธ
One can also control which GCC is seen by the Intel compiler by adding flags to the icc command:
- Identify the location of the compiler you just installed:
$ spack location --install-dir gcc
~/spack/opt/spack/linux-centos7-x86_64/gcc-4.9.3-iy4rw...- Set up
compilers.yaml, for example:
compilers:
- compiler:
modules: [intel-15.0.24]
operating_system: centos7
paths:
cc: /opt/intel-15.0.24/bin/icc-15.0.24-beta
cxx: /opt/intel-15.0.24/bin/icpc-15.0.24-beta
f77: /opt/intel-15.0.24/bin/ifort-15.0.24-beta
fc: /opt/intel-15.0.24/bin/ifort-15.0.24-beta
flags:
cflags: -gcc-name ~/spack/opt/spack/linux-centos7-x86_64/gcc-4.9.3-iy4rw.../bin/gcc
cxxflags: -gxx-name ~/spack/opt/spack/linux-centos7-x86_64/gcc-4.9.3-iy4rw.../bin/g++
fflags: -gcc-name ~/spack/opt/spack/linux-centos7-x86_64/gcc-4.9.3-iy4rw.../bin/gcc
spec: intel@15.0.24.4.9.3๐ PGI ๐
PGI comes with two sets of compilers for C++ and Fortran, distinguishable by their names. "Old" compilers:
cc: /soft/pgi/15.10/linux86-64/15.10/bin/pgcc
cxx: /soft/pgi/15.10/linux86-64/15.10/bin/pgCC
f77: /soft/pgi/15.10/linux86-64/15.10/bin/pgf77
fc: /soft/pgi/15.10/linux86-64/15.10/bin/pgf90"New" compilers:
...and that's what the config looks like:
cc: /soft/pgi/15.10/linux86-64/15.10/bin/pgcc
cxx: /soft/pgi/15.10/linux86-64/15.10/bin/pgc++
f77: /soft/pgi/15.10/linux86-64/15.10/bin/pgfortran
fc: /soft/pgi/15.10/linux86-64/15.10/bin/pgfortranSo here's the tea ๐ซ โ older PGI installations only ship the old compilers, while newer installations ship both the old and the new ones. The new compiler is the one you want, because some packages (hdf) straight up will not build with the old compiler. Periodt.
Big heads-up for auto-detection ๐: when Spack auto-detects a PGI compiler, there are actual cases where it'll find the old compilers when you really wanted it to find the new ones. So bestie, please check <hotpink>compilers.yaml</hotpink> โ and if the old compilers are the ones being used, change pgf77 and pgf90 to pgfortran. Easy fix, iconic results โจ
Other issues, because of course there are ๐:
- There are reports that some packages will not build with PGI, including libpciaccess and openssl. The workaround is to build these packages with another compiler and then use them as dependencies for PGI-build packages. Like this:
$ spack install openmpi%pgi ^libpciaccess%gcc- PGI requires a license to use; see Licensed Compilers for more information on installation.
NOTE: the belief is that the problem with HDF 4 is that everything gets compiled with the F77 compiler, but at some point some Fortran 90 code slipped in there ๐. So compilers that can handle both FORTRAN 77 and Fortran 90 (gfortran, pgfortran, etc.) are fine ๐. But compilers specific to one or the other (pgf77, pgf90) won't work. Sorry, that's just the rules.
๐ธ NAG ๐ธ
The Numerical Algorithms Group provides a licensed Fortran compiler. Like Clang, this requires you to set up a Mixed Toolchains. And it is recommended to use GCC for your C/C++ compilers ๐ซถ
The NAG Fortran compilers are a bit more strict than other compilers (ngl they're giving strict professor energy ๐ญ) and many packages will fail to install with error messages like:
Error: mpi_comm_spawn_multiple_f90.f90: Argument 3 to MPI_COMM_SPAWN_MULTIPLE has data type DOUBLE PRECISION in reference from MPI_COMM_SPAWN_MULTIPLEN and CHARACTER in reference from MPI_COMM_SPAWN_MULTIPLEAIn order to convince the NAG compiler not to be too picky about calling conventions, you can use FFLAGS=-mismatch and FCFLAGS=-mismatch. Through the command line, it's giving:
$ spack install openmpi fflags="-mismatch"Or you can set it permanently in your <hotpink>compilers.yaml</hotpink>, lowkey the tidy option ๐
:
- compiler:
modules: []
operating_system: centos6
paths:
cc: /soft/spack/opt/spack/linux-x86_64/gcc-5.3.0/gcc-6.1.0-q2zosj3igepi3pjnqt74bwazmptr5gpj/bin/gcc
cxx: /soft/spack/opt/spack/linux-x86_64/gcc-5.3.0/gcc-6.1.0-q2zosj3igepi3pjnqt74bwazmptr5gpj/bin/g++
f77: /soft/spack/opt/spack/linux-x86_64/gcc-4.4.7/nag-6.1-jt3h5hwt5myezgqguhfsan52zcskqene/bin/nagfor
fc: /soft/spack/opt/spack/linux-x86_64/gcc-4.4.7/nag-6.1-jt3h5hwt5myezgqguhfsan52zcskqene/bin/nagfor
flags:
fflags: -mismatch
spec: nag@6.1๐ง System Packages ๐ง
Okay so once compilers are configured, you need to figure out which pre-installed system packages (if any) you want to use in builds. This lives in the file ~/.spack/packages.yaml. Say you've got an OpenMPI sitting in /opt/local โ here's the vibe you'd write:
packages:
openmpi:
externals:
- spec: openmpi@1.10.1
prefix: /opt/local
buildable: FalseIn general, Spack is easier to use and more reliable if it builds all of its own dependencies. But โ and there's always a but โ there are several packages where you commonly need to use system versions. Let's go through them ๐
๐ฅ MPI ๐ฅ
On supercomputers, sysadmins have already built MPI versions that take into account the specifics of that computer's hardware. Unless you know how they were built and can choose the correct Spack variants, you are unlikely to get a working MPI from Spack. Not me guessing โ that's just how it is ๐ญ. Instead, use an appropriate pre-installed MPI.
If you choose a pre-installed MPI, you should consider using the pre-installed compiler that was used to build that MPI; see above on <hotpink>compilers.yaml</hotpink>.
๐ OpenSSL ๐
The openssl package underlies much of modern security in a modern OS; an attacker can easily "pwn" any computer on which they can modify SSL. So any openssl used on a system should be created in a "trusted environment" โ for example, that of the OS vendor. Big deal, no cap ๐ซก
OpenSSL is also updated by the OS vendor from time to time, in response to security problems discovered in the wider community. It's literally in everyone's best interest to use any newly updated versions as soon as they come out. Modern Linux installations have standard procedures for security updates without user involvement. Slay.
Spack running at user-level is not a trusted environment, nor do Spack users generally keep up-to-date on the latest security holes in SSL. For these reasons, a Spack-installed OpenSSL should likely not be trusted. Yeah. That's the whole point.
As long as the system-provided SSL works, you can just use it instead ๐. You can check if it works by trying to download an https://. For example:
$ curl -O https://github.com/ImageMagick/ImageMagick/archive/7.0.2-7.tar.gzTo tell Spack to use the system-supplied OpenSSL, first determine what version you have:
$ openssl version
OpenSSL 1.0.2g 1 Mar 2016Then add the following to ~/.spack/packages.yaml:
packages:
openssl:
externals:
- spec: openssl@1.0.2g
prefix: /usr
buildable: False๐งฎ BLAS / LAPACK ๐งฎ
The recommended way to use system-supplied BLAS / LAPACK packages is to add the following to packages.yaml:
packages:
netlib-lapack:
externals:
- spec: netlib-lapack@3.6.1
prefix: /usr
buildable: False
all:
providers:
blas: [netlib-lapack]
lapack: [netlib-lapack]NOTE: above we pretend that the system-provided BLAS / LAPACK is netlib-lapack only because it is the only BLAS / LAPACK provider which uses standard names for libraries (as opposed to, for example,libopenblas.so). Although we specify the external package in/usr, Spack is smart enough not to add/usr/libto RPATHs, where it could cause unrelated system libraries to be used instead of their Spack equivalents.usr/binwill be present inPATH, however it will have lower precedence compared to paths from other dependencies. This ensures that binaries in Spack dependencies are preferred over system binaries. Iconic behaviour honestly ๐
๐ Git ๐
Some Spack packages use git to download, which might not work on some computers. For example, this error was encountered on a Macintosh during spack install julia@master:
==> Cloning git repository:
https://github.com/JuliaLang/julia.git
on branch master
Cloning into 'julia'...
fatal: unable to access 'https://github.com/JuliaLang/julia.git/':
SSL certificate problem: unable to get local issuer certificateThis problem is related to OpenSSL, and in some cases might be solved by installing a new version of git and openssl:
- Run
spack install git - Add the output of
spack module tcl loads gitto your.bashrc.
If that doesn't work, it's also possible to disable checking of SSL certificates by using:
$ spack --insecure installUsing --insecure makes Spack disable SSL checking when fetching from websites and from git.
WARNING: โ ๏ธ this workaround should be used ONLY as a last resort! Without SSL certificate verification, spack and git will download from sites you wouldn't normally trust. The code you download and run may then be compromised! ๐ While this is not a major issue for archives that will be checksummed, it is especially problematic when downloading from name Git branches or tags, which relies entirely on trusting a certificate for security (no verification). (and again for the people in the back: no verification ๐ญ)
๐ Utilities Configuration ๐
Spack doesn't need installation per se, but it does rely on other packages being available on its host system. If those packages are out of date or missing, then Spack will not work. Sometimes an appeal to the system's package manager can fix such problems. If not, the solution is to have Spack install the required packages, and then have Spack use them ๐
For example, if curl doesn't work, one could use the following steps to provide Spack a working curl:
$ spack install curl
$ spack load curlor alternately:
$ spack module tcl loads curl >>~/.bashrcor if environment modules don't work, go full manual:
$ export PATH=`spack location --install-dir curl`/bin:$PATHExternal commands are used by Spack in two places: within core Spack, and in the package recipes. The bootstrapping procedure for these two cases is somewhat different, and is treated separately below ๐ซถ
โญ Core Spack Utilities โญ
Core Spack uses the following packages, mainly to download and unpack source code: curl, env, git, go, hg, svn, tar, unzip, patch.
As long as the user's environment is set up to successfully run these programs from outside of Spack, they should work inside of Spack as well. They can generally be activated as in the curl example above; or some systems might already have an appropriate hand-built environment module that may be loaded. Either way works ๐ฆ
A few notes on specific programs in this list:
๐ cURL, git, Mercurial, etc. ๐
Spack depends on cURL to download tarballs, the format that most Spack-installed packages come in. Your system's cURL should always be able to download unencrypted http://. However, the cURL on some systems has problems with SSL-enabled https:// URLs, due to outdated / insecure versions of OpenSSL on those systems. This will prevent Spack from installing any software requiring https:// until a new cURL has been installed, using the technique above.
WARNING: remember that if you install curl via Spack that it may rely on a user-space OpenSSL that is not upgraded regularly. It may fall out of date faster than your system OpenSSL ๐
Some packages use source code control systems as their download method: git, hg, svn and occasionally go. If you had to install a new curl, then chances are the system-supplied version of these other programs will also not work, because they also rely on OpenSSL. Once curl has been installed, you can similarly install the others ๐
๐ฆ Package Utilities ๐ฆ
Spack may also encounter bootstrapping problems inside a package's install() method. In this case, Spack will normally be running inside a sanitized build environment. This includes all of the package's dependencies, but none of the environment Spack inherited from the user: if you load a module or modify $PATH before launching Spack, it will have no effect. Yeah โ it's giving "we don't know her" ๐
In that case you'll probs need the --dirty flag when you run spack install โ that tells Spack to not sanitize the build environment. Heads up though ๐ซถ: once you do that, you are the one responsible for making sure that environment isn't doing weird little gremlin things to Spack or its installs.
Another way to get Spack to use its own version of something is to just add that something to the package that needs it. Like:
depends_on('binutils', type='build')This is genuinely considered best practice for some of the common build deps, like autotools (specifically if the autoreconf command is needed) and cmake โ cmake especially, because different packages require a different version of CMake. iykyk ๐
binutils
Sometimes the strangest little error messages pop up while building a package. Maybe ld just crashes ๐ญ. Or you get something like:
ld: final link failed: Nonrepresentable section on outputor:
ld: .../_fftpackmodule.o: unrecognized relocation (0x2a) in section `.text'These problems are, like, usually caused by an outdated binutils on your system. But โ and this is the plot twist โ unlike CMake or Autotools, adding depends_on('binutils') to every single package is not considered best practice, because literally every package written in C/C++/Fortran would need it. ๐ A potential workaround is to load a recent binutils into your environment and then use the --dirty flag.
GPG Signing
spack gpg
Spack has support for signing and verifying packages using GPG keys. A separate keyring is used for Spack, so any keys you've got hanging out in your user's home directory are not used. Not me being exclusive ๐ โจ
spack gpg init
When Spack is first installed, its keyring is empty. Keys stored in var/spack/gpg are the default keys for a Spack installation. These keys may be imported by running spack gpg init. This imports the default keys into the keyring as trusted keys. ๐
Trusting keys
You can add more keys to the keyring with spack gpg trust <keyfile>. Once a key is trusted, packages signed by the owner of that key may be installed. Slay ๐ซถ
Creating keys
You may also create your own key so you can sign your own packages, using spack gpg create <name> <email>. By default the key has no expiration, but you can set one with the --expires <date> flag (check the gnupg2 documentation for the accepted date formats). It's also recommended to add a comment about what the key is for, using the --comment <comment> flag. The public half of the key can also be exported so you can share it with others and they can use packages you've signed โ that's the --export <keyfile> flag. Secret keys may also be exported later using the spack gpg export <location> [<key>...] command.
NOTE: Key creation speed Creating a new GPG key requires generating a lot of random numbers. Depending on how much entropy your system produces, the whole process may take a really long time (like, even appearing to hang ๐ญ). Virtual machines and cloud instances are especially likely to show this behaviour. To speed it up you may install tools likerngd, which is usually available as a package in the host OS. On, say, an Ubuntu machine you need to give the following commands:$ sudo apt-get install rng-tools $ sudo rngd -r /dev/urandom...before generating the keys. Another alternative ishaveged, which can be installed on RHEL/CentOS machines as follows:$ sudo yum install haveged $ sudo chkconfig haveged onThis Digital Ocean tutorial provides a good overview of sources of randomness.
Here's an example of creating a key. Note that we give the key a name first (which we can use to reference the key later) and then an email address:
$ spack gpg create dinosaur dinosaur@thedinosaurthings.comIf you want to export the key as you create it:
$ spack gpg create --export key.pub dinosaur dinosaur@thedinosaurthings.comOr the private key:
$ spack gpg create --export-secret key.priv dinosaur dinosaur@thedinosaurthings.comYou can include both --export and --export-secret, each with an output file of your choice, to export both at once. ๐
Listing keys
To list the keys available in the keyring, the spack gpg list command will list trusted keys with the --trusted flag, and keys available for signing with --signing. If you want to remove keys from your keyring: spack gpg untrust <keyid>. Key IDs can be email addresses, names, or (best) fingerprints. Here's an example of listing the key we just created:
gpgconf: socketdir is '/run/user/1000/gnupg'
/home/spackuser/spack/opt/spack/gpg/pubring.kbx
----------------------------------------------------------
pub rsa4096 2021-03-25 [SC]
60D2685DAB647AD4DB54125961E09BB6F2A0ADCB
uid [ultimate] dinosaur (GPG created for Spack) <dinosaur@thedinosaurthings.com>Note that the name "dinosaur" shows up under the uid, which is the unique id. We might need this reference if we want to export or otherwise reference the key. ๐
Signing and Verifying Packages
To sign a package, use spack gpg sign <file>. By default the signature gets written to <file>.asc, but you can change that with the --output <file> flag. If there's only one signing key available, it'll be used โ but if there's more than one, you must specify which key to use with the --key <keyid> flag. The --clearsign flag may also be used to create a signed file which contains the contents, but it's not recommended. Signed packages may be verified with spack gpg verify <file>. โจ
Exporting Keys
You'll probably want to export a public key at some point, and it looks like this. Let's use the previous example and ask spack to export the key with uid "dinosaur." We give an output location (typically a *.pub file) and the name of the key:
$ spack gpg export dinosaur.pub dinosaurYou can then look at the created file, dinosaur.pub, to see the exported key. If you want to include the private key, just add --secret:
$ spack gpg export --secret dinosaur.priv dinosaurThis writes the private key to the file dinosaur.priv.
WARNING: You should be very careful about exporting private keys ๐ญ. You likely only want to do this in the context of moving your spack installation to a different server, and wanting to preserve keys for a buildcache. If you're unsure about exporting, ask your local system administrator, or ask for help on an issue or on the Spack slack. ๐ซถ
Spack on Cray
Spack differs slightly when used on a Cray system. The architecture spec can differentiate between the front-end and back-end processor and operating system. For example, on Edison at NERSC, the back-end target processor is "Ivy Bridge", so you can specify that you want the back-end like this:
$ spack install zlib target=ivybridgeYou can also use the operating system to build against the back-end:
$ spack install zlib os=CNL10Notice the name includes both the operating system name and the major version number concatenated together. ๐ง
Alternatively, if you want to build something for the front-end, you can specify the front-end target processor. The processor for a login node on Edison is "Sandy bridge", so we specify it on the command line like so:
$ spack install zlib target=sandybridgeAnd the front-end operating system is:
$ spack install zlib os=SuSE11Cray compiler detection
Spack can detect compilers using two methods. For the front-end, we treat everything the same. The difference lies in back-end compiler detection. Back-end compiler detection is made via the Tcl module avail command. Once it detects the compiler, it writes the appropriate PrgEnv and compiler module name to compilers.yaml, and sets the paths to each compiler with Cray's compiler wrapper names (i.e. cc, CC, ftn). During build time, Spack will load the correct PrgEnv and compiler module and will call the appropriate wrapper. ๐
The compilers.yaml config file will also differ. There's a modules section that gets filled with the compiler's Programming Environment and module name. On other systems, this field is empty []:
- compiler:
modules:
- PrgEnv-intel
- intel/15.0.109As mentioned earlier, the compiler paths will look different on a Cray system. Since most compilers are invoked using cc, CC and ftn, the paths for each compiler are replaced with their respective Cray compiler wrapper names:
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftnAs opposed to an explicit path to the compiler executable. This allows Spack to call the Cray compiler wrappers during build time. ๐ฆ
For more on compiler configuration, check out Compiler configuration.
Spack sets the default Cray link type to dynamic, to better match other other platforms. Individual packages can enable static linking (which is the default outside of Spack on cray systems) using the -static flag.
Setting defaults and using Cray modules
If you want to use default compilers for each PrgEnv and also be able to load cray external modules, you'll need to set up a packages.yaml.
Here's an example of an external configuration for cray modules:
packages:
mpich:
externals:
- spec: "mpich@7.3.1%gcc@5.2.0 arch=cray_xc-haswell-CNL10"
modules:
- cray-mpich
- spec: "mpich@7.3.1%intel@16.0.0.109 arch=cray_xc-haswell-CNL10"
modules:
- cray-mpich
all:
providers:
mpi: [mpich]This tells Spack that for whatever package depends on mpi, it should load the cray-mpich module into the environment. Then you'll be able to use whatever environment variables, libraries, etc. get brought into the environment via module load. ๐
NOTE: For Cray-provided packages, it's best to usemodules:instead ofprefix:inpackages.yaml, because the Cray Programming Environment heavily relies on modules (e.g., loading thecray-mpichmodule adds MPI libraries to the compiler wrapper link line).
You can set the default compiler that Spack can use for each compiler type. If you want to use the Cray defaults, set them under all: in packages.yaml. In the compiler field, set the compiler specs in your order of preference. Whenever you build with that compiler type, Spack will concretize to that version. โญ
Here is an example of a full packages.yaml used at NERSC
packages:
mpich:
externals:
- spec: "mpich@7.3.1%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge"
modules:
- cray-mpich
- spec: "mpich@7.3.1%intel@16.0.0.109 arch=cray_xc-SuSE11-ivybridge"
modules:
- cray-mpich
buildable: False
netcdf:
externals:
- spec: "netcdf@4.3.3.1%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge"
modules:
- cray-netcdf
- spec: "netcdf@4.3.3.1%intel@16.0.0.109 arch=cray_xc-CNL10-ivybridge"
modules:
- cray-netcdf
buildable: False
hdf5:
externals:
- spec: "hdf5@1.8.14%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge"
modules:
- cray-hdf5
- spec: "hdf5@1.8.14%intel@16.0.0.109 arch=cray_xc-CNL10-ivybridge"
modules:
- cray-hdf5
buildable: False
all:
compiler: [gcc@5.2.0, intel@16.0.0.109]
providers:
mpi: [mpich]So like, here's what we're telling spack ๐ whenever we wanna build with gcc, use version 5.2.0, and if we wanna build with intel compilers, use version 16.0.0.109. We add a spec for each compiler type for each cray module โ that's the whole point. This ensures that for each compiler on our system we can actually use that external module. No cap, that's the goal โจ
For more on external packages, check out the section External Packages. ๐ซถ
๐ง Using Linux containers on Cray machines ๐ง
Okay so this one's a sneaky little gotcha bestie ๐ญ Spack uses environment variables that are particular to the Cray programming environment to figure out which systems are Cray platforms. And those env vars? They may get propagated into containers that are not using the Cray programming environment. Rude.
To make sure Spack does not autodetect the Cray programming environment, unset the environment variable MODULEPATH. That'll cause Spack to treat a linux container on a Cray system as a plain base linux distro. Slay ๐
๐ช Spack On Windows ๐ช
Windows support for Spack is currently under development ๐ง Like, this work is still in an early stage โ but it is currently possible to set up Spack and perform a few operations on Windows. This section is gonna walk you through the steps needed to install Spack and start running it on a fresh Windows machine. Let's gooo ๐
โจ Step 1: Install prerequisites โจ
To use Spack on Windows, you will need the following packages:
- Required:
- Microsoft Visual Studio
- Python
- Git
- Optional:
- Intel Fortran (needed for some packages)
NOTE: Currently MSVC is the only compiler tested for C/C++ projects. Intel OneAPI provides Fortran support. ๐
๐ฅ๏ธ Microsoft Visual Studio ๐ฅ๏ธ
Microsoft Visual Studio provides the only Windows C/C++ compiler that is currently supported by Spack. Spack additionally requires that the Windows SDK (including WGL) be installed as part of your Visual Studio installation, because it's required to build many packages from source. Fr fr, don't skip it ๐ซ
We require several specific components to be included in the Visual Studio installation:
- One is the C/C++ toolset, which can be selected as "Desktop development with C++" or "C++ build tools," depending on installation type (Professional, Build Tools, etc.)
- The other required component is "C++ CMake tools for Windows," which can be selected from among the optional packages. This provides CMake and Ninja for use during Spack configuration.
If you already have Visual Studio installed, you can make sure these components are installed by rerunning the installer. Next to your installation, select "Modify" and look at the "Installation details" pane on the right. ๐
๐ฅ Intel Fortran ๐ฅ
For Fortran-based packages on Windows, we strongly recommend Intel's oneAPI Fortran compilers. The suite is free to download from Intel's website, located at โ https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/fortran-compiler.html
The executable of choice for Spack will be Intel's Beta Compiler, ifx, which supports the classic compiler's (ifort's) frontend and runtime libraries by using LLVM. Iconic behaviour honestly ๐
๐ Python ๐
As Spack is a Python-based package, an installation of Python is needed to run it. Python 3 can be downloaded and installed from the Windows Store, and it'll be automatically added to your PATH in that case. Convenient bestie โจ
NOTE: Spack currently supports Python versions later than 3.2 inclusive.
๐ง Git ๐ง
A bash console and GUI can be downloaded from โ https://git-scm.com/downloads
If you're unfamiliar with Git, there are a myriad of resources online to help guide you through checking out repositories and switching development branches. You got this ๐ช
When given the option of adjusting your PATH, choose the Git from the command line and also from 3rd-party software option. This will automatically update your PATH variable to include the git command. ๐
Spack support on Windows is currently dependent on installing the Git for Windows project as the project providing Git support on Windows. This is additionally the recommended method for installing Git on Windows, a link to which can be found above. Spack requires the utilities vendored by this project.
utilities vendored by this project. โ yes bestie, it really does say it twice in a row ๐ we're just built different.
๐ฆ Step 2: Install and setup Spack ๐ฆ
We are now ready to get the Spack environment set up on our machine. We begin by using Git to clone the Spack repo, hosted at โ https://github.com/spack/spack.git
โฆinto a desired directory, for our purposes today, called spack_install. ๐ธ
In order to install Spack with Windows support, run the following one liner in a Windows CMD prompt:
git clone https://github.com/spack/spack.gitNOTE: If you chose to install Spack into a directory on Windows that is set up to require Administrative Privileges, Spack will require elevated privileges to run. Administrative Privileges can be denoted either by default, such asC:\Program Files, or administrator-applied administrative restrictions on a directory that spack installs files to, such asC:\Users. ๐
โก Step 3: Run and configure Spack โก
To use Spack, run bin\spack_cmd.bat (you may need to Run as Administrator) from the top-level spack directory. This will provide a Windows command prompt with an environment properly set up with Spack and its prerequisites. ๐
If you receive a warning message that Python is not in your PATH (which may happen if you installed Python from the website and not the Windows Store), add the location of the Python executable to your PATH now. You can permanently add Python to your PATH variable by using the Edit the system environment variables utility in Windows Control Panel. ๐ ๏ธ
NOTE: Alternatively, Powershell can be used in place of CMD. ๐
To configure Spack, first run the following command inside the Spack console:
spack compiler findThis creates a .staging directory in our Spack prefix, along with a windows subdirectory containing a compilers.yaml file. On a fresh Windows install with the above packages installed, this command should only detect Microsoft Visual Studio โ and the Intel Fortran compiler will be integrated within the first version of MSVC present in the compilers.yaml output. Obsessed with how tidy that is โจ
Spack provides a default config.yaml file for Windows that it will use unless overridden. This file is located at etc\spack\defaults\windows\config.yaml. You can read more on how to do this and write your own configuration files in the Configuration Files section of our documentation. If you do this, pay particular attention to the build_stage block of the file, because this specifies the directory that will temporarily hold the source code for the packages to be installed. This path name must be sufficiently short for compliance with cmd โ otherwise you will see build errors during installation (particularly with CMake) tied to long path names. ๐ญ Ask me how I know.
To allow Spack use of external tools and dependencies already on your system, the external pieces of software must be described in the packages.yaml file. There are two methods to populate this file:
The first and easiest choice is to use Spack to find installation on your system. In the Spack terminal, run the following commands:
spack external find cmake
spack external find ninjaThe spack external find <name> will find executables on your system with the same name given. The command will store the items found in packages.yaml in the .staging\ directory. ๐ฏ
Assuming that the command found CMake and Ninja executables in the previous step, continue to Step 4. If no executables were found, we may need to manually direct spack towards the CMake and Ninja installations we set up with Visual Studio. Therefore, your packages.yaml file will look something like this, with possibly slight variants in the paths to CMake and Ninja:
packages:
cmake:
externals:
- spec: cmake@3.19
prefix: 'c:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake'
buildable: False
ninja:
externals:
- spec: ninja@1.8.2
prefix: 'c:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja'
buildable: FalseYou can also use a separate installation of CMake if you have one and prefer to use it. If you don't have a path to Ninja analogous to the above, then you can obtain it by running the Visual Studio Installer and following the instructions at the start of this section. Also note that .yaml files use spaces for indentation and not tabs, so ensure that this is the case when editing one directly. ๐ซถ (tabs are lowkey the villain of every YAML file ngl)
NOTE: Cygwin โ The use of Cygwin is not officially supported by Spack and is not tested. However Spack will not throw an error, so if choosing to use Spack with Cygwin, know that no functionality is guaranteed. โ ๏ธ
๐ Step 4: Use Spack ๐
Once the configuration is complete, it is time to give the installation a test. Install a basic package through the Spack console via:
spack install cpuinfoIf in the previous step you did not have CMake or Ninja installed, running the command above should bootstrap both packages. Slay, it just handles it for you ๐
๐งฉ Windows Compatible Packages ๐งฉ
Not all spack packages currently have Windows support. Some are inherently incompatible with the platform, and others simply have yet to be ported. To view the current set of packages with Windows support, the list command should be used via:
spack list -t windowsIf there's a package you'd like to install on Windows but it's not in that list, feel free to reach out to request the port or contribute the port yourself. Main character energy, honestly ๐โจ
NOTE: This is by no means a comprehensive list โ some packages may have ports that were not tagged, while others may just work out of the box on Windows and have not been tagged as such. ๐
For developers ๐ ๏ธ
So the whole intent here, bestie, is to ship a Windows installer that automatically sets up Python, Git, and Spack for you โ so nobody has to do it by hand like it's 2009. Instructions for actually creating that installer live at https://github.com/spack/spack/blob/develop/lib/spack/spack/cmd/installer/README.md.
Alternatively โ and this is the lowkey genius part โ a pre-built copy of the Windows installer is available as an artifact of Spack's Windows CI, and it shows up at each run of the CI on develop or on any PR. โจ
๐ BASIC USAGE
Okay so spack has so many subcommands, but honestly? You'll only need a tiny little subset of them for typical usage. Like, a handful. Don't stress. ๐
Heads up: Spack colorizes its output! ๐ So you should pipe it through less -R to keep that colorization alive.
Like this:
$ spack find | less -RAnd the manual recommends you put this in your .bashrc file, no cap:
alias less='less -R'Now โ if you don't see colorized output even though you're using less -R? It's because color is being disabled in the piped output. ๐ In that case, tell spack to force colorized output with a flag:
$ spack --color always find | less -R...or with an environment variable:
$ SPACK_COLOR=always spack find | less -R๐ฆ Listing available packages
To install software with Spack, you need to know what software is available โ obviously. ๐โโ๏ธ You can see a list of available package names at the packages.spack.io website, or you can use the spack list command.
๐ธ spack list
The spack list command prints out a list of all of the packages Spack can install:
$ spack list
3dtk py-flawfinder
3proxy py-flax
7zip py-flexmock
abacus py-flexx
abduco py-flit
abi-compliance-checker py-flit-core
abi-dumper py-flit-scm
abinit py-flufl-lock
abseil-cpp py-flye
abyss py-fn-py
...There are thousands of them ๐ญ so the output above is truncated, but you can find the full list here. Packages are listed by name in alphabetical order. And here's the tea on patterns, iykyk:
- A pattern to match with no wildcards,
*or?, will be treated as though it started and ended with*โ soutilis equivalent to*util*. - All patterns are treated as case-insensitive.
- You can also add the
-dto search the description of the package in addition to the name. ๐
Some examples, bestie:
All packages whose names contain "sql":
$ spack list sql
mysql perl-sql-translator py-mysqldb1 py-sqlparse
mysql-connector-c postgresql py-pygresql r-rmysql
mysqlpp py-agate-sql py-pymysql r-rpostgresql
perl-datetime-format-mysql py-aiosqlite py-pysqlite3 r-rsqlite
perl-dbd-mysql py-azure-mgmt-sql py-sqlalchemy r-sqldf
perl-dbd-sqlite py-azure-mgmt-sqlvirtualmachine py-sqlalchemy-migrate sqlcipher
perl-sql-abstract py-flask-sqlalchemy py-sqlalchemy-stubs sqlite
perl-sql-abstract-classic py-mysql-connector-python py-sqlalchemy-utils sqlite-jdbc
perl-sql-reservedwords py-mysqlclient py-sqlitedict sqlitebrowserAll packages whose names or descriptions contain documentation:
$ spack list --search-description documentation
asciidoc-py3 oommf py-furo py-sphinx-immaterial r-rinside
byacc perl-bioperl py-griffe py-sphinx-multiversion r-roxygen2
compositeproto perl-db-file py-interface-meta py-sphinx-tabs r-satellite
damageproto perl-io-prompt py-markdown py-sphinxautomodapi r-spam
double-conversion perl-pod-coverage py-mkdocs py-sphinxcontrib-websupport r-stanheaders
doxygen py-alabaster py-mkdocs-material py-sphinxemoji r-units
gflags py-astropy-helpers py-mkdocstrings r-downlit r-uwot
glow py-dask-sphinx-theme py-myst-parser r-lifecycle sowing
gtk-doc py-docutils py-param r-modeltools texinfo
libxfixes py-epydoc py-pdoc3 r-pkgdown totalview
libxpresent py-exhale py-python-docs-theme r-quadprog xorg-docs
man-db py-fastai py-recommonmark r-rcpp xorg-sgml-doctools
ntl py-ford py-sphinx r-rdpack๐ฆ spack info
To get more information on a particular package that you spotted in spack list, use spack info. Just supply the name of a package, bestie:
$ spack info --all mpich
AutotoolsPackage: mpich
Description:
MPICH is a high performance and widely portable implementation of the
Message Passing Interface (MPI) standard.
Homepage: https://www.mpich.org
Maintainers: @raffenet @yfguo
Externally Detectable:
True (version, variants)
Tags:
detectable e4s
Preferred version:
4.2.1 https://www.mpich.org/static/downloads/4.2.1/mpich-4.2.1.tar.gz
Safe versions:
develop [git] https://github.com/pmodels/mpich.git
4.2.1 https://www.mpich.org/static/downloads/4.2.1/mpich-4.2.1.tar.gz
4.2.0 https://www.mpich.org/static/downloads/4.2.0/mpich-4.2.0.tar.gz
4.1.2 https://www.mpich.org/static/downloads/4.1.2/mpich-4.1.2.tar.gz
4.1.1 https://www.mpich.org/static/downloads/4.1.1/mpich-4.1.1.tar.gz
4.1 https://www.mpich.org/static/downloads/4.1/mpich-4.1.tar.gz
4.0.3 https://www.mpich.org/static/downloads/4.0.3/mpich-4.0.3.tar.gz
4.0.2 https://www.mpich.org/static/downloads/4.0.2/mpich-4.0.2.tar.gz
4.0.1 https://www.mpich.org/static/downloads/4.0.1/mpich-4.0.1.tar.gz
4.0 https://www.mpich.org/static/downloads/4.0/mpich-4.0.tar.gz
3.4.3 https://www.mpich.org/static/downloads/3.4.3/mpich-3.4.3.tar.gz
3.4.2 https://www.mpich.org/static/downloads/3.4.2/mpich-3.4.2.tar.gz
3.4.1 https://www.mpich.org/static/downloads/3.4.1/mpich-3.4.1.tar.gz
3.4 https://www.mpich.org/static/downloads/3.4/mpich-3.4.tar.gz
3.3.2 https://www.mpich.org/static/downloads/3.3.2/mpich-3.3.2.tar.gz
3.3.1 https://www.mpich.org/static/downloads/3.3.1/mpich-3.3.1.tar.gz
3.3 https://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz
3.2.1 https://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1.tar.gz
3.2 https://www.mpich.org/static/downloads/3.2/mpich-3.2.tar.gz
3.1.4 https://www.mpich.org/static/downloads/3.1.4/mpich-3.1.4.tar.gz
3.1.3 https://www.mpich.org/static/downloads/3.1.3/mpich-3.1.3.tar.gz
3.1.2 https://www.mpich.org/static/downloads/3.1.2/mpich-3.1.2.tar.gz
3.1.1 https://www.mpich.org/static/downloads/3.1.1/mpich-3.1.1.tar.gz
3.1 https://www.mpich.org/static/downloads/3.1/mpich-3.1.tar.gz
3.0.4 https://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz
Deprecated versions:
None
Variants:
argobots [false] false, true
Enable Argobots support
build_system [autotools] autotools
Build systems supported by the package
cuda [false] false, true
Build with CUDA
device [ch4] ch3, ch3:sock, ch4
Abstract Device Interface (ADI)
implementation. The ch4 device is in experimental state for versions
before 3.4.
fortran [true] false, true
Enable Fortran support
hwloc [true] false, true
Use external hwloc package
hydra [true] false, true
Build the hydra process manager
libxml2 [true] false, true
Use libxml2 for XML support instead of the custom minimalistic implementation
netmod [ofi] mxm, ofi, tcp, ucx
Network module. Only single netmod builds are
supported, and netmod is ignored if device is ch3:sock.
pci [true] false, true
Support analyzing devices on PCI bus
pmi [default] cray, default, pmi, pmi2, pmix
PMI interface.
rocm [false] false, true
Enable ROCm support
romio [true] false, true
Enable ROMIO MPI I/O implementation
slurm [false] false, true
Enable Slurm support
verbs [false] false, true
Build support for OpenFabrics verbs.
wrapperrpath [true] false, true
Enable wrapper rpath
when +cuda
cuda_arch [none] none, 10, 11, 12, 13, 20, 21, 30, 32, 35, 37, 50, 52, 53, 60, 61, 62, 70, 72, 75,
80, 86, 87, 89, 90
CUDA architecture
when +rocm
amdgpu_target [none] none, gfx1010, gfx1011, gfx1012, gfx1013, gfx1030, gfx1031, gfx1032, gfx1033,
gfx1034, gfx1035, gfx1036, gfx1100, gfx1101, gfx1102, gfx1103, gfx701, gfx801,
gfx802, gfx803, gfx900, gfx900:xnack-, gfx902, gfx904, gfx906, gfx906:xnack-,
gfx908, gfx908:xnack-, gfx909, gfx90a, gfx90a:xnack+, gfx90a:xnack-, gfx90c,
gfx940, gfx941, gfx942
AMD GPU architecture
when @3.3: device=ch4 netmod=ucx
hcoll [false] false, true
Enable support for Mellanox HCOLL accelerated collective operations library
when @3.4:
datatype-engine [auto] auto, dataloop, yaksa
controls the datatype engine to use
xpmem [false] false, true
Enable XPMEM support when @4: device=ch4
vci [false] false, true
Enable multiple VCI (virtual communication interface) critical sections to improve performance of
applications that do heavy concurrent MPIcommunications. Set MPIR_CVAR_CH4_NUM_VCIS=<N> to enable multiple
vcis at runtime.ok so that vci one is the last variant in the list, and bestie, it's lowkey niche โ it's about enabling multiple VCI critical sections so apps doing heavy concurrent MPI communications go faster, and you turn the extra vcis on at runtime with MPIR_CVAR_CH4_NUM_VCIS=<N>. that's the whole vibe there ๐
next up, spack info dumps a bunch of phase / dependency / license blocks at you, and they look like this:
Installation Phases:
autoreconf configure build install
Build Dependencies:
argobots cray-pmi gmake hip libfabric libxml2 mxm python xpmem
autoconf cuda gnuconfig hsa-rocr-dev libpciaccess llvm-amdgpu pkgconfig slurm yaksa
automake findutils hcoll hwloc libtool m4 pmix ucx
Link Dependencies:
argobots cuda hip hwloc libpciaccess llvm-amdgpu pmix ucx yaksa
cray-pmi hcoll hsa-rocr-dev libfabric libxml2 mxm slurm xpmem
Run Dependencies:
None
Virtual Packages:
mpich provides mpi@:4.0
mpich@:3.2 provides mpi@:3.1
mpich@:3.1 provides mpi@:3.0
mpich@:1.2 provides mpi@:2.2
mpich@:1.1 provides mpi@:2.1
mpich@:1.0 provides mpi@:2.0
Available Build Phase Test Methods:
None
Available Install Phase Test Methods:
None
Stand-Alone/Smoke Test Methods:
None
Licenses:
UnlicenseMost of that is genuinely self-explanatory, no cap. The one thing worth flagging: the safe versions are the versions Spack actually has a checksum for, and it'll use that checksum to verify the download happened without errors or viruses. so like... safe = verified, basically ๐ซถ
Dependencies and virtual dependencies get their own deeper explanation later on, so don't stress about them yet.
๐ธ spack versions
wanna see more available versions of a package? run spack versions, easy:
$ spack versions libelf
0.8.13There are two sections in that output, bestie:
- Safe versions โ versions Spack has a checksum on file for, so it can verify they downloaded correctly โจ
- Remote versions โ in many cases Spack can also show you what versions are chillin' out on the web. It gets this by scraping it directly from the package web pages, and depending on the package and how its releases are organized, Spack may or may not be able to find these. so it's giving "best effort", not "guaranteed" ๐
๐ Installing and uninstalling
spack install
spack install will install any package shown by spack list. For example, to install the latest version of the mpileaks package you might type this:
$ spack install mpileaksIf mpileaks depends on other packages, Spack installs the dependencies first. Then it fetches the mpileaks tarball, expands it, verifies that it was downloaded without errors, builds it, and installs it in its own directory under $SPACK_ROOT/opt. You'll see a number of messages from Spack, a lot of build output, and a message that the package is installed. like this:
$ spack install mpileaks
... dependency build output ...
==> Installing mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2
==> No binary for mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2 found: installing from source
==> mpileaks: Executing phase: 'autoreconf'
==> mpileaks: Executing phase: 'configure'
==> mpileaks: Executing phase: 'build'
==> mpileaks: Executing phase: 'install'
[+] ~/spack/opt/linux-rhel7-broadwell/gcc-8.1.0/mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2The last line, with the [+], tells you where the package is installed. iconic little breadcrumb ๐
Add the Spack debug option (one or more times) โspack -d install mpileaksโ to get additional (and even more verbose) output. yes you read that right, you can stack-dfor extra chaos ๐
Building a specific version
Spack can also build specific versions of a package. To do this, just add @ after the package name, followed by a version:
$ spack install mpich@3.0.4Any number of versions of the same package can be installed at once without interfering with each other. this is so good for multi-user sites โ installing a version that one user needs will not disrupt existing installations for other users. main character energy for shared clusters ๐
In addition to different versions, Spack can customize the compiler, compile-time options (variants), compiler flags, and platform (for cross compiles) of an installation. And Spack is unique in that it can also configure the dependencies a package is built with. For example, two configurations of the same version of a package โ one built with boost 1.39.0, and the other version built with version 1.43.0 โ can coexist. literally no drama between them ๐ฆ
This can all be done on the command line using the spec syntax. Spack calls the descriptor used to refer to a particular package configuration a spec. In the commands above, mpileaks and mpileaks@3.0.4 are both valid specs. We'll talk more about how you can use them to customize an installation in Specs & dependencies.
Reusing installed dependencies
By default, when you run spack install, Spack tries hard to reuse existing installations as dependencies โ either from a local store or from remote buildcaches if configured. This minimizes unwanted rebuilds of common dependencies, in particular if you update Spack frequently. it's giving "recycle, don't recompile" โป๏ธ
But say you want the latest versions and configurations to be installed instead โ then you can add the --fresh option:
$ spack install --fresh mpichIn this mode, reusing installations is "accidental", and happens only if there's a match between existing installations and what Spack would have installed anyhow. so no promises ๐
You can use the spack spec -I mpich command to see what will be reused and what will be built before you install. highly recommend, bestie ๐
And you can configure Spack to use the --fresh behavior by default in concretizer.yaml:
concretizer:
reuse: falsespack uninstall
To uninstall a package, type spack uninstall <package>. This will ask the user for confirmation before completely removing the directory in which the package was installed.
$ spack uninstall mpichIf there are still installed packages that depend on the package to be uninstalled, Spack will refuse to uninstall it. nope. not today. ๐ โโ๏ธ
To uninstall a package and every package that depends on it, you may give the --dependents option:
$ spack uninstall --dependents mpich...which will display a list of all the packages that depend on mpich and, upon confirmation, will uninstall them in the right order. orderly, like it should be โจ
A command like
$ spack uninstall mpichmay be ambiguous if multiple mpich configurations are installed. For example, if both mpich@3.0.2 and mpich@3.1 are installed, mpich could refer to either one. Because it cannot determine which one to uninstall, Spack will ask you either to provide a version number to remove the ambiguity or use the --all option to uninstall all of the matching packages. pick a lane ๐ซ
You may force uninstall a package with the --force option:
$ spack uninstall --force mpichbut you risk breaking other installed packages. In general, it is safer to remove dependent packages before removing their dependencies, or use the--dependentsoption. seriously,--forceis that risky ex you don't text back ๐ฅ๐
๐ง Garbage collection
When Spack builds software from sources, it often installs tools that are needed just to build or test other software. These are not necessary at runtime. To support cases where removing these tools can be a benefit, Spack provides the spack gc ("garbage collector") command, which will uninstall all unneeded packages:
$ spack find
==> 24 installed packages
-- linux-ubuntu18.04-broadwell / gcc@9.0.1 ----------------------
autoconf@2.69 findutils@4.6.0 libiconv@1.16 libszip@2.1.1 m4@1.4.18 openjpeg@2.3.1 pkgconf@1.6.3 util-macros@1.19.1
automake@1.16.1 gdbm@1.18.1 libpciaccess@0.13.5 libtool@2.4.6 mpich@3.3.2 openssl@1.1.1d readline@8.0 xz@5.2.4
cmake@3.16.1 hdf5@1.10.5 libsigsegv@2.12 libxml2@2.9.9 ncurses@6.1 perl@5.30.0 texinfo@6.5 zlib@1.2.11
$ spack gc
==> The following packages will be uninstalled:
-- linux-ubuntu18.04-broadwell / gcc@9.0.1 ----------------------
vn47edz autoconf@2.69 6m3f2qn findutils@4.6.0 ubl6bgk libtool@2.4.6 pksawhz openssl@1.1.1d urdw22a readline@8.0
ki6nfw5 automake@1.16.1 fklde6b gdbm@1.18.1 b6pswuo m4@1.4.18 k3s2csy perl@5.30.0 lp5ya3t texinfo@6.5
ylvgsov cmake@3.16.1 5omotir libsigsegv@2.12 leuzbbh ncurses@6.1 5vmfbrq pkgconf@1.6.3 5bmv4tg util-macros@1.19.1
==> Do you want to proceed? [y/N] y
[ ... ]
$ spack find
==> 9 installed packages
-- linux-ubuntu18.04-broadwell / gcc@9.0.1 ----------------------
hdf5@1.10.5 libiconv@1.16 libpciaccess@0.13.5 libszip@2.1.1 libxml2@2.9.9 mpich@3.3.2 openjpeg@2.3.1 xz@5.2.4 zlib@1.2.11In the example above Spack went through all the packages in the package database and removed everything that is not either:
- A package installed upon explicit request of the user
- A link or run dependency, even transitive, of one of the packages at point 1.
You can check Viewing more metadata to see how to query for explicitly installed packages, or Dependency types for a more thorough treatment of dependency types.
Marking packages explicit or implicit
By default, Spack marks packages a user installs as explicitly installed, while all of its dependencies get marked as implicitly installed. You can flip that manually with spack mark โ mark a package explicitly or implicitly installed. This is chef's kiss combined with spack gc for cleaning up packages that are no longer required ๐
$ spack install m4
==> 29005: Installing libsigsegv
[...]
==> 29005: Installing m4
[...]
$ spack install m4 ^libsigsegv@2.11
==> 39798: Installing libsigsegv
[...]
==> 39798: Installing m4
[...]
$ spack find -d
==> 4 installed packages
-- linux-fedora32-haswell / gcc@10.1.1 --------------------------
libsigsegv@2.11
libsigsegv@2.12
m4@1.4.18
libsigsegv@2.12
m4@1.4.18
libsigsegv@2.11
$ spack gc
==> There are no unused specs. Spack's store is clean.
$ spack mark -i m4 ^libsigsegv@2.11
==> m4@1.4.18 : marking the package implicit
$ spack gc
==> The following packages will be uninstalled:
-- linux-fedora32-haswell / gcc@10.1.1 ----------------------
5fj7p2o libsigsegv@2.11 c6ensc6 m4@1.4.18
==> Do you want to proceed? [y/N]In the example above, we ended up with two versions of m4 since they depend on different versions of libsigsegv. spack gc will not remove any of the packages, since both versions of m4 have been installed explicitly and both versions of libsigsegv are required by the m4 packages. see how that works? the โจdependency graphโจ said no ๐
okay so spack mark can also be used to implement upgrade workflows. The following example demonstrates how spack mark and spack gc can be used to only keep the current version of a package installed.
When updating Spack via git pull, new versions for either libsigsegv or m4 might be introduced. This will cause Spack to install duplicates. Since we only want to keep one version, we mark everything as implicitly installed before updating Spack. If there is no new version for either of the packages, spack install will simply mark them as explicitly installed and spack gc will not remove them.
$ spack install m4
==> 62843: Installing libsigsegv
[...]
==> 62843: Installing m4
[...]
$ spack mark -i -a
==> m4@1.4.18 : marking the package implicit
$ git pull
[...]
$ spack install m4
[...]
==> m4@1.4.18 : marking the package explicit
[...]
$ spack gc
==> There are no unused specs. Spack's store is clean.When using this workflow for installations that contain more packages, care has to be taken to either only mark selected packages or issue spack install for all packages that should be kept.
You can check Viewing more metadata to see how to query for explicitly or implicitly installed packages.
๐ Non-Downloadable Tarballs ๐
The tarballs for some packages cannot be automatically downloaded by Spack. This could be for a number of reasons:
- The author requires users to manually accept a license agreement before downloading (
jdkandgalahad). - The software is proprietary and cannot be downloaded on the open Internet.
To install these packages, one must create a mirror and manually add the tarballs in question to it (see Mirrors (mirrors.yaml)):
- Create a directory for the mirror. You can create this directory anywhere you like, it does not have to be inside
~/.spack:
$ mkdir ~/.spack/manual_mirror- Register the mirror with Spack by creating
~/.spack/mirrors.yaml:
mirrors:
manual: file://~/.spack/manual_mirror- Put your tarballs in it. Tarballs should be named
<package>/<package>-<version>.tar.gz. For example:
$ ls -l manual_mirror/galahad
-rw-------. 1 me me 11657206 Jun 21 19:25 galahad-2.60003.tar.gz- Install as usual:
$ spack install galahad๐ Seeing installed packages ๐
We know that spack list shows you the names of available packages, but how do you figure out which are already installed?
๐ spack find ๐
spack find shows the specs of installed packages. A spec is like a name, but it has a version, compiler, architecture, and build options associated with it. In spack, you can have many installations of the same package with different specs.
Running spack find with no arguments lists installed packages:
$ spack find
==> 74 installed packages.
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
ImageMagick@6.8.9-10 libdwarf@20130729 py-dateutil@2.4.0
adept-utils@1.0 libdwarf@20130729 py-ipython@2.3.1
atk@2.14.0 libelf@0.8.12 py-matplotlib@1.4.2
boost@1.55.0 libelf@0.8.13 py-nose@1.3.4
bzip2@1.0.6 libffi@3.1 py-numpy@1.9.1
cairo@1.14.0 libmng@2.0.2 py-pygments@2.0.1
callpath@1.0.2 libpng@1.6.16 py-pyparsing@2.0.3
cmake@3.0.2 libtiff@4.0.3 py-pyside@1.2.2
dbus@1.8.6 libtool@2.4.2 py-pytz@2014.10
dbus@1.9.0 libxcb@1.11 py-setuptools@11.3.1
dyninst@8.1.2 libxml2@2.9.2 py-six@1.9.0
fontconfig@2.11.1 libxml2@2.9.2 python@2.7.8
freetype@2.5.3 llvm@3.0 qhull@1.0
gdk-pixbuf@2.31.2 memaxes@0.5 qt@4.8.6
glib@2.42.1 mesa@8.0.5 qt@5.4.0
graphlib@2.0.0 mpich@3.0.4 readline@6.3
gtkplus@2.24.25 mpileaks@1.0 sqlite@3.8.5
harfbuzz@0.9.37 mrnet@4.1.0 stat@2.1.0
hdf5@1.8.13 ncurses@5.9 tcl@8.6.3
icu@54.1 netcdf@4.3.3 tk@src
jpeg@9a openssl@1.0.1h vtk@6.1.0
launchmon@1.0.1 pango@1.36.8 xcb-proto@1.11
lcms@2.6 pixman@0.32.6 xz@5.2.0
libdrm@2.4.33 py-dateutil@2.4.0 zlib@1.2.8
-- linux-debian7-x86_64 / gcc@4.9.2 --------------------------------
libelf@0.8.10 mpich@3.0.4Packages are divided into groups according to their architecture and compiler. Within each group, Spack tries to keep the view simple, and only shows the version of installed packages.
๐ซถ Viewing more metadata ๐ซถ
spack find can filter the package list based on the package name, spec, or a number of properties of their installation status. For example, missing dependencies of a spec can be shown with --missing, deprecated packages can be included with --deprecated, packages which were explicitly installed with spack install <package> can be singled out with --explicit and those which have been pulled in only as dependencies with --implicit.
In some cases, there may be different configurations of the same version of a package installed. For example, there are two installations of libdwarf@20130729 above. We can look at them in more detail using spack find --deps, and by asking only to show libdwarf packages:
$ spack find --deps libdwarf
==> 2 installed packages.
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
libdwarf@20130729-d9b90962
^libelf@0.8.12
libdwarf@20130729-b52fac98
^libelf@0.8.13Now we see that the two instances of libdwarf depend on different versions of libelf: 0.8.12 and 0.8.13. This view can become complicated for packages with many dependencies. If you just want to know whether two packages' dependencies differ, you can use spack find --long:
$ spack find --long libdwarf
==> 2 installed packages.
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
libdwarf@20130729-d9b90962 libdwarf@20130729-b52fac98Now the libdwarf installs have hashes after their names. These are hashes over all of the dependencies of each package. If the hashes are the same, then the packages have the same dependency configuration.
If you want to know the path where each package is installed, you can use spack find --paths:
$ spack find --paths
==> 74 installed packages.
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
ImageMagick@6.8.9-10 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/ImageMagick@6.8.9-10-4df950dd
adept-utils@1.0 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/adept-utils@1.0-5adef8da
atk@2.14.0 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/atk@2.14.0-3d09ac09
boost@1.55.0 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/boost@1.55.0
bzip2@1.0.6 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/bzip2@1.0.6
cairo@1.14.0 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/cairo@1.14.0-fcc2ab44
callpath@1.0.2 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/callpath@1.0.2-5dce4318
...You can restrict your search to a particular package by supplying its name:
$ spack find --paths libelf
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
libelf@0.8.11 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/libelf@0.8.11
libelf@0.8.12 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/libelf@0.8.12
libelf@0.8.13 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/libelf@0.8.13๐ฆ Spec queries ๐ฆ
spack find actually does a lot more than this. You can use specs to query for specific configurations and builds of each package. If you want to find only libelf versions greater than version 0.8.12, you could say:
$ spack find libelf@0.8.12:
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
libelf@0.8.12 libelf@0.8.13Finding just the versions of libdwarf built with a particular version of libelf would look like this:
$ spack find --long libdwarf ^libelf@0.8.12
==> 1 installed packages.
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
libdwarf@20130729-d9b90962We can also search for packages that have a certain attribute. For example, spack find libdwarf +debug will show only installations of libdwarf with the 'debug' compile-time option enabled.
The full spec syntax is discussed in detail in Specs & dependencies.
๐ Machine-readable output ๐
If you only want to see very specific things about installed packages, Spack has some options for you. spack find --format can be used to output only specific fields:
$ spack find --format "{name}-{version}-{hash}"
autoconf-2.69-icynozk7ti6h4ezzgonqe6jgw5f3ulx4
automake-1.16.1-o5v3tc77kesgonxjbmeqlwfmb5qzj7zy
bzip2-1.0.6-syohzw57v2jfag5du2x4bowziw3m5p67
bzip2-1.0.8-zjny4jwfyvzbx6vii3uuekoxmtu6eyuj
cmake-3.15.1-7cf6onn52gywnddbmgp7qkil4hdoxpcb
...or:
$ spack find --format "{hash:7}"
icynozk
o5v3tc7
syohzw5
zjny4jw
7cf6onn
...This uses the same syntax as described in documentation for format() -- you can use any of the options there. This is useful for passing metadata about packages to other command-line tools.
Alternately, if you want something even more machine readable, you can output each spec as JSON records using spack find --json. This will output metadata on specs and all dependencies as json:
$ spack find --json sqlite@3.28.0
[
{
"name": "sqlite",
"hash": "3ws7bsihwbn44ghf6ep4s6h4y2o6eznv",
"version": "3.28.0",
"arch": {
"platform": "darwin",
"platform_os": "mojave",
"target": "x86_64"
},
"compiler": {
"name": "apple-clang",
"version": "10.0.0"
},
"namespace": "builtin",
"parameters": {
"fts": true,
"functions": false,
"cflags": [],
"cppflags": [],
"cxxflags": [],
"fflags": [],
"ldflags": [],
"ldlibs": []
},
"dependencies": {
"readline": {
"hash": "722dzmgymxyxd6ovjvh4742kcetkqtfs",
"type": [
"build",
"link"
]
}
}
},
...
]You can use this with tools like jq to quickly create JSON records structured the way you want:
$ spack find --json sqlite@3.28.0 | jq -C '.[] | { name, version, hash }'
{
"name": "sqlite",
"version": "3.28.0",
"hash": "3ws7bsihwbn44ghf6ep4s6h4y2o6eznv"
}
{
"name": "readline",
"version": "7.0",
"hash": "722dzmgymxyxd6ovjvh4742kcetkqtfs"
}
{
"name": "ncurses",
"version": "6.1",
"hash": "zvaa4lhlhilypw5quj3akyd3apbq5gap"
}๐ฆ spack diff ๐ฆ
okay so picture this: you've got two versions of the same spec and you literally cannot tell them apart. classic. let's say you installed two variants of zlib โ one with the optimize variant and one without:
$ spack install zlib
$ spack install zlib -optimizeand when you run spack find you see both of them, bestie:
$ spack find zlib
==> 2 installed packages
-- linux-ubuntu20.04-skylake / gcc@9.3.0 ------------------------
zlib@1.2.11 zlib@1.2.11so now you're like "cool, imma uninstall zlib" and you hit a wall immediately because there are literally two of them ๐ญ:
$ spack uninstall zlib
==> Error: zlib matches multiple packages:
-- linux-ubuntu20.04-skylake / gcc@9.3.0 ------------------------
efzjziy zlib@1.2.11 sl7m27m zlib@1.2.11
==> Error: You can either:
a) use a more specific spec, or
b) specify the spec by its hash (e.g. `spack uninstall /hash`), or
c) use `spack uninstall --all` to uninstall ALL matching specs.not the error message dragging us like that ๐ but fr, look at it โ two different versions of zlib are installed, and the only visible difference between them is the hash. this is literally the moment spack diff was born for: it shows you the "diff", aka the set difference, between the properties of two packages. let's try it. the spack find view only gave us hashes, so we feed those two hashes in:
$ spack diff /efzjziy /sl7m27m
==> Warning: This interface is subject to change.
--- zlib@1.2.11efzjziyc3dmb5h5u5azsthgbgog5mj7g
+++ zlib@1.2.11sl7m27mzkbejtkrajigj3a3m37ygv4u2
@@ variant_value @@
- zlib optimize False
+ zlib optimize Truethe output is colored, and it's written in the style of a git diff โ which means you can straight-up copy-paste it into a GitHub markdown code block with language diff and it renders gorgeously! here's that same thing as an example:
--- zlib@1.2.11/efzjziyc3dmb5h5u5azsthgbgog5mj7g
+++ zlib@1.2.11/sl7m27mzkbejtkrajigj3a3m37ygv4u2
@@ variant_value @@
- zlib optimize False
+ zlib optimize Trueiconic. so let's actually read the diff: it tells us our first zlib was built with ~optimize (False) and the second one was built with +optimize (True). you can't see it in these docs, but the output above is also colored based on whether the content is an addition (+) or a subtraction (-).
this is a tiny example, but you can see differences for any attributes on the installation spec. running spack diff A B means we'll see which spec attributes are on B but not on A (green) and which are on A but not on B (red). here's another example with an extra difference type โ version:
$ spack diff python@2.7.8 python@3.8.11
==> Warning: This interface is subject to change.
--- python@2.7.8/tsxdi6gl4lihp25qrm4d6nys3nypufbf
+++ python@3.8.11/yjtseru4nbpllbaxb46q7wfkyxbuvzxx
@@ variant_value @@
- python patches a8c52415a8b03c0e5f28b5d52ae498f7a7e602007db2b9554df28cd5685839b8
+ python patches 0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87
@@ version @@
- openssl 1.0.2u
+ openssl 1.1.1k
- python 2.7.8
+ python 3.8.11say you only cared about one kind of attribute up there โ version. you can tell the command to only output that one. just add the --attribute parameter (it defaults to all). filtering to just versions looks like this:
$ spack diff --attribute version python@2.7.8 python@3.8.11
==> Warning: This interface is subject to change.
--- python@2.7.8/tsxdi6gl4lihp25qrm4d6nys3nypufbf
+++ python@3.8.11/yjtseru4nbpllbaxb46q7wfkyxbuvzxx
@@ version @@
- openssl 1.0.2u
+ openssl 1.1.1k
- python 2.7.8
+ python 3.8.11and you can stack as many attributes as you want with multiple --attribute arguments โ and if you've got a lot of attributes, you can use -a for short ๐ซถ. finally, if you want the data as json (and maybe pipe it into an output file), just add --json:
$ spack diff --json python@2.7.8 python@3.8.11this data is gonna be way longer, because along with the differences for A vs. B and B vs. A, the JSON output also shows the intersection. ๐
๐ธ Using installed packages ๐ธ
there are several different ways to actually use your Spack packages once they're installed. like you've seen, Spack installs packages into long paths with hashes in them, so you need a way to get them onto your path. the easiest way is spack load, which we're getting into right now.
some more advanced ways to use Spack packages include:
- environments, which you can use to bundle a bunch of related packages so you can "activate" them all at once, and
- environment modules, which are super common on supercomputing clusters. Spack generates module files for every installation automatically, and you can customize how that goes.
๐ spack load / unload ๐
if you have shell support enabled, you can use the spack load command to quickly get a package onto your PATH.
for example, this adds the mpich package built with gcc to your path:
$ spack install mpich %gcc@4.4.7
# ... wait for install ...
$ spack load mpich %gcc@4.4.7
$ which mpicc
~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/mpich@3.0.4/bin/mpiccthese commands add the appropriate directories to your PATH and MANPATH according to the prefix inspections defined in your modules configuration. and when you're done with a package, you can type unload or unuse the same way:
$ spack unload mpich %gcc@4.4.7๐ญ Ambiguous specs ๐ญ
if a spec you use with load/unload is ambiguous โ meaning more than one installed package matches it โ then Spack will warn you:
$ spack load libelf
==> Error: libelf matches multiple packages.
Matching packages:
qmm4kso libelf@0.8.13%gcc@4.4.7 arch=linux-debian7-x86_64
cd2u6jt libelf@0.8.13%intel@15.0.0 arch=linux-debian7-x86_64
Use a more specific specyou've got two options, bestie: either type the spack load command again with a fully qualified argument, or add just enough extra constraints to pin down exactly one package. in the example above, the key differentiator is that one libelf was built with the Intel compiler while the other used gcc. so you could just type:
$ spack load libelf %intelto grab the one built with the Intel compiler. and if you want to be extremely specific, you can load it by its hash. for example, to load the first libelf above you'd run:
$ spack load /qmm4ksoto see which packages you've loaded into your environment, you'd use spack find --loaded:
$ spack find --loaded
==> 2 installed packages
-- linux-debian7 / gcc@4.4.7 ------------------------------------
libelf@0.8.13
-- linux-debian7 / intel@15.0.0 ---------------------------------
libelf@0.8.13you can also use spack load --list to get that same output, but it doesn't have the full set of query options that spack find offers.
we'll learn more about Spack's spec syntax in a later section โจ.
๐ Python packages and virtual environments ๐
Spack can install a huge number of Python packages. their names are typically prefixed with py-, and installing and using them is no different from any other package:
$ spack install py-numpy
$ spack load py-numpy
$ python3
>>> import numpythe spack load command sets the PATH variable so the right Python executable gets used, and makes sure that numpy and its dependencies can be found in the PYTHONPATH.
here's where Spack is different from other Python package managers: it installs every package into its own prefix. that's in contrast to pip, which installs all packages into the same prefix โ virtual environment or not.
for lots of users, virtual environments are more convenient than running spack load over and over, especially when you're working with multiple Python packages. and good news: Spack supports environments itself, which โ together with a view โ are no different from Python virtual environments ๐.
the recommended way of working with Python extensions like py-numpy is through Environments. this next example creates a Spack environment with numpy in the current working directory. it also drops a filesystem view in ./view, which is a more traditional combined prefix for all the packages in the environment:
$ spack env create --with-view view --dir .
$ spack -e . add py-numpy
$ spack -e . concretize
$ spack -e . installnow you can activate the environment and start using the packages:
$ spack env activate .
$ python3
>>> import numpythe environment view is also a virtual environment, which is super useful if you're sharing the environment with people who aren't familiar with Spack. they can either use the Python executable directly:
$ ./view/bin/python3
>>> import numpyor use the activation script:
$ source ./view/bin/activate
$ python3
>>> import numpyngl, in general there shouldn't be much difference between spack env activate and using the virtual environment. the main advantage of spack env activate is that it knows about way more packages than just Python packages, and it may set additional runtime variables that the virtual environment activation script doesn't cover.
see Environments (spack.yaml) for a much more in-depth description of Spack environments and customizations to views.
๐ Specs & dependencies ๐
so we know that spack install, spack uninstall and other commands take a package name with an optional version specifier. in Spack, that descriptor is called a spec. Spack uses specs to refer to a particular build configuration (or configurations) of a package. and specs are so much more than a package name and version โ you can use them to specify the compiler, compiler version, architecture, compile options, and dependency options for a build. this section is the full syntax tour, buckle up ๐.
here's an example of a much longer spec than anything we've seen so far:
mpileaks @1.2:1.4 %gcc@4.7.5 +debug -qt target=x86_64 ^callpath @1.1 %gcc@4.7.2you can put all the same modifiers on dependency specs that you would put on the root spec โ like, literally the same ones. versions, compilers, variants, architectures, all of it, just like any other spec. the rule is: specifiers stick to the nearest package name to their left. so in the example above, @1.1 and %gcc@4.7.2 are all about the callpath package, while @1.2:1.4, %gcc@4.7.5, +debug, -qt, and target=haswell os=CNL10 are all mpileaks's business. iykyk ๐
in the diagram above, mpileaks depends on mpich with an unspecified version, but packages can absolutely depend on other packages with constraints by adding more specifiers. for example, mpileaks could depend on mpich@1.2: if it can only build with version 1.2 or higher of mpich. slay, boundaries โจ
below are more details about the specifiers that you can add to specs.
๐ Version specifier ๐
a version specifier pkg@<specifier> comes after a package name and starts with @. it can be something abstract that matches multiple known versions, or a specific version. during concretization, Spack will pick the optimal version within the spec's constraints according to policies set for the particular Spack installation.
the version specifier can be a specific version, such as @=1.0.0 or @=1.2a7. or, it can be a range of versions, such as @1.0:1.5. version ranges are inclusive, so this example includes both 1.0 and any 1.5.x version. version ranges can be unbounded, e.g. @:3 means any version up to and including 3. this would include 3.4 and 3.4.2. similarly, @4.2: means any version above and including 4.2. as a short-hand, @3 is equivalent to the range @3:3 and includes any version with major version 3. bestie, it's giving "math but make it fashion" ๐ง
versions are ordered lexicograpically by its components. for more details on the order, see the packaging guide.
notice that you can distinguish between the specific version @=3.2 and the range @3.2. this is useful for packages that follow a versioning scheme that omits the zero patch version number: 3.2, 3.2.1, 3.2.2, etc. in general it is preferable to use the range syntax @3.2, since ranges also match versions with one-off suffixes, such as 3.2-custom. lowkey smarter, no cap
a version specifier can also be a list of ranges and specific versions, separated by commas. for example, @1.0:1.5,=1.7.1 matches any version in the range 1.0:1.5 and the specific version 1.7.1. obsessed with the range ๐
๐ฆ Git versions ๐ฆ
for packages with a git attribute, git references may be specified instead of a numerical version i.e. branches, tags and commits. Spack will stage and build based off the git reference provided. acceptable syntaxes for this are:
# commit hashes
foo@abcdef1234abcdef1234abcdef1234abcdef1234 # 40 character hashes are automatically treated as git commits
foo@git.abcdef1234abcdef1234abcdef1234abcdef1234
# branches and tags
foo@git.develop # use the develop branch
foo@git.0.19 # use the 0.19 tagSpack always needs to associate a Spack version with the git reference, which is used for version comparison. this Spack version is heuristically taken from the closest valid git tag among ancestors of the git ref.
once a Spack version is associated with a git ref, it always printed with the git ref. for example, if the commit @git.abcdefg is tagged 0.19, then the spec will be shown as @git.abcdefg=0.19.
if the git ref is not exactly a tag, then the distance to the nearest tag is also part of the resolved version. @git.abcdefg=0.19.git.8 means that the commit is 8 commits away from the 0.19 tag. it's giving "how far are we from home" ๐ญ
in cases where Spack cannot resolve a sensible version from a git ref, users can specify the Spack version to use for the git ref. this is done by appending = and the Spack version to the git ref. for example:
foo@git.my_ref=3.2 # use the my_ref tag or branch, but treat it as version 3.2 for version comparisons
foo@git.abcdef1234abcdef1234abcdef1234abcdef1234=develop # use the given commit, but treat it as develop for version comparisonsdetails about how versions are compared and how Spack determines if one version is less than another are discussed in the developer guide.
๐ Compiler specifier ๐
a compiler specifier comes somewhere after a package name and starts with %. it tells Spack what compiler(s) a particular package should be built with. after the % should come the name of some registered Spack compiler. this might include gcc, or intel, but the specific compilers available depend on the site. you can run spack compilers to get a list; more on this below.
the compiler spec can be followed by an optional compiler version. a compiler version specifier looks exactly like a package version specifier. version specifiers will associate with the nearest package name or compiler specifier to their left in the spec. main character energy for that nearest-left rule, honestly โจ
if the compiler spec is omitted, Spack will choose a default compiler based on site policies.
๐ Variants ๐
variants are named options associated with a particular package. they are optional, as each package must provide default values for each variant it makes available. variants can be specified using a flexible parameter syntax name=<value>. for example, spack install mercury debug=True will install mercury built with debug flags. the names of particular variants available for a package depend on what was provided by the package author. spack info <package> will provide information on what build variants are available. go look it up bestie, the info is right there ๐ซถ
for compatibility with earlier versions, variants which happen to be boolean in nature can be specified by a syntax that represents turning options on and off. for example, in the previous spec we could have supplied mercury +debug with the same effect of enabling the debug compile time option for the libelf package. (yes, the manpage really says libelf there โ not me, i'm just the messenger ๐)
depending on the package a variant may have any default value. for mercury here, debug is False by default, and we turned it on with debug=True or +debug. if a variant is True by default you can turn it off by either adding -name or ~name to the spec.
there are two syntaxes here because, depending on context, ~ and - may mean different things. in most shells, the following will result in the shell performing home directory substitution:
mpileaks ~debug # shell may try to substitute this!
mpileaks~debug # use this insteadโ ๏ธ ok but seriously โ that space before~debugis a trap. the shell will try to expand~debuginto some home directory and your spec will get absolutely cooked. no space, no problem. ๐
Okay so there's one little gremlin to watch out for with ~ โ if there is a user called debug on the system, the ~ will be incorrectly expanded. ๐ญ In that situation, you'd want to write libelf -debug. But plot twist: - can be ambiguous when it's shoved in after a package name with no spaces:
mpileaks-debug # wrong!
mpileaks -debug # rightSpack allows the - character to be part of package names, so the first line up there gets read as a request for the mpileaks-debug package, not a request for mpileaks built without debug options. ๐ In this scenario you should write mpileaks~debug to dodge the ambiguity entirely.
When spack normalizes specs, it prints them out with no spaces for boolean variants using the backwards compatibility syntax, and uses only ~ for disabled boolean variants. The - and the spaces on the command line are there purely for convenience and legibility. ๐
Spack also lets variants propagate their value to the package's dependency by using ++, --, and ~~ for boolean variants. For example, for a debug variant:
mpileaks ++debug # enabled debug will be propagated to dependencies
mpileaks +debug # only mpileaks will have debug enabledTo propagate the value of non-boolean variants, Spack uses name==value. For example, for the stackstart variant:
mpileaks stackstart==4 # variant will be propagated to dependencies
mpileaks stackstart=4 # only mpileaks will have this variant value๐ ๏ธ Compiler Flags
Compiler flags use the exact same syntax as non-boolean variants, but bestie they serve a totally different purpose. A variant's function is set by the package, whereas compiler flags are used by the compiler wrappers to inject flags into the compile line of the build. โจ Plus, compiler flags can be inherited by dependencies by using ==. So spack install libdwarf cppflags=="-g" will install both libdwarf and libelf with the -g flag injected into their compile line. ๐ซถ
NOTE: versions of spack prior to0.19.0will propagate compiler flags using the=syntax.
Notice that the value of the compiler flags must be quoted if it contains any spaces. Any of cppflags=-O3, cppflags="-O3", cppflags='-O3', and cppflags="-O3 -fPIC" are acceptable, but cppflags=-O3 -fPIC is not. ๐ค Additionally, if the value of the compiler flags is not the last thing on the line, it must be followed by a space. The command spack install libelf cppflags="-O3"%intel will be interpreted as an attempt to set cppflags="-O3%intel". ๐
The six compiler flags are injected in the order of implicit make commands in GNU Autotools. If all flags are set, the order is $cppflags $cflags|$cxxflags $ldflags <command> $ldlibs for C and C++ and $fflags $cppflags $ldflags <command> $ldlibs for Fortran. ๐
๐ง Compiler Environment Variables and Additional RPATHs
Sometimes compilers require setting special environment variables to operate correctly. Spack handles these cases by allowing custom environment modifications in the environment attribute of the compiler configuration section. See also the Environment Modifications section of the configuration files docs for more info. ๐
It's also possible to specify additional RPATHs that the compiler will add to all executables generated by that compiler. This is useful for forcing certain compilers to RPATH their own runtime libraries, so that executables will run without the need to set LD_LIBRARY_PATH. ๐ธ
compilers:
- compiler:
spec: gcc@4.9.3
paths:
cc: /opt/gcc/bin/gcc
c++: /opt/gcc/bin/g++
f77: /opt/gcc/bin/gfortran
fc: /opt/gcc/bin/gfortran
environment:
unset:
- BAD_VARIABLE
set:
GOOD_VARIABLE_NUM: 1
GOOD_VARIABLE_STR: good
prepend_path:
PATH: /path/to/binutils
append_path:
LD_LIBRARY_PATH: /opt/gcc/lib
extra_rpaths:
- /path/to/some/compiler/runtime/directory
- /path/to/some/other/compiler/runtime/directory๐๏ธ Architecture Specifiers
Each node in the dependency graph of a spec has an architecture attribute. That attribute is a triplet of platform, operating system and processor. You can specify the elements either separately, by using the reserved keywords platform, os and target:
$ spack install libelf platform=linux
$ spack install libelf os=ubuntu18.04
$ spack install libelf target=broadwellor together by using the reserved keyword arch:
$ spack install libelf arch=cray-CNL10-haswellNormally you don't have to bother specifying the architecture if you're installing software for your current host โ in that case the values get detected automatically, no stress. ๐ If you need fine-grained control over which packages use which targets (or over all packages' default target), see Package Preferences.
๐ฅ๏ธ Cray machines
The situation is a little bit different for Cray machines, and a detailed explanation on how the architecture can be set on them can be found at Spack on Cray.
๐ฌ Support for Specific Microarchitectures
Spack knows how to detect and optimize for many specific microarchitectures (including recent Intel, AMD and IBM chips) and encodes this information in the target portion of the architecture specification. A complete list of the microarchitectures known to Spack can be obtained in the following way:
$ spack arch --known-targets
Generic architectures (families)
aarch64 armv8.1a armv8.3a armv8.5a ppc ppc64le riscv64 sparc64 x86_64 x86_64_v3
arm armv8.2a armv8.4a armv9.0a ppc64 ppcle sparc x86 x86_64_v2 x86_64_v4
GenuineIntel - x86
i686 pentium2 pentium3 pentium4 prescott
GenuineIntel - x86_64
nocona nehalem sandybridge haswell skylake cannonlake cascadelake sapphirerapids
core2 westmere ivybridge broadwell mic_knl skylake_avx512 icelake
AuthenticAMD - x86_64
k10 bulldozer piledriver zen steamroller zen2 zen3 excavator zen4
IBM - ppc64
power7 power8 power9 power10
IBM - ppc64le
power8le power9le power10le
Cavium - aarch64
thunderx2
Fujitsu - aarch64
a64fx
ARM - aarch64
cortex_a72 neoverse_n1 neoverse_v1 neoverse_v2
Apple - aarch64
m1 m2
SiFive - riscv64
u74mcWhen a spec is installed, Spack matches the compiler being used with the microarchitecture being targeted to inject appropriate optimization flags at compile time. Giving a command such as:
$ spack install zlib%gcc@9.0.1 target=icelakewill produce compilation lines similar to:
$ /usr/bin/gcc-9 -march=icelake-client -mtune=icelake-client -c ztest10532.c
$ /usr/bin/gcc-9 -march=icelake-client -mtune=icelake-client -c -fPIC -O2 ztest10532.
...where the flags -march=icelake-client -mtune=icelake-client are injected by Spack based on the requested target and compiler. ๐ฅ
If Spack knows that the requested compiler can't optimize for the current target or can't build binaries for that target at all, it will exit with a meaningful error message:
$ spack install zlib%gcc@5.5.0 target=icelake
==> Error: cannot produce optimized binary for micro-architecture "icelake" with gcc@5.5.0 [supported compiler versions are 8:]When instead an old compiler is selected on a recent enough microarchitecture but there's no explicit target specification, Spack will optimize for the best match it can find instead of failing:
$ spack arch
linux-ubuntu18.04-broadwell
$ spack spec zlib%gcc@4.8
Input spec
--------------------------------
zlib%gcc@4.8
Concretized
--------------------------------
zlib@1.2.11%gcc@4.8+optimize+pic+shared arch=linux-ubuntu18.04-haswell
$ spack spec zlib%gcc@9.0.1
Input spec
--------------------------------
zlib%gcc@9.0.1
Concretized
--------------------------------
zlib@1.2.11%gcc@9.0.1+optimize+pic+shared arch=linux-ubuntu18.04-broadwellIn the snippet above, for instance, the microarchitecture was demoted to haswell when compiling with gcc@4.8 since support to optimize for broadwell starts from gcc@4.9:. ๐ง
Finally, if Spack has no information to match compiler and target, it will proceed with the installation but avoid injecting any microarchitecture specific flags.
WARNING: Currently, Spack doesn't print any warning to the user if it has no information on which optimization flags should be used for a given compiler. This behavior might change in the future. ๐
๐ฎ Virtual Dependencies
So the dependency graph for mpileaks we saw above wasn't quite accurate, bestie. mpileaks uses MPI, which is an interface that has many different implementations. Above, we showed mpileaks and callpath depending on mpich, which is one particular implementation of MPI. But we could build either with another implementation, such as openmpi or mvapich. ๐คฏ
Spack represents interfaces like this using virtual dependencies. The real dependency DAG for mpileaks looks like this: [graph]
Notice that mpich has now been replaced with mpi. There is no real MPI package, but some packages provide the MPI interface, and these packages can be substituted in for mpi when mpileaks is built. ๐
You can see what virtual packages a particular package provides by getting info on it:
$ spack info --virtuals mpich
AutotoolsPackage: mpich
Description:
MPICH is a high performance and widely portable implementation of the
Message Passing Interface (MPI) standard.
Homepage: https://www.mpich.org
Preferred version:
4.2.1 https://www.mpich.org/static/downloads/4.2.1/mpich-4.2.1.tar.gzokay so here's what spack info mpich literally dumps at you โ and yes we're reading the whole thing because it's giving โจloreโจ and also bestie needs to see the receipts:
Safe versions:
develop [git] https://github.com/pmodels/mpich.git
4.2.1 https://www.mpich.org/static/downloads/4.2.1/mpich-4.2.1.tar.gz
4.2.0 https://www.mpich.org/static/downloads/4.2.0/mpich-4.2.0.tar.gz
4.1.2 https://www.mpich.org/static/downloads/4.1.2/mpich-4.1.2.tar.gz
4.1.1 https://www.mpich.org/static/downloads/4.1.1/mpich-4.1.1.tar.gz
4.1 https://www.mpich.org/static/downloads/4.1/mpich-4.1.tar.gz
4.0.3 https://www.mpich.org/static/downloads/4.0.3/mpich-4.0.3.tar.gz
4.0.2 https://www.mpich.org/static/downloads/4.0.2/mpich-4.0.2.tar.gz
4.0.1 https://www.mpich.org/static/downloads/4.0.1/mpich-4.0.1.tar.gz
4.0 https://www.mpich.org/static/downloads/4.0/mpich-4.0.tar.gz
3.4.3 https://www.mpich.org/static/downloads/3.4.3/mpich-3.4.3.tar.gz
3.4.2 https://www.mpich.org/static/downloads/3.4.2/mpich-3.4.2.tar.gz
3.4.1 https://www.mpich.org/static/downloads/3.4.1/mpich-3.4.1.tar.gz
3.4 https://www.mpich.org/static/downloads/3.4/mpich-3.4.tar.gz
3.3.2 https://www.mpich.org/static/downloads/3.3.2/mpich-3.3.2.tar.gz
3.3.1 https://www.mpich.org/static/downloads/3.3.1/mpich-3.3.1.tar.gz
3.3 https://www.mpich.org/static/downloads/3.3/mpich-3.3.tar.gz
3.2.1 https://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1.tar.gz
3.2 https://www.mpich.org/static/downloads/3.2/mpich-3.2.tar.gz
3.1.4 https://www.mpich.org/static/downloads/3.1.4/mpich-3.1.4.tar.gz
3.1.3 https://www.mpich.org/static/downloads/3.1.3/mpich-3.1.3.tar.gz
3.1.2 https://www.mpich.org/static/downloads/3.1.2/mpich-3.1.2.tar.gz
3.1.1 https://www.mpich.org/static/downloads/3.1.1/mpich-3.1.1.tar.gz
3.1 https://www.mpich.org/static/downloads/3.1/mpich-3.1.tar.gz
3.0.4 https://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz
Deprecated versions:
None
Variants:
argobots [false] false, true
Enable Argobots support
build_system [autotools] autotools
Build systems supported by the package
cuda [false] false, true
Build with CUDA
device [ch4] ch3, ch3:sock, ch4
Abstract Device Interface (ADI)
implementation. The ch4 device is in experimental state for versions
before 3.4.
fortran [true] false, true
Enable Fortran support
hwloc [true] false, true
Use external hwloc package
hydra [true] false, true
Build the hydra process manager
libxml2 [true] false, true
Use libxml2 for XML support instead of the custom minimalistic implementation
netmod [ofi] mxm, ofi, tcp, ucx
Network module. Only single netmod builds are
supported, and netmod is ignored if device is ch3:sock.
pci [true] false, true
Support analyzing devices on PCI bus
pmi [default] cray, default, pmi, pmi2, pmix
PMI interface.
rocm [false] false, true
Enable ROCm support
romio [true] false, true
Enable ROMIO MPI I/O implementation
slurm [false] false, true
Enable Slurm support
verbs [false] false, true
Build support for OpenFabrics verbs.
wrapperrpath [true] false, true
Enable wrapper rpath
when +cuda
cuda_arch [none] none, 10, 11, 12, 13, 20, 21, 30, 32, 35, 37, 50, 52, 53, 60, 61, 62, 70, 72, 75,
80, 86, 87, 89, 90
CUDA architecture
when +rocm
amdgpu_target [none] none, gfx1010, gfx1011, gfx1012, gfx1013, gfx1030, gfx1031, gfx1032, gfx1033,
gfx1034, gfx1035, gfx1036, gfx1100, gfx1101, gfx1102, gfx1103, gfx701, gfx801,
gfx802, gfx803, gfx900, gfx900:xnack-, gfx902, gfx904, gfx906, gfx906:xnack-,
gfx908, gfx908:xnack-, gfx909, gfx90a, gfx90a:xnack+, gfx90a:xnack-, gfx90c,
gfx940, gfx941, gfx942
AMD GPU architecture
when @3.3: device=ch4 netmod=ucx
hcoll [false] false, true
Enable support for Mellanox HCOLL accelerated collective operations library
when @3.4:
datatype-engine [auto] auto, dataloop, yaksa
controls the datatype engine to use
xpmem [false] false, true
Enable XPMEM support
when @4: device=ch4
vci [false] false, true
Enable multiple VCI (virtual communication interface) critical sections to improve performance of
applications that do heavy concurrent MPIcommunications. Set MPIR_CVAR_CH4_NUM_VCIS=<N> to enable multiple
vcis at runtime.
Build Dependencies:
argobots cray-pmi gmake hip libfabric libxml2 mxm python xpmem
autoconf cuda gnuconfig hsa-rocr-dev libpciaccess llvm-amdgpu pkgconfig slurm yaksa
automake findutils hcoll hwloc libtool m4 pmix ucx
Link Dependencies:
argobots cuda hip hwloc libpciaccess llvm-amdgpu pmix ucx yaksa
cray-pmi hcoll hsa-rocr-dev libfabric libxml2 mxm slurm xpmem
Run Dependencies:
None
Virtual Packages:
mpich provides mpi@:4.0
mpich@:3.2 provides mpi@:3.1
mpich@:3.1 provides mpi@:3.0
mpich@:1.2 provides mpi@:2.2
mpich@:1.1 provides mpi@:2.1
mpich@:1.0 provides mpi@:2.0
Licenses:
Unlicenselike, that's a lot of text but every single line is doing its job ๐
Spack is iconic because its virtual packages can be versioned, just like regular packages. bestie that's the whole plot twist. A particular version of a package may provide a particular version of a virtual package โ and if you look up at that output, you can literally clock it: mpich versions 1 and above provide all mpi interface versions up to 1, and mpich versions 3 and above provide mpi versions up to 3. ๐
So a package can depend on a particular version of a virtual package. example so it clicks: if an application needs MPI-2 functions, it can depend on mpi@2: to indicate that it needs some implementation that provides MPI-2 functions. โจ
constraining virtual packages ๐ซถ
When you're installing a package that depends on a virtual package, you have choices: you can opt to specify the particular provider you want to use, or you can let Spack pick. iconic either way.
For example, if you just type this:
$ spack install mpileaksThen spack will pick a provider for you according to site policies. lowkey handing the aux to your sysadmin. If you really want a particular version โ say mpich โ then you could run this instead:
$ spack install mpileaks ^mpichThis forces spack to use some version of mpich for its implementation. And as always, you can be even more specific and require a particular mpich version:
$ spack install mpileaks ^mpich@3The mpileaks package in particular only needs MPI-1 commands, so any MPI implementation will do. but ok listen ๐ โ if another package depends on mpi@2 and you try to give it an insufficient MPI implementation (e.g., one that provides only mpi@:1), then Spack will raise an error. Likewise, if you try to plug in some package that doesn't provide MPI, Spack will raise an error. it's giving rejection and honestly? deserved ๐
explicit binding of virtual dependencies ๐ฅ
There are packages that provide more than just one virtual dependency. When interacting with them, users might want to utilize just a subset of what they could provide, and use other providers for virtuals they need.
It is possible to be more explicit and tell Spack which dependency should provide which virtual, using a special syntax:
$ spack spec strumpack ^[virtuals=mpi] intel-parallel-studio+mkl ^[virtuals=lapack] openblasConcretizing the spec above produces the following DAG:
[image]
where intel-parallel-studio could provide mpi, lapack, and blas but is used only for the former. The lapack and blas dependencies are satisfied by openblas. see how that works? one package, but it's only pulling one of its jobs. main character behavior honestly ๐
specifying specs by hash ๐ฆ
Complicated specs can become cumbersome to enter on the command line, especially when many of the qualifications are necessary to distinguish between similar installs. To avoid this, when referencing an existing spec, Spack allows you to reference specs by their hash. We previously discussed the spec hash that Spack computes. In place of a spec in any command, substitute /<hash> where <hash> is any amount from the beginning of a spec hash.
For example, lets say that you accidentally installed two different mvapich2 installations. If you want to uninstall one of them but don't know what the difference is, you can run:
$ spack find --long mvapich2
==> 2 installed packages.
-- linux-centos7-x86_64 / gcc@6.3.0 ----------
qmt35td mvapich2@2.2%gcc
er3die3 mvapich2@2.2%gccYou can then uninstall the latter installation using:
$ spack uninstall /er3die3Or, if you want to build with a specific installation as a dependency, you can use:
$ spack install trilinos ^/er3die3If the given spec hash is sufficiently long as to be unique, Spack will replace the reference with the spec to which it refers. Otherwise, it will prompt for a more qualified hash. (so like, give it more characters bestie, it's asking nicely ๐)
ngl, one caveat that matters: note that this will not work to reinstall a dependency uninstalled by spack uninstall --force. ๐จ
spack providers ๐ธ
You can see what packages provide a particular virtual package using spack providers. If you wanted to see what packages provide mpi, you would just run:
$ spack providers mpi
cray-mpich intel-mpi mpich@:1.0 mpich@:3.2 mpt mvapich mvapich2-gdr openmpi@1.6.5
cray-mvapich2 intel-oneapi-mpi mpich@:1.1 mpich mpt@1: mvapich2 mvapich2x openmpi@1.7.5:
fujitsu-mpi intel-parallel-studio mpich@:1.2 mpilander mpt@3: mvapich2@2.1: nvhpc openmpi@2.0.0:
hpcx-mpi mpi-serial mpich@:3.1 mpitrampoline msmpi mvapich2@2.3: openmpi spectrum-mpiAnd if you only wanted to see packages that provide MPI-2, you would add a version specifier to the spec:
So like, check out what happens when you bump the version filter up โ the packages that only do old MPI justโฆ don't show up anymore:
$ spack providers mpi@2
hpcx-mpi mpi-serial mpich@:3.1 mpt mvapich2 mvapich2x openmpi@1.7.5:
intel-mpi mpich@:1.0 mpich@:3.2 mpt@3: mvapich2@2.1: nvhpc openmpi@2.0.0:
intel-oneapi-mpi mpich@:1.1 mpich msmpi mvapich2@2.3: openmpi spectrum-mpi
intel-parallel-studio mpich@:1.2 mpilander mvapich mvapich2-gdr openmpi@1.6.5Notice that the package versions that provide insufficient MPI versions are now filtered out. โจ Like, the vibe check literally removed them. Gone. Not invited to the group chat anymore. ๐
Deprecating insecure packages ๐ฉ
Okay bestie, this is the big girl command. spack deprecate lets you yeet an insecure package out of existence while keeping its dependents mostly okay. ๐ธ
โ ๏ธ WARNING: The spack deprecate command is designed for use only in extraordinary circumstances. This is a VERY big hammer to be used with care. ๐จ๐ญ
So how does it actually work? The spack deprecate command will remove one package and replace it with another by replacing the deprecated package's prefix with a link to the deprecator package's prefix. It's giving "we moved your stuff into a new apartment and left a forwarding address" vibes. ๐ โก๏ธ๐
โ ๏ธ WARNING: The spack deprecate command makes no promises about binary compatibility. It is up to the user to ensure the deprecator is suitable for the deprecated package. ๐ (yes, you have to double-check it, not Spack, sorry bestie)
Spack tracks concrete deprecated specs and ensures that no future packages concretize to a deprecated spec. ๐ซถ So nothing new will accidentally latch onto the deprecated one.
Now for the arguments, because order matters here:
- The first spec you hand to
spack deprecateis the package to deprecate. It's an abstract spec that must describe a single installed package. - The second spec argument is the deprecator spec. By default it must be an abstract spec that describes a single installed package too โ but with
-i/--install-deprecatorit can be any abstract spec that Spack will install and then use as the deprecator. โจ - The
-I/--no-install-deprecatoroption will ensure the default behavior. (iykyk, it's the "no thanks, keep it classic" flag)
By default, spack deprecate will deprecate all dependencies of the deprecated spec, replacing each by the dependency of the same name in the deprecator spec. And the flags for that:
-d/--dependenciesโ ensures the default (deprecate the deps too).-D/--no-dependenciesโ deprecates only the root of the deprecate spec in favor of the root of the deprecator spec. Just the main character, no entourage. ๐
And for the linking situation โ spack deprecate can use symbolic links or hard links. Default behavior is symbolic links, but the -l/--link-type flag can take options hard or soft. ๐
Verifying installations ๐
The spack verify command can be used to verify the validity of Spack-installed packages any time after installation. Like a wellness check for your installs. ๐ฉบ
At installation time, Spack creates a manifest of every file in the installation prefix. And here's exactly what it keeps tabs on, no cap:
- For links, Spack tracks the
mode,ownership, anddestination. - For directories, Spack tracks the
mode, andownership. - For files, Spack tracks the
mode,ownership,modification time,hash, andsize. ๐
Then the spack verify command will check, for every file in each package, whether any of those attributes have changed. It will also check for newly added files or deleted files from the installation prefix. Spack can either check all installed packages using the -a,--all or accept specs listed on the command line to verify. ๐
The spack verify command can also verify for individual files that they haven't been altered since installation time. If the given file is not in a Spack installation prefix, Spack will report that it is not owned by any package. To check individual files instead of specs, use the -f,--files option. ๐
Spack installation manifests are part of the tarball signed by Spack for binary package distribution. When installed from a binary package, Spack uses the packaged installation manifest instead of creating one at install time. ๐
The spack verify command also accepts:
-l,--localโ to check only local packages (as opposed to those used transparently from upstream spack instances).-j,--jsonโ to output machine-readable json data for any errors. ๐ค
Filesystem requirements ๐พ
By default, Spack needs to be run from a filesystem that supports flock locking semantics. Nearly all local filesystems and recent versions of NFS support this, but parallel filesystems or NFS volumes may be configured without flock support enabled. ๐ฌ
You can determine how your filesystems are mounted with mount. The output for a Lustre filesystem might look like this:
$ mount | grep lscratch
mds1-lnet0@o2ib100:/lsd on /p/lscratchd type lustre (rw,nosuid,lazystatfs,flock)
mds2-lnet0@o2ib100:/lse on /p/lscratche type lustre (rw,nosuid,lazystatfs,flock)Note the flock option on both Lustre mounts. ๐ That's the good stuff.
If you do not see this or a similar option for your filesystem, you have a few options:
- First, you can move your Spack installation to a filesystem that supports locking. ๐ฆโก๏ธ
- Second, you could ask your system administrator to enable
flockfor your filesystem. ๐
If none of those work, you can disable locking in one of two ways:
- Run Spack with the
-Lor--disable-locksoption to disable locks on a call-by-call basis. - Edit
config.yamland set thelocksoption tofalseto always disable locking.
๐จ WARNING: If you disable locking, concurrent instances of Spack will have no way to avoid stepping on each other. You must ensure that there is only one instance of Spack running at a time. Otherwise, Spack may end up with a corrupted database file, or you may not be able to see all installed packages in commands like spack find. ๐
๐ฉน If you are unfortunate enough to run into this situation, you may be able to fix it by running spack reindex.
This issue typically manifests with the error below:
$ ./spack find
Traceback (most recent call last):
File "./spack", line 176, in <module>
main()
File "./spack", line 154,' in main
return_val = command(parser, args)
File "./spack/lib/spack/spack/cmd/find.py", line 170, in find
specs = set(spack.installed_db.query(\**q_args))
File "./spack/lib/spack/spack/database.py", line 551, in query
with self.read_transaction():
File "./spack/lib/spack/spack/database.py", line 598, in __enter__
if self._enter() and self._acquire_fn:
File "./spack/lib/spack/spack/database.py", line 608, in _enter
return self._db.lock.acquire_read(self._timeout)
File "./spack/lib/spack/llnl/util/lock.py", line 103, in acquire_read
self._lock(fcntl.LOCK_SH, timeout) # can raise LockError.
File "./spack/lib/spack/llnl/util/lock.py", line 64, in _lock
fcntl.lockf(self._fd, op | fcntl.LOCK_NB)
IOError: [Errno 38] Function not implementedA nicer error message is TBD in future versions of Spack. ๐ญ (we know, bestie, we know)
Troubleshooting ๐ ๏ธ
Meet the spack audit command:
$ spack audit -h
usage: spack audit [-h] SUBCOMMAND ...
audit configuration files, packages, etc.
positional arguments:
SUBCOMMAND
configs audit configuration files
externals check external detection in packages
packages-https check https in packages
packages audit package recipes
list list available checks and exits
options:
-h, --help show this help message and exitIt can be used to detect a number of configuration issues. โจ This command detects configuration settings which might not be strictly wrong but are not likely to be useful outside of special cases. (like, technically valid but why, bestie ๐ )
It can also be used to detect dependency issues with packages โ for example cases where a package constrains a dependency with a variant that doesn't exist (in this case Spack could report the problem ahead of time but automatically performing the check would slow down most runs of Spack). ๐ So yeah, it's opt-in for a reason.
A detailed list of the checks currently implemented for each subcommand can be printed with:
$ spack -v audit list
generic:
Generic checks relying on global variables
configs:
Sanity checks on compilers.yaml
1. Report compilers with the same spec and two different definitions
Sanity checks on packages.yaml
1. Search for duplicate specs declared as externals
2. Search package preferences deprecated in v0.21 (and slated for removal in v0.23)
3. Warns if variant preferences have mismatched types or names.
4. Warns if the wrong name is used for an external spec
packages:
Sanity checks on specs used in directives
1. Ensure stand-alone test method is not included in build-time callbacks
2. Ensure that patches fetched from GitHub and GitLab have stable sha256
hashes.
3. Report unknown or wrong variants in directives for this package
4. Reports issues with 'depends_on' directives.
Issues might be unknown dependencies, unknown variants or variant values, or declaration
of nested dependencies.
5. Ensures that variant defaults are present and parsable from cli
6. Ensures that all variants have a description.
7. Report if version constraints used in directives are not satisfiable
8. Reports named specs in the 'when=' attribute of a directive.
Note that 'conflicts' is the only directive allowing that.
Sanity checks on reserved attributes of packages
1. Ensure that packages don't override reserved names
Sanity checks on properties a package should maintain
1. Ensure package names are lowercase and consistent
2. Ensure that package objects are pickleable
3. Ensure that all packages can unparse and that unparsed code is valid Python
4. Ensure all versions in a package can produce a fetcher
5. Ensure the package has a docstring and no fixmes
6. Ensure no packages use md5 checksums
7. Ensure that methods modifying the build environment are ported to builder classes.
packages-https:
Sanity checks on https checks of package urls, etc.
1. Check for correctness of links
externals:
Sanity checks for external software detection
1. Test drive external detection for packagesDepending on the use case, users might run the appropriate subcommands to obtain diagnostics. Issues, if found, are reported to stdout:
% spack audit packages lammps
PKG-DIRECTIVES: 1 issue found
1. lammps: wrong variant in "conflicts" directive
the variant 'adios' does not exist
in /home/spack/spack/var/spack/repos/builtin/packages/lammps/package.pySee? It literally snitched on the package. ๐ต๏ธโโ๏ธ๐ฅ
Getting Help ๐ซถ
spack help
If you don't find what you need here, the help subcommand will print out out a list of all of spack's options and subcommands: ๐
$ spack help
usage: spack [-hkV] [--color {always,never,auto}] COMMAND ...
A flexible package manager that supports multiple versions,
configurations, platforms, and compilers.
These are common spack commands:- query packages:
listโ lists out all the available packages and searches through theminfoโ spills the full tea on one specific packagefindโ lists and searches the packages you've already got installed
- build packages:
installโ builds and installs packagesuninstallโ yeets the installed onesgcโ clears out specs that are no longer neededspecโ shows you what would get installed, given a spec
- configuration:
externalโ manages external packages in the Spack configuration
- environments:
envโ manages virtual environmentsviewโ projects packages to a compact naming scheme on the filesystem
- create packages:
createโ makes a brand new package fileeditโ opens package files in$EDITOR
- system:
archโ prints architecture info about this machineauditโ audits configuration files, packages, etc.compilersโ lists the available compilers
- user environment:
loadโ adds a package to the user environmentmoduleโ generates/manages module filesunloadโ removes a package from the user environment
- options:
--color {always,never,auto}โ when to colorize output (default:auto)-V,--versionโ shows the version number and dips-h,--helpโ shows this help message and exits-k,--insecureโ does not check ssl certificates when downloading
- more help:
spack help --allโ lists every single command and optionspack help <command>โ help on one specific commandspack help --specโ help on the package specification syntaxspack docsโ openshttps://spack.rtfd.io/in a browser ๐ฆ
Adding an argument, like spack help <subcommand>, prints out the usage info for that particular subcommand. Obsessed with a self-explaining CLI, honestly:
$ spack help install
usage: spack install [-h] [--only {package,dependencies}] [-u UNTIL] [-j JOBS] [--overwrite] [--fail-fast]
[--keep-prefix] [--keep-stage] [--dont-restage] [--use-cache | --no-cache | --cache-only |
--use-buildcache [{auto,only,never},][package:{auto,only,never},][dependencies:{auto,only,never}]]
[--include-build-deps] [--no-check-signature] [--show-log-on-error] [--source] [-n] [-v] [--fake]
[--only-concrete] [--add | --no-add] [-f SPEC_YAML_FILE] [--clean | --dirty] [--test {root,all}]
[--log-format {junit,cdash}] [--log-file LOG_FILE] [--help-cdash] [-y] [-U] [--reuse]
[--fresh-roots] [--deprecated]
...build and install packages โ that's the whole vibe of install, bestie.
๐ positional arguments
specโ the package spec
๐ options
--addโ (with environment) adds the spec to the environment as a root--cache-onlyโ only installs the package from binary mirrors--cleanโ unsets harmful variables in the build environment (default)--dirtyโ preserves your user environment inside spack's build environment (danger! ๐ญ)--dont-restageโ if a partial install gets detected, doesn't delete the prior state--fail-fastโ stops all the builds if any build fails (the default is best effort)--fakeโ a fake install, purely for debug purposes--help-cdashโ shows usage instructions for CDash reporting--include-build-depsโ includes build deps when installing from cache, which is useful for CI pipeline troubleshooting--keep-prefixโ doesn't remove the install prefix if the installation fails--keep-stageโ doesn't remove the build stage if the installation succeeds--log-file LOG_FILEโ filename for the log file--log-format {junit,cdash}โ the format used for log files--no-addโ (with environment) does not add the spec to the environment as a root--no-cacheโ doesn't check for pre-built Spack packages in mirrors--no-check-signatureโ doesn't check signatures of binary packages (overrides mirror config)--only {package,dependencies}โ picks the mode of installation
The default is to install the package along with all of its dependencies. Alternatively you can choose to install only the package, or only the dependencies. Pick your fighter ๐
--only-concreteโ (with environment) only installs specs that are already concretized--overwriteโ reinstalls an existing spec, even if it has dependents--show-log-on-errorโ prints the full build log to stderr if the build fails--sourceโ installs source files in the prefix--test {root,all}โ runs tests on only root packages, or on all packages--use-buildcache [{auto,only,never},][package:{auto,only,never},][dependencies:{auto,only,never}]โ selects the mode of buildcache for thepackageand for thedependencies
default:package:auto,dependencies:auto-autobehaves like--use-cache-onlybehaves like--cache-only-neverbehaves like--no-cache
--use-cacheโ checks for pre-built Spack packages in mirrors (default)-f,--file SPEC_YAML_FILEโ reads specs to install from.yamlfiles-h,--helpโ shows this help message and exits-j,--jobs JOBSโ explicitly sets the number of parallel jobs-n,--no-checksumโ does not use checksums to verify downloaded files (unsafe ๐)-u,--until UNTILโ the phase to stop after when installing (defaultNone)-v,--verboseโ displays verbose build output while installing-y,--yes-to-allโ assumes"yes"is the answer to every confirmation request
โจ concretizer arguments
--deprecatedโ allows the concretizer to select deprecated versions--fresh-roots,--reuse-depsโ concretizes with fresh roots and reused dependencies--reuseโ reuses installed packages/buildcaches when possible-U,--freshโ doesn't reuse installed deps; builds the newest configuration
Alternately, you can use spack --help instead of spack help, or spack <subcommand> --help to get help on a particular subcommand. Same energy, different spelling, iykyk ๐ซถ
๐ SPACK FOR HOMEBREW/CONDA USERS ๐
okay so Spack is an incredibly powerful package manager, designed for supercomputers where users have diverse installation needs ๐
. But bestie โ Spack can also handle simple single-user installations on your laptop, no supercomputer required ๐ซถ. Most macOS users are already familiar with package managers like Homebrew and Conda, where all installed packages are symlinked to a single central location like /usr/local. In this section we're gonna show you how to emulate the behavior of Homebrew/Conda using Environments (spack.yaml)! โจ
๐ Setup
First, let's create a new environment. We're gonna assume Spack is already set up correctly, and that you've already sourced the setup script for your shell. To create a new environment, simply run:
$ spack env create myenvHere, myenv can be anything you want to name your environment ๐. Next, we can add a list of packages we'd like to install into our environment. Let's say we want a newer version of Bash than the one that comes with macOS, and we want a few Python libraries. We can run:
$ spack -e myenv add bash@5 python py-numpy py-scipy py-matplotlibEach package can be listed on a separate line, or combined into a single line like we did above ๐. Notice that we're explicitly asking for Bash 5 here. You can use any spec you would normally use on the command line with other Spack commands, periodt.
Next, we wanna manually configure a couple of things:
$ spack -e myenv config edit
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs: [bash@5, python, py-numpy, py-scipy, py-matplotlib]
view: trueYou can see the packages we added earlier in the specs: section ๐. If you ever wanna add more packages, you can either use spack add or manually edit this file โ both slay ๐
.
We also need to change the concretizer:unify option. By default, Spack concretizes each spec separately, allowing multiple versions of the same package to coexist. Since we want a single consistent environment, we want to concretize all of the specs together ๐ง.
Here's what your spack.yaml looks like with this new setting:
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs: [bash@5, python, py-numpy, py-scipy, py-matplotlib]
view: true
concretizer:
unify: true๐ Symlink location
Spack symlinks all installations to /Users/me/spack/var/spack/environments/myenv/.spack-env/view, which is the default when view: true. You can actually change this to any directory you want ๐. For example, Homebrew uses /usr/local, while Conda uses /Users/me/anaconda. In order to access files in these locations, you need to update PATH and other environment variables to point to them. Activating the Spack environment does this automatically, but you can also manually set them in your .bashrc โจ.
โ ๏ธ WARNING โ bestie, read this one, fr fr โ there are several reasons why you shouldn't use /usr/local:
- If you are on macOS 10.11+ (El Capitan and newer), Apple makes it hard for you. You may notice permissions issues on
/usr/localdue to their System Integrity Protection. By default, users don't have permissions to install anything in/usr/local, and you can't even change this usingsudo chownorsudo chmod๐ญ. - Other package managers like Homebrew will try to install things to the same directory. If you plan on using Homebrew in conjunction with Spack, don't symlink things to
/usr/local๐. - If you are on a shared workstation, or don't have
sudoprivileges, you can't do this.
If you still want to do this anyway, there are several ways around SIP. You could disable SIP by booting into recovery mode and runningcsrutil disable, but this is not recommended, as it can open up your OS to security vulnerabilities ๐. Another technique is to runspack concretizeandspack installusingsudo. This is also not recommended.
The safest way I've found is to create your installation directories using sudo, then change ownership back to the user like so:
for directory in .spack bin contrib include lib man share
do
sudo mkdir -p /usr/local/$directory
sudo chown $(id -un):$(id -gn) /usr/local/$directory
doneDepending on the packages you install in your environment, the exact list of directories you need to create may vary. You may also find some packages like Java libraries that install a single file to the installation prefix instead of in a subdirectory. In this case, the action is the same, just replacemkdir -pwithtouchin the for-loop above ๐ซถ.
But again โ it's safer just to use the default symlink location ๐ .
๐ฆ Installation
To actually concretize the environment, run:
$ spack -e myenv concretizeThis will tell you which if any packages are already installed, and alert you to any conflicting specs ๐.
To actually install these packages and symlink them to your view: directory, simply run:
$ spack -e myenv install
$ spack env activate myenvNow, when you type which python3, it should find the one you just installed โจ.
In order to change the default shell to our newer Bash installation, we first need to add it to this list of acceptable shells. Run:
$ sudo vim /etc/shellsand add the absolute path to your bash executable. Then run:
$ chsh -s /path/to/bashNow, when you log out and log back in, echo $SHELL should point to the newer version of Bash ๐๐.
๐ Updating Installed Packages
Let's say you upgraded to a new version of macOS, or a new version of Python was released, and you want to rebuild your entire software stack. To do this, simply run the following commands:
$ spack env activate myenv
$ spack concretize --fresh --force
$ spack installThe --fresh flag tells Spack to use the latest version of every package where possible instead of trying to optimize for reuse of existing installed packages โญ.
The --force flag in addition tells Spack to overwrite its previous concretization decisions, allowing you to choose a new version of Python ๐
. If any of the new packages like Bash are already installed, spack install won't re-install them, it will keep the symlinks in place ๐.
๐งน Updating & Cleaning Up Old Packages
If you're looking to mimic the behavior of Homebrew, you may also want to clean up out-of-date packages from your environment after an upgrade. To upgrade your entire software stack within an environment and clean up old package versions, simply run the following commands:
$ spack env activate myenv
$ spack mark -i --all
$ spack concretize --fresh --force
$ spack install
$ spack gcRunning spack mark -i --all tells Spack to mark all of the existing packages within an environment as "implicitly" installed. This tells spack's garbage collection system that these packages should be cleaned up ๐๏ธโจ.
Don't worry though โ this will not remove your entire environment ๐. Running spack install will reexamine your spack environment after a fresh concretization and will re-mark any packages that should remain installed as "explicitly" installed ๐.
Note: if you use multiple spack environments you should re-runspack installin each of your environments prior to runningspack gcto prevent spack from uninstalling any shared packages that are no longer required by the environment you just upgraded โ ๏ธ.
๐๏ธ Uninstallation
If you decide that Spack isn't right for you, uninstallation is simple. Just run:
$ spack env activate myenv
$ spack uninstall --allThis will uninstall all packages in your environment and remove the symlinks ๐.
๐ฌ FREQUENTLY ASKED QUESTIONS ๐ฌ
This page contains answers to frequently asked questions about Spack. If you have questions that are not answered here, feel free to ask on Slack or GitHub Discussions ๐ซถ. If you've learned the answer to a question that you think should be here, please consider contributing to this page โจ.
๐ค Why does Spack pick particular versions and variants?
This question comes up in a variety of forms:
- Why does Spack seem to ignore my package preferences from
packages.yamlconfig? - Why does Spack toggle a variant instead of using the default from the
package.pyfile?
The short answer is that Spack always picks an optimal configuration based on a complex set of criteria<sup>[1]</sup> ๐ฅ. These criteria are more nuanced than always choosing the latest versions or default variants, iykyk.
๐ NOTE: as a rule of thumb โ requirements + constraints > reuse > preferences > defaults ๐
The following set of criteria (from lowest to highest precedence) explain common cases where concretization output may seem surprising at first ๐:
- Package preferences configured in
packages.yamloverride variant defaults frompackage.pyfiles, and influence the optimal ordering of versions. Preferences are specified as follows:
packages:
foo:
version: [1.0, 1.1]
variants: ~mpi- Reuse concretization configured in
concretizer.yamloverrides preferences, since it's typically faster to reuse an existing spec than to build a preferred one from sources. When build caches are enabled, specs may be reused from a remote location too ๐. Reuse concretization is configured as follows:
concretizer:
reuse: dependencies # other options are 'true' and 'false'- Package requirements configured in
packages.yaml, and constraints from the command line as well aspackage.pyfiles override all of the above ๐ . Requirements are specified as follows:
packages:
foo:
require:
- "@1.2: +mpi"Requirements and constraints restrict the set of possible solutions, while reuse behavior and preferences influence what an optimal solution looks like ๐งโจ.
๐ FOOTNOTES ๐
[1] The exact list of criteria can be retrieved with the spack solve command ๐ซถ
๐๏ธ CONFIGURATION FILES ๐๏ธ
Spack has many configuration files. Here's a quick list of them, in case you want to skip directly to specific docs ๐
compilers.yamlconcretizer.yamlconfig.yamlmirrors.yamlmodules.yamlpackages.yamlrepos.yaml
You can also add any of these as inline configuration in the YAML manifest file (spack.yaml) describing an environment โจ๐.
YAML Format
okay so ๐ Spack configuration files are written in YAML. Why YAML? Because it's human readable (like, you can actually open one and not cry) but also versatile โ it supports dictionaries, lists, and nested sections. For more details on the format, see yaml.org and libyaml. Here's an example config.yaml file:
config:
install_tree: $spack/opt/spack
build_stage:
- $tempdir/$user/spack-stage
- ~/.spack/stageEvery Spack configuration file is nested under a top-level section that matches its name. So config.yaml starts with config:, mirrors.yaml starts with mirrors:, etc. consistency is the moment โจ
Configuration Scopes
Spack pulls configuration data from files in several directories. There are six configuration scopes (and yes, the list below has eight entries โ I'm just reporting what the page says, don't come for me ๐). From lowest to highest:
- defaults: stored in
$(prefix)/etc/spack/defaults/. These are the "factory" settings. Users should generally not modify the settings here โ override them in other configuration scopes instead. The defaults here will change from version to version of Spack. - system: stored in
/etc/spack/. These are settings for this machine, or for all machines on which this file system is mounted. The site scope can be used for settings idiosyncratic to a particular machine, such as the locations of compilers or external packages. These settings are presumably controlled by someone with root access on the machine. They override the defaults scope. - site: stored in
$(prefix)/etc/spack/. Settings here affect only this instance of Spack, and they override the defaults and system scopes. The site scope can be used for per-project settings (one Spack instance per project) or for site-wide settings on a multi-user machine (e.g., for a common Spack instance). ๐ซถ - plugin: read from a Python project's entry points. Settings here affect all instances of Spack running with the same Python installation. This scope takes higher precedence than site, system, and default scopes.
- user: stored in the home directory:
~/.spack/. These settings affect all instances of Spack and take higher precedence than site, system, plugin, or defaults scopes. this is your personal era ๐ - custom: stored in a custom directory specified by
--config-scope. If multiple scopes are listed on the command line, they are ordered from lowest to highest precedence. - environment: when using Spack Environments (
spack.yaml), Spack reads additional configuration from the environment file. See Configuring Environments for further details on these scopes. Environment scopes can be referenced from the command line asenv:name(to reference environment foo, useenv:foo). - command line: build settings specified on the command line take precedence over all other scopes. main character energy obviously ๐
Each configuration directory may contain several configuration files, such as config.yaml, compilers.yaml, or mirrors.yaml. When configurations conflict, settings from higher-precedence scopes override lower-precedence settings. no cap ๐ง
Commands that modify scopes (e.g., spack compilers, spack repo, etc.) take a --scope=<name> parameter that you can use to control which scope is modified. By default, they modify the highest-precedence scope.
Custom scopes
On top of the defaults, system, site, and user scopes, you can add configuration scopes directly on the command line with the --config-scope argument, or -C for short. slay, extra scopes ๐
For example, the following adds two configuration scopes, named scopea and scopeb, to a spack spec command:
$ spack -C ~/myscopes/scopea -C ~/myscopes/scopeb spec ncursesCustom scopes come after the spack command and before the subcommand, and they specify a single path to a directory full of configuration files. You can add the same configuration files to that directory that you can add to any other scope (config.yaml, packages.yaml, etc.).
If multiple scopes are provided:
- Each must be preceded with the
--config-scopeor-Cflag. - They must be ordered from lowest to highest precedence.
Example: scopes for release and development
Suppose you need to support simultaneous building of release and development versions of mypackage, where mypackage -> A -> B. You could create the following files ๐
~/myscopes/release/packages.yaml
packages:
mypackage:
version: [1.7]
A:
version: [2.3]
B:
version: [0.8]~/myscopes/develop/packages.yaml
packages:
mypackage:
version: [develop]
A:
version: [develop]
B:
version: [develop]You can switch between release and develop configurations using configuration arguments. You'd type spack -C ~/myscopes/release when you want to build the designated release versions of mypackage, A, and B, and you'd type spack -C ~/myscopes/develop when you want to build all of these packages at the develop version. literally a whole vibe switcheroo ๐ฆ
Example: swapping MPI providers
Suppose you need to build two software packages, packagea and packageb. packagea is Python 2-based and packageb is Python 3-based. packagea only builds with OpenMPI and packageb only builds with MPICH. You can create different configuration scopes for use with packagea and packageb:
~/myscopes/packgea/packages.yaml
packages:
python:
version: [2.7.11]
all:
providers:
mpi: [openmpi]~/myscopes/packageb/packages.yaml
packages:
python:
version: [3.5.2]
all:
providers:
mpi: [mpich]yes, that first path really says packgea in the docs โ spelling it exactly as-is, bestie, not my typo, not my problem ๐
Plugin scopes
NOTE: Python version >= 3.8 is required to enable plugin configuration. ๐
Spack can be made aware of configuration scopes that are installed as part of a python package. To do so, register a function that returns the scope's path to the "spack.config" entry point. Consider the Python package my_package that includes Spack configurations:
my-package/
โโโ src
โ โโโ my_package
โ โ โโโ __init__.py
โ โ โโโ spack/
โ โ โ โโโ config.yaml
โโโ pyproject.tomladding the following to my_package's pyproject.toml will make my_package's spack/ configurations visible to Spack when my_package is installed:
[project.entry_points."spack.config"]
my_package = "my_package:get_config_path"The function my_package.get_extension_path in my_package/__init__.py might look like
import importlib.resources
def get_config_path():
dirname = importlib.resources.files("my_package").joinpath("spack")
if dirname.exists():
return str(dirname)Platform-specific Scopes
For each scope above (excluding environment scopes), there can also be platform-specific settings ๐ธ. For example, on most platforms, GCC is the preferred compiler. However, on macOS (darwin), Clang often works for more packages, and is set as the default compiler. This configuration is set in $(prefix)/etc/spack/defaults/darwin/packages.yaml. It will take precedence over settings in the defaults scope, but can still be overridden by settings in system, system/darwin, site, site/darwin, user, user/darwin, custom, or custom/darwin. So, the full scope precedence is:
defaultsdefaults/<platform>systemsystem/<platform>sitesite/<platform>useruser/<platform>customcustom/<platform>
You can get the name to use for <platform> by running spack arch --platform. The system config scope has a <platform> section for sites at which /etc is mounted on multiple heterogeneous machines. go off, heterogeneous sites ๐
Scope Precedence
When spack queries for configuration parameters, it searches in higher-precedence scopes first. So, settings in a higher-precedence file can override those with the same key in a lower-precedence one. For list-valued settings, Spack prepends higher-precedence settings to lower-precedence settings. Completely ignoring higher-level configuration options is supported with the :: notation for keys (see Overriding entire sections below).
There are also special notations for string concatenation and precedence override:
+: will force prepending strings or lists. For lists, this is the default behavior.-: works similarly, but for appending values.
String Concatenation
(heading's here, actual content a couple sections down โ keep scrolling bestie ๐)
Simple keys
Let's look at an example of overriding a single key in a Spack file. If your configurations look like this:
$(prefix)/etc/spack/defaults/config.yaml
config:
install_tree: $spack/opt/spack
build_stage:
- $tempdir/$user/spack-stage
- ~/.spack/stage~/.spack/config.yaml
config:
install_tree: /some/other/directorySpack will only override install_tree in the config section, and will take the site preferences for other settings. You can see the final, combined configuration with the spack config get <configtype> command:
$ spack config get config
config:
install_tree: /some/other/directory
build_stage:
- $tempdir/$user/spack-stage
- ~/.spack/stagesee how build_stage survived untouched? obsessed ๐ญ๐
String Concatenation
Above, the user config.yaml completely overrides specific settings in the default config.yaml. Sometimes, it is useful to add a suffix/prefix to a path or name. To do this, you can use the -: notation for append string concatenation at the end of a key in a configuration file. For example:
~/.spack/config.yaml
config:
install_tree-: /my/custom/suffix/Spack will then append to the lower-precedence configuration under the install_tree-: section:
$ spack config get config
config:
install_tree: /some/other/directory/my/custom/suffix
build_stage:
- $tempdir/$user/spack-stage
- ~/.spack/stageSimilarly, +: can be used to prepend to a path or name:
~/.spack/config.yaml
config:
install_tree+: /my/custom/suffix/prefix era vs suffix era, pick your fighter โจ
Overriding entire sections
Above, the user config.yaml only overrides specific settings in the default config.yaml. Sometimes, it is useful to completely override lower-precedence settings. To do this, you can use two colons at the end of a key in a configuration file. For example:
~/.spack/config.yaml
config::
install_tree: /some/other/directorySpack will ignore all lower-precedence configuration under the config:: section:
$ spack config get config
config:
install_tree: /some/other/directorydouble colon = "I said what I said", nothing below gets a say ๐ ๐ฅ
๐ List-valued settings
okay bestie, let's circle back to that config.yaml example one more time, because lists are such a mood. The build_stage setting's value is literally an ordered list of directories:
$(prefix)/etc/spack/defaults/config.yaml
build_stage:
- $tempdir/$user/spack-stage
- ~/.spack/stageNow imagine the user config adds its own lil list of build_stage paths:
~/.spack/config.yaml
build_stage:
- /lustre-scratch/$user/spack
- ~/mystageSpack will first look at the paths in the defaults config.yaml, then the paths in the user's ~/.spack/config.yaml. The list in the higher-precedence scope is prepended to the defaults (yes, it goes on the front, main character energy). spack config get config shows the result:
$ spack config get config
config:
install_tree: /some/other/directory
build_stage:
- /lustre-scratch/$user/spack
- ~/mystage
- $tempdir/$user/spack-stage
- ~/.spack/stageAnd just like in Overriding entire sections, the higher-precedence scope can completely override the lower-precedence scope using ::. So if the user config looked like this:
~/.spack/config.yaml
build_stage::
- /lustre-scratch/$user/spack
- ~/mystageThe merged configuration would look like this:
$ spack config get config
config:
install_tree: /some/other/directory
build_stage:
- /lustre-scratch/$user/spack
- ~/mystageโจ Config File Variables โจ
Spack understands several variables which can be used in config file paths wherever they appear. There are three whole sets of these cuties: Spack-specific variables, environment variables, and user path variables. Spack-specific variables and environment variables are both indicated by prefixing the variable name with $. User path variables are indicated at the start of the path with ~ or ~user.
๐ Spack-specific variables
Spack understands over a dozen special variables. These are:
$env: name of the currently active environment$spack: path to the prefix of this Spack installation$tempdir: default system temporary directory (as specified in Python'stempfile.tempdirvariable.$user: name of the current user$user_cache_path: user cache directory (~/.spackunless overridden)$architecture: the architecture triple of the current host, as detected by Spack.$arch: alias for$architecture.$platform: the platform of the current host, as detected by Spack.$operating_system: the operating system of the current host, as detected by thedistropython module.$os: alias for$operating_system.$target: the ISA target for the current host, as detected by ArchSpec. E.g.skylakeorneoverse-n1.$target_family. The target family for the current host, as detected by ArchSpec. E.g.x86_64oraarch64.$date: the current date in the formatYYYY-MM-DD
Note that, as with shell variables, you can write these as $varname or with braces to distinguish the variable from surrounding characters: ${varname}. Their names are also case insensitive, meaning that $SPACK works just as well as $spack. These special variables are substituted first, so any environment variables with the same name will not be used. iykyk ๐
๐ธ Environment variables
After Spack-specific variables are evaluated, environment variables are expanded. These are formatted like Spack-specific variables, e.g., ${varname}. You can use this to insert environment variables in your Spack configuration. slay.
๐ฆ User home directories
Spack performs Unix-style tilde expansion on paths in configuration files. This means that tilde (~) will expand to the current user's home directory, and ~user will expand to a specified user's home directory. The ~ must appear at the beginning of the path, or Spack will not expand it. no cap, that's the rule.
๐ง Environment Modifications
Spack allows to prescribe custom environment modifications in a few places within its configuration files. Every time these modifications are allowed they are specified as a dictionary, like in the following example:
environment:
set:
LICENSE_FILE: '/path/to/license'
unset:
- CPATH
- LIBRARY_PATH
append_path:
PATH: '/new/bin/dir'The possible actions that are permitted are set, unset, append_path, prepend_path and finally remove_path. They all require a dictionary of variable names mapped to the values used for the modification. The only exception is unset that requires just a list of variable names. No particular order is ensured on the execution of each of these modifications. ๐
๐ Seeing Spack's Configuration
With so many scopes overriding each other, it can sometimes be difficult to understand what Spack's final configuration looks like. honestly, same โ it's giving layers on layers.
Spack provides two useful ways to view the final "merged" version of any configuration file: spack config get and spack config blame. bestie, these are your new faves.
โญ spack config get
spack config get shows a fully merged configuration file, taking into account all scopes. For example, to see the fully merged config.yaml, you can type:
$ spack config get config
config:
debug: false
checksum: true
verify_ssl: true
dirty: false
build_jobs: 8
install_tree: $spack/opt/spack
template_dirs:
- $spack/templates
directory_layout: {architecture}/{compiler.name}-{compiler.version}/{name}-{version}-{hash}
build_stage:
- $tempdir/$user/spack-stage
- ~/.spack/stage
- $spack/var/spack/stage
source_cache: $spack/var/spack/cache
misc_cache: ~/.spack/cache
locks: trueLikewise, this will show the fully merged packages.yaml:
$ spack config get packagesYou can use this in conjunction with the -C / --config-scope argument to see how your scope will affect Spack's configuration:
$ spack -C /path/to/my/scope config get packages๐ฅ spack config blame
spack config blame functions much like spack config get, but it shows exactly which configuration file each preference came from. If you do not know why Spack is behaving a certain way, this can help you track down the problem โ it's literally the receipts:
$ spack --insecure -C ./my-scope -C ./my-scope-2 config blame config
==> Warning: You asked for --insecure. Will NOT check SSL certificates.
--- config:
_builtin debug: False
/home/myuser/spack/etc/spack/defaults/config.yaml:72 checksum: True
command_line verify_ssl: False
./my-scope-2/config.yaml:2 dirty: False
_builtin build_jobs: 8
./my-scope/config.yaml:2 install_tree: /path/to/some/tree
/home/myuser/spack/etc/spack/defaults/config.yaml:23 template_dirs:
/home/myuser/spack/etc/spack/defaults/config.yaml:24 - $spack/templates
/home/myuser/spack/etc/spack/defaults/config.yaml:28 directory_layout: {architecture}/{compiler.name}-{compiler.version}/{name}-{version}-{hash}
/home/myuser/spack/etc/spack/defaults/config.yaml:49 build_stage:
/home/myuser/spack/etc/spack/defaults/config.yaml:50 - $tempdir/$user/spack-stage
/home/myuser/spack/etc/spack/defaults/config.yaml:51 - ~/.spack/stage
/home/myuser/spack/etc/spack/defaults/config.yaml:52 - $spack/var/spack/stage
/home/myuser/spack/etc/spack/defaults/config.yaml:57 source_cache: $spack/var/spack/cache
/home/myuser/spack/etc/spack/defaults/config.yaml:62 misc_cache: ~/.spack/cache
/home/myuser/spack/etc/spack/defaults/config.yaml:86 locks: TrueYou can see above that the build_jobs and debug settings are built in and are not overridden by a configuration file. The verify_ssl setting comes from the --insecure option on the command line. dirty and install_tree come from the custom scopes ./my-scope and ./my-scope-2, and all other configuration options come from the default configuration files that ship with Spack. ๐
๐ Overriding Local Configuration
Spack's system and user scopes provide ways for administrators and users to set global defaults for all Spack instances, but for use cases where one wants a clean Spack installation, these scopes can be undesirable. For example, users may want to opt out of global system configuration, or they may want to ignore their own home directory settings when running in a continuous integration environment.
Spack also, by default, keeps various caches and user data in ~/.spack, but users may want to override these locations.
Spack provides three environment variables that allow you to override or opt out of configuration locations:
SPACK_USER_CONFIG_PATH: Override the path to use for the user scope (~/.spackby default).SPACK_SYSTEM_CONFIG_PATH: Override the path to use for the system scope (/etc/spackby default).SPACK_DISABLE_LOCAL_CONFIG: set this environment variable to completely disable both the system and user configuration directories. Spack will only consider its own defaults and site configuration locations.
And one that allows you to move the default cache location:
SPACK_USER_CACHE_PATH: Override the default path to use for user data (misc_cache,tests,reports, etc.)
With these settings, if you want to isolate Spack in a CI environment, you can do this:
export SPACK_DISABLE_LOCAL_CONFIG=true
export SPACK_USER_CACHE_PATH=/tmp/spack๐ SPACK SETTINGS (CONFIG.YAML) ๐
Spack's basic configuration options are set in config.yaml. You can see the default settings by looking at etc/spack/defaults/config.yaml:
okay so here's the actual default config.yaml that ships with Spack โ like, the whole thing, comments and all โ and bestie I'm gonna walk you through every single line because this is the file that decides where stuff gets installed, where builds happen, and honestly how much Spack trusts the internet. buckle up ๐
# -------------------------------------------------------------------------
# This is the default spack configuration file.
#
# Settings here are versioned with Spack and are intended to provide
# sensible defaults out of the box. Spack maintainers should edit this
# file to keep it current.
#
# Users can override these settings by editing the following files.
#
# Per-spack-instance settings (overrides defaults):
# $SPACK_ROOT/etc/spack/config.yaml
#
# Per-user settings (overrides default and site settings):
# ~/.spack/config.yaml
# -------------------------------------------------------------------------
config:
# This is the path to the root of the Spack install tree.
# You can use $spack here to refer to the root of the spack instance.
install_tree:
root: $spack/opt/spack
projections:
all: "{architecture}/{compiler.name}-{compiler.version}/{name}-{version}-{hash}"
# install_tree can include an optional padded length (int or boolean)
# default is False (do not pad)
# if padded_length is True, Spack will pad as close to the system max path
# length as possible
# if padded_length is an integer, Spack will pad to that many characters,
# assuming it is higher than the length of the install_tree root.
# padded_length: 128so install_tree is literally where everything lands โ the root defaults to $spack/opt/spack and $spack means "root of this spack instance." the projections line is the naming pattern for each install dir, and padded_length is optional: leave it alone (False, no padding), set it to True to pad as close to the system max path length as possible, or give it an integer like 128 to pad to that many characters (as long as it's higher than the length of the install_tree root). ๐
# Locations where templates should be found
template_dirs:
- $spack/share/spack/templates
# Directory where licenses should be located
license_dir: $spack/etc/spack/licensestemplate_dirs = where templates live. license_dir = where licenses live. simple, iconic, no notes. โจ
# Temporary locations Spack can try to use for builds.
#
# Recommended options are given below.
#
# Builds can be faster in temporary directories on some (e.g., HPC) systems.
# Specifying `$tempdir` will ensure use of the default temporary directory
# (i.e., ``$TMP` or ``$TMPDIR``).
#
# Another option that prevents conflicts and potential permission issues is
# to specify `$user_cache_path/stage`, which ensures each user builds in their
# home directory.
#
# A more traditional path uses the value of `$spack/var/spack/stage`, which
# builds directly inside Spack's instance without staging them in a
# temporary space. Problems with specifying a path inside a Spack instance
# are that it precludes its use as a system package and its ability to be
# pip installable.
#
# In Spack environment files, chaining onto existing system Spack
# installations, the $env variable can be used to download, cache and build
# into user-writable paths that are relative to the currently active
# environment.
#
# In any case, if the username is not already in the path, Spack will append
# the value of `$user` in an attempt to avoid potential conflicts between
# users in shared temporary spaces.
#
# The build stage can be purged with `spack clean --stage` and
# `spack clean -a`, so it is important that the specified directory uniquely
# identifies Spack staging to avoid accidentally wiping out non-Spack work.
build_stage:
- $tempdir/$user/spack-stage
- $user_cache_path/stage
# - $spack/var/spack/stageokay build_stage is a whole saga so let me break it down, because this is where builds actually happen and you do NOT wanna get it wrong ๐ญ
$tempdir= the default temp dir (like$TMPor$TMPDIR), and builds can be faster in temp dirs on some systems (HPC bestie, I'm looking at you).$user_cache_path/stage= builds each user inside their own home dir, which dodges conflicts and permission drama.$spack/var/spack/stageis the more traditional one โ builds right inside Spack's instance with no temp staging โ but the catch is it precludes using Spack as a system package and its ability to be pip installable.- in Spack environment files chaining onto existing system Spack installs,
$envgets you user-writable download/cache/build paths relative to the currently active environment. - if the username isn't already in the path, Spack appends
$userto avoid collisions between users sharing temp space. - and โ ๏ธ the build stage can be wiped by
spack clean --stageandspack clean -a, so make sure the dir uniquely identifies Spack staging or you might nuke non-Spack work. dangerous, periodt.
# Directory in which to run tests and store test results.
# Tests will be stored in directories named by date/time and package
# name/hash.
test_stage: $user_cache_path/test
# Cache directory for already downloaded source tarballs and archived
# repositories. This can be purged with `spack clean --downloads`.
source_cache: $spack/var/spack/cache
## Directory where spack managed environments are created and stored
# environments_root: $spack/var/spack/environments
# Cache directory for miscellaneous files, like the package index.
# This can be purged with `spack clean --misc-cache`
misc_cache: $user_cache_path/cachetest_stage= where tests run and results get stored, in dirs named by date/time plus package name/hash.source_cache= already-downloaded source tarballs + archived repos. purgeable withspack clean --downloads.environments_root= where Spack-managed environments get created and stored (commented out, so it's the default unless you uncomment).misc_cache= random extra files like the package index. purgeable withspack clean --misc-cache. ๐ง
# Timeout in seconds used for downloading sources etc. This only applies
# to the connection phase and can be increased for slow connections or
# servers. 0 means no timeout.
connect_timeout: 10
# If this is false, tools like curl that use SSL will not verify
# certifiates. (e.g., curl will use use the -k option)
verify_ssl: true
# This is where custom certs for proxy/firewall are stored.
# It can be a path or environment variable. To match ssl env configuration
# the default is the environment variable SSL_CERT_FILE
ssl_certs: $SSL_CERT_FILE
# Suppress gpg warnings from binary package verification
# Only suppresses warnings, gpg failure will still fail the install
# Potential rationale to set True: users have already explicitly trusted the
# gpg key they are using, and may not want to see repeated warnings that it
# is self-signed or something of the sort.
suppress_gpg_warnings: falseconnect_timeout: 10= 10 seconds for the connection phase of downloads. bump it for slow connections/servers.0means no timeout.verify_ssl: true= curl and friends verify certificates. set it false and curl will use-k(aka stops verifying โ your call bestie ๐).ssl_certs: $SSL_CERT_FILE= where custom certs for proxy/firewall live, as a path or env var, defaulting to theSSL_CERT_FILEenv var to match ssl env config.suppress_gpg_warnings: false= only hides gpg warnings from binary package verification; a gpg failure still fails the install. reasoning forTrue: you already explicitly trusted that gpg key and don't want repeated self-signed-ish warnings. valid ngl.
# If set to true, Spack will attempt to build any compiler on the spec
# that is not already available. If set to False, Spack will only use
# compilers already configured in compilers.yaml
install_missing_compilers: false
# If set to true, Spack will always check checksums after downloading
# archives. If false, Spack skips the checksum step.
checksum: true
# If set to true, Spack will fetch deprecated versions without warning.
# If false, Spack will raise an error when trying to install a deprecated version.
deprecated: false
# If set to true, `spack install` and friends will NOT clean
# potentially harmful variables from the build environment. Use wisely.
dirty: falseinstall_missing_compilers: false= only build a compiler not already available if you flip this to true; otherwise Spack sticks to compilers already configured incompilers.yaml.checksum: true= always check checksums after downloading archives. false = skip the checksum step.deprecated: false= false means Spack raises an error when you try to install a deprecated version; true means it fetches deprecated versions without warning.dirty: false= true makesspack installand friends NOT clean potentially harmful variables from the build environment. use wisely, I'm serious ๐
# The language the build environment will use. This will produce English
# compiler messages by default, so the log parser can highlight errors.
# If set to C, it will use English (see man locale).
# If set to the empty string (''), it will use the language from the
# user's environment.
build_language: C
# When set to true, concurrent instances of Spack will use locks to
# avoid modifying the install tree, database file, etc. If false, Spack
# will disable all locking, but you must NOT run concurrent instances
# of Spack. For filesystems that don't support locking, you should set
# this to false and run one Spack at a time, but otherwise we recommend
# enabling locks.
locks: true
# The default url fetch method to use.
# If set to 'curl', Spack will require curl on the user's system
# If set to 'urllib', Spack will use python built-in libs to fetch
url_fetch_method: urllibbuild_language: C= English compiler messages by default so the log parser can highlight errors.Cโ English (seeman locale). empty string''โ whatever language the user's environment is set to.locks: true= concurrent Spack instances use locks so they don't stomp the install tree, database file, etc. false disables all locking BUT you must not run concurrent instances then. filesystems without locking support: set false and run one Spack at a time โ otherwise locks are recommended, fr fr.url_fetch_method: urllib= Python's built-in libs do the fetching.curlwould require curl on the user's system. ๐ฆ
# The maximum number of jobs to use for the build system (e.g. `make`), when
# the -j flag is not given on the command line. Defaults to 16 when not set.
# Note that the maximum number of jobs is limited by the number of cores
# available, taking thread affinity into account when supported. For instance:
# - With `build_jobs: 16` and 4 cores available `spack install` will run `make -j4`
# - With `build_jobs: 16` and 32 cores available `spack install` will run `make -j16`
# - With `build_jobs: 2` and 4 cores available `spack install -j6` will run `make -j6`
# build_jobs: 16
# If set to true, Spack will use ccache to cache C compiles.
ccache: falsebuild_jobs is commented out but defaults to 16 when not set, and it's the max jobs for the build system (think make) when no -j is given on the command line. but! the max is limited by cores available, thread affinity included, so:
build_jobs: 16+ 4 cores โspack installrunsmake -j4build_jobs: 16+ 32 cores โspack installrunsmake -j16build_jobs: 2+ 4 cores โspack install -j6runsmake -j6
and ccache: false means no ccache for C compiles unless you flip it to true. โญ
# The concretization algorithm to use in Spack. Options are:
#
# 'clingo': Uses a logic solver under the hood to solve DAGs with full
# backtracking and optimization for user preferences. Spack will
# try to bootstrap the logic solver, if not already available.
#
# 'original': Spack's original greedy, fixed-point concretizer. This
# algorithm can make decisions too early and will not backtrack
# sufficiently for many specs. This will soon be deprecated in
# favor of clingo.
#
# See `concretizer.yaml` for more settings you can fine-tune when
# using clingo.
concretizer: clingothe concretizer choice, aka the brain of the operation:
'clingo'โ uses a logic solver under the hood to solve DAGs with full backtracking and optimization for user preferences. Spack will try to bootstrap the logic solver if it's not already available. this is the default and it's iconic.'original'โ the original greedy, fixed-point concretizer. it can make decisions too early and won't backtrack enough for many specs. soon to be deprecated in favor of clingo. so likeโฆ bestie, don't get attached ๐ญ
and head to concretizer.yaml for more settings you can fine-tune when using clingo. ๐ฅ
# How long to wait to lock the Spack installation database. This lock is used
# when Spack needs to manage its own package metadata and all operations are
# expected to complete within the default time limit. The timeout should
# therefore generally be left untouched.
db_lock_timeout: 60
# How long to wait when attempting to modify a package (e.g. to install it).
# This value should typically be 'null' (never time out) unless the Spack
# instance only ever has a single user at a time, and only if the user
# anticipates that a significant delay indicates that the lock attempt will
# never succeed.
package_lock_timeout: nulldb_lock_timeout: 60= seconds to wait locking the install database. it's used when Spack manages its own package metadata and operations are expected to finish within the default limit, so generally leave it untouched.package_lock_timeout: null= how long to wait when modifying a package (e.g. installing it). typicallynull(never time out) unless the instance only ever has a single user at a time and you expect that a big delay means the lock attempt will never succeed. ๐ซถ
# Control how shared libraries are located at runtime on Linux. See the
# the Spack documentation for details.
shared_linking:
# Spack automatically embeds runtime search paths in ELF binaries for their
# dependencies. Their type can either be "rpath" or "runpath". For glibc, rpath is
# inherited and has precedence over LD_LIBRARY_PATH; runpath is not inherited
# and of lower precedence. DO NOT MIX these within the same install tree.
type: rpath
# (Experimental) Embed absolute paths of dependent libraries directly in ELF
# binaries to avoid runtime search. This can improve startup time of
# executables with many dependencies, in particular on slow filesystems.
bind: falseshared_linking controls how shared libraries are found at runtime on Linux (docs have the details):
type: rpathโ Spack auto-embeds runtime search paths in ELF binaries for their dependencies, and the type is either"rpath"or"runpath". for glibc: rpath is inherited and has precedence overLD_LIBRARY_PATH; runpath is not inherited and has lower precedence. โ ๏ธ DO NOT MIX these within the same install tree.bind: falseโ experimental. embeds absolute paths of dependent libraries straight into ELF binaries to avoid runtime search. can improve startup time for executables with many dependencies, especially on slow filesystems.
# Set to 'false' to allow installation on filesystems that doesn't allow setgid bit
# manipulation by unprivileged user (e.g. AFS)
allow_sgid: true
# Whether to show status information during building and installing packages.
# This gives information about Spack's current progress as well as the current
# and total number of packages. Information is shown both in the terminal
# title and inline.
install_status: true
# Number of seconds a buildcache's index.json is cached locally before probing
# for updates, within a single Spack invocation. Defaults to 10 minutes.
binary_index_ttl: 600allow_sgid: trueโ set tofalseto allow installation on filesystems that don't let unprivileged users manipulate the setgid bit (e.g. AFS).install_status: trueโ shows status while building/installing: current progress plus current and total package counts, in both the terminal title and inline.binary_index_ttl: 600โ seconds a buildcache'sindex.jsonis cached locally before probing for updates within a single Spack invocation. defaults to 10 minutes. ๐ธ
flags:
# Whether to keep -Werror flags active in package builds.
keep_werror: 'none'
# A mapping of aliases that can be used to define new commands. For instance,
# `sp: spec -I` will define a new command `sp` that will execute `spec` with
# the `-I` argument. Aliases cannot override existing commands.
aliases:
concretise: concretize
containerise: containerize
rm: removelast bit, and it's cute:
flags.keep_werror: 'none'โ whether to keep-Werrorflags active in package builds.aliasesโ maps aliases to new commands. likesp: spec -Idefines a newspcommand that runsspecwith the-Iargument. โ ๏ธ aliases cannot override existing commands.- the shipped ones:
concretise: concretize,containerise: containerize,rm: removeโ british spelling besties andrmgirlies, Spack's got you ๐
okay that's the whole default config.yaml end to end, every comment and every default, nothing skipped. but bestie we're not even close to done with the settings files โ the next one has its own whole personality and I am obsessed with how much it controls. stay right here, we're going straight into it ๐
These settings can be overridden in etc/spack/config.yaml or ~/.spack/config.yaml. See Configuration Scopes for details.
๐ฆ install_tree:root
The location where Spack will install packages and their dependencies. Default is $spack/opt/spack. That's the home base, the main character address, the place everything moves into ๐
๐งต install_hash_length and install_path_scheme
Okay so the default Spack installation path can be very long, and that lowkey creates problems for scripts with hardcoded shebangs. Also โ and this is the messy part โ when you're using the Intel compiler and there's also a long list of dependencies, the compiler may segfault ๐ญ If you see the following:
: internal error: ** The compiler has encountered an unexpected problem.
** Segmentation violation signal raised. **
Access violation or stack overflow. Please contact Intel Support for assistance.it may be because variables containing dependency specs may be too long. There are two parameters to help with long path names. Firstly, the install_hash_length parameter can set the length of the hash in the installation path from 1 to 32. The default path uses the full 32 characters.
Secondly, it is also possible to modify the entire installation scheme. By default Spack uses
{architecture}/{compiler.name}-{compiler.version}/{name}-{version}-{hash}where the tokens that are available for use in this directive are the same as those understood by the format() method. Using this parameter it is possible to use a different package layout or reduce the depth of the installation paths. For example
config:
install_path_scheme: '{name}/{version}/{hash:7}'would install packages into sub-directories using only the package name, version and a hash length of 7 characters. Short and sweet, we love a compact queen ๐
When using either parameter to set the hash length it only affects the representation of the hash in the installation directory. You should be aware that the smaller the hash length the more likely naming conflicts will occur. These parameters are independent of those used to configure module names.
WARNING: Modifying the installation hash length or path scheme after packages have been installed will prevent Spack from being able to find the old installation directories. Fr fr, do this before, bestie โ not after ๐
๐๏ธ build_stage
Spack is designed to run out of a user home directory, and on many systems the home directory is a (slow) network file system. On most systems, building in a temporary file system is faster. Usually, there is also more space available in the temporary location than in the home directory. If the username is not already in the path, Spack will append the value of $user to the selected build_stage path.
WARNING: We highly recommend specifyingbuild_stagepaths that distinguish between staging and other activities to ensurespack cleandoes not inadvertently remove unrelated files. Spack prependsspack-stage-to temporary staging directory names to reduce this risk. Using a combination ofspackand orstagein each specified path, as shown in the default settings and documented examples, will add another layer of protection.
By default, Spack's build_stage is configured like this:
build_stage:
- $tempdir/$user/spack-stage
- ~/.spack/stageThis can be an ordered list of paths that Spack should search when trying to find a temporary directory for the build stage. The list is searched in order, and Spack will use the first directory to which it has write access. First come, first served โจ
Specifying ~/.spack/stage first will ensure each user builds in their home directory. The historic Spack stage path $spack/var/spack/stage will build directly inside the Spack instance. See Config File Variables for more on $tempdir and $spack.
When Spack builds a package, it creates a temporary directory within the build_stage. After the package is successfully installed, Spack deletes the temporary directory it used to build. Unsuccessful builds are not deleted, but you can manually purge them with spack clean --stage ๐งน
NOTE: The build will fail if there is no writable directory in the build_stage list, where any user- and site-specific setting will be searched first.
๐๏ธ source_cache
Location to cache downloaded tarballs and repositories. By default these are stored in $spack/var/spack/cache. These are stored indefinitely by default. Can be purged with spack clean --downloads. It's giving hoarder closet but make it tarballs ๐
๐งบ misc_cache
Temporary directory to store long-lived cache files, such as indices of packages available in repositories. Defaults to ~/.spack/cache. Can be purged with spack clean --misc-cache.
๐ verify_ssl
When set to true (default) Spack will verify certificates of remote hosts when making ssl connections. Set to false to disable, and tools like curl will use their --insecure options. Disabling this can expose you to attacks. Use at your own risk. Bestie please just leave this one alone ๐ญ
๐ ssl_certs
Path to custom certificates for SSL verification. The value can be a filesystem path, or an environment variable that expands to an absolute file path. The default value is set to the environment variable SSL_CERT_FILE to use the same syntax used by many other applications that automatically detect custom certificates. When url_fetch_method:curl the config:ssl_certs should resolve to a single file. Spack will then set the environment variable CURL_CA_BUNDLE in the subprocess calling curl. If url_fetch_method:urllib then files and directories are supported i.e. config:ssl_certs:$SSL_CERT_FILE or config:ssl_certs:$SSL_CERT_DIR will work. In all cases the expanded path must be absolute for Spack to use the certificates. Certificates relative to an environment can be created by prepending the path variable with the Spack configuration variable $env.
โ checksum
When set to true, Spack verifies downloaded source code using a checksum, and will refuse to build packages that it cannot verify. Set to false to disable these checks. Disabling this can expose you to attacks. Use at your own risk. (Same energy as verify_ssl โ safety girlies stay winning ๐ก๏ธ)
๐ locks
When set to true, concurrent instances of Spack will use locks to avoid modifying the install tree, database file, etc. If false, Spack will disable all locking, but you must not run concurrent instances of Spack. For file systems that don't support locking, you should set this to false and run one Spack at a time, but otherwise we recommend enabling locks. One Spack at a time is such a vibe check ๐
๐ซง dirty
By default, Spack unsets variables in your environment that can change the way packages build. This includes LD_LIBRARY_PATH, CPATH, LIBRARY_PATH, DYLD_LIBRARY_PATH, and others.
By default, builds are clean, but on some machines, compilers and other tools may need custom LD_LIBRARY_PATH settings to run. You can set dirty to true to skip the cleaning step and make all builds "dirty" by default. Be aware that this will reduce the reproducibility of builds. Messy builds, messy life ๐
๐งต build_jobs
Unless overridden in a package or on the command line, Spack builds all packages in parallel. The default parallelism is equal to the number of cores available to the process, up to 16 (the default of build_jobs). For a build system that uses Makefiles, this spack install runs:
make -j<build_jobs>, whenbuild_jobsis less than the number of cores availablemake -j<ncores>, whenbuild_jobsis greater or equal to the number of cores available
If you work on a shared login node or have a strict ulimit, it may be necessary to set the default to a lower value. By setting build_jobs to 4, for example, commands like spack install will run make -j4 instead of hogging every core. To build all software in serial, set build_jobs to 1.
Note that specifying the number of jobs on the command line always takes priority, so spack install -j<n> always runs make -j<n>, even when that exceeds the number of cores available. Command line stays the main character, periodt ๐
โก ccache
When set to true Spack will use ccache to cache compiles. This is useful specifically in two cases: (1) when using spack dev-build, and (2) when building the same package with many different variants. The default is false.
When enabled, Spack will look inside your PATH for a ccache executable and stop if it is not found. Some systems come with ccache, but it can also be installed using spack install ccache. ccache comes with reasonable defaults for cache size and location. (See the Configuration settings section of man ccache to learn more about the default settings and how to change them). Please note that we currently disable ccache's hash_dir feature to avoid an issue with the stage directory (see https://github.com/spack/spack/pull/3761#issuecomment-294352232).
๐ shared_linking:type
Control whether Spack embeds RPATH or RUNPATH attributes in ELF binaries so that they can find their dependencies. Has no effect on macOS. Two options are allowed:
rpathuses RPATH and forces the--disable-new-tagsflag to be passed to the linkerrunpathuses RUNPATH and forces the--enable-new-tagsflag to be passed to the linker
RPATH search paths have higher precedence than LD_LIBRARY_PATH and ld.so will search for libraries in transitive RPATHs of parent objects.
RUNPATH search paths have lower precedence than LD_LIBRARY_PATH, and ld.so will ONLY search for dependencies in the RUNPATH of the loading object.
DO NOT MIX the two options within the same install tree. I'm so serious, pick a lane bestie ๐ค
๐งฒ shared_linking:bind
This is an experimental option that controls whether Spack embeds absolute paths to needed shared libraries in ELF executables and shared libraries on Linux. Setting this option to true has two advantages:
- Improved startup time: when running an executable, the dynamic loader does not have to perform a search for needed libraries, they are loaded directly.
- Reliability: libraries loaded at runtime are those that were linked to. This minimizes the risk of accidentally picking up system libraries.
In the current implementation, Spack sets the soname (shared object name) of libraries to their install path upon installation. This has two implications:
- binding does not apply to libraries installed before the option was enabled;
- toggling the option off does not prevent binding of libraries installed when the option was still enabled.
It is also worth noting that:
- Applications relying on
dlopen(3)will continue to work, even when they open a library by name. This is because RPATHs are retained in binaries also when bind is enabled. LD_PRELOADcontinues to work for the typical use case of overriding symbols, such as preloading a library with a more efficient malloc. However, the preloaded library will be loaded additionally to, instead of in place of another library with the same name โ this can be problematic in very rare cases where libraries rely on a particular init or fini order.
NOTE: lowkey sometimes packages ship stub libraries โ they've got the interface for linking but no actual implementation for runtime, bestie. The classic example is libcuda.so, which comes with the CUDA toolkit: you can link against it, but the library you need at runtime is the one installed with the CUDA driver ๐ญ. To stop Spack from binding those, mark them non-bindable with a property in the package:
class Example(Package):
non_bindable_shared_objects = ["libinterface.so"]install_status โจ
Set this to true and Spack will show you info about its current progress, plus the current and total package numbers ๐ซถ. The progress shows up in both the terminal title and inline. Set it to false and you get zero progress info, silence, nothing, the void ๐.
To work properly this needs your terminal to reset its title after Spack has finished its work โ otherwise Spack's status info just chills in your terminal title forever ๐ซ . Most terminals are already set up this way and clear Spack's status information on their own, so like, you're probably fine.
aliases ๐
Aliases let you define brand new Spack commands, no cap. They can be shortcuts for longer commands, or they can bake in specific arguments for convenience โจ. Say you want spack install's -v argument literally every single time โ make a new alias called inst that always calls install -v:
aliases:
inst: install -v๐ PACKAGE SETTINGS (PACKAGES.YAML) ๐
Spack lets you customize how your software gets built through the packages.yaml file, and honestly it's kind of the main character of build config ๐
. With it you can make Spack prefer particular implementations of virtual dependencies (think MPI or BLAS/LAPACK), or make it prefer building with particular compilers. You can also tell Spack to use external software installations that already exist on your system ๐ฅ.
At a high level, packages.yaml is structured like this:
packages:
package1:
# settings for package1
package2:
# settings for package2
# ...
all:
# settings that apply to all packages.So you can either set build preferences specifically for one package, or say "these settings apply to literally all packages" โ either way works ๐. The types of settings you can customize are described in detail below.
Spack's build defaults live in the default etc/spack/defaults/packages.yaml file. You can override them in ~/.spack/packages.yaml or etc/spack/packages.yaml. For more details on how that works, see Configuration Scopes ๐
External Packages ๐ธ
Spack can be configured to use externally-installed packages instead of building its own, which is a whole vibe when machines ship with system packages โ like a customized MPI you want used instead of Spack building its own MPI ๐ฆ.
External packages are configured through the packages.yaml file. Here's an example external config:
packages:
openmpi:
externals:
- spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
prefix: /opt/openmpi-1.4.3
- spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
prefix: /opt/openmpi-1.4.3-debug
- spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
prefix: /opt/openmpi-1.6.5-intelThis example lists three installations of OpenMPI: one built with GCC, one built with GCC plus debug information, and another built with Intel โจ. If Spack is asked to build a package that uses one of these MPIs as a dependency, it'll use the pre-installed OpenMPI in the given directory. And pssst โ the specified path is the top-level install prefix, not the bin subdirectory ๐.
packages.yaml can also specify modules to load instead of installation prefixes. This next example says module CMake/3.7.2 provides cmake version 3.7.2:
cmake:
externals:
- spec: cmake@3.7.2
modules:
- CMake/3.7.2Every packages.yaml begins with a packages: attribute, followed by a list of package names ๐. To specify externals, add an externals: attribute under the package name, which lists externals. Each external should have a spec: string that's as well-defined as reasonably possible โ fr fr, be thorough. If a package lacks a spec component, like a missing compiler or package version, Spack will guess the missing component based on its most-favored packages... and it may guess incorrectly ๐ญ.
Each package version and compiler listed in an external should have entries in Spack's packages and compiler configuration, even though that package and compiler may never actually be built ๐ซถ.
Extra attributes for external packages ๐ฆ
Sometimes external packages need extra attributes to be used effectively (iconic, but high maintenance ๐
). This info can be defined on a per-package basis and stored in the extra_attributes section of the external package configuration. On top of per-package info, this section can define environment modifications performed whenever the package is used โจ. For example, if an external package is built without rpath support, it may need LD_LIBRARY_PATH settings to find its dependencies. That would be configured like this:
packages:
mpich:
externals:
- spec: mpich@3.3 %clang@12.0.0 +hwloc
prefix: /path/to/mpich
extra_attributes:
environment:
prepend_path:
LD_LIBRARY_PATH: /path/to/hwloc/lib64See Environment Modifications for more on how to configure environment modifications in Spack config files ๐
Prevent packages from being built from sources ๐
Okay so here's the plot twist: adding an external spec in packages.yaml lets Spack use an external location, but it does not prevent Spack from building packages from sources ๐ฑ. In the example above, Spack might choose โ for many valid reasons โ to start building and linking with the latest version of OpenMPI instead of continuing to use the pre-installed OpenMPI versions.
To prevent that, packages.yaml also lets packages be flagged as non-buildable ๐. The previous example could be modified to:
packages:
openmpi:
externals:
- spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
prefix: /opt/openmpi-1.4.3
- spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
prefix: /opt/openmpi-1.4.3-debug
- spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
prefix: /opt/openmpi-1.6.5-intel
buildable: FalseAdding that buildable flag tells Spack it should never build its own version of OpenMPI from sources, and it will instead always rely on a pre-built OpenMPI ๐ฅ periodt.
NOTE: Ifconcretizer:reuseis on (see Concretization Settings (concretizer.yaml) for more info on that flag), pre-built specs include specs already available from a local store, an upstream store, a registered buildcache, or specs marked as externals inpackages.yaml. Ifconcretizer:reuseis off, only external specs inpackages.yamlare included in the list of pre-built specs ๐.
If an external module is specified as not buildable, then Spack will load the external module into the build environment, which can be used for linking โจ.
And the buildable flag does not need to be paired with external packages ๐
โ you can use it alone to forbid packages that may be buggy or otherwise undesirable. Gatekeeping, but make it config.
Non-buildable virtual packages ๐
Virtual packages in Spack can also be specified as not buildable, and external implementations can be provided ๐ซถ. In the example above, OpenMPI is configured as not buildable, but Spack will often prefer other MPI implementations over the externally available OpenMPI. You could configure every MPI provider as not buildable individually, but there's a more convenient way:
packages:
mpi:
buildable: False
openmpi:
externals:
- spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
prefix: /opt/openmpi-1.4.3
- spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
prefix: /opt/openmpi-1.4.3-debug
- spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
prefix: /opt/openmpi-1.6.5-intelSpack can then use any of the listed external implementations of MPI to satisfy a dependency, and it'll choose depending on the compiler and architecture โจ iykyk.
In cases where the concretizer is configured to reuse specs, and other MPI providers (available via stores or buildcaches) are not wanted, Spack can be configured to require specs matching only the available externals:
packages:
mpi:
buildable: False
require:
- one_of: [
"openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64",
"openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug",
"openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
]
openmpi:
externals:
- spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
prefix: /opt/openmpi-1.4.3
- spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
prefix: /opt/openmpi-1.4.3-debug
- spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
prefix: /opt/openmpi-1.6.5-intelThis configuration prevents any spec using MPI and originating from stores or buildcaches from being reused โ unless it matches the requirements under packages:mpi:require ๐. For more info on requirements, see Package Requirements.
Automatically Find External Packages ๐
You can run the spack external find command to search for system-provided packages and add them to packages.yaml ๐. After running this command your packages.yaml may include shiny new entries like:
packages:
cmake:
externals:
- spec: cmake@3.17.2
prefix: /usrGenerally this is useful for detecting a small set of commonly-used packages; for now it's generally limited to finding build-only dependencies ๐ฆ. Specific limitations include:
- Packages are not discoverable by default: for a package to be discoverable with
spack external find, it needs to add special logic. See here for more details. - The logic does not search through module files, it can only detect packages with executables defined in
PATH; you can help Spack locate externals that use module files by loading any associated modules for packages you want Spack to know about before runningspack external find๐ . - Spack will not overwrite entries that already exist in your package config โ if there's an external defined for a spec at any config scope, Spack is not adding a new external entry for it, periodt. (If you're like "where did all these externals even come from??" run
spack config blame packagesand it'll point at exactly which file each one lives in. iykyk ๐ )
๐ธ Package Requirements ๐ธ
Okay so bestie, Spack can be told to always use certain compilers, package versions, and variants while concretizing โ that's what package requirements are for. ๐
They're lowkey perfect for when you keep typing the same constraints on the command line over and over and you're just like "please remember this for me." Another total vibe: constraints that should apply to all root specs in an environment, without repeating the thing everywhere like a clown. ๐
Also, requirements config is more flexible than command-line constraints because it can specify constraints on packages when they show up as a dependency. On the command line you literally cannot constrain dependencies while keeping those dependencies optional. Not possible. Requirements ate. ๐
SEE ALSO: FAQ: Why does Spack pick particular versions and variants?
๐ฆ Requirements syntax ๐ฆ
So the package requirements config goes in packages.yaml, keyed by package name, and expressed using the Spec syntax. Easiest case: you give a single spec string and that's what you always want the package to have:
packages:
libfabric:
require: "@1.13.2"In the example above, libfabric will always build with version 1.13.2. Iconic. โจ
Need to compose multiple config scopes? require also accepts a list of strings:
packages:
libfabric:
require:
- "@1.13.2"
- "%gcc"In this case libfabric will always build with version 1.13.2 and using GCC as a compiler. Two for one, slay.
For the more complex situations, require also accepts a list of objects. These objects must have either an any_of or a one_of field, containing a list of spec strings, and they can optionally have a when and a message attribute:
packages:
openmpi:
require:
- any_of: ["@4.1.5", "%gcc"]
message: "in this example only 4.1.5 can build with other compilers"any_of is a list of specs. One of those specs must be satisfied, and it's also allowed for the concretized spec to match more than one. So in the example above that means you could build openmpi@4.1.5%gcc, openmpi@4.1.5%clang or openmpi@3.9%gcc โ but not openmpi@3.9%clang. ๐ซถ
If you gave a custom message and the requirement isn't satisfiable, Spack prints your custom error message:
$ spack spec openmpi@3.9%clang
==> Error: in this example only 4.1.5 can build with other compilersWe could express a similar requirement with the when attribute:
packages:
openmpi:
require:
- any_of: ["%gcc"]
when: "@:4.1.4"
message: "in this example only 4.1.5 can build with other compilers"In the example above, if the version turns out to be 4.1.4 or less, we require the compiler to be GCC.
For readability, Spack also allows a spec key accepting a string when there's only a single constraint:
packages:
openmpi:
require:
- spec: "%gcc"
when: "@:4.1.4"
message: "in this example only 4.1.5 can build with other compilers"This code snippet and the one before it are semantically equivalent. Same thing, different outfit. ๐
Finally, instead of any_of you can use one_of, which also takes a list of specs. The final concretized spec must match one and only one of them:
packages:
mpich:
require:
- one_of: ["+cuda", "+rocm"]In the example above, that means you could build mpich+cuda or mpich+rocm but not mpich+cuda+rocm. Pick a lane bestie ๐ญ
NOTE: Forany_ofandone_of, the order of specs indicates a preference: items that appear earlier in the list are preferred (note that these preferences can be ignored in favor of others).
NOTE: When using a conditional requirement, Spack is allowed to actively avoid the triggering condition (thewhen=...spec) if that leads to a concrete spec with better scores in the optimization criteria. To check the current optimization criteria and their priorities you can runspack solve zlib.
โญ Setting default requirements โญ
You can also set default requirements for all packages under all, like this:
packages:
all:
require: '%clang'which means every spec will be required to use clang as a compiler. Everyone's getting clang. No exceptions. ๐ซก
Requirements on variants for all packages are possible too โ but heads up, they're only enforced for those packages that actually define these variants, otherwise they're just disregarded. Like:
packages:
all:
require:
- "+shared"
- "+cuda"will just enforce +shared on zlib, which has a boolean shared variant but no cuda variant. zlib is sitting there like "I don't do cuda??" ๐
Constraints in a single spec literal are always considered as a whole, so in a case like:
packages:
all:
require: "+shared +cuda"the default requirement will be enforced only if a package has both a cuda and a shared variant, and will never be partially enforced. All or nothing, babe.
And finally โ all represents a default set of requirements. If there are specific package requirements, then the default requirements under all are disregarded. So with a config like this:
packages:
all:
require:
- 'build_type=Debug'
- '%clang'
cmake:
require:
- 'build_type=Debug'
- '%gcc'Spack requires cmake to use gcc and all other nodes (including cmake dependencies) to use clang. If enforcing build_type=Debug is needed also on cmake, it must be repeated in the specific cmake requirements. Yeah it's a little repetitive, that's just how it is ๐
๐ง Setting requirements on virtual specs ๐ง
A requirement on a virtual spec applies whenever that virtual is present in the DAG. Which is super useful for fixing which virtual provider you want to use:
packages:
mpi:
require: 'mvapich2 %gcc'With the configuration above, the only allowed mpi provider is mvapich2 %gcc. Locked in. ๐ฅ
Requirements on the virtual spec and on the specific provider are both applied, if present. For instance, with a configuration like:
packages:
mpi:
require: 'mvapich2 %gcc'
mvapich2:
require: '~cuda'you will use mvapich2~cuda %gcc as an mpi provider. Both rules showed up and both got honored. ๐
๐ Conflicts and strong preferences ๐
If the semantics of requirements are too strong for you, you can also express "strong preferences" and "conflicts" right from the config files:
packages:
all:
prefer:
- '%clang'
conflict:
- '+shared'The prefer and conflict sections can be used whenever a require section is allowed. The argument is always a list of constraints, and each constraint can be either a simple string, or a more complex object:
packages:
all:
conflict:
- spec: '%clang'
when: 'target=x86_64_v3'
message: 'reason why clang cannot be used'The spec attribute is mandatory, while both when and message are optional. (Don't skip the spec, it will not be chill about it.)
NOTE: Requirements allow for expressing both "strong preferences" and "conflicts". The syntax for doing so, though, may not be immediately clear. For instance, if we want to prevent any package from using%clang, we can set:yaml packages: all: require: - one_of: ['%clang', '@:']Since only one of the requirements must hold, and@:is always true, the rule above is equivalent to a conflict. For "strong preferences" we need to substitute theone_ofpolicy withany_of.
๐ Package Preferences ๐
Okay sometimes package requirements are just too strong and package preferences are the better option. Preferences don't impose constraints on packages for particular versions or variant values โ they only set defaults. The concretizer is free to change them if it must, due to other constraints, and it also prefers reusing installed packages over building new ones that are a better match for preferences. It's giving "suggestion, not a demand." ๐
SEE ALSO: FAQ: Why does Spack pick particular versions and variants?
Most package preferences (compilers, target and providers) can only be set globally under the all section of packages.yaml:
packages:
all:
compiler: [gcc@12.2.0, clang@12:, oneapi@2023:]
target: [x86_64_v3]
providers:
mpi: [mvapich2, mpich, openmpi]These preferences override Spack's default and effectively reorder priorities when looking for the best compiler, target or virtual package provider. Each preference takes an ordered list of spec constraints, with earlier entries in the list preferred over later ones. First place gets the crown ๐
In the example above, all packages prefer to be compiled with gcc@12.2.0, to target the x86_64_v3 microarchitecture, and to use mvapich2 if they depend on mpi.
The variants and version preferences can be set under package-specific sections of the packages.yaml file:
packages:
opencv:
variants: +debug
gperftools:
version: [2.2, 2.4, 2.3]In this case, the preference for opencv is to build with debug options, while gperftools prefers version 2.2 over 2.4. (2.3 is there too, just chillin' at the end of the list.)
Any preference can be overwritten on the command line if explicitly requested. Your word on the command line wins, always.
Preferences cannot overcome explicit constraints โ they only set a preferred ordering among homogeneous attribute values. Going back to the example, if gperftools@2.3: was requested, then Spack will install version 2.4 since the most preferred version 2.2 is prohibited by the version constraint. The 2.2 dream is dead, we move. ๐ญ
๐ธ Package Permissions ๐ธ
Spack can be configured to assign permissions to the files installed by a package. Yes, actual file permissions, we're being responsible today ๐ซถ
In the packages.yaml file under permissions, the attributes read, write, and group control the package permissions. These attributes can be set per-package, or for all packages under all. If permissions are set under all and for a specific package, the package-specific settings take precedence.
The read and write attributes take one of user, group, and world.
packages:
all:
permissions:
write: group
group: spack
my_app:
permissions:
read: group
group: my_teamSo the permissions settings are basically describing the broadest level of access that installations of the packages you listed can get. And here's a cute lil detail: for files that are executable, the execute permissions get set to the same level as the read permissions. ๐
The default for read is world, and the default for write is user โ that's just how it ships out of the box.
So in the example above (the one with the YAML, you remember ๐), installations of my_app are gonna be installed with user and group permissions but no world permissions, and they're owned by the group my_team. Slay, private app. Meanwhile all the other packages get installed with user and group write privileges plus world read privileges, and those packages are owned by the group spack. Bestie, that's the default vibe for everyone else.
The group attribute is the one that assigns a Unix-style group to a package. Lowkey important: all files installed by the package will be owned by the assigned group, and the sticky group bit gets set on the install prefix and all directories inside the install prefix. Why do we care? Because that makes sure even files you dropped in there manually, inside the install prefix, end up owned by the assigned group. ๐ซถ Teamwork. If no group is assigned, Spack just lets the OS default behavior go as expected โ no meddling, no drama.
Assigning Package Attributes
Okay so you can also assign class-level attributes right in the configuration, like this:
packages:
mpileaks:
package_attributes:
# Override existing attributes
url: http://www.somewhereelse.com/mpileaks-1.0.tar.gz
# ... or add new ones
x: 1Attributes you set this way are accessible to any method executed in the package.py file (yes, even the install() method โ it can peek at them ๐). And the values for these attributes can be literally any value that's parseable by yaml. Anything. Iconic.
Heads up bestie: these can only be applied to specific packages โ not to "all" and not to virtual packages. Sorry, no blanket rules here. ๐ญ
โจ CONCRETIZATION SETTINGS (CONCRETIZER.YAML) โจ
okay so concretizer.yaml is the config file that lets you customize how Spack picks your dependencies when it concretizes. it's giving "the algorithm's vibe settings" ๐ง. the default config is literally this:
# -------------------------------------------------------------------------
# This is the default spack configuration file.
#
# Settings here are versioned with Spack and are intended to provide
# sensible defaults out of the box. Spack maintainers should edit this
# file to keep it current.
#
# Users can override these settings by editing
# `$SPACK_ROOT/etc/spack/concretizer.yaml`, `~/.spack/concretizer.yaml`,
# or by adding a `concretizer:` section to an environment.
# -------------------------------------------------------------------------
concretizer:
# Whether to consider installed packages or packages from buildcaches when
# concretizing specs. If `true`, we'll try to use as many installs/binaries
# as possible, rather than building. If `false`, we'll always give you a fresh
# concretization. If `dependencies`, we'll only reuse dependencies but
# give you a fresh concretization for your root specs.
reuse: true
# Options that tune which targets are considered for concretization. The
# concretization process is very sensitive to the number targets, and the time
# needed to reach a solution increases noticeably with the number of targets
# considered.
targets:
# Determine whether we want to target specific or generic
# microarchitectures. Valid values are: "microarchitectures" or "generic".
# An example of "microarchitectures" would be "skylake" or "bulldozer",
# while an example of "generic" would be "aarch64" or "x86_64_v4".
granularity: microarchitectures
# If "false" allow targets that are incompatible with the current host (for
# instance concretize with target "icelake" while running on "haswell").
# If "true" only allow targets that are compatible with the host.
host_compatible: true
# When "true" concretize root specs of environments together, so that each unique
# package in an environment corresponds to one concrete spec. This ensures
# environments can always be activated. When "false" perform concretization separately
# on each root spec, allowing different versions and variants of the same package in
# an environment.
unify: true
# Option to deal with possible duplicate nodes (i.e. different nodes from the same package) in the DAG.
duplicates:
# "none": allows a single node for any package in the DAG.
# "minimal": allows the duplication of 'build-tools' nodes only (e.g. py-setuptools, cmake etc.)
# "full" (experimental): allows separation of the entire build-tool stack (e.g. the entire "cmake" subDAG)
strategy: minimal
# Option to specify compatiblity between operating systems for reuse of compilers and packages
# Specified as a key: [list] where the key is the os that is being targeted, and the list contains the OS's
# it can reuse. Note this is a directional compatibility so mutual compatibility between two OS's
# requires two entries i.e. os_compatible: {sonoma: [monterey], monterey: [sonoma]}
os_compatible: {}like. it's a whole essay in there but every comment is a bop, read them ๐
๐ซถ Reuse already installed packages
the reuse attribute is the one that controls how aggressively Spack reuses binary packages during concretization. it can be either a single value, or an object if you're feeling fancy and want more complex configurations. two modes bestie ๐
in the "single value" case, it lets Spack:
- reuse installed packages and buildcaches for all the specs to be concretized, when
true - reuse installed packages and buildcaches only for the dependencies of the root specs, when
dependencies - straight up disregard reusing installed packages and buildcaches, when
false
and if you need finer control over which specs get reused, the value of this attribute can be an object, with these keys:
roots: iftrueroot specs are reused, iffalseonly dependencies of root specs are reusedfrom: list of sources from which reused specs are taken
each source in from is itself an object ๐
โจ Attributes for a source or reusable specs
type(mandatory, string) โ can belocal,buildcache, orexternalinclude(optional, list of specs) โ if present, reusable specs must match at least one of the constraint in the listexclude(optional, list of specs) โ if present, reusable specs must not match any of the constraint in the list
for instance, this configuration:
concretizer:
reuse:
roots: true
from:
- type: local
include:
- "%gcc"
- "%clang"tells the concretizer to reuse all specs compiled with either gcc or clang, that are installed in the local store. any spec from remote buildcaches is disregarded ๐ซ
to reduce the boilerplate in configuration files, default values for the include and exclude options can be pushed up one level:
concretizer:
reuse:
roots: true
include:
- "%gcc"
from:
- type: local
- type: buildcache
- type: local
include:
- "foo %oneapi"in the example above we reuse all specs compiled with gcc from the local store and remote buildcaches, and we also reuse foo %oneapi. note that the last source of specs override the default include attribute ๐
for one-off concretizations, there are command line arguments for each of the simple "single value" configurations. this means a user can:
% spack install --reuse <spec>to enable reuse for a single installation, or:
spack install --fresh <spec>to do a fresh install if reuse is enabled by default ๐ซ
SEE ALSO:
- FAQ: Why does Spack pick particular versions and variants?
๐ฆ Selection of the target microarchitectures
the options under the targets attribute control which targets are considered during a solve. currently the options in this section are only configurable from the concretizer.yaml file โ there are no corresponding command line arguments to enable them for a single solve. so like, config file only, no shortcuts ๐
the granularity option can take two possible values: microarchitectures and generic. if set to:
concretizer:
targets:
granularity: microarchitecturesSpack will consider all the microarchitectures known to archspec to label nodes for compatibility. if instead the option is set to:
concretizer:
targets:
granularity: genericSpack will consider only generic microarchitectures. for instance, when running on an Haswell node, Spack will consider haswell as the best target in the former case and x86_64_v3 as the best target in the latter case ๐
the host_compatible option is a Boolean option that determines whether or not the microarchitectures considered during the solve are constrained to be compatible with the host Spack is currently running on. for instance, if this option is set to true, a user cannot concretize for target=icelake while running on an Haswell node ๐
๐ Duplicate nodes
the duplicates attribute controls whether the DAG can contain multiple configurations of the same package. this is mainly relevant for build dependencies, which may have their version pinned by some nodes, and thus be required at different versions by different nodes in the same DAG. messy but valid โจ
the strategy option controls how the solver deals with duplicates. if the value is none, then a single configuration per package is allowed in the DAG. this means, for instance, that only a single cmake or a single py-setuptools version is allowed. the result would be a slightly faster concretization, at the expense of making a few specs unsolvable ๐ญ
if the value is minimal Spack will allow packages tagged as build-tools to have duplicates. this allows, for instance, to concretize specs whose nodes require different, and incompatible, ranges of some build tool. for instance, in the figure below the latest py-shapely requires a newer py-setuptools, while py-numpy still needs an older version:
[image]up to Spackv0.20duplicates:strategy:nonewas the default (and only) behavior. from Spackv0.21the default behavior isduplicates:strategy:minimal. ๐ฅ
๐ ENVIRONMENTS (SPACK.YAML) ๐
an environment is used to group together a set of specs for the purpose of building, rebuilding and deploying in a coherent fashion. environments provide a number of advantages over the ร la carte approach of building and loading individual Spack modules:
- environments separate the steps of (a) choosing what to install, (b) concretizing, and (c) installing. this allows Environments to remain stable and repeatable, even if Spack packages are upgraded: specs are only re-concretized when the user explicitly asks for it. it is even possible to reliably transport environments between different computers running different versions of Spack! ๐
- environments allow several specs to be built at once; a more robust solution than ad-hoc scripts making multiple calls to
spack install๐ซถ - an Environment that is built as a whole can be loaded as a whole into the user environment. an Environment can be built to maintain a filesystem view of its packages, and the environment can load that view into the user environment at activation time. Spack can also generate a script to load all modules related to an environment ๐
okay so like other packaging systems also ship environments that are similar in some ways to Spack environments โ Conda environments and Python Virtual Environments are the classic examples, iykyk. But Spack environments come with some genuinely distinctive features:
- A spec installed "in" an environment is no different from the same spec installed anywhere else in Spack. Environments are assembled simply by collecting together a set of specs.
- Spack Environments may contain more than one spec of the same package โ bestie, that's fully allowed and honestly iconic.
Spack uses a "manifest and lock" model similar to Bundler gemfiles and other package managers. The user input file is named spack.yaml and the lock file is named spack.lock.
๐ธ Using Environments
Here we follow a typical use case of creating, concretizing, installing and loading an environment. Slay, let's go.
๐ Creating a managed Environment
An environment is created by:
$ spack env create myenvSpack then creates the directory var/spack/environments/myenv.
NOTE: All managed environments by default are stored in thevar/spack/environmentsfolder. This location can be changed by setting theenvironments_rootvariable inconfig.yaml.
In the var/spack/environments/myenv directory, Spack creates the file spack.yaml and the hidden directory .spack-env.
Spack stores metadata in the .spack-env directory. User interaction will occur through the spack.yaml file and the Spack commands that affect it. When the environment is concretized, Spack will create a file spack.lock with the concrete information for the environment.
In addition to being the default location for the view associated with an Environment, the .spack-env directory also contains:
repo/: A repo consisting of the Spack packages used in this environment. This allows the environment to build the same, in theory, even on different versions of Spack with different packages! ๐คฏlogs/: A directory containing the build logs for the packages in this Environment.
Spack Environments can also be created from either a manifest file (usually but not necessarily named, spack.yaml) or a lockfile. To create an Environment from a manifest:
$ spack env create myenv spack.yamlTo create an Environment from a spack.lock lockfile:
$ spack env create myenv spack.lockEither of these commands can also take a full path to the initialization file.
A Spack Environment created from a spack.yaml manifest is guaranteed to have the same root specs as the original Environment, but may concretize differently. A Spack Environment created from a spack.lock lockfile is guaranteed to have the same concrete specs as the original Environment. Either may obviously then differ as the user modifies it. Mood, honestly.
โจ Activating an Environment
To activate an environment, use the following command:
$ spack env activate myenvBy default, spack env activate will load the view associated with the Environment into the user environment. The -v, --with-view argument ensures this behavior, and the -V, --without-view argument activates the environment without changing the user environment variables.
The -p option to the spack env activate command modifies the user's prompt to begin with the environment name in brackets.
$ spack env activate -p myenv
[myenv] $ ...The activate command can also be used to create a new environment if it does not already exist.
$ spack env activate --create -p myenv
# ...
# [creates if myenv does not exist yet]
# ...
[myenv] $ ...To deactivate an environment, use the command:
$ spack env deactivateor the shortcut alias
$ despacktivateIf the environment was activated with its view, deactivating the environment will remove the view from the user environment.
๐ฆ Anonymous Environments
Apart from managed environments, Spack also supports anonymous environments. Lowkey mysterious, highkey useful.
Anonymous environments can be placed in any directory of choice.
NOTE: When uninstalling packages, Spack asks the user to confirm the removal of packages that are still used in a managed environment. This is not the case for anonymous environments.
To create an anonymous environment, use one of the following commands:
$ spack env create --dir my_env
$ spack env create ./my_envAs a shorthand, you can also create an anonymous environment upon activation if it does not already exist:
$ spack env activate --create ./my_envFor convenience, Spack can also place an anonymous environment in a temporary directory for you:
$ spack env activate --temp๐ Environment Sensitive Commands
Spack commands are environment sensitive. For example, the find command shows only the specs in the active Environment if an Environment has been activated. Similarly, the install and uninstall commands act on the active environment.
$ spack find
==> 0 installed packages
$ spack install zlib@1.2.11
==> Installing zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv
==> No binary for zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv found: installing from source
==> zlib: Executing phase: 'install'
[+] ~/spack/opt/spack/linux-rhel7-broadwell/gcc-8.1.0/zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv
$ spack env activate myenv
$ spack find
==> In environment myenv
==> No root specs
==> 0 installed packages
$ spack install zlib@1.2.8
==> Installing zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x
==> No binary for zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x found: installing from source
==> zlib: Executing phase: 'install'
[+] ~/spack/opt/spack/linux-rhel7-broadwell/gcc-8.1.0/zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x
==> Updating view at ~/spack/var/spack/environments/myenv/.spack-env/view
$ spack find
==> In environment myenv
==> Root specs
zlib@1.2.8
==> 1 installed package
-- linux-rhel7-broadwell / gcc@8.1.0 ----------------------------
zlib@1.2.8
$ despacktivate
$ spack find
==> 2 installed packages
-- linux-rhel7-broadwell / gcc@8.1.0 ----------------------------
zlib@1.2.8 zlib@1.2.11Note that when we installed the abstract spec zlib@1.2.8, it was presented as a root of the Environment. All explicitly installed packages will be listed as roots of the Environment.
All of the Spack commands that act on the list of installed specs are Environment-sensitive in this way, including install, uninstall, find, extensions, and more. In the Configuring Environments section we will discuss Environment-sensitive commands further.
๐ Adding Abstract Specs
An abstract spec is the user-specified spec before Spack has applied any defaults or dependency information.
Users can add abstract specs to an Environment using the spack add command. The most important component of an Environment is a list of abstract specs.
Adding a spec adds to the manifest (the spack.yaml file), which is used to define the roots of the Environment, but does not affect the concrete specs in the lockfile, nor does it install the spec.
The spack add command is environment aware. It adds to the currently active environment. All environment aware commands can also be called using the spack -e flag to specify the environment.
$ spack env activate myenv
$ spack add mpileaksor
$ spack -e myenv add python๐ง Concretizing
Once some user specs have been added to an environment, they can be concretized. There are at the moment three different modes of operation to concretize an environment, which are explained in details in Spec concretization. Regardless of which mode of operation has been chosen, the following command will ensure all the root specs are concretized according to the constraints that are prescribed in the configuration:
[myenv]$ spack concretizeIn the case of specs that are not concretized together, the command above will concretize only the specs that were added and not yet concretized. Forcing a re-concretization of all the specs can be done instead with this command:
[myenv]$ spack concretize -fWhen the -f flag is not used to reconcretize all specs, Spack guarantees that already concretized specs are unchanged in the environment.
The concretize command does not install any packages. For packages that have already been installed outside of the environment, the process of adding the spec and concretizing is identical to installing the spec assuming it concretizes to the exact spec that was installed outside of the environment.
The spack find command can show concretized specs separately from installed specs using the -c (--concretized) flag.
[myenv]$ spack add zlib
[myenv]$ spack concretize
[myenv]$ spack find -c
==> In environment myenv
==> Root specs
zlib
==> Concretized roots
-- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
zlib@1.2.11
==> 0 installed packages๐ Installing an Environment
In addition to installing individual specs into an Environment, one can install the entire Environment at once using the command
[myenv]$ spack installIf the Environment has been concretized, Spack will install the concretized specs. Otherwise, spack install will first concretize the Environment and then install the concretized specs.
NOTE: Everyspack installprocess builds one package at a time with multiple build jobs, controlled by the-jflag and theconfig:build_jobsoption (see build_jobs). To speed up environment builds further, independent packages can be installed in parallel by launching more Spack instances. For example, the following will build at most four packages in parallel using three background jobs:
[myenv]$ spack install & spack install & spack install & spack installAnother option is to generate a Makefile and run make -j<N> to control the number of parallel install processes. See Generating Depfiles from Environments for details.
As it installs, spack install creates symbolic links in the logs/ directory in the Environment, allowing for easy inspection of build logs related to that environment. The spack install command also stores a Spack repo containing the package.py file used at install time for each package in the repos/ directory in the Environment.
The --no-add option? Okay so bestie, in a concrete environment this one tells spack to install specs that are already chillin' in the environment but to not add any new root specs to it. For root specs you hand to spack install on the command line, --no-add is literally the default. For dependency specs, though? It's optional. In other words: if there's an unambiguous match in the active concrete environment for a root spec you gave spack install, spack does not require you to say --no-add to keep the spec from being added again. But at the same time, a spec that already exists in the environment โ but only as a dependency โ will get added to the environment as a root spec without the --no-add option. ๐
๐ ๏ธ Developing Packages in a Spack Environment
The spack develop command lets you, like, develop Spack packages right inside an environment. It needs a spec with a concrete version, and it'll configure Spack to install the package from local source. By default it also clones the package into a subdirectory in the environment. That package gets a special variant called dev_path, and Spack will make sure the package and its dependents get rebuilt any time the environment is installed if the package's local source code has been modified. Spack also ensures that every instance of a developed package in the environment is concretized to match the version (and other constraints) you passed as the spec argument to the spack develop command. ๐ฆ
For packages with git attributes, git branches, tags, and commits can also be valid concrete versions (check the Version specifier section). So for a package foo, spack develop foo@git.main will clone the main branch of the package, and spack install will install from that git clone if foo is in the environment. You can test further development on foo by reinstalling the environment, and eventually commit and push it to the upstream git repo. โจ
If the package you're developing supports out-of-source builds, you can use the --build_directory flag to control the location and name of the build directory. This is a shortcut to set package_attributes:build_directory in the packages configuration (see Assigning Package Attributes). Whatever location you supply becomes the build-directory for that package in all future builds.
WARNING: potential pitfalls of setting the build directory โ ๏ธ Spack does not check for out-of-source build compatibility with the packages, so the onus of making sure the package supports out-of-source builds is on you. For example, most autotool and makefile packages do not support out-of-source builds, while all CMake packages do. Understanding these nuances is on the software developers, and we strongly encourage developers to only redirect the build directory if they understand their package's build-system. ๐ฌ
๐ฆ Loading
Once an environment has been installed, the following creates a load script for it:
$ spack env loads -rThis creates a file called loads in the environment directory. Sourcing that file in Bash makes the environment available to you, and it can be included in .bashrc files, etc. The loads file can also be copied out of the environment, renamed, whatever you're feeling. ๐
๐ Included Concrete Environments
Spack environments can create an environment based off of information in already established environments. Think of it as a combination of existing environments. It gathers information from the existing environment's spack.lock and uses that during the creation of this included concrete environment. When an included concrete environment is created, it generates a spack.lock file for the newly created environment.
๐ฑ Creating included environments
To create a combined concrete environment, you need at least one existing concrete environment. You'll use the command spack env create with the argument --include-concrete followed by the name or path of the environment you'd like to include. Here's an example of creating a combined environment from the command line:
$ spack env create myenv
$ spack -e myenv add python
$ spack -e myenv concretize
$ spack env create --include-concrete myenv included_envYou can also include an environment directly in the spack.yaml file. It involves adding the include_concrete heading in the yaml followed by the absolute path to the independent environments:
spack:
specs: []
concretizer:
unify: true
include_concrete:
- /absolute/path/to/environment1
- /absolute/path/to/environment2Once the spack.yaml has been updated, you must concretize the environment to get the concrete specs from the included environments.
๐ Updating an included environment
If changes were made to the base environment and you want them reflected in the included environment, you'll need to reconcretize both the base environment and the included environment for the change to be implemented. For example:
$ spack env create myenv
$ spack -e myenv add python
$ spack -e myenv concretize
$ spack env create --include-concrete myenv included_env
$ spack -e myenv find
==> In environment myenv
==> Root specs
python
==> 0 installed packages
$ spack -e included_env find
==> In environment included_env
==> No root specs
==> Included specs
python
==> 0 installed packagesHere we see that included_env has access to the python package through the myenv environment. But if we were to add another spec to myenv, included_env will not be able to access the new information:
$ spack -e myenv add perl
$ spack -e myenv concretize
$ spack -e myenv find
==> In environment myenv
==> Root specs
perl python
==> 0 installed packages
$ spack -e included_env find
==> In environment included_env
==> No root specs
==> Included specs
python
==> 0 installed packagesIt isn't until you run the spack concretize command that the combined environment will get the updated information from the reconcretized base environment:
$ spack -e included_env concretize
$ spack -e included_env find
==> In environment included_env
==> No root specs
==> Included specs
perl python
==> 0 installed packagesโ๏ธ Configuring Environments
A whole variety of Spack behaviors get changed through Spack configuration files โ that's covered in more detail in the Configuration Files section.
Spack Environments give you an additional level of configuration scope between the custom scope and the user scope discussed in the configuration documentation.
There are two ways to include configuration information in a Spack Environment:
- Inline in the
spack.yamlfile - Included in the
spack.yamlfile from another file.
Many Spack commands also affect configuration information in files automatically. Those commands take a --scope argument, and the environment can be specified by env:NAME (to affect environment foo, set --scope env:foo). These commands will automatically manipulate configuration inline in the spack.yaml file.
๐ Inline configurations
Inline Environment-scope configuration is done using the same yaml format as standard Spack configuration scopes, covered in the Configuration Files section. Each section is contained under a top-level yaml object with its name. For example, a spack.yaml manifest file containing some package preference configuration (as in a packages.yaml file) could contain:
spack:
# ...
packages:
all:
compiler: [intel]
# ...This configuration sets the default compiler for all packages to intel.
๐ Included configurations
Spack environments allow an include heading in their yaml schema. This heading pulls in external configuration files and applies them to the Environment.
spack:
include:
- relative/path/to/config.yaml
- https://github.com/path/to/raw/config/compilers.yaml
- /absolute/path/to/packages.yamlEnvironments can include files or URLs. File paths can be relative or absolute. URLs include the path to the text for individual files or can be the path to a directory containing configuration files.
๐ฅ Configuration precedence
Inline configurations take precedence over included configurations, so you don't have to change shared configuration files just to make small changes to an individual environment. Included configurations listed earlier have higher precedence, since the included configs are applied in reverse order. ๐
โ๏ธ Manually Editing the Specs List
The list of abstract/root specs in the Environment is maintained in the spack.yaml manifest under the heading specs.
spack:
specs:
- ncview
- netcdf
- nco
- py-sphinxAppending to this list in the yaml is identical to using the spack add command from the command line. However, there's more power available from the yaml file. ๐ช
๐งฉ Spec concretization
An environment can be concretized in three different modes, and the behavior active under any environment is determined by the concretizer:unify configuration option.
The default mode is to unify all specs:
spack:
specs:
- hdf5+mpi
- zlib@1.2.8
concretizer:
unify: trueThis means any package in the environment corresponds to a single concrete spec. In the above example, when hdf5 depends down the line of zlib, it's required to take zlib@1.2.8 instead of a newer version. This mode of concretization is particularly useful when environment views are used: if every package occurs in only one flavor, it's usually possible to merge all install directories into a view.
A downside of unified concretization is that it can be overly strict. For example, a concretization error would happen when both hdf5+mpi and hdf5~mpi are specified in an environment.
The second mode is to unify when possible: this makes concretization of root specs more independent. Instead of requiring reuse of dependencies across different root specs, it's only maximized:
spack:
specs:
- hdf5~mpi
- hdf5+mpi
- zlib@1.2.8
concretizer:
unify: when_possibleThis means both hdf5 installations will use zlib@1.2.8 as a dependency even if newer versions of that library are available.
The third mode of operation is to concretize root specs entirely independently by disabling unified concretization: ๐
spack:
specs:
- hdf5~mpi
- hdf5+mpi
- zlib@1.2.8
concretizer:
unify: falseSo in this example, hdf5 gets concretized all by itself, and it literally does not consider zlib@1.2.8 as a constraint or a preference. Instead it just goes and grabs the latest possible version. Independent queen behavior โจ
The last two concretization options are typically useful for system admins and the user-support girlies who are out here providing a whole big software stack for their HPC center. Iconic infrastructure work tbh ๐
NOTE: theconcretizer:unifyconfig option was introduced in Spack 0.18 to replace theconcretizationproperty. For reference,concretization: togetheris replaced byconcretizer:unify:true, andconcretization: separatelyis replaced byconcretizer:unify:false.
Re-concretization of user specs โ okay this part matters, pay attention bestie ๐
Thespack concretizecommand without additional arguments will not change any previously concretized specs. This may prevent it from finding a solution when usingunify: true, and it may prevent it from finding a minimal solution when usingunify: when_possible. You can force Spack to ignore the existing concrete environment withspack concretize -f.
๐ Spec Matrices ๐
Entries in the specs list can be individual abstract specs or a spec matrix.
A spec matrix is a yaml object containing multiple lists of specs, and it evaluates to the cross-product of those specs. Spec matrices also contain an excludes directive, which eliminates certain combinations from the evaluated result. It's giving "mix and match but make it math" ๐ง
The following two Environment manifests are identical:
spack:
specs:
- zlib %gcc@7.1.0
- zlib %gcc@4.9.3
- libelf %gcc@7.1.0
- libelf %gcc@4.9.3
- libdwarf %gcc@7.1.0
- cmakespack:
specs:
- matrix:
- [zlib, libelf, libdwarf]
- ['%gcc@7.1.0', '%gcc@4.9.3']
exclude:
- libdwarf%gcc@4.9.3
- cmakeSpec matrices can be used to install swaths of software across various toolchains. So many packages, so little time ๐ซถ
๐ Spec List References ๐
The last type of possible entry in the specs list is a reference.
The Spack Environment manifest yaml schema contains an additional heading definitions. Under definitions is an array of yaml objects. Each object has one or two fields. The one required field is a name, and the optional field is a when clause.
The named field is a spec list. The spec list uses the same syntax as the specs entry. Each entry in the spec list can be a spec, a spec matrix, or a reference to an earlier named list. References are specified using the $ sigil, and are "splatted" into place (i.e. the elements of the referent are at the same level as the elements listed separately). As an example, the following two manifest files are identical. Splat is such a fun word ngl ๐
spack:
definitions:
- first: [libelf, libdwarf]
- compilers: ['%gcc', '%intel']
- second:
- $first
- matrix:
- [zlib]
- [$compilers]
specs:
- $second
- cmakespack:
specs:
- libelf
- libdwarf
- zlib%gcc
- zlib%intel
- cmakeNOTE: named spec lists in the definitions section may only refer to a named list defined above itself. Order matters. No time travel allowed, bestie โณ
In short files like the example, it may be easier to simply list the included specs. However for more complicated examples involving many packages across many toolchains, separately factored lists make Environments substantially more manageable. Like, your future self will thank you ๐
Additionally, the -l option to the spack add command allows one to add to named lists in the definitions section of the manifest file directly from the command line. Chef's kiss ๐
The when directive can be used to conditionally add specs to a named list. The when directive takes a string of Python code referring to a restricted set of variables, and evaluates to a boolean. The specs listed are appended to the named list if the when string evaluates to True. In the following snippet, the named list compilers is ['%gcc', '%clang', '%intel'] on x86_64 systems and ['%gcc', '%clang'] on all other systems. Conditional slay ๐ธ
spack:
definitions:
- compilers: ['%gcc', '%clang']
- when: arch.satisfies('x86_64:')
compilers: ['%intel']NOTE: any definitions with the same named list with truewhenclauses (or absentwhenclauses) will be appended together.
The valid variables for a when clause are:
platform. The platform string of the default Spack architecture on the system.os. The os string of the default Spack architecture on the system.target. The target string of the default Spack architecture on the system.architectureorarch. A Spack spec satisfying the default Spack architecture on the system. This supports querying via thesatisfiesmethod, as shown above.arch_str. The architecture string of the default Spack architecture on the system.re. The standard regex module inPython.env. The user environment (usuallyos.environinPython).hostname. The hostname of the system (ifhostnameis an executable in the user'sPATH).
๐ SpecLists as Constraints ๐
Dependencies and compilers in Spack can be both packages in an environment and constraints on other packages. References to SpecLists allow a shorthand to treat packages in a list as either a compiler or a dependency using the $% or $^ syntax respectively. Double duty, we love a multitasker ๐ฆ
For example, the following environment has three root packages: gcc@8.1.0, mvapich2@2.3.1 %gcc@8.1.0, and hdf5+mpi %gcc@8.1.0 ^mvapich2@2.3.1.
spack:
definitions:
- compilers: [gcc@8.1.0]
- mpis: [mvapich2@2.3.1]
- packages: [hdf5+mpi]
specs:
- $compilers
- matrix:
- [$mpis]
- [$%compilers]
- matrix:
- [$packages]
- [$^mpis]
- [$%compilers]This allows for a much-needed reduction in redundancy between packages and constraints. DRY code is main character energy fr fr โจ
๐ฆ Filesystem Views ๐ฆ
Spack Environments can define filesystem views, which provide a direct access point for software similar to the directory hierarchy that might exist under /usr/local. Filesystem views are updated every time the environment is written out to the lock file spack.lock, so the concrete environment and the view are always compatible. The files of the view's installed packages are brought into the view by symbolic or hard links, referencing the original Spack installation, or by copy. Lowkey it's a whole little curated neighborhood of software ๐๏ธ
โญ Configuration in spack.yaml โญ
The Spack Environment manifest file has a top-level keyword view. Each entry under that heading is a view descriptor, headed by a name. Any number of views may be defined under the view heading. The view descriptor contains the root of the view, and optionally the projections for the view, select and exclude lists for the view and link information via link and link_type.
For example, in the following manifest file snippet we define a view named mpis, rooted at /path/to/view in which all projections use the package name, version, and compiler name to determine the path for a given package. This view selects all packages that depend on MPI, and excludes those built with the PGI compiler at version 18.5. The root specs with their (transitive) link and run type dependencies will be put in the view due to the link: all option, and the files in the view will be symlinks to the spack install directories. Obsessed with how precise this is ๐
spack:
# ...
view:
mpis:
root: /path/to/view
select: [^mpi]
exclude: ['%pgi@18.5']
projections:
all: '{name}/{version}-{compiler.name}'
link: all
link_type: symlinkThe default for the select and exclude values is to select everything and exclude nothing. The default projection is the default view projection ({}). The link attribute allows the following values:
link: allinclude root specs with their transitive run and link type dependencies (default);link: runinclude root specs with their transitive run type dependencies;link: rootsinclude root specs without their dependencies.
The link_type defaults to symlink but can also take the value of hardlink or copy.
TIP: the optionlink: runcan be used to create small environment views forPythonpackages.Pythonwill be able to import packages inside of the view even when the environment is not activated, and linked libraries will be located outside of the view thanks to rpaths.
There are two shorthands for environments with a single view. If the environment at /path/to/env has a single view, with a root at /path/to/env/.spack-env/view, with default selection and exclusion and the default projection, we can put view: True in the environment manifest. Similarly, if the environment has a view with a different root, but default selection, exclusion, and projections, the manifest can say view: /path/to/view. These views are automatically named default, so that
spack:
# ...
view: Trueis equivalent to
spack:
# ...
view:
default:
root: .spack-env/viewand
spack:
# ...
view: /path/to/viewis equivalent to
spack:
# ...
view:
default:
root: /path/to/viewBy default, Spack environments are configured with view: True in the manifest. Environments can be configured without views using view: False. For backwards compatibility reasons, environments with no view key are treated the same as view: True. Nostalgic defaults, we respect them ๐
From the command line, the spack env create command takes an argument --with-view [PATH] that sets the path for a single, default view. If no path is specified, the default path is used (view: True). The argument --without-view can be used to create an environment without any view configured.
The spack env view command can be used to change the manage views of an Environment. The subcommand spack env view enable will add a view named default to an environment. It takes an optional argument to specify the path for the new default view. The subcommand spack env view disable will remove the view named default from an environment if one exists. The subcommand spack env view regenerate will regenerate the views for the environment. This will apply any updates in the environment configuration that have not yet been applied. Go off, view manager ๐ซถ
๐ญ View Projections
ok so by default, the projection into a view is literally "link every package into the root of the view" โ simple, no thoughts, iconic ๐
but you can absolutely switch that up with the projections attribute: it's a mapping of partial specs โ spec format strings, defined by the format() function, like this:
projections:
zlib: "{name}-{version}"
^mpi: "{name}-{version}/{^mpi.name}-{^mpi.version}-{compiler.name}-{compiler.version}"
all: "{name}-{version}/{compiler.name}-{compiler.version}"Projections also permit environment and spack configuration variable expansions, bestie:
projections:
all: "{name}-{version}/{compiler.name}-{compiler.version}/$date/$SYSTEM_ENV_VARIBLE"where $date is the spack configuration variable that will expand with the YYYY-MM-DD format and $SYSTEM_ENV_VARIABLE is an environment variable defined in the shell. โจ
The entries in the projections configuration file must all be either specs or the keyword all. For each spec, the projection used will be the first non-all entry that the spec satisfies, or all if there is an entry for all and no other entry is satisfied by the spec. Where the keyword all appears in the file does not matter โ it's not about location, it's about vibes.
Given the example above, the spec zlib@1.2.8 will be linked into /my/view/zlib-1.2.8/, the spec hdf5@1.8.10+mpi %gcc@4.9.3 ^mvapich2@2.2 will be linked into /my/view/hdf5-1.8.10/mvapich2-2.2-gcc-4.9.3, and the spec hdf5@1.8.10~mpi %gcc@4.9.3 will be linked into /my/view/hdf5-1.8.10/gcc-4.9.3. see how that works?? chef's kiss ๐งโ๐ณ
If the keyword all does not appear in the projections configuration file, any spec that does not satisfy any entry in the file will be linked into the root of the view as in a single-prefix view. Any entries that appear below the keyword all in the projections configuration file will not be used, as all specs will use the projection under all before reaching those entries. so likeโฆ don't put stuff under all and expect it to matter, because it will not. ๐
๐ฌ Activating environment views
The spack env activate command will put the default view for the environment into the user's path, in addition to activating the environment for Spack commands. The arguments -v,--with-view and -V,--without-view can be used to tune this behavior. The default behavior is to activate with the environment view if there is one. ๐ซถ
The environment variables affected by the spack env activate command and the paths that are used to update them are determined by the prefix inspections defined in your modules configuration; the defaults are summarized as follows:
PATHโbinMANPATHโman,share/manACLOCAL_PATHโshare/aclocalPKG_CONFIG_PATHโlib/pkgconfig,lib64/pkgconfig,share/pkgconfigCMAKE_PREFIX_PATHโ.
Each of these paths are appended to the view root, and added to the relevant variable if the path exists. For this reason, it is not recommended to use non-default projections with the default view of an environment. like, genuinely, don't do it. ๐ญ
The spack env deactivate command will remove the default view of the environment from the user's path. bye bestie ๐
๐ฆ Generating Depfiles from Environments
Spack can generate Makefiles to make it easier to build multiple packages in an environment in parallel. Generated Makefiles expose targets that can be included in existing Makefiles, to allow other targets to depend on the environment installation. parallel builds are the moment, fr fr ๐ฅ
A typical workflow is as follows:
spack env create -d .
spack -e . add perl
spack -e . concretize
spack -e . env depfile -o Makefile
make -j64This generates a Makefile from a concretized environment in the current working directory, and make -j64 installs the environment, exploiting parallelism across packages as much as possible. Spack respects the Make jobserver and forwards it to the build environment of packages, meaning that a single -j flag is enough to control the load, even when packages are built in parallel. one flag. that's it. that's the whole thing. ๐
By default the following phony convenience targets are available:
make all: installs the environment (default target);make clean: cleans files used bymake, but does not uninstall packages.
TIP: GNU Make version 4.3 and above have great support for output synchronization through the-Oand--output-syncflags, which ensure that output is printed orderly per package install. To get synchronized output with colors, usemake -j<N> SPACK_COLOR=always --output-sync=recurse. ๐
๐ Specifying dependencies on generated make targets
An interesting question is how to include generated Makefiles in your own Makefiles. This comes up when you want to install an environment that provides executables required in a command for a make target of your own. relatable problem honestly.
The example below shows how to accomplish this: the env target specifies the generated spack/env target as a prerequisite, meaning that the environment gets installed and is available for use in the env target.
SPACK ?= spack
.PHONY: all clean env
all: env
spack.lock: spack.yaml
$(SPACK) -e . concretize -f
env.mk: spack.lock
$(SPACK) -e . env depfile -o $@ --make-prefix spack
env: spack/env
$(info Environment installed!)
clean:
rm -rf spack.lock env.mk spack/
ifeq (,$(filter clean,$(MAKECMDGOALS)))
include env.mk
endifThis works as follows: when make is invoked, it first "remakes" the missing include env.mk as there is a target for it. This triggers concretization of the environment and makes spack output env.mk. At that point the generated target spack/env becomes available through include env.mk. it's giving sleight of hand but make it build system ๐ช
As it is typically undesirable to remake env.mk as part of make clean, the include is conditional.
NOTE: When including generated Makefiles, it is important to use the--make-prefixflag and use the non-phony target<prefix>/envas prerequisite, instead of the phony target<prefix>/all.
๐ฏ Building a subset of the environment
The generated Makefiles contain install targets for each spec, identified by <name>-<version>-<hash>. This allows you to install only a subset of the packages in the environment. When packages are unique in the environment, it's enough to know the name and let tab-completion fill out the version and hash. tab-completion said "i gotchu" โจ
The following phony targets are available: install/<spec> to install the spec with its dependencies, and install-deps/<spec> to only install its dependencies. This can be useful when certain flags should only apply to dependencies. Below we show a use case where a spec is installed with verbose output (spack install --verbose) while its dependencies are installed silently:
$ spack env depfile -o Makefile
# Install dependencies in parallel, only show a log on error.
$ make -j16 install-deps/python-3.11.0-<hash> SPACK_INSTALL_FLAGS=--show-log-on-error
# Install the root spec with verbose output.
$ make -j16 install/python-3.11.0-<hash> SPACK_INSTALL_FLAGS=--verbose๐ช Adding post-install hooks
Another advanced use-case of generated Makefiles is running a post-install command for each package. These "hooks" could be anything from printing a post-install message, running tests, or pushing just-built binaries to a buildcache. obsessed with a good hook ngl ๐
This can be accomplished through the generated [<prefix>/]SPACK_PACKAGE_IDS variable. Assuming we have an active and concrete environment, we generate the associated Makefile with a prefix example:
$ spack env depfile -o env.mk --make-prefix exampleAnd we now include it in a different Makefile, in which we create a target example/push/% with % referring to a package identifier. This target depends on the particular package installation. In this target we automatically have the target-specific HASH and SPEC variables at our disposal. They are respectively the spec hash (excluding leading /), and a human-readable spec. Finally, we have an entrypoint target push that will update the buildcache index once every package is pushed. Note how this target uses the generated example/SPACK_PACKAGE_IDS variable to define its prerequisites.
SPACK ?= spack
BUILDCACHE_DIR = $(CURDIR)/tarballs
.PHONY: all
all: push
include env.mk
example/push/%: example/install/%
@mkdir -p $(dir $@)
$(info About to push $(SPEC) to a buildcache)
$(SPACK) -e . buildcache push --allow-root --only=package $(BUILDCACHE_DIR) /$(HASH)
@touch $@
push: $(addprefix example/push/,$(example/SPACK_PACKAGE_IDS))
$(info Updating the buildcache index)
$(SPACK) -e . buildcache update-index $(BUILDCACHE_DIR)
$(info Done!)
@touch $@๐ฆ CONTAINER IMAGES
Spack Environments (spack.yaml) can easily be turned into container images. This page outlines two ways in which this can be done: ๐ณ
- By installing the environment on the host system, and copying the installations into the container image. This approach does not require any tools like Docker or Singularity to be installed.
- By generating a Docker or Singularity recipe that can be used to build the container image. In this approach, Spack builds the software inside the container runtime, not on the host system.
The first approach is easiest if you already have an installed environment, the second approach gives more control over the container image. pick your fighter ๐
๐ผ๏ธ From existing installations
If you already have a Spack environment installed on your system, you can share the binaries as an OCI compatible container image. To get started you just have to configure and OCI registry and run spack buildcache push.
okay so picture the whole pipeline start to finish โ env, registry, image, done โจ it's literally four commands and you're a container girlie now:
# Create and install an environment in the current directory
spack env create -d .
spack -e . add pkg-a pkg-b
spack -e . install
# Configure the registry
spack -e . mirror add --oci-username ... --oci-password ... container-registry oci://example.com/name/image
# Push the image
spack -e . buildcache push --update-index --base-image ubuntu:22.04 --tag my_env container-registryand then the resulting container image is justโฆ a normal image you run, bestie:
$ docker run -it example.com/name/image:my_envwhat's actually inside the box ๐ โ the image Spack generates = your specified base image, plus each package from the environment as a separate layer stacked on top. It's minimal by construction fr fr: it only contains the environment roots and their runtime dependencies. nothing extra, no clutter, we love a clean build ๐
๐ซถ NOTE: when you're using registries like GHCR and Docker Hub, the --oci-password flag is not the password for your account โ it's a personal access token you have to generate separately. don't get caught out ๐
okay and now the libc lore, because this part matters ๐ญ the --base-image you pick should have a libc that's compatible with the host system. example, straight from the source: if your host system is Ubuntu 20.04, you can use ubuntu:20.04, ubuntu:22.04 or newer โ the libc in the container image must be at least the version of the host system, assuming ABI compatibility. and ngl it's also perfectly fine to use a completely different Linux distribution, as long as the libc is compatible. iconic, flexible, no gatekeeping ๐
for convenience (because Spack is a sweetie ๐), Spack also turns the OCI registry into a build cache, so future spack install of the environment will simply pull the binaries from the registry instead of doing source builds. and the --update-index flag is what's needed to make Spack take the build cache into account when concretizing. don't skip it, it's kind of the whole point ๐
โ ๏ธ NOTE: when generating container images in CI, the approach above is recommended when CI jobs already run in a sandboxed environment. you can simply use spack directly in the CI job and push the resulting image to a registry. subsequent CI jobs should run faster because Spack can install from the same registry instead of rebuilding from sources.็งๆๆๆ = speed ๐ฅ
๐ณ Generating recipes for Docker and Singularity
so apart from copying existing installations into container images, Spack can also generate recipes for container images. this is the glow-up you want when you'd rather run Spack itself in a sandboxed environment instead of on the host system. main character energy, honestly โจ
the thing is, recipes need a little more boilerplate than this:
COPY spack.yaml /environment
RUN spack -e /environment installso Spack hands you a command to generate customizable recipes for container images ๐ and the customizations include:
- minimizing the size of the image
- installing packages in the base image using the system package manager
- setting up a proper entrypoint to run the image
๐ธ A Quick Introduction
okay imagine you have a Spack environment that looks like this:
spack:
specs:
- gromacs+mpi
- mpichproducing a Dockerfile from it is literally as simple as changing directories to where the spack.yaml file is stored and running:
$ spack containerize > Dockerfileand the Dockerfile that gets created uses multi-stage builds and other techniques to minimize the size of the final image. look at her:
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-bionic:latest as builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo "spack:" \
&& echo " specs:" \
&& echo " - gromacs+mpi" \
&& echo " - mpich" \
&& echo " concretizer:" \
&& echo " unify: true" \
&& echo " config:" \
&& echo " install_tree: /opt/software" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y
# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
# Bare OS image to run the installed executables
FROM ubuntu:18.04
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]and the image itself can then be built and run in the usual way, with any of the tools suitable for the task ๐ ๏ธ for instance, if we decided to use docker:
$ spack containerize > Dockerfile
$ docker build -t myimage .
[ ... ]
$ docker run -it myimagethe various components involved in the generation of the recipe and their configuration are discussed in details in the sections below ๐ (iykyk, keep reading)
๐ฆ Spack Images on Docker Hub
Docker images with Spack preinstalled and ready to be used are built when a release is tagged, or nightly on develop ๐ the images are then pushed both to Docker Hub and to GitHub Container Registry. the OSes currently supported are summarized in the table below:
Supported operating systems ๐
- Ubuntu 20.04 โ base image
ubuntu:20.04, spack imagespack/ubuntu-focal - Ubuntu 22.04 โ base image
ubuntu:22.04, spack imagespack/ubuntu-jammy - Ubuntu 24.04 โ base image
ubuntu:24.04, spack imagespack/ubuntu-noble - CentOS 7 โ base image
centos:7, spack imagespack/centos7 - CentOS Stream โ base image
quay.io/centos/centos:stream, spack imagespack/centos-stream - openSUSE Leap โ base image
opensuse/leap, spack imagespack/leap15 - Amazon Linux 2 โ base image
amazonlinux:2, spack imagespack/amazon-linux - AlmaLinux 8 โ base image
almalinux:8, spack imagespack/almalinux8 - AlmaLinux 9 โ base image
almalinux:9, spack imagespack/almalinux9 - Rocky Linux 8 โ base image
rockylinux:8, spack imagespack/rockylinux8 - Rocky Linux 9 โ base image
rockylinux:9, spack imagespack/rockylinux9 - Fedora Linux 39 โ base image
fedora:39, spack imagespack/fedora39 - Fedora Linux 40 โ base image
fedora:40, spack imagespack/fedora40
all the images are tagged with the corresponding release of Spack: [image]
โฆwith the exception of the latest tag, which points to the HEAD of the develop branch โญ
these images are available for anyone to use and take care of all the repetitive tasks that are necessary to set up Spack within a container ๐ซถ the container recipes generated by Spack use them as default base images for their build stage, even though handles to use custom base images provided by users are available to accommodate complex use cases. so like, defaults that just work, but the door's wide open for your weird niche setup, periodt ๐
๐ง Configuring the Container Recipe
any Spack Environment can be used for the automatic generation of container recipes ๐ฅ sensible defaults are provided for things like the base image or the version of Spack used in the image. and if a finer tuning is needed, it can be obtained by adding the relevant metadata under the container attribute of environments:
spack:
specs:
- gromacs+mpi
- mpich
container:
# Select the format of the recipe e.g. docker,
# singularity or anything else that is currently supported
format: dockerand the rest of the knobs inside that container: block are coming right up ๐
# Sets the base images for the stages where Spack builds the
# software or where the software gets installed after being built..
images:
os: "centos:7"
spack: develop
# Whether or not to strip binaries
strip: true
# Additional system packages that are needed at runtime
os_packages:
final:
- libgomp
# Labels for the image
labels:
app: "gromacs"
mpi: "mpich"okay so if you want the full tea on every single knob here, the Configuration Reference section has the detailed description of all the options ๐
๐ Setting Base Images ๐
The images subsection is, like, the โจmain characterโจ of this whole thing โ it picks BOTH the image where Spack builds your software AND the image where that built software actually gets installed. You can set this attribute a few different ways, and which one you pick is entirely about your use case, bestie. Let's go ๐ฆ
๐ณ Use Official Spack Images From Dockerhub ๐ณ
If you just want a recipe that uses an official Docker image from the Spack org to build the thing, plus the matching official OS image to install the built thing โ literally all you have to do is specify:
- An operating system under
images:os - A Spack version under
images:spack
Any combination of those two values that maps to one of the images discussed in Spack Images on Docker Hub is allowed, no cap. For instance, this spack.yaml:
spack:
specs:
- gromacs+mpi
- mpich
container:
images:
os: centos:7
spack: 0.15.4uses spack/centos7:0.15.4 and centos:7 for the stages where the software is respectively built and installed:
# Build stage with Spack pre-installed and ready to be used
FROM spack/centos7:0.15.4 as builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo "spack:" \
&& echo " specs:" \
&& echo " - gromacs+mpi" \
&& echo " - mpich" \
&& echo " concretizer:" \
&& echo " unify: true" \
&& echo " config:" \
&& echo " install_tree: /opt/software" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
[ ... ]
# Bare OS image to run the installed executables
FROM centos:7
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]This is the simplest available method of selecting base images, and we advise to use it whenever possible. ๐ That said โ there are cases where using Spack official images isn't enough to fit production needs. In those situations, users can extend the recipe to start with the bootstrapping of Spack at a certain pinned version, or manually select which base image to start from in the recipe, as we'll see next ๐
๐ง Use a Bootstrap Stage for Spack ๐ง
Sometimes you wanna pin the commit sha used for Spack so that later reproducibility is, like, actually guaranteed โ or you wanna start from a fork of the official Spack repo to try a bugfix or a feature in the early stage of development. Totally possible! You just have to be a little more verbose when specifying info about Spack in the spack.yaml file, iykyk:
images:
os: amazonlinux:2
spack:
# URL of the Spack repository to be used in the container image
url: <to-use-a-fork>
# Either a commit sha, a branch name or a tag
ref: <sha/tag/branch>
# If true turn a branch name or a tag into the corresponding commit
# sha at the time of recipe generation
resolve_sha: <true/false>url specifies the URL from which to clone Spack, and it defaults to https://github.com/spack/spack. The ref attribute can be either a commit sha, a branch name or a tag โ the default value in this case is to use the develop branch, but it may change in the future to point to the latest stable release. Finally, resolve_sha transforms branch names or tags into the corresponding commit shas at the time of recipe generation, to allow for a greater reproducibility of the results at a later time. โจ
The list of operating systems that can be used to bootstrap Spack can be obtained with:
$ spack containerize --list-os
==> The following operating systems can be used to bootstrap Spack:
alpine:3 amazonlinux:2 fedora:40 fedora:39 rockylinux:9 rockylinux:8 almalinux:9 almalinux:8 centos:stream centos:7 opensuse/leap:15 suse/sle:15 nvidia/cuda:11.2.1 ubuntu:24.04 ubuntu:22.04 ubuntu:20.04NOTE: Theresolve_shaoption usesgit rev-parseunder the hood and thus it requires to checkout the corresponding Spack repository in a temporary folder before generating the recipe. Recipe generation may take longer when this option is set totruebecause of this additional step. โณ
๐ฆ Use Custom Images Provided by Users ๐ฆ
Okay, picture this: you're building a production grade image for a CUDA application. The best strategy would probably be to build on top of images provided by the vendor and regard CUDA as an external package. Chef's kiss reasoning, honestly. ๐
Spack doesn't currently provide an official image with CUDA configured this way, but users can build it on their own and then configure the environment to explicitly pull it. This requires users to:
- Specify the image used to build the software under
images:build - Specify the image used to install the built software under
images:final
A spack.yaml like the following:
spack:
specs:
- gromacs@2019.4+cuda build_type=Release
- mpich
- fftw precision=float
packages:
cuda:
buildable: False
externals:
- spec: cuda%gcc
prefix: /usr/local/cuda
container:
images:
build: custom/cuda-10.1-ubuntu18.04:latest
final: nvidia/cuda:10.1-base-ubuntu18.04produces, for instance, the following Dockerfile:
# Build stage with Spack pre-installed and ready to be used
FROM custom/cuda-10.1-ubuntu18.04:latest as builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo "spack:" \
&& echo " specs:" \
&& echo " - gromacs@2019.4+cuda build_type=Release" \
&& echo " - mpich" \
&& echo " - fftw precision=float" \
&& echo " packages:" \
&& echo " cuda:" \
&& echo " buildable: false" \
&& echo " externals:" \
&& echo " - spec: cuda%gcc" \
&& echo " prefix: /usr/local/cuda" \
&& echo " concretizer:" \
&& echo " unify: true" \
&& echo " config:" \
&& echo " install_tree: /opt/software" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y
# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
# Bare OS image to run the installed executables
FROM nvidia/cuda:10.1-base-ubuntu18.04
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]where the base images for both stages are completely custom. ๐จ
This second mode of selection for base images is more flexible than just choosing an operating system and a Spack version, but it's also more demanding. Users may need to generate by themselves their base images, and it's also their responsibility to ensure that:
- Spack is available in the build stage and set up correctly to install the required software
- The artifacts produced in the build stage can be executed in the final stage
Therefore we don't recommend its use in cases that can be otherwise covered by the simplified mode shown first. ๐
๐ธ Singularity Definition Files ๐ธ
Plot twist: Spack can also produce Singularity Definition Files, not just Dockerfile-format recipes โ you literally just change the value of the format attribute and it's giving versatility โจ:
$ cat spack.yaml
spack:
specs:
- hdf5~mpi
container:
format: singularity
$ spack containerize > hdf5.def
$ sudo singularity build hdf5.sif hdf5.defThe minimum version of Singularity required to build a SIF (Singularity Image Format) image from the recipes generated by Spack is 3.5.3. Don't @ me, just upgrade bestie ๐
๐ Extending the Jinja2 Templates ๐
The Dockerfile and the Singularity definition file that Spack can generate are based on a few Jinja2 templates that are rendered according to the environment being containerized. Even though Spack allows a great deal of customization by just setting appropriate values for the configuration options, sometimes that is not enough โ and honestly, that's so valid of you.
In those cases, a user can directly extend the template that Spack uses to render the image to e.g. set additional environment variables or perform specific operations either before or after a given stage of the build. ๐ Let's consider as an example the following structure:
$ tree /opt/environment
/opt/environment
โโโ data
โ โโโ data.csv
โโโ spack.yaml
โโโ data
โโโ templates
โโโ container
โโโ CustomDockerfilecontaining both the custom template extension and the environment manifest file. To use a custom template, the environment must register the directory containing it, and declare its use under the container configuration:
spack:
specs:
- hdf5~mpi
concretizer:
unify: true
config:
template_dirs:
- /opt/environment/templates
container:
format: docker
depfile: true
template: container/CustomDockerfileThe template extension can override two blocks, named build_stage and final_stage, kinda like this little example here:
{% extends "container/Dockerfile" %}
{% block build_stage %}
RUN echo "Start building"
{{ super() }}
{% endblock %}
{% block final_stage %}
{{ super() }}
COPY data /share/myapp/data
{% endblock %}Then you generate the Dockerfile by running:
$ spack -e /opt/environment containerizeHeads up bestie ๐ โ the environment has to be active or spack literally can't read the template. No active env, no template, no vibes.
The recipe that gets generated contains the two extra instructions we added in our template extension, and it looks exactly like this:
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-jammy:latest as builder
RUN echo "Start building"
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo "spack:" \
&& echo " specs:" \
&& echo " - hdf5~mpi" \
&& echo " concretizer:" \
&& echo " unify: true" \
&& echo " config:" \
&& echo " template_dirs:" \
&& echo " - /tmp/environment/templates" \
&& echo " install_tree: /opt/software" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && spack env activate . && spack concretize && spack env depfile -o Makefile && make -j $(nproc) && spack gc -y
# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
# Bare OS image to run the installed executables
FROM ubuntu:22.04
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/._view /opt/._view
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
COPY data /share/myapp/data
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l", "-c", "$*", "--" ]
CMD [ "/bin/bash" ]๐ Configuration Reference ๐
Okay so here's the part where we spill all the tea โ โ the lists below describe every configuration option that's currently supported to customize how container recipes get generated. No gatekeeping, we're giving you the whole menu.
โจ General configuration options for the container section of spack.yaml โจ
formatโ the format of the recipe. Allowed values:dockerorsingularity. Required? Yes ๐depfileโ whether to use a depfile for installation, or not. Allowed values:TrueorFalse(default). Required? No.images:osโ the operating system used as a base for the image. Allowed values: see Supported operating systems. Required? Yes, if you're using constrained selection of base images.images:spackโ the version of Spack to use in the build stage. Allowed values: valid tags forbase:image. Required? Yes, if you're using constrained selection of base images.images:spack:urlโ the repository from which Spack is cloned. Allowed values: any fork of Spack. Required? No.images:spack:refโ the reference for the checkout of Spack. Allowed values: either a commit sha, a branch name or a tag. Required? No.images:spack:resolve_shaโ resolve branches and tags inspack.yamlto commits in the generated recipe. Allowed values:TrueorFalse(default:False). Required? No.images:buildโ the image to be used in the build stage. Allowed values: any valid container image. Required? Yes, if you're using custom selection of base images.images:finalโ the image to be used in the build stage. Allowed values: any valid container image. Required? Yes, if you're using custom selection of base images.stripโ whether to strip binaries. Allowed values:true(default) orfalse. Required? No.os_packages:commandโ the tool used to manage system packages. Allowed values:apt,yum,dnf,dnf_epel,zypper,apk,yum_amazon. Required? Only with custom base images.os_packages:updateโ whether or not to update the list of available packages. Allowed values:TrueorFalse(default:True). Required? No.os_packages:buildโ system packages needed at build-time. Allowed values: valid packages for the current OS. Required? No.os_packages:finalโ system packages needed at run-time. Allowed values: valid packages for the current OS. Required? No.
That's the full roster, no cap ๐ซถ โ every single one of those is in the table above, nothing skipped, nothing hidden.
labelsโ the lil tags you slap on your image so it knows what it is ๐ โ value is a set of Pairs of key-value strings โ required? No. You're free to leave it blank, bestie.
Configuration options specific to Singularity
Okay so these ones are Singularity-only, no other image format gets them ๐ซถ each one just hands over the content of a script, or the image description:
singularity:runscriptโ the actual content of%runscriptโ allowed value: Any valid script โ required: Nosingularity:startscriptโ the content of%startscriptโ allowed value: Any valid script โ required: Nosingularity:testโ the content of%testโ allowed value: Any valid script โ required: Nosingularity:helpโ the description of the image โ allowed value: Description string โ required: No
โจ Best Practices โจ
MPI
- Because of the dependency on Fortran for OpenMPI (which is the spack default implementation, iykyk), consider adding
gfortranto theapt-get installlist. ๐ง - Recent versions of OpenMPI will require you to pass
--allow-run-as-rootto yourmpiruncalls if started as root user inside Docker. Like, it will literally refuse otherwise. - For execution on HPC clusters, it can be helpful to import the docker image into Singularity in order to start a program with an external MPI. Otherwise, also add
openssh-serverto theapt-get installlist. Both options are valid, just depends on your vibe ๐ฅ
CUDA
Starting from CUDA 9.0, Nvidia provides minimal CUDA images based on Ubuntu. Please see their instructions. Avoid double-installing CUDA by adding, e.g.
packages:
cuda:
externals:
- spec: "cuda@9.0.176%gcc@5.4.0 arch=linux-ubuntu16-x86_64"
prefix: /usr/local/cuda
buildable: Falseto your spack.yaml. ๐
Users will either need nvidia-docker or e.g. Singularity to execute device kernels. No cap, you need one of them โ pick your fighter ๐
Docker on Windows and OSX
On Mac OS and Windows, docker runs on a hypervisor that is not allocated much memory by default, and some spack packages may fail to build due to lack of memory. ๐ญ To work around this issue, consider configuring your docker installation to use more of your host memory. In some cases, you can also ease the memory pressure on parallel builds by limiting the parallelism in your config.yaml.
config:
build_jobs: 2Lowkey the memory thing gets everyone on Mac and Windows. Give the hypervisor more RAM and you're golden ๐ธ
๐ MIRRORS (MIRRORS.YAML) ๐
okay so picture this bestie: some sites literally cannot get to the internet to fetch packages. like, no wifi, no nothing, the vibes are offline-only ๐ญ. Those sites need a local repository of tarballs to pull their files from, and that's exactly what mirrors are for. A mirror is a URL pointing at a directory โ either on the local filesystem or on some server โ that contains tarballs for all of Spack's packages. iconic behavior honestly ๐
Here's what a mirror's directory structure looks like:
mirror/
cmake/
cmake-2.8.10.2.tar.gz
dyninst/
dyninst-8.1.1.tgz
dyninst-8.1.2.tgz
libdwarf/
libdwarf-20130126.tar.gz
libdwarf-20130207.tar.gz
libdwarf-20130729.tar.gz
libelf/
libelf-0.8.12.tar.gz
libelf-0.8.13.tar.gz
libunwind/
libunwind-1.1.tar.gz
mpich/
mpich-3.0.4.tar.gz
mvapich2/
mvapich2-1.9.tgzThe structure is genuinely so simple it's giving minimalist queen โจ. There's a top-level directory. The second-level directories are named after packages, and the third level holds the tarballs for each package, named after each package. that's it, that's the whole architecture ๐
NOTE: ๐จ Archives are not named exactly the way they were in the package's fetch URL. They have the form<name>-<version>.<extension>, where<name>is Spack's name for the package,<version>is the version of the tarball, and<extension>is whatever format the package's fetch URL contains. And real talk: to make mirror creation reasonably fast, we copy the tarball in its original format to the mirror directory, but we do not standardize on a particular compression algorithm โ because that would potentially require expanding and re-compressing each archive. no thank you, we're not doing all that ๐
๐ช spack mirror
Mirrors are managed with the spack mirror command, and the help for spack mirror looks exactly like this:
$ spack help mirror
usage: spack mirror [-hn] SUBCOMMAND ...
manage mirrors (source and binary)
positional arguments:
SUBCOMMAND
create create a directory to be used as a spack mirror, and fill it with package archives
destroy given a url, recursively delete everything under it
add add a mirror to Spack
remove (rm) remove a mirror by name
set-url change the URL of a mirror
set configure the connection details of a mirror
list print out available mirrors to the console
options:
-h, --help show this help message and exit
-n, --no-checksum do not use checksums to verify downloaded files (unsafe)The create command is the one that actually builds a mirror, by fetching all of its packages from the internet and checksumming them โ
.
The other three commands are for managing mirror configuration โ they control the URL(s) from which Spack downloads its packages. the more you know ๐
๐ ๏ธ spack mirror create
You can create a mirror using the spack mirror create command, assuming you're on a machine where you can actually access the internet (sorry to the offline girlies, this step needs the wifi ๐).
The command will iterate through all of Spack's packages and download the safe ones into a directory structure like the one above. Here's what it looks like in action:
$ spack mirror create libelf libdwarf
==> Created new mirror in spack-mirror-2014-06-24
==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.13.tar.gz
########################################################## 81.6%
==> Checksum passed for libelf@0.8.13
==> Added libelf@0.8.13
==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.12.tar.gz
###################################################################### 98.6%
==> Checksum passed for libelf@0.8.12
==> Added libelf@0.8.12
==> Trying to fetch from http://www.prevanders.net/libdwarf-20130207.tar.gz
###################################################################### 97.3%
==> Checksum passed for libdwarf@20130207
==> Added libdwarf@20130207
==> Trying to fetch from http://www.prevanders.net/libdwarf-20130126.tar.gz
######################################################## 78.9%
==> Checksum passed for libdwarf@20130126
==> Added libdwarf@20130126
==> Trying to fetch from http://www.prevanders.net/libdwarf-20130729.tar.gz
############################################################# 84.7%
==> Added libdwarf@20130729
==> Added spack-mirror-2014-06-24/libdwarf/libdwarf-20130729.tar.gz to mirror
==> Added python@2.7.8.
==> Successfully updated mirror in spack-mirror-2015-02-24.
Archive stats:
0 already present
5 added
0 failed to fetch.Once this is done, you can tar up the spack-mirror-2014-06-24 directory and copy it over to the machine you want it hosted on. so easy, so slay ๐
๐ฏ Custom package sets
Normally, spack mirror create downloads all the archives it has checksums for. But if you only want to create a mirror for a subset of packages, you can totally do that โ just supply a list of package specs on the command line after spack mirror create. For example, this command:
$ spack mirror create libelf@0.8.12: boost@1.44:Will create a mirror for libelf versions greater than or equal to 0.8.12 and boost versions greater than or equal to 1.44. selective queen behavior โจ
๐ Mirror files
If you have a very large number of packages you want to mirror, you can supply a file with specs in it, one per line:
$ cat specs.txt
libdwarf
libelf@0.8.12:
boost@1.44:
boost@1.39.0
...
$ spack mirror create --file specs.txt
...This is useful if there is a specific suite of software managed by your site. lowkey love a tidy little spec list ngl ๐ฆ
๐ฑ Mirror environment
To create a mirror of all packages required by a concrete environment, activate the environment and call spack mirror create -a. This is especially useful to create a mirror of an environment concretized on another machine.
[remote] $ spack env create myenv
[remote] $ spack env activate myenv
[remote] $ spack add ...
[remote] $ spack concretize
$ sftp remote:/spack/var/environment/myenv/spack.lock
$ spack env create myenv spack.lock
$ spack env activate myenv
$ spack mirror create -alook at that cross-machine workflow โ it's giving main character energy ๐
โ spack mirror add
Once you have a mirror, you need to let spack know about it. This is relatively simple. First, figure out the URL for the mirror. If it's a directory, you can use a file URL like this one:
file://$HOME/spack-mirror-2014-06-24That points to the directory on the local filesystem. If it were on a web server, you could use a URL like this one:
https://example.com/some/web-hosted/directory/spack-mirror-2014-06-24
Spack will use the URL as the root for all of the packages it fetches. You can tell your Spack installation to use that mirror like this:
$ spack mirror add local_filesystem file://$HOME/spack-mirror-2014-06-24Each mirror has a name so that you can refer to it again later. name your mirrors like you name your playlists bestie ๐
๐ spack mirror list
To see all the mirrors Spack knows about, run spack mirror list:
$ spack mirror list
local_filesystem file:///home/username/spack-mirror-2014-06-24๐๏ธ spack mirror remove
To remove a mirror by name, run:
$ spack mirror remove local_filesystem
$ spack mirror list
==> No mirrors configured.clean slate, no notes ๐
โ๏ธ Mirror precedence
Adding a mirror really adds a line in ~/.spack/mirrors.yaml:
mirrors:
local_filesystem: file:///home/username/spack-mirror-2014-06-24
remote_server: https://example.com/some/web-hosted/directory/spack-mirror-2014-06-24If you want to change the order in which mirrors are searched for packages, you can edit this file and reorder the sections. Spack will search the topmost mirror first and the bottom-most mirror last. order matters, bestie โ first come first served โจ
๐พ Local Default Cache
Spack caches resources that are downloaded as part of installs. The cache is a valid spack mirror: it uses the same directory structure and naming scheme as other Spack mirrors (so it can be copied anywhere and referenced with a URL like other mirrors). The mirror is maintained locally (within the Spack installation directory) at var/spack/cache/. It is always enabled (and is always searched first when attempting to retrieve files for an installation) but can be cleared with clean; the cache directory can also be deleted manually without issue ๐ซถ
Caching includes retrieved tarball archives and source control repositories, but only resources with an associated digest or commit ID (e.g. a revision number for SVN) will be cached. iykyk ๐ฅ
๐ธ MODULES (MODULES.YAML) ๐ธ
okay bestie so picture this ๐ โ using module systems to manage your user environment in a controlled way? that's a super common practice at HPC centers, and honestly some individual programmers do it on their own dev machines too. and to support that whole common practice, Spack integrates with Environment Modules and Lmod by giving you post-install hooks that generate module files and the commands to manipulate them. โจ
modules are just one of several ways you can use Spack packages, tho. for other options that might fit your use case better, you should also look at spack load and environments. ๐ซถ
๐ Quick start ๐
in the current version of Spack, module files are not generated by default, no cap. to get started, you can generate module files for all currently installed packages by running either
$ spack module tcl refreshor
$ spack module lmod refreshSpack can also generate module files for all future installations automatically through this configuration:
$ spack config add modules:default:enable:[tcl]or
$ spack config add modules:default:enable:[lmod]assuming you've got a module system installed, you should now be able to use the module command to interact with them:
$ module avail
--------------------------------------------------------------- ~/spack/share/spack/modules/linux-ubuntu14-x86_64 ---------------------------------------------------------------
autoconf/2.69-gcc-4.8-qextxkq hwloc/1.11.6-gcc-6.3.0-akcisez m4/1.4.18-gcc-4.8-ev2znoc openblas/0.2.19-gcc-6.3.0-dhkmed6 py-setuptools/34.2.0-gcc-6.3.0-fadur4s
automake/1.15-gcc-4.8-maqvukj isl/0.18-gcc-4.8-afi6taq m4/1.4.18-gcc-6.3.0-uppywnz openmpi/2.1.0-gcc-6.3.0-go2s4z5 py-six/1.10.0-gcc-6.3.0-p4dhkaw
binutils/2.28-gcc-4.8-5s7c6rs libiconv/1.15-gcc-4.8-at46wg3 mawk/1.3.4-gcc-4.8-acjez57 openssl/1.0.2k-gcc-4.8-dkls5tk python/2.7.13-gcc-6.3.0-tyehea7
bison/3.0.4-gcc-4.8-ek4luo5 libpciaccess/0.13.4-gcc-6.3.0-gmufnvh mawk/1.3.4-gcc-6.3.0-ostdoms openssl/1.0.2k-gcc-6.3.0-gxgr5or readline/7.0-gcc-4.8-xhufqhn
bzip2/1.0.6-gcc-4.8-iffrxzn libsigsegv/2.11-gcc-4.8-pp2cvte mpc/1.0.3-gcc-4.8-g5mztc5 pcre/8.40-gcc-4.8-r5pbrxb readline/7.0-gcc-6.3.0-zzcyicg
bzip2/1.0.6-gcc-6.3.0-bequudr libsigsegv/2.11-gcc-6.3.0-7enifnh mpfr/3.1.5-gcc-4.8-o7xm7az perl/5.24.1-gcc-4.8-dg5j65u sqlite/3.8.5-gcc-6.3.0-6zoruzj
cmake/3.7.2-gcc-6.3.0-fowuuby libtool/2.4.6-gcc-4.8-7a523za mpich/3.2-gcc-6.3.0-dmvd3aw perl/5.24.1-gcc-6.3.0-6uzkpt6 tar/1.29-gcc-4.8-wse2ass
curl/7.53.1-gcc-4.8-3fz46n6 libtool/2.4.6-gcc-6.3.0-n7zmbzt ncurses/6.0-gcc-4.8-dcpe7ia pkg-config/0.29.2-gcc-4.8-ib33t75 tcl/8.6.6-gcc-4.8-tfxzqbr
expat/2.2.0-gcc-4.8-mrv6bd4 libxml2/2.9.4-gcc-4.8-ryzxnsu ncurses/6.0-gcc-6.3.0-ucbhcdy pkg-config/0.29.2-gcc-6.3.0-jpgubk3 util-macros/1.19.1-gcc-6.3.0-xorz2x2
flex/2.6.3-gcc-4.8-yf345oo libxml2/2.9.4-gcc-6.3.0-rltzsdh netlib-lapack/3.6.1-gcc-6.3.0-js33dog py-appdirs/1.4.0-gcc-6.3.0-jxawmw7 xz/5.2.3-gcc-4.8-mew4log
gcc/6.3.0-gcc-4.8-24puqve lmod/7.4.1-gcc-4.8-je4srhr netlib-scalapack/2.0.2-gcc-6.3.0-5aidk4l py-numpy/1.12.0-gcc-6.3.0-oemmoeu xz/5.2.3-gcc-6.3.0-3vqeuvb
gettext/0.19.8.1-gcc-4.8-yymghlh lua/5.3.4-gcc-4.8-im75yaz netlib-scalapack/2.0.2-gcc-6.3.0-hjsemcn py-packaging/16.8-gcc-6.3.0-i2n3dtl zip/3.0-gcc-4.8-rwar22d
gmp/6.1.2-gcc-4.8-5ub2wu5 lua-luafilesystem/1_6_3-gcc-4.8-wkey3nl netlib-scalapack/2.0.2-gcc-6.3.0-jva724b py-pyparsing/2.1.10-gcc-6.3.0-tbo6gmw zlib/1.2.11-gcc-4.8-pgxsxv7
help2man/1.47.4-gcc-4.8-kcnqmau lua-luaposix/33.4.0-gcc-4.8-mdod2ry netlib-scalapack/2.0.2-gcc-6.3.0-rgqfr6d py-scipy/0.19.0-gcc-6.3.0-kr7nat4 zlib/1.2.11-gcc-6.3.0-7cqp6cjthe names should look familiar, since they resemble the output from spack find. ๐ like, for example, you could type this command to load the cmake module:
$ module load cmake/3.7.2-gcc-6.3.0-fowuubyngl neither of these is particularly pretty, easy to remember, or easy to type ๐ญ. luckily, Spack offers many facilities for customizing the module scheme used at your site. ๐
๐ Module file customization ๐
the table below summarizes the essential info associated with the different file formats that can be generated by Spack (it's a list now, but every single detail is still here, promise ๐ซถ):
tclโ Hierarchical: No ยท default root directory:share/spack/modulesยท default template file:share/spack/templates/modules/modulefile.tclยท compatible tools: Env. Modules/Lmodlmodโ Hierarchical: Yes ยท default root directory:share/spack/lmodยท default template file:share/spack/templates/modules/modulefile.luaยท compatible tools: Lmod
Spack ships with sensible defaults for the generation of module files, but you can customize so many aspects of it to accommodate package- or site-specific needs. โจ in general you can override or extend the default behavior by:
- overriding certain callback APIs in the Python packages
- writing specific rules in the
modules.yamlconfiguration file - writing your own templates to override or extend the defaults
the former method lets you express changes in the run-time environment that are needed to use the installed software properly โ e.g. injecting variables from language interpreters into their extensions. ๐ the latter two instead let you fine-tune the filesystem layout, content, and creation of module files to meet site-specific conventions. ๐ฆ
๐ง Setting environment variables dynamically in package.py ๐ง
there are two methods you can implement in any package.py to dynamically affect the content of the module files generated by Spack.
the most important one is setup_run_environment, which you can use to set environment variables in the module file that depend on the spec:
def setup_run_environment(self, env):
if self.spec.satisfies("+foo"):
env.set("FOO", "bar")the second one, less commonly used, is setup_dependent_run_environment(self, env, dependent_spec), which lets a dependency set variables in the module file of its dependents. this is typically used in packages like python, r, or perl to prepend the dependent's prefix to the search path of the interpreter (PYTHONPATH, R_LIBS, PERL5LIB resp.), so it can locate the packages at runtime. ๐
for example, a simplified version of the python package could look like this:
def setup_dependent_run_environment(self, env, dependent_spec):
if dependent_spec.package.extends(self.spec):
env.prepend_path("PYTHONPATH", dependent_spec.prefix.lib.python)and that would make any package that extends("python") have its library directory added to the PYTHONPATH environment variable in the module file. ๐ it's so much more convenient to set this variable here than to repeat it in every Python extension's setup_run_environment method. fr fr.
๐ The modules.yaml config file and module sets ๐
the configuration files that control module generation behavior are named modules.yaml. the default config looks like this:
# -------------------------------------------------------------------------
# This is the default configuration for Spack's module file generation.
#
# Settings here are versioned with Spack and are intended to provide
# sensible defaults out of the box. Spack maintainers should edit this
# file to keep it current.
#
# Users can override these settings by editing the following files.
#
# Per-spack-instance settings (overrides defaults):
# $SPACK_ROOT/etc/spack/modules.yaml
#
# Per-user settings (overrides default and site settings):
# ~/.spack/modules.yaml
# -------------------------------------------------------------------------
modules:
# This maps paths in the package install prefix to environment variables
# they should be added to. For example, <prefix>/bin should be in PATH.
prefix_inspections:
./bin:
- PATH
./man:
- MANPATH
./share/man:
- MANPATH
./share/aclocal:
- ACLOCAL_PATH
./lib/pkgconfig:
- PKG_CONFIG_PATH
./lib64/pkgconfig:
- PKG_CONFIG_PATH
./share/pkgconfig:
- PKG_CONFIG_PATH
./:
- CMAKE_PREFIX_PATH
# These are configurations for the module set named "default"
default:
# Where to install modules
roots:
tcl: $spack/share/spack/modules
lmod: $spack/share/spack/lmod
# What type of modules to use ("tcl" and/or "lmod")
enable:
- lmod
tcl:
all:
autoload: direct
# Default configurations if lmod is enabled
lmod:
all:
autoload: direct
hierarchy:
- mpiyou can define one or more module sets, each of which can be configured separately with regard to install location, naming scheme, inclusion and exclusion, autoloading, et cetera. โจ
the default module set is aptly named default. all Spack commands that operate on modules apply to the default module set, unless another module set is specified explicitly (with the --name flag). periodt. ๐
๐ Changing the modules root
Okay so real talk โ the default module root for lmod is $spack/share/spack/lmod, and the default root for tcl is $spack/share/spack/modules. But bestie, you can override that for any module set by changing the roots key of the configuration. It's giving customization ๐
modules:
default:
roots:
tcl: /path/to/install/tcl/modules
my_custom_lmod_modules:
roots:
lmod: /path/to/install/custom/lmod/modules
# ...So this config? It creates two module sets, periodt. The default module set installs its tcl modules to /path/to/install/tcl/modules (and still installs its lmod modules, if any, to the default location). The set my_custom_lmod_modules installs its lmod modules to /path/to/install/custom/lmod/modules (and still installs its tcl modules, if any, to the default location). Everyone keeps their own lane ๐
By default, an architecture-specific directory is added to the root directory. A module set can override that behavior by setting the arch_folder config value to False:
modules:
default:
roots:
tcl: /path/to/install/tcl/modules
arch_folder: falseObviously (and I say this with love), having multiple module sets install modules to the default location could be confusing to users of your modules. Fr fr. In the next section, we discuss enabling and disabling module types (module file generators) for each module set. Stay with me ๐ซถ
โจ Automatically generating module files
Spack can be configured to automatically generate module files as part of package installation. Iconic behavior honestly. You do it by adding the desired module systems to the enable list:
modules:
default:
enable:
- tcl
- lmod๐ธ Configuring tcl and lmod modules
You can configure the behavior of either module system separately, under a key corresponding to the generator being customized:
modules:
default:
tcl:
# contains environment modules specific customizations
lmod:
# contains lmod specific customizationsIn general, the configuration options you can use in modules.yaml will either change the layout of the module files on the filesystem, or they'll affect their content. For that latter point โ and this is the cute part โ it's possible to use anonymous specs to fine tune the set of packages on which the modifications should be applied. Chef's kiss ๐ง
๐ Autoloading and hiding dependencies
A module file should set the variables that are needed for an application to work. But since an application often has many dependenciesโฆ where should all the environment variables for those be set? ๐ค
In Spack the rule is that each package sets the runtime variables that are needed by the package itself, and no more. No overachieving, no scope creep. This way, dependencies can be loaded standalone too, and duplication of environment variables is avoided. Efficiency queen behavior.
That means however that if you want to use an application, you need to load the modules for all its dependencies. And of course, that is not something you'd want users to do manually. Imagine ๐ญ
Since Spack knows the dependency graph of every package, it can easily generate module files that automatically load the modules for its dependencies recursively. It's enabled by default for both Lmod and Environment Modules under the autoload: direct config option. The former system has builtin support through the depends_on function, the latter simply uses a module load statement. Both module systems (at least in newer versions) do reference counting, so that if a module is loaded by two different modules, it will only be unloaded after the others are. Teamwork makes the dream work ๐
The autoload key accepts the values:
none: no autoloadingrun: autoload direct run type dependenciesdirect: autoload direct link and run type dependenciesall: autoload all dependencies
In case of run and direct, a module load triggers a recursive load.
The direct option is most correct: there are cases where pure link dependencies need to set variables for themselves, or need to have variables of their own dependencies set. It's the responsible one of the group ๐ซก
In practice however, run is often sufficient, and may make module load snappier. Lowkey the practical pick.
The all option is discouraged and seldomly used. We don't really do that one ๐
A common complaint about autoloading is the large number of modules that are visible to the user. Spack has a solution for this as well: hide_implicits: true. This ensures that only those packages you've explicitly installed are exposed by module avail, but still allows for autoloading of hidden dependencies. Lmod should support hiding implicits in general, while Environment Modules requires version 4.7 or higher.
NOTE: If supported by your module system, we highly encourage the following configuration that enables autoloading and hiding of implicits. It ensures all runtime variables are set correctly, including those for dependencies, without overwhelming the user with a large number of available modules. Further, it makes it easier to get readable module names without collisions โ see the section below on Customize the naming of modules. ๐
modules:
default:
tcl:
hide_implicits: true
all:
autoload: direct # or `run`
lmod:
hide_implicits: true
all:
autoload: direct # or `run`๐ฆ Setting environment variables for selected packages in config
In the configuration file you can filter particular specs, and make further changes to the environment variables that go into their module files. This is very powerful when you want to avoid modifying the package itself, or when you want to set certain variables on multiple selected packages at once. Big main character energy for a config file, ngl.
For instance, in the snippet below:
modules:
default:
tcl:
# The keyword `all` selects every package
all:
environment:
set:
BAR: 'bar'
# This anonymous spec selects any package that
# depends on mpi. The double colon at the
# end clears the set of rules that matched so far.
^mpi::
environment:
prepend_path:
PATH: '{^mpi.prefix}/bin'
set:
BAR: 'baz'
# Selects any zlib package
zlib:
environment:
prepend_path:
LD_LIBRARY_PATH: 'foo'
# Selects zlib compiled with gcc@4.8
zlib%gcc@4.8:
environment:
unset:
- FOOBARโฆyou are instructing Spack to set the environment variable BAR=bar for every module, unless the associated spec satisfies the abstract dependency ^mpi โ in which case BAR=baz, and the directory containing the respective MPI executables is prepended to the PATH variable. In addition, in any spec that satisfies zlib the value foo will be prepended to LD_LIBRARY_PATH, and in any spec that satisfies zlib%gcc@4.8 the variable FOOBAR will be unset. Look at her go ๐
NOTE: Order does matter โ the modifications associated with theallkeyword are always evaluated first, no matter where they appear in the configuration file. All the other changes to environment variables for matching specs are evaluated from top to bottom. So like, don't come for me about file order,allalways goes first ๐
WARNING: As general advice, it's often better to set as few unnecessary variables as possible. For example, the following seemingly innocent and potentially useful configurationall: environment: set: "{name}_ROOT": "{prefix}"setsBINUTILS_ROOTto its prefix in modules forbinutils, which happens to break thegcccompiler: it uses this variable as its default search path for certain object files and libraries, and by merely setting it, everything fails to link. Yeah. Everything. Fails. To. Link. ๐ Do not do this bestie, I am begging ๐
๐ Exclude or include specific module files
You can use anonymous specs also to prevent module files from being written, or to force them to be written. Consider the case where you want to hide from users all the boilerplate software that you had to build in order to bootstrap a new compiler. Nobody needs to see the scaffolding.
Suppose for instance that gcc@4.4.7 is the compiler provided by your system. If you write a configuration file like:
modules:
default:
tcl:
include: ['gcc', 'llvm'] # include will have precedence over exclude
exclude: ['%gcc@4.4.7'] # Assuming gcc@4.4.7 is the system compilerโฆyou will prevent the generation of module files for any package that is compiled with gcc@4.4.7, with the only exception of any gcc or any llvm installation. include said "not on my watch" โจ
It is safe to combine exclude and autoload mentioned above. When exclude prevents a module file from being generated for a dependency, the autoload feature will simply not generate a statement to load it. No ghost references, we love a clean break ๐ซถ
๐ Customize the naming of modules
The names of environment modules generated by Spack are not always easy to fully comprehend due to the long hash in the name. It's giving cryptic. There are three module configuration options to help with that.
The first is a global setting to adjust the hash length. It can be set anywhere from 0 to 32 and has a default length of 7. This is the representation of the hash in the module file name and does not affect the size of the package hash. Be aware that the smaller the hash length, the more likely naming conflicts will occur. Foreshadowing ๐
The following snippet shows how to set hash length in the module file names:
modules:
default:
tcl:
hash_length: 7TIP: Usinghide_implicits: true(see Autoloading and hiding dependencies) vastly reduces the number modules exposed to the user. The hidden modules always contain the hash in their name, and are not influenced by thehash_lengthsetting. Hidden implicits thus make it easier to use a short hash length or no hash at all, without risking name conflicts. Literally the cheat code ๐ฅ
To help make module names more readable, and to help alleviate name conflicts with a short hash, one can use the suffixes option in the modules configuration file. This option will add strings to modules that match a spec. For instance, the following config options,
modules:
default:
tcl:
all:
suffixes:
^python@3.12: 'python-3.12'
^openblas: 'openblas'will add a python-3.12 version string to any packages compiled with Python matching the spec, python@3.12. This is genuinely so useful for knowing which version of Python a set of Python extensions is associated with, no cap. Likewise, the openblas string gets attached to any program that has openblas in the spec โ most likely via the +blas variant spec. ๐
The most heavyweight solution to module naming is to change the entire naming convention for module files. This uses the projections format covered in View Projections. Big brain mode:
modules:
default:
tcl:
projections:
all: '{name}/{version}-{compiler.name}-{compiler.version}-module'
^mpi: '{name}/{version}-{^mpi.name}-{^mpi.version}-{compiler.name}-{compiler.version}-module'will create module files that are nested in directories by package name, contain the version and compiler name and version, and have the word module before the hash for all specs that do not depend on mpi โ and will have the same information plus the MPI implementation name and version for all packages that depend on mpi. ๐
When specifying module names by projection for Lmod modules, we recommend NOT including names of dependencies (e.g., MPI, compilers) that are already in the Lmod hierarchy. just trust us on this one bestie
NOTE: Tcl and Lua modules also allow for explicit conflicts between modulefiles. ๐ฅ
modules:
default:
enable:
- tcl
tcl:
projections:
all: '{name}/{version}-{compiler.name}-{compiler.version}'
all:
conflict:
- '{name}'
- 'intel/14.0.1'will create module files that will conflict with intel/14.0.1 and with the base directory of the same module โ effectively preventing the possibility to load two or more versions of the same software at the same time. no double-dipping, periodt The tokens that are available for use in this directive are the same understood by the format() method.
For Lmod and Environment Modules versions prior 4.2, it is important to express the conflict on both modulefiles conflicting with each other. like both sides have to say it, it's giving mutual beef ๐
NOTE: Lmod hierarchical module files Whenlmodis activated Spack will generate a set of hierarchical lua module files that are understood by Lmod. The hierarchy will always contain the two layers Core / Compiler but can be further extended to any of the virtual dependencies present in Spack. A case that could be useful in practice is for instance:yaml modules: default: enable: - lmod lmod: core_compilers: - 'gcc@4.8' core_specs: - 'python' hierarchy: - 'mpi' - 'lapack'that will generate a hierarchy in which thelapackandmpilayer can be switched independently. This allows a site to build the same libraries or applications against different implementations ofmpiandlapack, and let Lmod switch safely from one to the other. ๐ฆ All packages built with a compiler incore_compilersand all packages that satisfy a spec incore_specswill be put in the Core hierarchy of the lua modules.
WARNING: Consistency of Core packages The user is responsible for maintining consistency among core packages, as core_specs bypasses the hierarchy that allows Lmod to safely switch between coherent software stacks. ๐ซ you're on your own for that vibe check
WARNING: Deep hierarchies and lmod spider For hierarchies that are deeper than three layers lmod spider may have some issues. See this discussion on the Lmod project. ๐ท๏ธ
๐ Customize environment modifications ๐
You can control which prefixes in a Spack package are added to environment variables with the prefix_inspections section โ this section maps relative prefixes to the list of environment variables which should be updated with those prefixes. ๐
The prefix_inspections configuration is different from other settings in that a prefix_inspections configuration at the modules level of the configuration file applies to all module sets. This allows users to make general overrides to the default inspections and customize them per-module-set. iykyk โจ
modules:
prefix_inspections:
./bin:
- PATH
./man:
- MANPATH
./:
- CMAKE_PREFIX_PATHPrefix inspections are only applied if the relative path inside the installation prefix exists. In this case, for a Spack package foo installed to /spack/prefix/foo, if foo installs executables to bin but no manpages in man, the generated module file for foo would update PATH to contain /spack/prefix/foo/bin and CMAKE_PREFIX_PATH to contain /spack/prefix/foo, but would not update MANPATH. conditional queen behavior ๐
The default list of environment variables in this config section includes PATH, MANPATH, ACLOCAL_PATH, PKG_CONFIG_PATH and CMAKE_PREFIX_PATH, as well as DYLD_FALLBACK_LIBRARY_PATH on macOS. On Linux however, the corresponding LD_LIBRARY_PATH variable is not set, because it affects the behavior of system executables too. ๐ญ
NOTE: In general, theLD_LIBRARY_PATHvariable is not required when using packages built with Spack, thanks to the use of RPATH. Some packages may still need the variable, which is best handled on a per-package basis instead of globally, as explained in Setting environment variables dynamically inpackage.py.
There is a special case for prefix inspections relative to environment views. If all of the following conditions hold for a module set configuration:
- The configuration is for an environment and will never be applied outside the environment,
- The environment in question is configured to use a view,
- The environment view is configured with a projection that ensures every package is linked to a unique directory,
then the module set may be configured to create modules relative to the environment view. This is specified by the use_view configuration option in the module set. If True, the module set is constructed relative to the default view of the environment. Otherwise, the value must be the name of the environment view relative to which to construct modules, or False-ish to disable the feature explicitly (the default is False). ๐ง
If the use_view value is set in the config, then the prefix inspections for the package are done relative to the package's path in the view. ๐
spack:
modules:
view_relative_modules:
use_view: my_view
prefix_inspections:
./bin:
- PATH
view:
my_view:
projections:
root: /path/to/my/view
all: '{name}-{hash}'The spack key is relevant to environment configuration, and the view key is discussed in detail in the section on Configuring environment views. With this configuration the generated module for package foo would set PATH to include /path/to/my/view/foo-<hash>/bin instead of /spack/prefix/foo/bin. ๐ฆ
The use_view option is useful when deploying a large software stack to users who are likely to inspect the modules to find full paths to software โ when it is desirable to present the users with a simpler set of paths than those generated by the Spack install tree. cleaner paths, happier users, we love to see it
๐ธ Filter out environment modifications ๐ธ
Modifications to certain environment variables in module files are there by default โ for instance because they are generated by prefix inspections. If you want to prevent modifications to some environment variables, you can do so by using the exclude_env_vars: ๐
modules:
default:
tcl:
all:
filter:
# Exclude changes to any of these variables
exclude_env_vars: ['CPATH', 'LIBRARY_PATH']The configuration above will generate module files that will not contain modifications to either CPATH or LIBRARY_PATH. blocked. denied. not today. ๐
โญ Select default modules โญ
By default, when multiple modules of the same name share a directory, the highest version number will be the default module. This behavior of the module command can be overridden with a symlink named default to the desired default module. If you wish to configure default modules with Spack, add a defaults key to your modules configuration:
modules:
my-module-set:
tcl:
defaults:
- gcc@10.2.1
- hdf5@1.2.10+mpi+hl%gccThese defaults may be arbitrarily specific. For any package that satisfies a default, Spack will generate the module file in the appropriate path, and will generate a default symlink to the module file as well. ๐
WARNING: If Spack is configured to generate multiple default packages in the same directory, the last modulefile to be generated will be the default module. last one wins, so order matters bestie
๐ Maintaining Module Files ๐
Each type of module file has a command with the same name associated with it. The actions these commands permit are usually associated with the maintenance of a production environment. Here's, for instance, a sample of the features of the spack module tcl command:
$ spack module tcl --help
usage: spack module tcl [-h] [-n MODULE_SET_NAME] SUBCOMMAND ...
positional arguments:
SUBCOMMAND
refresh regenerate module files
find find module files for packages
rm remove module files
loads prompt the list of modules associated with a constraint
setdefault set the default module file for a package
options:
-h, --help show this help message and exit
-n, --name MODULE_SET_NAME
named module set to use from modules configuration๐ Refresh the set of modules ๐
The subcommand that regenerates module files to update their content or their layout is refresh:
$ spack module tcl refresh --help
usage: spack module tcl refresh [-hy] [--delete-tree] [--upstream-modules] ...
positional arguments:
installed_specs constraint to select a subset of installed packages
options:
--delete-tree delete the module file tree before refresh
--upstream-modules generate modules for packages installed upstream
-h, --help show this help message and exit
-y, --yes-to-all assume "yes" is the answer to every confirmation requestfresh modules, fresh start, slay โจ
Okay so if you want, like, a specific set of packages instead of the whole tree, you can pick them with anonymous specs in the optional constraint positional argument. And if the layout change is truly unhinged/radical, you can nuke the entire tree before regenerating. Your call bestie ๐
๐๏ธ Delete module files
But wait โ if all you actually need is to delete a few module files, don't do the dramatic thing. The right subcommand is rm:
$ spack module tcl rm --help
usage: spack module tcl rm [-hy] ...
positional arguments:
installed_specs constraint to select a subset of installed packages
options:
-h, --help show this help message and exit
-y, --yes-to-all assume "yes" is the answer to every confirmation requestNOTE: We care about your module files! ๐ Every modification done on modules that are already existing will ask for a confirmation by default. If the command is used in a script it is possible though to pass the -y argument, that will skip this safety measure.
Like, it's giving "are you SURE" energy, and honestly? Iconic of it. But if you're scripting, -y is your girl.
๐ Using Spack modules in shell scripts
The easiest way to enable additional Spack commands for loading and unloading module files โ and to add the correct path to MODULEPATH โ is to source the appropriate setup file. Assuming Spack is installed in $SPACK_ROOT, run the appropriate command for your shell:
# For bash/zsh/sh
$ . $SPACK_ROOT/share/spack/setup-env.sh
# For tcsh/csh
$ source $SPACK_ROOT/share/spack/setup-env.csh
# For fish
$ . $SPACK_ROOT/share/spack/setup-env.fishIf you want to have Spack's shell support available on the command line at any login, you can put this source line in one of the files that are sourced at startup (like .profile, .bashrc or .cshrc). Just be aware that the shell startup time may increase slightly as a result โ tiny price to pay, ngl ๐ซถ
๐ spack module tcl loads
Sometimes you genuinely want to use a Spack-generated module rather than relying on Spack's built-in user-environment modification capabilities. Mood. To translate a spec into a module name, use spack module tcl loads or spack module lmod loads, depending on which module system you're feeling that day.
And if you want to load not just a module but also, like, literally all the modules it depends on, use the --dependencies option. This is not required for most modules, because Spack builds binaries with RPATH support. But not all packages use RPATH to find their dependencies โ this can be true in particular for Python extensions, which are currently not built with RPATH. So yeah, keep that in mind โจ
Scripts to load modules recursively may be made with the command:
$ spack module tcl loads --dependencies <spec>An equivalent alternative using process substitution is:
$ source <( spack module tcl loads --dependencies <spec> )๐ Module Commands for Shell Scripts
Spack is flexible and all, but the module command is much faster. This could become an issue when emitting a series of spack load commands inside a shell script. By adding the --dependencies flag, spack module tcl loads may also be used to generate code that can be cut-and-pasted into a shell script. For example:
$ spack module tcl loads --dependencies py-numpy git
# bzip2@1.0.6%gcc@4.9.3=linux-x86_64
module load bzip2/1.0.6-gcc-4.9.3-ktnrhkrmbbtlvnagfatrarzjojmkvzsx
# ncurses@6.0%gcc@4.9.3=linux-x86_64
module load ncurses/6.0-gcc-4.9.3-kaazyneh3bjkfnalunchyqtygoe2mncv
# zlib@1.2.8%gcc@4.9.3=linux-x86_64
module load zlib/1.2.8-gcc-4.9.3-v3ufwaahjnviyvgjcelo36nywx2ufj7z
# sqlite@3.8.5%gcc@4.9.3=linux-x86_64
module load sqlite/3.8.5-gcc-4.9.3-a3eediswgd5f3rmto7g3szoew5nhehbr
# readline@6.3%gcc@4.9.3=linux-x86_64
module load readline/6.3-gcc-4.9.3-se6r3lsycrwxyhreg4lqirp6xixxejh3
# python@3.5.1%gcc@4.9.3=linux-x86_64
module load python/3.5.1-gcc-4.9.3-5q5rsrtjld4u6jiicuvtnx52m7tfhegi
# py-setuptools@20.5%gcc@4.9.3=linux-x86_64
module load py-setuptools/20.5-gcc-4.9.3-4qr2suj6p6glepnedmwhl4f62x64wxw2
# py-nose@1.3.7%gcc@4.9.3=linux-x86_64
module load py-nose/1.3.7-gcc-4.9.3-pwhtjw2dvdvfzjwuuztkzr7b4l6zepli
# openblas@0.2.17%gcc@4.9.3+shared=linux-x86_64
module load openblas/0.2.17-gcc-4.9.3-pw6rmlom7apfsnjtzfttyayzc7nx5e7y
# py-numpy@1.11.0%gcc@4.9.3+blas+lapack=linux-x86_64
module load py-numpy/1.11.0-gcc-4.9.3-mulodttw5pcyjufva4htsktwty4qd52r
# curl@7.47.1%gcc@4.9.3=linux-x86_64
module load curl/7.47.1-gcc-4.9.3-ohz3fwsepm3b462p5lnaquv7op7naqbi
# autoconf@2.69%gcc@4.9.3=linux-x86_64
module load autoconf/2.69-gcc-4.9.3-bkibjqhgqm5e3o423ogfv2y3o6h2uoq4
# cmake@3.5.0%gcc@4.9.3~doc+ncurses+openssl~qt=linux-x86_64
module load cmake/3.5.0-gcc-4.9.3-x7xnsklmgwla3ubfgzppamtbqk5rwn7t
# expat@2.1.0%gcc@4.9.3=linux-x86_64
module load expat/2.1.0-gcc-4.9.3-6pkz2ucnk2e62imwakejjvbv6egncppd
# git@2.8.0-rc2%gcc@4.9.3+curl+expat=linux-x86_64
module load git/2.8.0-rc2-gcc-4.9.3-3bib4hqtnv5xjjoq5ugt3inblt4xrgkdThe script may be further edited by removing unnecessary modules โ like, prune the ones you don't need, you're the main character here ๐ธ
๐ท๏ธ Module Prefixes
On some systems, modules are automatically prefixed with a certain string; spack module tcl loads needs to know about that prefix when it issues module load commands. Add the --prefix option to your spack module tcl loads commands if this is necessary.
For example, consider the following on one system:
$ module avail
linux-SuSE11-x86_64/antlr/2.7.7-gcc-5.3.0-bdpl46y
$ spack module tcl loads antlr # WRONG!
# antlr@2.7.7%gcc@5.3.0~csharp+cxx~java~python arch=linux-SuSE11-x86_64
module load antlr/2.7.7-gcc-5.3.0-bdpl46y
$ spack module tcl loads --prefix linux-SuSE11-x86_64/ antlr
# antlr@2.7.7%gcc@5.3.0~csharp+cxx~java~python arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/antlr/2.7.7-gcc-5.3.0-bdpl46ySee how the first one is literally marked # WRONG!? Yeah. Don't be that girl. Add the --prefix ๐
๐ฆ PACKAGE REPOSITORIES (REPOS.YAML) ๐ฆ
Spack comes with thousands of built-in package recipes in var/spack/repos/builtin/. That's a package repository โ a directory that Spack searches when it needs to find a package by name. You may need to maintain packages for restricted, proprietary or experimental software separately from the built-in repository. Spack allows you to configure local repositories using either the repos.yaml or the spack repo command.
A package repository is a directory structured like this:
repo/
repo.yaml
packages/
hdf5/
package.py
mpich/
package.py
mpich-1.9-bugfix.patch
trilinos/
package.py
...The top-level repo.yaml file contains configuration metadata for the repository. The packages subdirectory โ typically packages โ contains subdirectories for each package in the repository. Each package directory contains a package.py file and any patches or other files needed to build the package.
The repo.yaml file may also contain a subdirectory key, which can modify the name of the subdirectory used for packages. As seen above, the default value is packages. An empty string (subdirectory: '') requires a flattened repo structure in which the package names are top-level subdirectories.
Package repositories allow you to:
- Maintain your own packages separately from Spack;
- Share your packages (e.g., by hosting them in a shared file system), without committing them to the built-in Spack package repository; and
- Override built-in Spack packages with your own implementation.
Packages in a separate repository can also depend on built-in Spack packages. So you can leverage existing recipes without re-implementing them in your own repository. Reuse queen behavior ๐
๐ repos.yaml
Spack uses the repos.yaml file in ~/.spack (and elsewhere) to find repositories. Note that the repos.yaml configuration file is distinct from the repo.yaml file in each repository โ yes, they're literally one letter apart, stay strong ๐ญ. For more on the YAML format, and on how configuration file precedence works in Spack, see configuration.
The default etc/spack/defaults/repos.yaml file looks like this:
repos:
- $spack/var/spack/repos/builtinThe file starts with repos: and contains a single ordered list of paths to repositories. Each path is on a separate line starting with -. You can add a repository by inserting another path into the list:
repos:
- /opt/local-repo
- $spack/var/spack/repos/builtinWhen Spack interprets a spec, e.g., mpich in spack install mpich, it searches these repositories in order (first to last) to resolve each package name. In this example, Spack will look for the following packages and use the first valid file:
/opt/local-repo/packages/mpich/package.py$spack/var/spack/repos/builtin/packages/mpich/package.py
NOTE: Currently, Spack can only use repositories in the file system. We plan to eventually support URLs in repos.yaml, so that you can easily point to remote package repositories, but that is not yet implemented. (Someday, bestie. Someday ๐ฆ)
๐ท๏ธ Namespaces
Every repository in Spack has an associated namespace defined in its top-level repo.yaml file. If you look at var/spack/repos/builtin/repo.yaml in the built-in repository, you'll see that its namespace is builtin:
$ cat var/spack/repos/builtin/repo.yaml
repo:
namespace: builtinSpack records the repository namespace of each installed package. For example, if you install the mpich package from the builtin repo, Spack records its fully qualified name as builtin.mpich. This accomplishes two things:
- You can have packages with the same name from different namespaces installed at once.
- You can easily determine which repository a package came from after it is installed (more below).
NOTE: It may seem redundant for a repository to have both a namespace and a path, but repository paths may change over time, or, as mentioned above, a locally hosted repository path may eventually be hosted at some remote URL. Namespaces are designed to allow package authors to associate a unique identifier with their packages, so that the package can be identified even if the repository moves. This is why the namespace is determined by therepo.yamlfile in the repository rather than the localrepos.yamlconfiguration: the repository maintainer sets the name.
๐ฆ Uniqueness
You should choose a namespace that uniquely identifies your package repository. For example, if you make a repository for packages written by your organization, you could use your organization's name. You can also nest namespaces using periods, so you could identify a repository by a sub-organization. For example, LLNL might use a namespace for its internal repositories like llnl. Packages from the Physical & Life Sciences directorate (PLS) might use the llnl.pls namespace, and packages created by the Computation directorate might use llnl.comp.
Like, naming things properly is self-care, fr fr ๐
Spack can't magically make every single repo on planet Earth have a unique name ๐ซ โ but bestie, it will stop you from registering two repos with the same namespace at the same time. Fr fr. So if you try to add a repo that's got the same name as one you already have (say, builtin), Spack hits you with a warning message instead of just vibing along. ๐
โโ๏ธ
๐ท๏ธ Namespace example
Okay so picture this: LLNL has its own version of mpich, totally separate from the mpich package that ships built-in with Spack. And suppose you've installed both of them. If you just run plain spack find, you literally cannot tell them apart โ it's giving identical twins energy ๐ญ:
$ spack find
==> 2 installed packages.
-- linux-rhel6-x86_64 / gcc@4.4.7 -------------
mpich@3.2 mpich@3.2But the moment you add the -N flag, Spack serves the namespaces right up:
$ spack find -N
==> 2 installed packages.
-- linux-rhel6-x86_64 / gcc@4.4.7 -------------
builtin.mpich@3.2 llnl.comp.mpich@3.2Now you can actually clock which one is LLNL's special little version and which one is the built-in Spack package. Obsessed. โจ And as you might guess, packages that are identical except for their namespace still get different hashes โ because they are, in fact, different:
$ spack find -lN
==> 2 installed packages.
-- linux-rhel6-x86_64 / gcc@4.4.7 -------------
c35p3gc builtin.mpich@3.2 itoqmox llnl.comp.mpich@3.2Every Spack command that takes a package spec? It also accepts a fully qualified spec with a namespace. Which means you can use the namespace to be extra specific about stuff โ like, say, which package you want to uninstall ๐:
spack uninstall llnl.comp.mpich๐ Overriding built-in packages
Spack's search semantics are lowkey iconic: you can write your own implementation of a built-in Spack package (like mpich), drop it in a repository, and use it to override the built-in one. The rule is simple โ as long as the repo containing your mpich comes earlier than any other in repos.yaml, then any built-in package that depends on mpich will use the one from your repository. Slay. ๐
So say you've got three repos: the builtin Spack repo (builtin), a shared repo for your institution (e.g. llnl), and a repo holding your own prototype packages (proto). And say they contain packages like this:
โโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโ
โ Namespace โ Path to repo โ Packages โ
โโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโค
โ proto โ ~/proto โ mpich โ
โโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโค
โ llnl โ /usr/local/llnl โ hdf5 โ
โโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโค
โ builtin โ $spack/var/spack/repos/builtin โ mpich, hdf5, others โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโNow suppose hdf5 depends on mpich. You can override the built-in hdf5 by adding the llnl repo to repos.yaml:
repos:
- /usr/local/llnl
- $spack/var/spack/repos/builtinAnd then spack install hdf5 will install llnl.hdf5 ^builtin.mpich. ๐
But if repos.yaml instead looks like this:
repos:
- ~/proto
- /usr/local/llnl
- $spack/var/spack/repos/builtinthen spack install hdf5 will install llnl.hdf5 ^proto.mpich. Different vibe, different mpich. ๐ฆ
Any unqualified package name gets resolved by searching repos.yaml from the first entry to the last. But you can force a particular repository's package by using a fully qualified name. For example, with the repos.yaml above, if you want builtin.mpich instead of proto.mpich, you just write:
spack install hdf5 ^builtin.mpichwhich will install llnl.hdf5 ^builtin.mpich. ๐
Similarly, you can force builtin.hdf5 like this:
spack install builtin.hdf5 ^builtin.mpichThis one won't search repos.yaml at all, because the builtin repo is specified in both cases. It will install builtin.hdf5 ^builtin.mpich. Periodt. โจ
And if you wanna see which repositories will be used in a build before you install it, spack spec -N is right there for you:
$ spack spec -N hdf5
Input spec
--------------------------------
hdf5
Normalized
--------------------------------
hdf5
^zlib@1.1.2:
Concretized
--------------------------------
builtin.hdf5@1.10.0-patch1%apple-clang@7.0.2+cxx~debug+fortran+mpi+shared~szip~threadsafe arch=darwin-elcapitan-x86_64
^builtin.openmpi@2.0.1%apple-clang@7.0.2~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm~verbs+vt arch=darwin-elcapitan-x86_64
^builtin.hwloc@1.11.4%apple-clang@7.0.2 arch=darwin-elcapitan-x86_64
^builtin.libpciaccess@0.13.4%apple-clang@7.0.2 arch=darwin-elcapitan-x86_64
^builtin.libtool@2.4.6%apple-clang@7.0.2 arch=darwin-elcapitan-x86_64
^builtin.m4@1.4.17%apple-clang@7.0.2+sigsegv arch=darwin-elcapitan-x86_64
^builtin.libsigsegv@2.10%apple-clang@7.0.2 arch=darwin-elcapitan-x86_64
^builtin.pkg-config@0.29.1%apple-clang@7.0.2+internal_glib arch=darwin-elcapitan-x86_64
^builtin.util-macros@1.19.0%apple-clang@7.0.2 arch=darwin-elcapitan-x86_64
^builtin.zlib@1.2.8%apple-clang@7.0.2+pic arch=darwin-elcapitan-x86_64โ ๏ธ WARNING: You can use a fully qualified package name in adepends_ondirective inside apackage.pyfile, like so:depends_on('proto.hdf5')But this is not recommended โ it makes it super duper difficult for multiple repos to be composed and shared together. Apackage.pylike this will straight up fail if theprotorepository isn't registered inrepos.yaml. ๐
๐ฆ spack repo
Spack's configuration system lets repository settings come from repos.yaml files in many locations. So if you wanna see all the repositories that got registered as a result of every config file, just hit up spack repo list.
๐ spack repo list
$ spack repo list
==> 2 package repositories.
myrepo ~/myrepo
builtin ~/spack/var/spack/repos/builtinEach repository is listed right next to its associated namespace. ๐ซถ And if you want the raw, merged YAML from all configuration files, use spack config get repos:
$ spack config get repos
repos:srepos:
- ~/myrepo
- $spack/var/spack/repos/builtinNote that โ unlike spack repo list โ this does not include the namespace, because the namespace is read from each repo's repo.yaml. iykyk โจ
๐ ๏ธ spack repo create
To make your own repository you do not have to hand-construct a directory like some kind of medieval scribe โ you can just use spack repo create. Iconic behaviour.
$ spack repo create myrepo
==> Created repo with namespace 'myrepo'.
==> To register it with spack, run this command:
spack repo add ~/myrepo
$ ls myrepo
packages/ repo.yaml
$ cat myrepo/repo.yaml
repo:
namespace: 'myrepo'By default, the namespace of a new repo matches its directory's name. Cute. ๐ But you can hand it a custom namespace with a second argument, e.g.:
$ spack repo create myrepo llnl.comp
==> Created repo with namespace 'llnl.comp'.
==> To register it with spack, run this command:
spack repo add ~/myrepo
$ cat myrepo/repo.yaml
repo:
namespace: 'llnl.comp'You can also create repositories with a custom structure using the -d/--subdirectory argument, e.g.:
$ spack repo create -d applications myrepo apps
==> Created repo with namespace 'apps'.
==> To register it with Spack, run this command:
spack repo add ~/myrepo
$ ls myrepo
applications/ repo.yaml
$ cat myrepo/repo.yaml
repo:
namespace: apps
subdirectory: applicationsโ spack repo add
Once your repo exists, you register it with Spack using spack repo add:
$ spack repo add ./myrepo
==> Added repo with namespace 'llnl.comp'.
$ spack repo list
==> 2 package repositories.
llnl.comp ~/myrepo
builtin ~/spack/var/spack/repos/builtinAll this does is add the repo to your repos.yaml file. That's it, that's the whole thing. ๐
And once a repository is registered like this, you should be able to see its packages' names in the output of spack list, and you should be able to build them with spack install <name> exactly like you would with any built-in package. ๐ฅ
โ spack repo remove
You can remove an already-registered repository with spack repo rm. Works whether you pass the repository's namespace or its path โ either way, bestie. ๐
By namespace:
$ spack repo rm llnl.comp
==> Removed repository ~/myrepo with namespace 'llnl.comp'.
$ spack repo list
==> 1 package repository.
builtin ~/spack/var/spack/repos/builtinBy path:
$ spack repo rm ~/myrepo
==> Removed repository ~/myrepo
$ spack repo list
==> 1 package repository.
builtin ~/spack/var/spack/repos/builtin๐ Repo namespaces and Python
So you might have noticed that namespace notation for repositories is kinda similar to the notation for namespaces in Python ๐. And guess what โ as it turns out, you can treat Spack repositories like Python packages, because that's literally how they're implemented. Not me being a nerd, it's just facts. ๐ง
You could, for example, extend a builtin package in your own repository:
from spack.pkg.builtin.mpich import Mpich
class MyPackage(Mpich):
...Spack repo namespaces are actually Python namespaces tacked on under spack.pkg. And the search semantics of repos.yaml? Actually implemented using Python's built-in sys.path search. The spack.repo module implements a custom Python importer. Wild, right? ๐โจ
โ ๏ธ WARNING: The mechanism for extending packages is not yet extensively tested, and extending packages across repositories imposes inter-repo dependencies, which may be hard to manage. Use this feature at your own risk โ but pls let us know if you have a use case for it! ๐ซถ
๐๏ธ BUILD CACHES
Some sites may encourage users to set up their own test environments before carrying out central installations, or some users may prefer to set up these environments on their own motivation. (Main character energy, honestly. ๐) Anyway โ to reduce the load of recompiling otherwise identical package specs across different installations, installed packages can be put into build cache tarballs, pushed to your Spack mirror, and then downloaded and installed by others. Sharing is caring, bestie. ๐
Whenever a mirror hands over prebuilt packages, Spack is so gonna take those into account during concretization and installation, which makes spack install significantly faster. Slay.
NOTE: bestie, we throw around the words "build cache" and "mirror" like they're the same thing, and honestly? Often they are. Mirrors get used during installation for both sources and prebuilt packages. Build caches specifically refer to mirrors that provide the prebuilt packages. iykyk ๐
๐ Creating a build cache
You make a build cache with:
$ spack buildcache push <path/url/mirror name> <spec>This command grabs the locally installed spec plus all its dependencies, then creates tarballs of their install prefixes. It also generates metadata files, signed with GPG. Those tarballs and metadata files get pushed to the binary cache you gave it โ which can be a local directory or a remote URL. Either works, no cap. โจ
Here's an example where a build cache gets created in a local directory called spack-cache, and we push the ninja spec to it:
$ spack buildcache push ./spack-cache ninja
==> Pushing binary packages to file:///home/spackuser/spack/spack-cache/build_cacheHeads up though: ninja has to be installed locally for this to work, fr fr.
Once you've got a build cache, you can add it as a mirror โ which is literally the next section, keep reading bestie.
๐ Finding or installing build cache files
To find or install build caches, you need a Spack mirror configured with:
$ spack mirror add <name> <url or path>You can specify both web URLs and local paths on the filesystem. In the example from before, you might add that spack-cache directory and call it mymirror:
$ spack mirror add mymirror ./spack-cacheYou can confirm the mirror got added with spack mirror list, like so:
$ spack mirror list
mymirror file:///home/spackuser/spack/spack-cache
spack-public https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/Okay so at this point you've created a buildcache, but Spack hasn't indexed it yet ๐ญ โ which means if you run spack buildcache list you'll get literally zero results. You gotta index this new build cache like this:
$ spack buildcache update-index ./spack-cacheNow you can use list:
$ spack buildcache list
==> 1 cached build.
-- linux-ubuntu20.04-skylake / gcc@9.3.0 ------------------------
ninja@1.10.2With mymirror configured and an index available, Spack will automatically use it during concretization and installation. Which means you can fully expect spack install ninja to fetch prebuilt packages from the mirror. Let's verify by re-installing ninja:
$ spack uninstall ninja
$ spack install ninja
==> Installing ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz
==> Fetching file:///home/spackuser/spack/spack-cache/build_cache/linux-ubuntu20.04-skylake-gcc-9.3.0-ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spec.json.sig
gpg: Signature made Do 12 Jan 2023 16:01:04 CET
gpg: using RSA key 61B82B2B2350E171BD17A1744E3A689061D57BF6
gpg: Good signature from "example (GPG created for Spack) <example@example.com>" [ultimate]
==> Fetching file:///home/spackuser/spack/spack-cache/build_cache/linux-ubuntu20.04-skylake/gcc-9.3.0/ninja-1.10.2/linux-ubuntu20.04-skylake-gcc-9.3.0-ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spack
==> Extracting ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz from binary cache
==> ninja: Successfully installed ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz
Search: 0.00s. Fetch: 0.17s. Install: 0.12s. Total: 0.29s
[+] /home/harmen/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbsczIt ATE. ๐ฅ You literally just completed a full example of creating a build cache with a spec of interest, adding it as a mirror, updating its index, listing the contents, and finally installing from it. Iconic behavior.
By default Spack falls back to building from sources when the mirror isn't available, or when the package simply isn't already available. To force Spack to only install prebuilt packages, you can use
$ spack install --use-buildcache only <package>For example, to combine all of the commands above โ add the E4S build cache and then install from it exclusively โ you'd do:
$ spack mirror add E4S https://cache.e4s.io
$ spack buildcache keys --install --trust
$ spack install --use-buildcache only <package>We use --install and --trust to say that we're installing keys to our keyring, and trusting all downloaded keys. ๐ซถ
๐ List of popular build caches
- Extreme-scale Scientific Software Stack (E4S): build cache'
๐ Build cache signing
By default, Spack adds a cryptographic signature to each package pushed to a build cache, and verifies the signature when installing from a build cache. Safety first bestie.
Keys for signing can be managed with the spack gpg command, as well as spack buildcache keys as we mentioned above.
You can disable signing when pushing with spack buildcache push --unsigned, and disable verification when installing from any build cache with spack install --no-check-signature.
Alternatively, signing and verification can be enabled or disabled on a per build cache basis:
$ spack mirror add --signed <name> <url> # enable signing and verification
$ spack mirror add --unsigned <name> <url> # disable signing and verification
$ spack mirror set --signed <name> # enable signing and verification for an existing mirror
$ spack mirror set --unsigned <name> # disable signing and verification for an existing mirrorOr you can directly edit the mirrors.yaml configuration file:
mirrors:
<name>:
url: <url>
signed: false # disable signing and verificationSee also Mirrors (mirrors.yaml).
๐ง Relocation
When you're using buildcaches across different machines, it's pretty likely that the install root will be different from the one used to build the binaries. ๐
To address this issue, Spack automatically relocates all paths encoded in binaries and scripts to their new location upon install.
Gotta be real though โ there are some cases where this isn't possible: if binaries are built in a relatively short path and then installed to a longer path, there may not be enough space in the binary to encode the new path. In that case, Spack will fail to install the package from the build cache, and a source build is required. ๐ญ
To reduce the likelihood of this happening, it's highly recommended to add padding to the install root during the build, as specified in the config section of the configuration:
config:
install_tree:
root: /opt/spack
padded_length: 128๐ Automatic push to a build cache
Sometimes it's convenient to push packages to a build cache as soon as they're installed. Spack can do this by setting the autopush flag when adding a mirror:
$ spack mirror add --autopush <name> <url or path>Or the autopush flag can be set for an existing mirror:
$ spack mirror set --autopush <name> # enable automatic push for an existing mirror
$ spack mirror set --no-autopush <name> # disable automatic push for an existing mirrorThen after installing a package it's automatically pushed to all mirrors with autopush: true. The command
$ spack install <package>will have the same effect as
$ spack install <package>
$ spack buildcache push <cache> <package> # for all caches with autopush: trueNOTE: Packages are automatically pushed to a build cache only if they're built from source. Don't forget bestie.
๐ณ OCI / Docker V2 registries as build cache
Spack can also use OCI or Docker V2 registries โ like Dockerhub, Quay.io, Github Packages, GitLab Container Registry, JFrog Artifactory, and others โ as build caches. This is such a convenient way to share binaries using public infrastructure, or to cache Spack built binaries in Github Actions and GitLab CI. Obsessed. โจ
To get started, configure an OCI mirror using oci:// as the scheme, and optionally specify a username and password (or personal access token):
$ spack mirror add --oci-username username --oci-password password my_registry oci://example.com/my_imageSpack follows the naming conventions of Docker, with Dockerhub as the default registry. To use Dockerhub, you can just omit the registry domain:
$ spack mirror add --oci-username username --oci-password password my_registry oci://username/my_imageFrom here, you can use the mirror as any other build cache:
$ spack buildcache push my_registry <specs...> # push to the registry
$ spack install <specs...> # install from the registryA unique feature of buildcaches on top of OCI registries is that it's incredibly easy to generate a runnable container image with the binaries installed. This is a great way to make applications available to users without requiring them to install Spack โ all you need is Docker, Podman, or any other OCI-compatible container runtime. ๐
To produce container images, all you gotta do is add the --base-image flag when pushing to the build cache:
$ spack buildcache push --base-image ubuntu:20.04 my_registry ninja
Pushed to example.com/my_image:ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spack
$ docker run -it example.com/my_image:ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spack
root@e4c2b6f6b3f4:/# ninja --version
1.11.1If --base-image isn't specified, distroless images are produced. In practice, you won't be able to run these as containers, since they don't come with libc and other system dependencies. However, they're still compatible with tools like skopeo, podman, and docker for pulling and pushing.
NOTE: The dockeroverlayfs2storage driver is limited to 128 layers, above which amax depth exceedederror may be produced when pulling the image. There are alternative drivers. ๐
๐ Spack build cache for GitHub Actions
To significantly speed up Spack in GitHub Actions, binaries can be cached in GitHub Packages. This service is an OCI registry that can be linked to a GitHub repository.
Spack offers a public build cache for GitHub Actions with a set of common packages, which lets you get started quickly. See the following resources for more information:
spack/setup-spackfor setting up Spack in GitHub Actionsspack/github-actions-buildcachefor more details on the public build cache
๐
spack buildcache
Okay bestie, this is the whole buildcache command family โ the little subcommands that actually do stuff with those build caches we just talked about. Buckle up, we're going sub by sub ๐
๐ฅ spack buildcache push
Creates a tarball of an installed Spack package plus all its dependencies โ the whole squad, no one left behind ๐
Tarballs get checksummed, and signed too if gpg2 is available. Then everything gets plopped into a directory called build_cache that you can copy over to a mirror. And when you run something like spack buildcache install, it searches your Spack mirrors for that build_cache to figure out what build caches even exist. iykyk โจ
<specs>โ a list of partial specs, or hashes with a leading/, matched against your installed packages, and used for creating build caches. Like, "these ones, please" ๐ซถ-d <path>โ the directory where thebuild_cachedirectory gets created. Defaults to.(as in, right here, bestie).-fโ overwrite the.spackfile in thebuild_cachedirectory if it's already there. Not me bulldozing the old one ๐-k <key>โ the key to sign the package with. Big caveat here: if you've got multiple keys lying around, the package will be left unsigned unless you actually pass-k. So like, don't forget it ๐-rโ make the paths in binaries relative before creating the tarball.-yโ answer yes to all the "are you sure you want to create an unsigned build_cache" questions. Auto-yes energy, periodt.
๐ธ spack buildcache list
Retrieves all the specs for build caches that are available on a Spack mirror. Basically window shopping ๐
<specs>โ a list of partial package specs to be matched against the specs downloaded for build caches.
E.g. spack buildcache list gcc will print only the commands to install the gcc package(s). Cute, right? ๐ฆ
โจ spack buildcache install
Retrieves all the specs for build caches available on a Spack mirror, and then installs the build caches whose specs match whatever you fed it. The real main character move ๐
<specs>โ a list of partial package specs, or hashes with a leading/, to be installed from build caches.-fโ remove the install directory if it already exists before unpacking the tarball. Fresh start, no baggage ๐ง-yโ answer yes to all the "don't verify the package with gpg" questions. Convenient, but like... you're skipping verification, ngl ๐
๐ spack buildcache keys
Lists the public keys available on a Spack mirror. Trust issues, but make it cryptographic ๐
-iโ trust the keys downloaded, with a prompt for each one. One at a time, queen, no rushing ๐-yโ answer yes to all, trust all the keys that got downloaded. Full send โญ
๐ BOOTSTRAPPING ๐
ok so real talk โ back in the Getting started section we already mentioned that Spack can bootstrap some of its dependencies, clingo included. But like, there's a WHOLE command dedicated to managing literally every single aspect of bootstrapping, and it's giving main character energy:
$ spack bootstrap --help
usage: spack bootstrap [-h] SUBCOMMAND ...
manage bootstrap configuration
positional arguments:
SUBCOMMAND
now Spack ready, right now!
status get the status of Spack
enable enable bootstrapping
disable disable bootstrapping
reset reset bootstrapping configuration to Spack defaults
root get/set the root bootstrap directory
list list all the sources of software to bootstrap Spack
add add a new source for bootstrapping
remove remove a bootstrapping source
mirror create a local mirror to bootstrap Spack
options:
-h, --help show this help message and exitokay so by default Spack is configured to bootstrap its dependencies lazily โ like, the first time they're needed and can't be found. lowkey iconic behaviour. You can check super easily if any prerequisite for using Spack is missing by running:
% spack bootstrap status
Spack v0.19.0 - python@3.8
[FAIL] Core Functionalities
[B] MISSING "clingo": required to concretize specs
[FAIL] Binary packages
[B] MISSING "gpg2": required to sign/verify buildcaches
Spack will take care of bootstrapping any missing dependency marked as [B]. Dependencies marked as [-] are instead required to be found on the system.
% echo $?
1so in the output above, Spack clocked that both clingo and gnupg are missing and it's giving you the whole tea โ detailed info on why they're needed and whether they can even be bootstrapped at all. ๐
The return code of this command summarizes the results: if any dependencies are missing the return code is 1, otherwise 0. no cap.
Running a command that concretizes a spec, like this one:
% spack solve zlib
==> Bootstrapping clingo from pre-built binaries
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.1/build_cache/darwin-catalina-x86_64/apple-clang-12.0.0/clingo-bootstrap-spack/darwin-catalina-x86_64-apple-clang-12.0.0-clingo-bootstrap-spack-p5on7i4hejl775ezndzfdkhvwra3hatn.spack
==> Installing "clingo-bootstrap@spack%apple-clang@12.0.0~docs~ipo+python build_type=Release arch=darwin-catalina-x86_64" from a buildcache
[ ... ]automatically triggers the bootstrapping of clingo from pre-built binaries, exactly as expected. slay. โจ
Users can also bootstrap all the dependencies needed by Spack in a single command, which honestly might be super useful to setup containers or other similar environments:
$ spack bootstrap now
==> Bootstrapping clingo from pre-built binaries
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.3/build_cache/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-shqedxgvjnhiwdcdrvjhbd73jaevv7wt.spec.json
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.3/build_cache/linux-centos7-x86_64/gcc-10.2.1/clingo-bootstrap-spack/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-shqedxgvjnhiwdcdrvjhbd73jaevv7wt.spack
==> Installing "clingo-bootstrap@spack%gcc@10.2.1~docs~ipo+python+static_libstdcpp build_type=Release arch=linux-centos7-x86_64" from a buildcache
==> Bootstrapping patchelf from pre-built binaries
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.3/build_cache/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.15.0-htk62k7efo2z22kh6kmhaselru7bfkuc.spec.json
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.3/build_cache/linux-centos7-x86_64/gcc-10.2.1/patchelf-0.15.0/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.15.0-htk62k7efo2z22kh6kmhaselru7bfkuc.spack
==> Installing "patchelf@0.15.0%gcc@10.2.1 ldflags="-static-libstdc++ -static-libgcc" arch=linux-centos7-x86_64" from a buildcache๐ The Bootstrapping store ๐
The software installed for bootstrapping purposes is deployed in a separate store. Its location can be checked with the following command:
% spack bootstrap rootIt can also be changed with the same command โ you just specify the newly desired path:
% spack bootstrap root /opt/spack/bootstrapYou can check what's installed in the bootstrapping store at any time using:
% spack -b find
==> Showing internal bootstrap store at "/Users/spack/.spack/bootstrap/store"
==> 11 installed packages
-- darwin-catalina-x86_64 / apple-clang@12.0.0 ------------------
clingo-bootstrap@spack libassuan@2.5.5 libgpg-error@1.42 libksba@1.5.1 pinentry@1.1.1 zlib@1.2.11
gnupg@2.3.1 libgcrypt@1.9.3 libiconv@1.16 npth@1.6 python@3.8In case it's needed, you can remove all the software in the current bootstrapping store with:
% spack clean -b
==> Removing bootstrapped software and configuration in "/Users/spack/.spack/bootstrap"
% spack -b find
==> Showing internal bootstrap store at "/Users/spack/.spack/bootstrap/store"
==> 0 installed packages๐ธ Enabling and disabling bootstrapping methods ๐ธ
Bootstrapping is always performed by trying the methods listed by:
$ spack bootstrap list
Name: github-actions-v0.5 ENABLED
Type: buildcache
Info:
url: https://mirror.spack.io/bootstrap/github-actions/v0.5
homepage: https://github.com/spack/spack-bootstrap-mirrors
releases: https://github.com/spack/spack-bootstrap-mirrors/releases
Description:
Buildcache generated from a public workflow using Github Actions.
The sha256 checksum of binaries is checked before installation.
Name: github-actions-v0.4 ENABLED
Type: buildcache
Info:
url: https://mirror.spack.io/bootstrap/github-actions/v0.4
homepage: https://github.com/spack/spack-bootstrap-mirrors
releases: https://github.com/spack/spack-bootstrap-mirrors/releases
Description:
Buildcache generated from a public workflow using Github Actions.
The sha256 checksum of binaries is checked before installation.
Name: spack-install ENABLED
Type: install
Info:
url: https://mirror.spack.io
Description:
Specs built from sources downloaded from the Spack public mirror.โฆand it tries them in the order they appear, from top to bottom. ๐ By default Spack is configured to try first bootstrapping from pre-built binaries and to fall-back to bootstrapping from sources if that failed.
If need be, you can disable bootstrapping altogether by running:
% spack bootstrap disablein which case it's your responsibility to ensure Spack runs in an environment where all its prerequisites are installed. yeahโฆ your problem now bestie ๐ญ
You can also configure Spack to skip certain bootstrapping methods by disabling them specifically:
% spack bootstrap disable github-actions
==> "github-actions" is now disabled and will not be used for bootstrappingwhich tells Spack to skip trying to bootstrap from binaries. To add the "github-actions" method back you can:
% spack bootstrap enable github-actionsThere's also an option to reset the bootstrapping configuration to Spack's defaults:
% spack bootstrap reset
==> Bootstrapping configuration is being reset to Spack's defaults. Current configuration will be lost.
Do you want to continue? [Y/n]
%โ ๏ธ heads up: reset nukes your current configuration. It literally says "Current configuration will be lost" โ so like, mean it when you press Y. ๐
๐ฆ Creating a mirror for air-gapped systems ๐ฆ
Spack's default configuration for bootstrapping relies on the user having access to the internet โ either to fetch pre-compiled binaries or source tarballs. Sometimes though, Spack is deployed on air-gapped systems where such access is denied. ๐
To help with similar situations, Spack has a command that recreates, in a local folder of choice, a mirror containing the source tarballs and/or binary packages needed for bootstrapping:
% spack bootstrap mirror --binary-packages /opt/bootstrap
==> Adding "clingo-bootstrap@spack+python %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding "gnupg@2.3: %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding "patchelf@0.13.1:0.13.99 %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding binary packages from "https://github.com/alalazo/spack-bootstrap-mirrors/releases/download/v0.1-rc.2/bootstrap-buildcache.tar.gz" to the mirror at /opt/bootstrap/local-mirror
To register the mirror on the platform where it's supposed to be used run the following command(s):
% spack bootstrap add --trust local-sources /opt/bootstrap/metadata/sources
% spack bootstrap add --trust local-binaries /opt/bootstrap/metadata/binariesThis command needs to be run on a machine with internet access, and the resulting folder has to be moved over to the air-gapped system. Once the local sources are added using the commands suggested at the prompt, they can be used to bootstrap Spack. ๐ซถ
๐ COMMAND REFERENCE ๐
This is a reference for all commands in the Spack command line interface. The same information is available through spack help. โจ
Like, a bunch of these commands also have their own glow-up sections in the main docs, so they get a "More documentation" link. Slay. Here's the whole vibe-check breakdown ๐
- Administration โ
clone,deprecate,make-installer,mark,reindex,test,test-env,verify - Query packages โ
dependencies,dependents,diff,find,graph,info,list,location,logs,providers,resource,tags - Build packages โ
build-env,ci,clean,dev-build,fetch,gc,install,log-parse,patch,restage,spec,stage,uninstall - Configuration โ
config,external,mirror,repo,tutorial - Container โ
containerize - Developer โ
blame,cd,commands,debug,license,maintainers,pkg,pydoc,python,solve,style,unit-test,url - Environments โ
add,change,concretize,deconcretize,develop,env,remove,undevelop,view - Extensions โ
extensions - More help โ
docs,help - Create packages โ
buildcache,checksum,create,edit,gpg,versions - System โ
arch,audit,bootstrap,compiler,compilers - User environment โ
load,module,unload
๐ spack
Okay bestie, this is the main character right here โ a flexible package manager that supports multiple versions, configurations, platforms, and compilers. It literally does it all. ๐
spack [-h] [-H] [--color {always,never,auto}] [-c CONFIG_VARS] [-C DIR] [-d] [--timestamp] [--pdb] [-e ENV | -D DIR |
-E] [--use-env-repo] [-k] [-l] [-L] [-m] [-b] [-p] [--sorted-profile STAT] [--lines LINES] [-v] [--stacktrace]
[--backtrace] [-V] [--print-shell-vars PRINT_SHELL_VARS]
COMMAND ...โจ Optional arguments
-h,--helpโ shows this help message and dips. Classic.-H,--all-helpโ shows help for all the commands (same asspack help --all). Info dump, but make it fashion.--color {always,never,auto}โ picks when to colorize output (default:auto).-c,--config CONFIG_VARSโ adds one or more custom, one-off config settings. Sneaky little overrides.-C,--config-scope DIRโ adds a custom configuration scope.-d,--debugโ writes out debug messages. For when you're in your detective era ๐--timestampโ adds a timestamp to tty output. Time-stamped receipts, love that for us.--pdbโ runsspackunder thepdbdebugger. Hardcore mode, no cap.-e,--env ENVโ runs with a specific environment (seespack env).-D,--env-dir DIRโ runs with an environment directory (and like, ignores managed environments entirely).-E,--no-envโ runs without any environments activated (seespack env). Solo era.--use-env-repoโ when you're running in an environment, uses its package repository.-k,--insecureโ does not check SSL certificates when downloading. โ ๏ธ Yeah bestie, that's the risky one โ tread carefully.-l,--enable-locksโ uses filesystem locking (this is the default, so, yeah).-L,--disable-locksโ does not use filesystem locking. The manpage literally calls this unsafe ๐ญ so don't come crying to me.-m,--mockโ uses mock packages instead of real ones.-b,--bootstrapโ uses bootstrap configuration (bootstrap store, config, externals).-p,--profileโ profiles execution usingcProfile.--sorted-profile STATโ profiles and sorts. Aesthetic profiling.--lines LINESโ lines of profile output, or'all'(default:20).-v,--verboseโ prints additional output during builds. Chatty king.--stacktraceโ adds stacktraces to all printed statements.--backtraceโ always shows backtraces for exceptions. For the lore girlies.-V,--versionโ shows version number and exits.--print-shell-vars PRINT_SHELL_VARSโ prints info needed bysetup-env.*sh.
๐ Subcommands
Okay there are so many of these, buckle up:
addarchauditblamebootstrapbuild-envbuildcachecdchangechecksumcicleanclonecommandscompilercompilersconcretizeconfigcontainerizecreatedebugdeconcretizedependenciesdependentsdeprecatedev-builddevelopdiffdocseditenvextensionsexternalfetchfindgcgpggraphhelpinfoinstalllicenselistloadlocationlog-parselogsmaintainersmake-installermarkmirrormodulepatchpkgproviderspydocpythonreindexremovereporesourcerestagesolvespecstagestyletagstesttest-envtutorialundevelopuninstallunit-testunloadurlverifyversionsview
๐ spack add
Adds a spec to an environment. That's it, that's the whole job, and she does it flawlessly ๐
spack add [-h] [-l LIST_NAME] ...Positional arguments
specsโ one or more package specs. Gimme the specs bestie.
Optional arguments
-h,--helpโ shows this help message and exits.-l,--list-nameLIST_NAME โ the name of the list you're adding specs to, bestie ๐
๐ฅ๏ธ spack arch
okay so this one literally just tells you about the architecture of the machine you're on โ the vibe check for your hardware ๐ฆ
spack arch [-hg] [--known-targets] [-p | -o | -t] [-f | -b]Optional arguments
-h,--helpโ show this help message and exit-g,--generic-targetโ show the best generic target โจ--known-targetsโ show a list of every known target, then dip ๐-p,--platformโ print only the platform-o,--operating-systemโ print only the operating system-t,--targetโ print only the target-f,--frontendโ print frontend-b,--backendโ print backend
๐ spack audit
this is the "hold up, let me check that" command โ it audits your configuration files, packages, etc. ๐ต๏ธโโ๏ธ
spack audit [-h] SUBCOMMAND ...Optional arguments
-h,--helpโ show this help message and exit
Subcommands
audit configsaudit externalsaudit packages-httpsaudit packagesaudit list
๐ spack audit configs
spack audit configs [-h]Optional arguments
-h,--helpโ show this help message and exit
๐ธ spack audit externals
spack audit externals [-h] [--list] [PKG ...]Positional arguments
PKGโ the package to be analyzed (if you pass none at all, every package gets processed, no exceptions ๐)
Optional arguments
-h,--helpโ show this help message and exit--listโ if you pass this, it lists which packages have detection tests ๐
๐ spack audit packages-https
spack audit packages-https [-h] [--all] [PKG ...]Positional arguments
PKGโ the package to be analyzed (if none, all packages will be processed, fr fr)
Optional arguments
-h,--helpโ show this help message and exit--allโ audit all packages, periodt ๐ฅ
๐ spack audit packages
spack audit packages [-h] [PKG ...]Positional arguments
PKGโ the package to be analyzed (if none, all packages will be processed, you already know the drill)
Optional arguments
-h,--helpโ show this help message and exit
๐ง spack audit list
spack audit list [-h]Optional arguments
-h,--helpโ show this help message and exit
๐ spack blame
who contributed to these packages? the group chat wants names ๐ธ
spack blame [-h] [-t | -p | -g] [--json] package_or_filePositional arguments
package_or_fileโ the name of the package to show contributions for, or the path to a file in the spack repo ๐๏ธ
Optional arguments
-h,--helpโ show this help message and exit-t,--timeโ sort by last modification date (this is the default โจ)-p,--percentโ sort by percent of code ๐-g,--gitโ showgit blameoutput instead of a summary--jsonโ output blame as machine-readable json records ๐ค
๐ spack bootstrap
manage the bootstrap configuration โ like, the whole "how do we even get our tools" situation ๐ซถ
spack bootstrap [-h] SUBCOMMAND ...Optional arguments
-h,--helpโ show this help message and exit
Subcommands
bootstrap nowbootstrap statusbootstrap enablebootstrap disablebootstrap resetbootstrap rootbootstrap listbootstrap addbootstrap removebootstrap mirror
โญ spack bootstrap now
spack bootstrap now [-h] [--dev]Optional arguments
-h,--helpโ show this help message and exit--devโ bootstrap the dev dependencies too ๐ค
๐ spack bootstrap status
spack bootstrap status [-h] [--optional] [--dev]Optional arguments
-h,--helpโ show this help message and exit--optionalโ show the status of the rarely used optional dependencies ๐--devโ show the status of the dependencies needed to develop Spack ๐ป
๐ฆ spack bootstrap enable
spack bootstrap enable [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT] [name]Positional arguments
nameโ the name of the source to be enabled โจ
Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ the configuration scope to read/modify ๐
๐ธ spack bootstrap disable
spack bootstrap disable [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT] [name]Positional arguments
nameโ the name of the source to be disabled ๐
Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ the configuration scope to read/modify
๐ spack bootstrap reset
spack bootstrap reset [-hy]Optional arguments
-h,--helpโ show this help message and exit-y,--yes-to-allโ assume "yes" is the answer to every confirmation request ๐ (great for not being asked nine times, bestie)
๐ฅ spack bootstrap root
spack bootstrap root [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT] [path]Positional arguments
pathโ set the bootstrap directory to this value ๐
Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ the configuration scope to read/modify
๐ง spack bootstrap list
spack bootstrap list [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ the configuration scope to read/modify
๐ spack bootstrap add
spack bootstrap add [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT] [--trust]
name metadata_dirPositional arguments
nameโ the name of the new source of software โจmetadata_dirโ the directory where the metadata files live ๐๏ธ
Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ the configuration scope to read/modify--trustโ enable the source immediately upon addition, no extra steps ๐ซถ
๐ spack bootstrap remove
spack bootstrap remove [-h] namePositional arguments
nameโ the name of the source to be removed ๐
Optional arguments
-h,--helpโ show this help message and exit
๐ช spack bootstrap mirror
spack bootstrap mirror [-h] [--binary-packages] [--dev] DIRECTORYPositional arguments
DIRECTORYโ the root directory in which to create the mirror and metadata ๐ฆ
Optional arguments
-h,--helpโ show this help message and exit--binary-packagesโ download public binaries in the mirror--devโ download dev dependencies too ๐ค
๐ ๏ธ spack build-env
run a command inside a spec's install environment, or dump that environment to your screen or a file โ it's giving "let me stand in your shoes" energy ๐
spack build-env [-hU] [--clean] [--dirty] [--reuse] [--fresh-roots] [--deprecated] [--dump FILE] [--pickle FILE] ...More documentation โ yes there's literally a whole other doc for this one ๐
Positional arguments
spec [--] [cmd]...โ the specs of the package environment to emulate ๐
Optional arguments
-h,--helpโ show this help message and exit--cleanโ unset harmful variables in the build environment (this is the default โจ)--dirtyโ preserve the user environment in spack's build environment (danger! โ ๏ธ like, actually be careful, bestie)-U,--freshโ do not reuse installed deps; build the newest configuration ๐ฅ--reuseโ reuse installed packages/buildcaches when possible--fresh-roots,--reuse-depsโ concretize with fresh roots and reused dependencies--deprecatedโ let the concretizer select deprecated versions ๐--dumpFILE โ dump a source-able environment to FILE ๐๏ธ--pickleFILE โ dump a pickled source-able environment to FILE
๐ฆ spack buildcache
create, download and install binary packages โ the "why compile when you can grab a snack" command ๐
spack buildcache [-h] SUBCOMMAND ...More documentation โ yeah, again, separate doc, go look ๐ซถ
Optional arguments
-h,--helpโ show this help message and exit
Subcommands
buildcache pushbuildcache installbuildcache listbuildcache keysbuildcache previewbuildcache checkbuildcache downloadbuildcache get-buildcache-namebuildcache save-specfilebuildcache syncbuildcache update-index
๐ spack buildcache push
push those binaries, bestie ๐
spack buildcache push [-h] [-f] [--allow-root] [--unsigned | --signed | --key key] [--update-index]
[--spec-file SPEC_FILE] [--only {package,dependencies}] [--fail-fast] [--base-image BASE_IMAGE]
[--tag TAG] [--private] [-j JOBS]
mirror ...Positional arguments
mirrorโ the mirror name, path, or URL ๐specsโ one or more package specs ๐ฏ
Optional arguments
-h,--helpโ show this help message and exit-f,--forceโ overwrite the tarball if it already exists ๐ค--allow-root,-aโ allow install root string in binary files after RPATH substitution--unsigned,-uโ push unsigned buildcache tarballs--signedโ push signed buildcache tarballs โ๏ธ--key,-kkey โ the key for signing ๐--update-index,--rebuild-indexโ regenerate the buildcache index after building package(s)--spec-fileSPEC_FILE โ create a buildcache entry for the spec from a json or yaml file ๐๏ธ--only {package,dependencies}โ select the buildcache mode. The default is to build a cache for the package along with all its dependencies. Alternatively, one can decide to build a cache for only the package or only the dependencies ๐--fail-fastโ stop pushing on the first failure (default is best effort) ๐--base-imageBASE_IMAGE โ specify the base image for the buildcache ๐ผ๏ธ--tag,-tTAG โ when pushing to an OCI registry, tag an image containing all root specs and their runtime dependencies ๐ท๏ธ--privateโ for a private mirror, include non-redistributable packages ๐-j,--jobsJOBS โ explicitly set number of parallel jobs, no guessing bestie โก
spack buildcache install ๐พ
spack buildcache install [-hfmuo] ...Positional arguments
- specs โ one or more package specs, give me the goods ๐ฆ
Optional arguments
-h,--helpโ show this help message and exit, the classic escape hatch โจ-f,--forceโ overwrite install directory if it exists, no thoughts just vibes ๐ซฃ-m,--multipleโ allow all matching packages, not just one, greedy but iconic ๐-u,--unsignedโ install unsigned buildcache tarballs for testing (testing only, okurrr) ๐งช-o,--otherarchโ install specs from other architectures instead of default platform and OS ๐
spack buildcache list ๐
spack buildcache list [-hlLNva] ...Positional arguments
- specs โ one or more package specs ๐ธ
Optional arguments
-h,--helpโ show this help message and exit ๐-l,--longโ show dependency hashes as well as versions, the full receipt ๐งพ-L,--very-longโ show full dependency hashes as well as versions, very long, she's extra ๐-N,--namespacesโ show fully qualified package names ๐ท๏ธ-v,--variantsโ show variants in output (can be long โ fair warning bestie, it's giving scroll marathon) ๐ญ-a,--allarchโ list specs for all available architectures instead of default platform and OS ๐ฆ
spack buildcache keys ๐
spack buildcache keys [-hitf]Optional arguments
-h,--helpโ show this help message and exit โญ-i,--installโ install Keys pulled from mirror ๐-t,--trustโ trust all downloaded keys, trust fall fr fr ๐ซถ-f,--forceโ force new download of keys, no caching, just fresh ๐
spack buildcache preview ๐
spack buildcache preview [-h] ...Positional arguments
- installed_specs โ one or more installed package specs ๐
Optional arguments
-h,--helpโ show this help message and exit ๐
spack buildcache check โ
spack buildcache check [-h] [-m MIRROR_URL] [-o OUTPUT_FILE]
[--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT] [-s SPEC |
--spec-file SPEC_FILE]
...Positional arguments
- specs โ one or more package specs ๐
Optional arguments
-h,--helpโ show this help message and exit โจ-m,--mirror-urlMIRROR_URL โ override any configured mirrors with this mirror URL, we're going rogue ๐ช-o,--output-fileOUTPUT_FILE โ file where rebuild info should be written ๐--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope containing mirrors to check ๐-s,--specSPEC โ check single spec instead of release specs file ๐ฏ--spec-fileSPEC_FILE โ check single spec from json or yaml file instead of release specs file ๐
spack buildcache download โฌ๏ธ
spack buildcache download [-h] (-s SPEC | --spec-file SPEC_FILE) -p PATHOptional arguments
-h,--helpโ show this help message and exit ๐-s,--specSPEC โ download built tarball for spec from mirror ๐ฆ--spec-fileSPEC_FILE โ download built tarball for spec (from json or yaml file) from mirror ๐-p,--pathPATH โ path to directory where tarball should be downloaded ๐๏ธ
spack buildcache get-buildcache-name ๐ท๏ธ
spack buildcache get-buildcache-name [-h] (-s SPEC | --spec-file SPEC_FILE)Optional arguments
-h,--helpโ show this help message and exit ๐-s,--specSPEC โ spec string for which buildcache name is desired ๐ค--spec-fileSPEC_FILE โ path to spec json or yaml file for which buildcache name is desired ๐
spack buildcache save-specfile ๐พ
spack buildcache save-specfile [-h] (--root-spec ROOT_SPEC | --root-specfile ROOT_SPECFILE) -s SPECS
--specfile-dir SPECFILE_DIROptional arguments
-h,--helpโ show this help message and exit ๐ธ--root-specROOT_SPEC โ root spec of dependent spec ๐ณ--root-specfileROOT_SPECFILE โ path to json or yaml file containing root spec of dependent spec ๐-s,--specsSPECS โ list of dependent specs for which saved yaml is desired ๐--specfile-dirSPECFILE_DIR โ path to directory where spec yamls should be saved ๐๏ธ
spack buildcache sync ๐
spack buildcache sync [-h] [--manifest-glob MANIFEST_GLOB] [source mirror] [destination mirror]Positional arguments
- source mirror โ source mirror name, path, or URL ๐ซ
- destination mirror โ destination mirror name, path, or URL ๐ฌ
Optional arguments
-h,--helpโ show this help message and exit โจ--manifest-globMANIFEST_GLOB โ a quoted glob pattern identifying CI rebuild manifest files, keep it quoted bestie ๐ง
spack buildcache update-index ๐๏ธ
spack buildcache update-index [-hk] mirrorPositional arguments
- mirror โ destination mirror name, path, or URL ๐ช
Optional arguments
-h,--helpโ show this help message and exit ๐-k,--keysโ if provided, key index will be updated as well as package index ๐
spack cd ๐ช
cd to spack directories in the shell โ yes it literally teleports you, main character energy ๐ง
spack cd [-h] [-m | -r | -i | -p | -P | -s | -S | -c | -b | -e [name]] [--first] ...More documentation
๐ (go read it, it's right there)
Positional arguments
- spec โ package spec ๐
Optional arguments
-h,--helpโ show this help message and exit ๐-m,--module-dirโ spack python module directory ๐-r,--spack-rootโ spack installation root ๐ณ-i,--install-dirโ install prefix for spec (spec need not be installed) ๐-p,--package-dirโ directory enclosing a spec'spackage.pyfile ๐ฆ-P,--packagesโ top-level packages directory for Spack ๐-s,--stage-dirโ stage directory for a spec ๐ญ-S,--stagesโ top level stage directory ๐ญ-c,--source-dirโ source directory for a spec (requires it to be staged first) ๐-b,--build-dirโ build directory for a spec (requires it to be staged first) ๐จ-e,--env[name] โ location of the named or current environment ๐--firstโ use the first match if multiple packages match the spec ๐ฅ
spack change ๐ง
change an existing spec in an environment โ glow-up but for specs ๐
spack change [-ha] [-l LIST_NAME] [--match-spec MATCH_SPEC] ...Positional arguments
- specs โ one or more package specs ๐ธ
Optional arguments
-h,--helpโ show this help message and exit โจ-l,--list-nameLIST_NAME โ name of the list to remove specs from ๐--match-specMATCH_SPEC โ if name is ambiguous, supply a spec to match ๐ฏ-a,--allโ change all matching specs (allow changing more than one spec) ๐
spack checksum ๐งฎ
checksum available versions of a package โ the vibe check for tarballs ๐
spack checksum [-h] [--keep-stage] [--batch] [--latest] [--preferred] [--add-to-package | --verify] [-j JOBS]
package [versions ...]More documentation
๐ (it exists, go look)
Positional arguments
- package โ name or spec (e.g.
cmakeorcmake@3.18) ๐ฆ - versions โ checksum these specific versions (if omitted, Spack searches for remote versions) ๐
Optional arguments
-h,--helpโ show this help message and exit ๐--keep-stageโ don't clean up staging area when command completes ๐งนโ--batch,-bโ don't ask which versions to checksum, just go ๐--latest,-lโ checksum the latest available version โญ--preferred,-pโ checksum the known Spack preferred version ๐--add-to-package,-aโ add new versions to package โ--verifyโ verify known package checksums โ-j,--jobsJOBS โ explicitly set number of parallel jobs โก
spack ci ๐ค
manage continuous integration pipelines โ the robot besties that build your stuff ๐
spack ci [-h] SUBCOMMAND ...More documentation
๐ (yes, again, go read it)
Optional arguments
-h,--helpโ show this help message and exit ๐
Subcommands
ci generateci rebuild-indexci rebuildci reproduce-build
spack ci generate โ๏ธ
generate jobs file from a CI-aware spack file ๐๏ธ
if you want to report the results on CDash, you will need to set the SPACK_CDASH_AUTH_TOKEN before invoking this command. the value must be the CDash authorization token needed to create a build group and register all generated jobs under it ๐
no token, no CDash report โ that's just the rules bestie ๐
spack ci generate [-h] [--output-file OUTPUT_FILE] [--copy-to COPY_TO] [--optimize] [--dependencies]
[--buildcache-destination BUILDCACHE_DESTINATION] [--prune-dag | --no-prune-dag]
[--check-index-only] [--artifacts-root ARTIFACTS_ROOT]More documentation
๐ (the docs are RIGHT THERE, click them)
Optional arguments
-h,--helpโ show this help message and exit โจ--output-fileOUTPUT_FILE โ pathname for the generated gitlab ci yaml file ๐--copy-toCOPY_TO โ path to additional directory for job files ๐--optimizeโ (experimental) optimize the gitlab yaml file for size ๐งช--dependenciesโ (experimental) disable DAG scheduling (use 'plain' dependencies) ๐งช--buildcache-destinationBUILDCACHE_DESTINATION โ override the mirror configured in the environment ๐ช--prune-dagโ skip up-to-date specs โ๏ธ--no-prune-dagโ process up-to-date specs ๐--check-index-onlyโ only check spec state from buildcache indices ๐--artifacts-rootARTIFACTS_ROOT โ path to the root of the artifacts directory ๐๏ธ
spack ci rebuild-index ๐
rebuild the buildcache index for the remote mirror ๐ช
use the active, gitlab-enabled environment to rebuild the buildcache index for the associated mirror ๐
spack ci rebuild-index [-h]More documentation
๐ (you know the drill)
Optional arguments
-h,--helpโ show this help message and exit ๐
spack ci rebuild ๐จ
rebuild a spec if it is not on the remote mirror ๐
check a single spec against the remote mirror, and rebuild it from source if the mirror does not contain the hash ๐
spack ci rebuild [-ht] [--fail-fast]More documentation
๐ (docs go brr)
Optional arguments
-h,--helpโ shows this help message and dips โจ-t,--testsโ runs stand-alone tests after the build, bestie. build first, vibe check second ๐--fail-fastโ the second one stand-alone test fails, we STOP. no pushing through, no suffering in silence ๐ญ
spack ci reproduce-build
okay so this one literally generates instructions for reproducing the spec rebuild job ๐ซถ
the artifacts from the gitlab pipeline rebuild job URL you give it get used to derive instructions for reproducing the build locally. at home. on your machine. iconic ๐
spack ci reproduce-build [-h] [--runtime {docker,podman}] [--working-dir WORKING_DIR] [-s] [--gpg-file GPG_FILE |
--gpg-url GPG_URL]
job_urlMore documentation ๐
Positional arguments
job_urlโ URL of the GitLab job web page or artifact
Optional arguments
-h,--helpโ show this help message and exit--runtime {docker,podman}โ which container runtime to use.--working-dir WORKING_DIRโ where to unpack artifacts-s,--autostartโ run the docker reproducer automatically, no extra clicking ๐โโ๏ธ--gpg-file GPG_FILEโ path to public GPG key for validating binary cache installs--gpg-url GPG_URLโ URL to public GPG key for validating binary cache installs
spack clean
remove temporary build files and/or downloaded archives ๐งนโจ the literal spring cleaning of your Spack life
spack clean [-hsdfmpba] ...More documentation ๐
Positional arguments
specsโ one or more package specs
Optional arguments
-h,--helpโ show this help message and exit-s,--stageโ remove all temporary build stages (this one is the default btw)-d,--downloadsโ remove cached downloads-f,--failuresโ force removal of all install failure tracking markers-m,--misc-cacheโ remove long-lived caches, like the virtual package index-p,--python-cacheโ remove.pyc,.pyofiles and__pycache__folders ๐-b,--bootstrapโ remove software and configuration needed to bootstrap Spack-a,--allโ equivalent to-sdfmp(and likeโฆ it does not include--bootstrap, noting that for the girlies who assume "all" means all ๐)
spack clone
create a new installation of spack in another prefix โ cloning yourself, main character energy only ๐
spack clone [-h] [-r REMOTE] prefixPositional arguments
prefixโ name of prefix where we should install spack
Optional arguments
-h,--helpโ show this help message and exit-r,--remote REMOTEโ name of the remote to clone from
spack commands
list available spack commands. yes this is the command that lists commands. we love a self-aware queen ๐
spack commands [-h] [--update-completion] [-a] [--format {subcommands,rst,names,bash,fish}] [--header FILE]
[--update FILE]
...Positional arguments
rst_filesโ list of rst files to search for_cmd-spack-<cmd>cross-refs
Optional arguments
-h,--helpโ show this help message and exit--update-completionโ regenerate spack's tab completion scripts โก (tab tab tab, slay)-a,--aliasesโ include command aliases--format {subcommands,rst,names,bash,fish}โ format to be used to print the output (default:names)--header FILEโ prepend contents ofFILEto the output (useful for rst format)--update FILEโ write output to the specified file, if any command is newer
spack compiler
manage compilers. your compilers are your children and this is the group chat where you keep track of them ๐ซถ
spack compiler [-h] SUBCOMMAND ...Optional arguments
-h,--helpโ show this help message and exit
Subcommands
compiler findcompiler removecompiler listcompiler info
spack compiler find
spack compiler find [-h] [--mixed-toolchain | --no-mixed-toolchain]
[--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]
...More documentation ๐
Positional arguments
add_paths
Optional arguments
-h,--helpโ show this help message and exit--mixed-toolchainโ allow mixed toolchains (for example:clang,clang++,gfortran)--no-mixed-toolchainโ do not allow mixed toolchains (for example:clang,clang++,gfortran)--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to modify
spack compiler remove
spack compiler remove [-h] [-a] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]
compiler_specPositional arguments
compiler_spec
Optional arguments
-h,--helpโ show this help message and exit-a,--allโ remove ALL compilers that match spec. all of them. no survivors ๐--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to modify
spack compiler list
spack compiler list [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to read from
spack compiler info
spack compiler info [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]
compiler_specMore documentation ๐
Positional arguments
compiler_spec
Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to read from
spack compilers
list available compilers (yes, plural, it's the vibe-y older sibling of spack compiler list) ๐
spack compilers [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]More documentation ๐
Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to read/modify
spack concretize
concretize an environment and write a lockfile ๐โจ this is the moment where all your little spec dreams become one actual plan. the way that this command decides your whole buildโฆ it's giving destiny
spack concretize [-hfqU] [--test {root,all}] [--reuse] [--fresh-roots] [--deprecated] [-j JOBS]Optional arguments
-h,--helpโ show this help message and exit-f,--forceโ re-concretize even if already concretized--test {root,all}โ concretize with test dependencies of only root packages, or of all packages-q,--quietโ don't print concretized specs ๐คซ-U,--freshโ do not reuse installed deps; build newest configuration--reuseโ reuse installed packages/buildcaches when possible--fresh-roots,--reuse-depsโ concretize with fresh roots and reused dependencies--deprecatedโ allow the concretizer to select deprecated versions โ ๏ธ (deprecated for a reason bestie, but it's your call)-j,--jobs JOBSโ explicitly set number of parallel jobs
spack config
get and set configuration options โ the settings app of the Spack universe ๐
spack config [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT] SUBCOMMAND ...Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to read/modify
Subcommands
config getconfig blameconfig editconfig listconfig addconfig changeconfig prefer-upstreamconfig removeconfig updateconfig revert
spack config get
spack config get [-h] [section]More documentation ๐
Positional arguments
sectionโ configuration section to print
Optional arguments
-h,--helpโ show this help message and exit
spack config blame
spack config blame [-h] [section]More documentation ๐
Positional arguments
sectionโ configuration section to print
Optional arguments
-h,--helpโ show this help message and exit
spack config edit
spack config edit [-h] [--print-file] [section]Positional arguments
sectionโ configuration section to edit
Optional arguments
-h,--helpโ show this help message and exit--print-fileโ print the file name that would be edited (peek before you edit โจ)
spack config list
spack config list [-h]Optional arguments
-h,--helpโ show this help message and exit
spack config add
spack config add [-h] [-f FILE] [path]Positional arguments
pathโ colon-separated path to config that should be added, e.g.'config:default:true'
Optional arguments
-h,--helpโ show this help message and exit-f,--file FILEโ file from which to set all config values
spack config change
spack config change [-h] [--match-spec MATCH_SPEC] pathPositional arguments
pathโ colon-separated path to config section with specs
Optional arguments
-h,--helpโ show this help message and exit--match-spec MATCH_SPECโ only change constraints that match this
spack config prefer-upstream
spack config prefer-upstream [-h] [--local]Optional arguments
-h,--helpโ show this help message and exit--localโ set packages preferences based on local installs, rather than upstream
spack config remove
spack config remove [-h] pathPositional arguments
pathโ colon-separated path to config that should be removed, e.g.'config:default:true'
Optional arguments
-h,--helpโ show this help message and exit
spack config update
spack config update [-hy] sectionPositional arguments
sectionโ section to update
Optional arguments
-h,--helpโ show this help message and exit-y,--yes-to-allโ assume"yes"is the answer to every confirmation request ๐ (careful, she will not ask twice)
spack config revert
spack config revert [-hy] sectionPositional arguments
sectionโ section to update
Last couple of options hanging off the command above ๐ (they're its "Optional arguments", promise):
-h,--helpโ shows that help message and dips โ๏ธ-y,--yes-to-allโ literally assumes "yes" is the answer to every single confirmation request. every one. bestie said consent-to-everything ๐
๐ณ spack containerize
It whips up recipes so you can build images for the different container runtimes out there. Container-flavored Spack, iykyk ๐ซโจ
spack containerize [-h] [--list-os] [--last-stage {bootstrap,build,final}]Optional arguments
-h,--helpโ show this help message and exit--list-osโ lists all the OS you can use in the bootstrap phase, then bounces ๐--last-stage {bootstrap,build,final}โ tells it which stage is the last one in the container recipe ๐
๐ฑ spack create
The origin story command โ makes you a brand-new package file ๐
spack create [-hfb] [--keep-stage] [-n NAME] [-t TEMPLATE] [-r REPO] [-N NAMESPACE] [--skip-editor] [url]More documentation exists for this one, so go peek if you're nosy ๐
Positional arguments
urlโ the url of the package archive ๐
Optional arguments
-h,--helpโ show this help message and exit--keep-stageโ does not clean up the staging area when the command finishes (so you can snoop around in there) ๐-n,--name NAMEโ the name of the package you're creating-t,--template TEMPLATEโ which build system template to use-r,--repo REPOโ path to the repository where this package should be born ๐ผ-N,--namespace NAMESPACEโ pick a namespace for the package-f,--forceโ overwrites any existing package file with the same name. it will truly just bulldoze it ๐--skip-editorโ skips the edit session for the package, say for automation ๐ค-b,--batchโ doesn't ask you which versions to checksum, just goes ๐โโ๏ธ
๐ spack debug
Your troubleshooting toolbox โ debugging commands for when things are giving chaos ๐ญ๐
spack debug [-h] SUBCOMMAND ...Optional arguments
-h,--helpโ show this help message and exit
Subcommands
debug create-db-tarballdebug report
๐ฆ spack debug create-db-tarball
spack debug create-db-tarball [-h]Optional arguments
-h,--helpโ show this help message and exit
๐ spack debug report
spack debug report [-h]Optional arguments
-h,--helpโ show this help message and exit
๐งน spack deconcretize
Un-concretizes stuff โ it removes specs from the concretized lockfile of an environment. the ultimate "never mind, take it back" ๐
spack deconcretize [-hya] [--root] ...Positional arguments
specsโ one or more package specs
Optional arguments
-h,--helpโ show this help message and exit--rootโ deconcretizes only specific environment roots, not the whole vibe-y,--yes-to-allโ assumes "yes" is the answer to every confirmation request ๐-a,--allโ deconcretizes ALL specs that match each supplied spec. all of them. no survivors ๐ฅ
๐ธ๏ธ spack dependencies
Shows you the dependencies of a package โ who it needs to survive fr fr
spack dependencies [-hitV] [--deptype DEPTYPE] ...Positional arguments
specโ package spec
Optional arguments
-h,--helpโ show this help message and exit-i,--installedโ lists installed dependencies of an installed spec instead of the possible dependencies of a package-t,--transitiveโ shows all transitive dependencies (the whole friend-of-a-friend chain)--deptype DEPTYPEโ comma-separated list of deptypes to traverse (default=build,link,run,test)-V,--no-expand-virtualsโ do not expand virtual dependencies
๐ช spack dependents
The reverse angle โ shows packages that depend on another one ๐ซถ
spack dependents [-hit] ...Positional arguments
specโ package spec
Optional arguments
-h,--helpโ show this help message and exit-i,--installedโ lists installed dependents of an installed spec instead of the possible dependents of a package-t,--transitiveโ shows all transitive dependents
๐ spack deprecate
Replaces one package with another via symlinks ๐
spack deprecate [-hy] [-d | -D] [-i | -I] [-l {soft,hard}] ...Positional arguments
specsโ the spec to deprecate, and the spec to use as deprecator
Optional arguments
-h,--helpโ show this help message and exit-y,--yes-to-allโ assumes "yes" is the answer to every confirmation request-d,--dependenciesโ deprecate dependencies (this is the default)-D,--no-dependenciesโ do not deprecate dependencies-i,--install-deprecatorโ concretizes and installs the deprecator spec-I,--no-install-deprecatorโ the deprecator spec must already be installed (this is the default)-l,--link-type {soft,hard}โ type of filesystem link to use for deprecation (defaultsoft)
๐ ๏ธ spack dev-build
Developer build time โ builds from code sitting in your current working directory ๐ปโจ
spack dev-build [-h] [-j JOBS] [-n] [-d SOURCE_PATH] [-i] [--keep-prefix] [--skip-patch] [-q] [--drop-in SHELL]
[--test {root,all}] [-b BEFORE | -u UNTIL] [--clean | --dirty] [-U] [--reuse] [--fresh-roots]
[--deprecated]
...Positional arguments
specโ package spec
Optional arguments
-h,--helpโ show this help message and exit-j,--jobs JOBSโ explicitly set the number of parallel jobs ๐ฅ-n,--no-checksumโ do not use checksums to verify downloaded files (unsafe โ ๏ธ)-d,--source-path SOURCE_PATHโ path to the source directory (defaults to the current directory)-i,--ignore-dependenciesโ do not try to install dependencies of requested packages--keep-prefixโ do not remove the install prefix if installation fails (keeps the crime scene intact ๐ต๏ธโโ๏ธ)--skip-patchโ skips patching for the developer build-q,--quietโ do not display verbose build output while installing ๐คซ--drop-in SHELLโ drops you into a build environment in a new shell, e.g.bash๐--test {root,all}โ run tests on only root packages, or on all packages-b,--before BEFOREโ phase to stop before when installing (defaultNone)-u,--until UNTILโ phase to stop after when installing (defaultNone)--cleanโ unsets harmful variables in the build environment (this is the default) ๐งผ--dirtyโ preserves the user environment in spack's build environment (danger! ๐)-U,--freshโ do not reuse installed deps; build the newest configuration--reuseโ reuse installed packages/buildcaches when possible โป๏ธ--fresh-roots,--reuse-depsโ concretize with fresh roots and reused dependencies--deprecatedโ lets the concretizer select deprecated versions ๐
๐ฉโ๐ป spack develop
Adds a spec to an environment's dev-build information โ your dev era starts here ๐ฌ
spack develop [-h] [-p PATH] [-b BUILD_DIRECTORY] [--no-clone | --clone] [-f FORCE] ...Positional arguments
specโ package spec
Optional arguments
-h,--helpโ show this help message and exit-p,--path PATHโ source location of the package-b,--build-directory BUILD_DIRECTORYโ build directory for the package--no-cloneโ do not clone, the package already exists at the source path--cloneโ clone the package even if the path already exists-f,--force FORCEโ removes any files or directories that block cloning the source code. ruthless. iconic. ๐ฅ
๐ spack diff
Compares two specs side by side, like a vibe check but for packages ๐
spack diff [-h] [--json] [--first] [-a ATTRIBUTE] [--ignore IGNORE] ...Positional arguments
specsโ one or more package specs
Optional arguments
-h,--helpโ show this help message and exit--jsonโ dumps json output instead of pretty printing--firstโ loads the first match if multiple packages match the spec-a,--attribute ATTRIBUTEโ selects the attributes to show (defaults to all)--ignore IGNOREโ omit diffs related to these dependencies
๐ spack docs
Opens the Spack documentation in a web browser โ one click and you're reading, bestie ๐๐
spack docs [-h]Optional arguments
-h,--helpโ show this help message and exit
โ๏ธ spack edit
Opens package files in $EDITOR โ yes, <em>that</em> $EDITOR ๐
spack edit [-h] [-b | -c | -d | -t | -m | -r REPO | -N NAMESPACE] [package]There's more documentation for this one too, go look ๐
Positional arguments
packageโ package name
Optional arguments
-h,--helpโ show this help message and exit-b,--build-systemโ edits the build system with the supplied name-c,--commandโ edits the command with the supplied name-d,--docsโ edits the docs with the supplied name-t,--testโ edits the test with the supplied name-m,--moduleโ edits the main spack module with the supplied name-r,--repo REPOโ path to the repo to edit the package in-N,--namespace NAMESPACEโ namespace of the package to edit
๐ spack env
Manages virtual environments โ the whole little ecosystem ๐ฆ
spack env [-h] SUBCOMMAND ...Optional arguments
-h,--helpโ show this help message and exit
Subcommands
env activateenv deactivateenv createenv removeenv renameenv listenv statusenv loadsenv viewenv updateenv revertenv depfile
๐ช spack env activate
spack env activate [-h] [--sh | --csh | --fish | --bat | --pwsh] [--with-view name | --without-view] [-p] [--temp]
[--create] [--envfile [ENVFILE]] [--keep-relative] [-d]
[env]Positional arguments
envโ name of the managed environment, or the directory of the anonymous env (when you're using--dir/-d) to activate
Optional arguments
-h,--helpโ show this help message and exit--shโ printsshcommands to activate the environment--cshโ printscshcommands to activate the environment--fishโ printsfishcommands to activate the environment ๐--batโ printsbatcommands to activate the environment--pwshโ prints powershell commands to activate the environment--with-view,-vname โ set runtime environment variables for a specific view--without-view,-Vโ do not set runtime environment variables for any view-p,--promptโ decorates the command line prompt when activating โจ--tempโ creates and activates an environment in a temporary directory ๐๏ธ--createโ creates and activates the environment if it doesn't exist
Heads up, there are still more options coming in the next bit โ--envfile [ENVFILE],--keep-relativeand-dare in the synopsis, so don't think I forgot, we're just mid-roll ๐
--envfileENVFILE โ okay so this gives you either a lockfile (and bestie, it must end with.jsonor.lock) or a manifest file ๐--keep-relativeโ when you're initializing fromenvfile, this copies relative develop paths verbatim into the new environment. Verbatim!! No rewriting, no vibes-based cleanup ๐-d,--dirโ activates the environment based on the directory you supplied, slay ๐
spack env deactivate ๐
spack env deactivate [-h] [--sh | --csh | --fish | --bat | --pwsh]Optional arguments โจ
-h,--helpโ show this help message and exit (the classic, the icon, the moment ๐)--shโ printshcommands to deactivate the environment--cshโ printcshcommands to deactivate the environment--fishโ printfishcommands to activate the environment--batโ printbatcommands to activate the environment--pwshโ printpwshcommands to activate the environment
ngl the last three say "activate" and this is the deactivate command ๐ญ that's what the page says, I'm just the messenger, no cap ๐
spack env create ๐ฃ
spack env create [-h] [-d] [--keep-relative] [--without-view | --with-view WITH_VIEW]
[--include-concrete INCLUDE_CONCRETE]
env [envfile]Positional arguments ๐ซถ
envโ name or directory of environmentenvfileโ either a lockfile (which, again, must end with.jsonor.lock) or a manifest file
Optional arguments โจ
-h,--helpโ show this help message and exit-d,--dirโ create an environment in a specific directory--keep-relativeโ copy relative develop paths verbatim into the new environment when initializing fromenvfile--without-viewโ do not maintain a view for this environment (no view, no problem ๐)--with-viewWITH_VIEW โ specify that this environment should maintain a view at the specified path (by default the view is maintained in the environment directory) โจ--include-concreteINCLUDE_CONCRETE โ name of old environment to copy specs from
spack env remove ๐๏ธ
spack env remove [-hyf] env [env ...]Positional arguments
envโ environment(s) to remove (yes, plural, you can nuke several at once ๐ฅ)
Optional arguments
-h,--helpโ show this help message and exit-y,--yes-to-allโ assume "yes" is the answer to every confirmation request. Living dangerously, I respect it ๐-f,--forceโ remove the environment even if it is included in another environment
removal is permanent, bestie โ this is not a drill, there's no undo button ๐ธ
spack env rename โ๏ธ
spack env rename [-hdf] from toPositional arguments
fromโ name (or path) of existing environmenttoโ new name (or path) for existing environment
Optional arguments
-h,--helpโ show this help message and exit-d,--dirโ the specified arguments correspond to directory paths-f,--forceโ allow overwriting of an existing environment
without-fit won't clobber an env that already exists. with-f? it absolutely will ๐ซถ
spack env list ๐
spack env list [-h]Optional arguments
-h,--helpโ show this help message and exit
that's it. that's the whole command. a little minimalist king ๐
spack env status ๐ฉบ
spack env status [-h]Optional arguments
-h,--helpโ show this help message and exit
vibe check for your environment, literally ๐
spack env loads ๐ฆ
spack env loads [-hr] [-n MODULE_SET_NAME] [-m {tcl,lmod}] [--input-only] [-p PREFIX] [-x EXCLUDE]Optional arguments โจ
-h,--helpโ show this help message and exit-n,--module-set-nameMODULE_SET_NAME โ module set for which to generate load operations-m,--module-type{tcl,lmod}โ type of module system to generate loads for--input-onlyโ generate input for themodulecommand (instead of a shell script)-p,--prefixPREFIX โ prepend to module names when issuingmodule loadcommands-x,--excludeEXCLUDE โ exclude package from output; may be specified multiple times (stack 'em up ๐ฅ)-r,--dependenciesโ recursively traverse spec dependencies
spack env view ๐
spack env view [-h] {regenerate,enable,disable} [view_path]Positional arguments
{regenerate,enable,disable}โ action to take for the environment's viewview_pathโ when enabling a view, optionally set the path manually
Optional arguments
-h,--helpโ show this help message and exit
three actions, one little command. iconic behavior tbh ๐
spack env update ๐
spack env update [-hy] envPositional arguments
envโ name or directory of the environment to activate
Optional arguments
-h,--helpโ show this help message and exit-y,--yes-to-allโ assume "yes" is the answer to every confirmation request
spack env revert โช
spack env revert [-hy] envPositional arguments
envโ name or directory of the environment to activate
Optional arguments
-h,--helpโ show this help message and exit-y,--yes-to-allโ assume "yes" is the answer to every confirmation request
yep,updateandreverthave literally the same positional-arg description. take it up with the page, not me ๐ญ
spack env depfile ๐งพ
spack env depfile [-h] [--make-prefix TARGET] [--make-disable-jobserver]
[--use-buildcache [{auto,only,never},][package:{auto,only,never},][dependencies:{auto,only,never}]]
[-o FILE] [-G {make}]
...Positional arguments
specsโ generate a depfile only for matching specs in the environment
Optional arguments โจ
-h,--helpโ show this help message and exit--make-prefix,--make-target-prefixTARGET โ prefix Makefile targets (and variables) with<TARGET>/<name>--make-disable-jobserverโ disable POSIX jobserver support--use-buildcache[{auto,only,never},][package:{auto,only,never},][dependencies:{auto,only,never}]โ when usingonly, redundant build dependencies are pruned from the DAG-o,--outputFILE โ write the depfile to FILE rather than to stdout-G,--generator{make}โ specify the depfile type
spack extensions ๐งฉ
list extensions for package
spack extensions [-hlLdp] [-s {packages,installed,all}] ...Positional arguments
extendableโ spec of package to list extensions for
Optional arguments
-h,--helpโ show this help message and exit-l,--longโ show dependency hashes as well as versions-L,--very-longโ show full dependency hashes as well as versions (go off, maximal detail ๐ )-d,--depsโ output dependencies along with found specs-p,--pathsโ show paths to package install directories-s,--show{packages,installed,all}โ show only part of output
spack external ๐
manage external packages in Spack configuration
spack external [-h] SUBCOMMAND ...Optional arguments
-h,--helpโ show this help message and exit
Subcommands ๐ซถ
external findexternal listexternal read-cray-manifest
spack external find ๐
spack external find [-h] [--not-buildable] [--exclude EXCLUDE] [-p PATH]
[--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT] [--all] [-t TAG]
[-j JOBS]
...More documentation ๐
Positional arguments
packages
Optional arguments โจ
-h,--helpโ show this help message and exit--not-buildableโ packages with detected externals won't be built with Spack--excludeEXCLUDE โ packages to exclude from search-p,--pathPATH โ one or more alternative search paths for finding externals--scope{defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to modify--allโ search for all packages that Spack knows about (all of them. every single one. ๐ค)-t,--tagTAG โ filter a package query by tag (multiple use allowed)-j,--jobsJOBS โ explicitly set number of parallel jobs
spack external list ๐
spack external list [-h]Optional arguments
-h,--helpโ show this help message and exit
spack external read-cray-manifest ๐๏ธ
spack external read-cray-manifest [-h] [--file FILE] [--directory DIRECTORY] [--ignore-default-dir] [--dry-run]
[--fail-on-error]Optional arguments
-h,--helpโ show this help message and exit--fileFILE โ specify a location other than the default--directoryDIRECTORY โ specify a directory storing a group of manifest files--ignore-default-dirโ ignore the default directory of manifest files--dry-runโ don't modify DB with files that are read (peek only, no touchy ๐)--fail-on-errorโ if a manifest file cannot be parsed, fail and report the full stack trace
without --fail-on-error a bad manifest file just... doesn't blow up the whole run. with it, you get the full stack trace and the whole drama ๐ฅ
spack fetch ๐ฅ
fetch archives for packages
spack fetch [-hnmDU] [--reuse] [--fresh-roots] [--deprecated] ...More documentation ๐
Positional arguments
specsโ one or more package specs
Optional arguments โจ
-h,--helpโ show this help message and exit-n,--no-checksumโ do not use checksums to verify downloaded files (unsafe โ ๏ธ)
bestie I am begging you โ this one is literally flagged unsafe in the page. checksums exist for a reason ๐ญ๐
-m,--missingโ fetch only missing (not yet installed) dependencies-D,--dependenciesโ also fetch all dependencies-U,--freshโ do not reuse installed deps; build newest configuration--reuseโ reuse installed packages/buildcaches when possible--fresh-roots,--reuse-depsโ concretize with fresh roots and reused dependencies--deprecatedโ allow concretizer to select deprecated versions
spack find ๐
list and search installed packages
spack find [-h] [--format FORMAT | -H | --json] [-d] [-p] [--groups] [--no-groups] [-l] [-L] [-t TAG] [-N] [-r] [-c]
[-f] [--show-full-compiler] [-x | -X] [-u] [-m] [-v] [--loaded] [-M] [--deprecated] [--only-deprecated]
[--install-tree INSTALL_TREE] [--start-date START_DATE] [--end-date END_DATE]
...More documentation ๐
Positional arguments
installed_specsโ constraint to select a subset of installed packages
Optional arguments โจ
-h,--helpโ show this help message and exit--formatFORMAT โ output specs with the specified format string-H,--hashesโ same as--format {/hash}; use withxargsor$()๐--jsonโ output specs as machine-readable json records-d,--depsโ output dependencies along with found specs-p,--pathsโ shows you the actual paths to where the packages are installed ๐ like, the receipts--groupsโ displays specs in arch/compiler groups (this is on by default, bestie)--no-groupsโ nope, don't group the specs by arch/compiler-l,--longโ shows dependency hashes as well as versions ๐-L,--very-longโ shows the full dependency hashes as well as versions, go off-t,--tagTAG โ filters a package query by tag (you can use this multiple times, no cap)-N,--namespacesโ shows fully qualified package names โจ-r,--only-rootsโ doesn't show the full list of installed specs in an environment-c,--show-concretizedโ shows concretized specs in an environment-f,--show-flagsโ shows the spec's compiler flags--show-full-compilerโ shows full compiler specs-x,--explicitโ shows only the specs that were installed explicitly, main character specs only-X,--implicitโ shows only the specs that got installed as dependencies-u,--unknownโ shows only the specs Spack doesn't have a package for ๐ญ-m,--missingโ shows missing dependencies as well as installed specs-v,--variantsโ shows variants in the output (heads up: this can get long, fr fr)--loadedโ shows only the packages loaded in the user environment-M,--only-missingโ shows only the missing dependencies--deprecatedโ shows deprecated packages as well as installed specs--only-deprecatedโ shows only the deprecated packages--install-treeINSTALL_TREE โ the install trees to query:'all'(default),'local','upstream', an upstream name, or a path--start-dateSTART_DATE โ earliest date of installation[YYYY-MM-DD]--end-dateEND_DATE โ latest date of installation[YYYY-MM-DD]
๐๏ธ spack gc
okay so this one removes specs that are now no longer needed โ it's giving cleanup era ๐งนโจ
spack gc [-hEby] [-e ENV]Optional arguments
-h,--helpโ show this help message and exit-E,--except-any-environmentโ remove everything unless it's needed by an environment-e,--except-environmentENV โ remove everything unless it's needed by the specified environment-b,--keep-build-dependenciesโ do not remove installed build-only dependencies of roots-y,--yes-to-allโ assume"yes"is the answer to every confirmation request ๐
๐ spack gpg
handles GPG actions for spack โ keys, signatures, the whole vibe ๐
spack gpg [-h] SUBCOMMAND ...More documentation
Optional arguments
-h,--helpโ show this help message and exit
Subcommands
gpg verifygpg trustgpg untrustgpg signgpg creategpg listgpg initgpg exportgpg publish
โ spack gpg verify
spack gpg verify [-h] ... [signature]Positional arguments
- installed_spec โ installed package spec
- signature โ the signature file
Optional arguments
-h,--helpโ show this help message and exit
๐ค spack gpg trust
spack gpg trust [-h] keyfilePositional arguments
- keyfile โ add a key to the trust store
Optional arguments
-h,--helpโ show this help message and exit
๐ spack gpg untrust
spack gpg untrust [-h] [--signing] keys [keys ...]Positional arguments
- keys โ remove keys from the trust store (it's over, sorry babe)
Optional arguments
-h,--helpโ show this help message and exit--signingโ allow untrusting signing keys
โ๏ธ spack gpg sign
spack gpg sign [-h] [--output DEST] [--key KEY] [--clearsign] ...Positional arguments
- installed_spec โ installed package spec
Optional arguments
-h,--helpโ show this help message and exit--outputDEST โ the directory to place signatures--keyKEY โ the key to use for signing--clearsignโ if specified, create a clearsign signature
๐ spack gpg create
spack gpg create [-h] [--comment COMMENT] [--expires EXPIRATION] [--export DEST] [--export-secret DEST] name emailPositional arguments
- name โ the name to use for the new key
- email โ the email address to use for the new key
Optional arguments
-h,--helpโ show this help message and exit--commentCOMMENT โ a description for the intended use of the key--expiresEXPIRATION โ when the key should expire--exportDEST โ export the public key to a file--export-secretDEST โ export the private key to a file ๐ฅ (careful with this one bestie)
๐ spack gpg list
spack gpg list [-h] [--trusted] [--signing]Optional arguments
-h,--helpโ show this help message and exit--trustedโ list trusted keys--signingโ list keys which may be used for signing
๐ฑ spack gpg init
spack gpg init [-h]Optional arguments
-h,--helpโ show this help message and exit--fromDIR
๐ค spack gpg export
spack gpg export [-h] [--secret] location [keys ...]Positional arguments
- location โ where to export keys
- keys โ the keys to export (all public keys if unspecified)
Optional arguments
-h,--helpโ show this help message and exit--secretโ export secret keys
๐ spack gpg publish
spack gpg publish [-h] (-d directory | -m mirror-name | --mirror-url mirror-url) [--rebuild-index] [keys ...]Positional arguments
- keys โ keys to publish (all public keys if unspecified)
Optional arguments
-h,--helpโ show this help message and exit-d,--directorydirectory โ local directory where keys will be published-m,--mirror-namemirror-name โ name of the mirror where keys will be published--mirror-urlmirror-url โ URL of the mirror where keys will be published--rebuild-indexโ regenerate buildcache key index after publishing key(s)
๐ธ๏ธ spack graph
generates graphs of package dependency relationships โ genuinely one of the prettiest outputs in spack, obsessed ๐ฆ
spack graph [-hsci] [-a | -d] [--deptype DEPTYPE] ...More documentation
Positional arguments
- specs โ one or more package specs
Optional arguments
-h,--helpโ show this help message and exit-a,--asciiโ draw graph as ascii to stdout (this is the default)-d,--dotโ generate graph in dot format and print to stdout-s,--staticโ graph static (possible) deps, don't concretize (implies--dot)-c,--colorโ use different colors for different dependency types ๐-i,--installedโ graph specs from the DB--deptypeDEPTYPE โ comma-separated list of deptypes to traverse (default=build,link,run,test)
โ spack help
get help on spack and its commands โ when you're lost, she's got you ๐ซถ
spack help [-ha] [--spec] [help_command]More documentation
Positional arguments
- help_command โ command to get help on
Optional arguments
-h,--helpโ show this help message and exit-a,--allโ list all available commands and options--specโ help on the package specification syntax
๐ spack info
get detailed information on a particular package โ like a whole wikipedia page for one spec, ngl it's iconic ๐
spack info [-h] [-a] [--detectable] [--maintainers] [--no-dependencies] [--no-variants] [--no-versions] [--phases]
[--tags] [--tests] [--virtuals] [--variants-by-name]
packageMore documentation
Positional arguments
- package โ package name
Optional arguments
-h,--helpโ show this help message and exit-a,--allโ output all package information--detectableโ output information on external detection--maintainersโ output package maintainers (appreciate them ๐)--no-dependenciesโ do not output build, link, and run package dependencies--no-variantsโ do not output variants--no-versionsโ do not output versions--phasesโ output installation phases--tagsโ output package tags--testsโ output relevant build-time and stand-alone tests--virtualsโ output virtual packages--variants-by-nameโ list variants in strict name order; don't group by condition
๐ ๏ธ spack install
build and install packages โ the big one, the main event, the moment ๐
spack install [-h] [--only {package,dependencies}] [-u UNTIL] [-j JOBS] [--overwrite] [--fail-fast] [--keep-prefix]
[--keep-stage] [--dont-restage] [--use-cache | --no-cache | --cache-only |
--use-buildcache [{auto,only,never},][package:{auto,only,never},][dependencies:{auto,only,never}]]
[--include-build-deps] [--no-check-signature] [--show-log-on-error] [--source] [-n] [-v] [--fake]
[--only-concrete] [--add | --no-add] [-f SPEC_YAML_FILE] [--clean | --dirty] [--test {root,all}]
[--log-format {junit,cdash}] [--log-file LOG_FILE] [--help-cdash] [-y] [-U] [--reuse] [--fresh-roots]
[--deprecated]
...More documentation
Positional arguments
- spec โ package spec
Optional arguments
-h,--helpโ show this help message and exit--only {package,dependencies}โ select the mode of installation-u,--untilUNTIL โ phase to stop after when installing (defaultNone)-j,--jobsJOBS โ explicitly set number of parallel jobs--overwriteโ reinstall an existing spec, even if it has dependents--fail-fastโ stop all builds if any build fails (default is best effort)--keep-prefixโ don't remove the install prefix if installation fails--keep-stageโ don't remove the build stage if installation succeeds--dont-restageโ if a partial install is detected, don't delete prior state--use-cacheโ check for pre-built Spack packages in mirrors (default)--no-cacheโ do not check for pre-built Spack packages in mirrors--cache-onlyโ only install package from binary mirrors--use-buildcache [{auto,only,never},][package:{auto,only,never},][dependencies:{auto,only,never}]โ select the mode of buildcache for the'package'and'dependencies'--include-build-depsโ when you're installing straight from a build cache, this pulls in the build deps too ๐ซถ honestly a lifesaver for CI pipeline debugging, like actually--no-check-signatureโ skips checking the signatures on binary packages (and yes, it overrides whatever your mirror config says) ๐--show-log-on-errorโ if the build flops, it dumps the entire build log straight to stderr so you can actually see what went down ๐--sourceโ drops the source files into your prefix โจ-n,--no-checksumโ doesn't use checksums to verify the downloaded files. unsafe, bestie, I'm warning you ๐ญ-v,--verboseโ shows the full verbose build output while it's installing ๐--fakeโ does a fake install, purely for debug purposes ๐ง--only-concreteโ (with an environment) only installs specs that are already concretized, no fresh concretizing--addโ (with an environment) adds the spec to the environment as a root--no-addโ (with an environment) does not add the spec to the environment as a root-f,--file SPEC_YAML_FILEโ reads the specs you wanna install from.yamlfiles ๐--cleanโ unsets the harmful variables in the build environment (this is the default btw)--dirtyโ preserves your user environment inside Spack's build environment. danger! โ ๏ธ--test {root,all}โ runs tests on only the root packages, or on all packages--log-format {junit,cdash}โ the format used for the log files--log-file LOG_FILEโ the filename for the log file--help-cdashโ shows you the usage instructions for CDash reporting--cdash-upload-url CDASH_UPLOAD_URL--cdash-build CDASH_BUILD--cdash-site CDASH_SITE--cdash-track CDASH_TRACK--cdash-buildstamp CDASH_BUILDSTAMP-y,--yes-to-allโ assumes "yes" is the answer to literally every confirmation request, no questions asked ๐-U,--freshโ doesn't reuse installed deps; builds the newest configuration โจ--reuseโ reuses installed packages and buildcaches whenever it can--fresh-roots,--reuse-depsโ concretizes with fresh roots and reused dependencies ๐--deprecatedโ lets the concretizer pick deprecated versions (iykyk)
๐ชช spack license
Lists and checks the license headers on files in spack, no cap ๐งพ
spack license [-h] [--root ROOT] SUBCOMMAND ...Optional arguments
-h,--helpโ shows this help message and exits--root ROOTโ scans a different prefix for license issues ๐
Subcommands
license list-fileslicense verifylicense update-copyright-year
spack license list-files
spack license list-files [-h]Optional arguments
-h,--helpโ shows this help message and exits
spack license verify
spack license verify [-h]Optional arguments
-h,--helpโ shows this help message and exits
spack license update-copyright-year
spack license update-copyright-year [-h]Optional arguments
-h,--helpโ shows this help message and exits
๐ spack list
Lists and searches the available packages ๐
spack list [-hdv] [-r REPOS] [--format {name_only,version_json,html}] [-t TAG] [--count | --update FILE] ...More documentation ๐
Positional arguments
- filter โ optional case-insensitive glob patterns to filter the results
Optional arguments
-h,--helpโ shows this help message and exits-r,--repo,-N,--namespace REPOSโ only lists packages from the specified repo/namespace ๐๏ธ-d,--search-descriptionโ filtering will also search the description for a match--format {name_only,version_json,html}โ the format used to print the output [default:name_only]-v,--virtualsโ includes virtual packages in the list ๐-t,--tag TAGโ filters a package query by tag (you can use it multiple times, go off)--countโ displays the number of packages that would be listed--update FILEโ writes output to the specified file, if any package is newer
๐ฆ spack load
Adds a package to the user environment, bestie ๐
spack load [-h] [--sh | --csh | --fish | --bat | --pwsh] [--first] [--list] ...More documentation ๐
Positional arguments
- installed_specs โ a constraint to select a subset of installed packages
Optional arguments
-h,--helpโ shows this help message and exits--shโ printsshcommands to load the package--cshโ printscshcommands to load the package--fishโ printsfishcommands to load the package--batโ printsbatcommands to load the package--pwshโ printspwshcommands to load the package--firstโ loads the first match if multiple packages match the spec ๐--listโ shows loaded packages: same asspack find --loaded
๐ spack location
Prints out the locations of packages and spack directories ๐บ๏ธ
spack location [-h] [-m | -r | -i | -p | -P | -s | -S | -c | -b | -e [name]] [--first] ...More documentation ๐
Positional arguments
- spec โ package spec
Optional arguments
-h,--helpโ shows this help message and exits-m,--module-dirโ the spack python module directory-r,--spack-rootโ the spack installation root-i,--install-dirโ install prefix for the spec (the spec doesn't even need to be installed โจ)-p,--package-dirโ the directory enclosing a spec'spackage.pyfile-P,--packagesโ the top-level packages directory for Spack-s,--stage-dirโ the stage directory for a spec-S,--stagesโ the top level stage directory-c,--source-dirโ the source directory for a spec (requires it to be staged first ๐)-b,--build-dirโ the build directory for a spec (requires it to be staged first ๐)-e,--env [name]โ location of the named or current environment--firstโ uses the first match if multiple packages match the spec
๐ชต spack log-parse
Filters the errors and warnings out of build logs ๐งน
spack log-parse [-hp] [--show SHOW] [-c CONTEXT] [-w WIDTH] [-j JOBS] filePositional arguments
- file โ a log file containing build output, or
-for stdin
Optional arguments
-h,--helpโ shows this help message and exits--show SHOWโ a comma-separated list of what to show; options:errors,warnings-c,--context CONTEXTโ how many lines of context to show around the lines of interest-p,--profileโ prints out a profile of time spent in regexes during the parse โฑ๏ธ-w,--width WIDTHโ wrap width: auto-sizes to the terminal by default;0for no wrap-j,--jobs JOBSโ number of jobs to parse the log file (default:1for short logs,ncpusfor long logs)
๐ spack logs
Prints out the logs for packages ๐๏ธ
spack logs [-h] ...Positional arguments
- spec โ package spec
Optional arguments
-h,--helpโ shows this help message and exits
๐ ๏ธ spack maintainers
Gets info about the package maintainers, the unsung heroes fr fr ๐ซถ
spack maintainers [-ha] [--maintained | --unmaintained] [--by-user] ...Positional arguments
- package_or_user โ names of packages or users to get info for
Optional arguments
-h,--helpโ shows this help message and exits--maintainedโ shows the names of maintained packages--unmaintainedโ shows the names of unmaintained packages-a,--allโ shows maintainers for all packages--by-userโ shows packages for users instead of users for packages (flip it ๐)
๐ฟ spack make-installer
Generates a Windows installer ๐ช
spack make-installer [-h] (-v SPACK_VERSION | -s SPACK_SOURCE) [-g {SILENT,VERYSILENT}] output_dirPositional arguments
- output_dir โ output directory
Optional arguments
-h,--helpโ shows this help message and exits-v,--spack-version SPACK_VERSIONโ downloads the given spack version-s,--spack-source SPACK_SOURCEโ full path to the spack source-g,--git-installer-verbosity {SILENT,VERYSILENT}โ level of verbosity provided by the bundled git installer (default is fully verbose ๐)
๐ท๏ธ spack mark
Marks packages as explicitly or implicitly installed โ๏ธ
spack mark [-ha] (-e | -i) ...Positional arguments
- installed_specs โ one or more installed package specs
Optional arguments
-h,--helpโ shows this help message and exits-a,--allโ marks ALL installed packages that match each supplied spec-e,--explicitโ marks packages as explicitly installed ๐-i,--implicitโ marks packages as implicitly installed
๐ช spack mirror
Manages mirrors (source AND binary, iconic) ๐
spack mirror [-hn] SUBCOMMAND ...More documentation ๐
Optional arguments
-h,--helpโ shows this help message and exits-n,--no-checksumโ doesn't use checksums to verify downloaded files (unsafe, bestie ๐ญ)
Subcommands
mirror createmirror destroymirror addmirror removemirror set-urlmirror setmirror list
๐ช spack mirror create
spack mirror create [-h] [-d DIRECTORY] [-a] [-f FILE] [--exclude-file EXCLUDE_FILE] [--exclude-specs EXCLUDE_SPECS]
[--skip-unstable-versions] [-D] [-n VERSIONS_PER_SPEC] [--private] [-U] [--reuse] [--fresh-roots]
[--deprecated]
...More documentation ๐
Positional arguments
- specs โ one or more package specs
Optional arguments
-h,--helpโ shows this help message and exits-d,--directory DIRECTORYโ the directory in which to create the mirror-a,--allโ mirrors all versions of all packages in Spack, or all packages in the current environment if there's an active environment. Heads up: this requires significant time and space ๐-f,--file FILEโ a file with specs of packages to put in the mirror--exclude-file EXCLUDE_FILEโ specs which Spack should not try to add to a mirror (listed in a file, one per line)--exclude-specs EXCLUDE_SPECSโ specs which Spack should not try to add to a mirror (specified on the command line)--skip-unstable-versionsโ don't cache versions unless they identify a stable (unchanging) source code ๐-D,--dependenciesโ also fetches all the dependencies ๐ฆ-n,--versions-per-spec VERSIONS_PER_SPECโ the number of versions to fetch for each spec, choose'all'to retrieve all versions of each package--privateโ for a private mirror, include non-redistributable packages-U,--freshโ do not reuse installed deps; build newest configuration--reuseโ reuse installed packages/buildcaches when possible--fresh-roots,--reuse-depsโ concretize with fresh roots and reused dependencies--deprecatedโ allow concretizer to select deprecated versions
๐ช spack mirror destroy
spack mirror destroy [-h] (-m mirror_name | --mirror-url mirror_url)๐ Optional arguments
-h,--helpโ show this help message and exit-m,--mirror-name mirror_nameโ find mirror to destroy by name--mirror-url mirror_urlโ find mirror to destroy by url
โ spack mirror add
spack mirror add [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]
[--type {binary,source}] [--autopush] [--unsigned | --signed] [--s3-access-key-id S3_ACCESS_KEY_ID]
[--s3-access-key-secret S3_ACCESS_KEY_SECRET] [--s3-access-token S3_ACCESS_TOKEN]
[--s3-profile S3_PROFILE] [--s3-endpoint-url S3_ENDPOINT_URL] [--oci-username OCI_USERNAME]
[--oci-password OCI_PASSWORD]
mirror url๐ More documentation
๐ Positional arguments
mirrorโ mnemonic name for mirror (your cute lil nickname for it โจ)urlโ url of mirror directory from'spack mirror create'
๐ Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to modify--type {binary,source}โ specify the mirror type: for both binary and source use--type binary --type source(default)--autopushโ set mirror to push automatically after installation (it just does the thing, no reminder needed ๐ )--unsignedโ do not require signing and signature verification when pushing and installing from this build cache--signedโ require signing and signature verification when pushing and installing from this build cache--s3-access-key-id S3_ACCESS_KEY_IDโ ID string to use to connect to this S3 mirror--s3-access-key-secret S3_ACCESS_KEY_SECRETโ secret string to use to connect to this S3 mirror (secret bestie, keep it secret ๐)--s3-access-token S3_ACCESS_TOKENโ access token to use to connect to this S3 mirror--s3-profile S3_PROFILEโ S3 profile name to use to connect to this S3 mirror--s3-endpoint-url S3_ENDPOINT_URLโ endpoint URL to use to connect to this S3 mirror--oci-username OCI_USERNAMEโ username to use to connect to this OCI mirror--oci-password OCI_PASSWORDโ password to use to connect to this OCI mirror
โ spack mirror remove
spack mirror remove [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT] mirror๐ More documentation
๐ Positional arguments
mirrorโ mnemonic name for mirror
๐ Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to modify
๐ spack mirror set-url
spack mirror set-url [-h] [--push | --fetch]
[--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]
[--s3-access-key-id S3_ACCESS_KEY_ID] [--s3-access-key-secret S3_ACCESS_KEY_SECRET]
[--s3-access-token S3_ACCESS_TOKEN] [--s3-profile S3_PROFILE] [--s3-endpoint-url S3_ENDPOINT_URL]
[--oci-username OCI_USERNAME] [--oci-password OCI_PASSWORD]
mirror url๐ Positional arguments
mirrorโ mnemonic name for mirrorurlโ url of mirror directory from'spack mirror create'
๐ Optional arguments
-h,--helpโ show this help message and exit--pushโ set only the URL used for uploading <butterfly>๐ฆ</butterfly>--fetchโ set only the URL used for downloading--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to modify--s3-access-key-id S3_ACCESS_KEY_IDโ ID string to use to connect to this S3 mirror--s3-access-key-secret S3_ACCESS_KEY_SECRETโ secret string to use to connect to this S3 mirror--s3-access-token S3_ACCESS_TOKENโ access token to use to connect to this S3 mirror--s3-profile S3_PROFILEโ S3 profile name to use to connect to this S3 mirror--s3-endpoint-url S3_ENDPOINT_URLโ endpoint URL to use to connect to this S3 mirror--oci-username OCI_USERNAMEโ username to use to connect to this OCI mirror--oci-password OCI_PASSWORDโ password to use to connect to this OCI mirror
โ๏ธ spack mirror set
spack mirror set [-h] [--push | --fetch] [--type {binary,source}] [--url URL] [--autopush | --no-autopush]
[--unsigned | --signed]
[--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]
[--s3-access-key-id S3_ACCESS_KEY_ID] [--s3-access-key-secret S3_ACCESS_KEY_SECRET]
[--s3-access-token S3_ACCESS_TOKEN] [--s3-profile S3_PROFILE] [--s3-endpoint-url S3_ENDPOINT_URL]
[--oci-username OCI_USERNAME] [--oci-password OCI_PASSWORD]
mirror๐ Positional arguments
mirrorโ mnemonic name for mirror
๐ Optional arguments
-h,--helpโ show this help message and exit--pushโ modify just the push connection details--fetchโ modify just the fetch connection details--type {binary,source}โ specify the mirror type: for both binary and source use--type binary --type source--url URLโ url of mirror directory from'spack mirror create'--autopushโ set mirror to push automatically after installation--no-autopushโ set mirror to not push automatically after installation--unsignedโ do not require signing and signature verification when pushing and installing from this build cache--signedโ require signing and signature verification when pushing and installing from this build cache--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to modify--s3-access-key-id S3_ACCESS_KEY_IDโ ID string to use to connect to this S3 mirror--s3-access-key-secret S3_ACCESS_KEY_SECRETโ secret string to use to connect to this S3 mirror--s3-access-token S3_ACCESS_TOKENโ access token to use to connect to this S3 mirror--s3-profile S3_PROFILEโ S3 profile name to use to connect to this S3 mirror--s3-endpoint-url S3_ENDPOINT_URLโ endpoint URL to use to connect to this S3 mirror--oci-username OCI_USERNAMEโ username to use to connect to this OCI mirror--oci-password OCI_PASSWORDโ password to use to connect to this OCI mirror
๐ spack mirror list
spack mirror list [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]๐ More documentation
๐ Optional arguments
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ configuration scope to read from (note: read from, not modify โ we're just looking bestie ๐)
๐งฉ spack module
generate/manage module files
spack module [-h] SUBCOMMAND ...๐ Optional arguments
-h,--helpโ show this help message and exit
๐ Subcommands
module lmodmodule tcl
๐ spack module lmod
spack module lmod [-h] [-n MODULE_SET_NAME] SUBCOMMAND ...๐ Optional arguments
-h,--helpโ show this help message and exit-n,--name MODULE_SET_NAMEโ named module set to use from modules configuration
๐ Subcommands
module lmod refreshmodule lmod findmodule lmod rmmodule lmod loadsmodule lmod setdefault
๐ spack module lmod refresh
spack module lmod refresh [-hy] [--delete-tree] [--upstream-modules] ...๐ Positional arguments
installed_specsโ constraint to select a subset of installed packages
๐ Optional arguments
-h,--helpโ show this help message and exit--delete-treeโ delete the module file tree before refresh (yep, it nukes the tree first โ heads up fr fr ๐)--upstream-modulesโ generate modules for packages installed upstream-y,--yes-to-allโ assume "yes" is the answer to every confirmation request
๐ spack module lmod find
spack module lmod find [-hr] [--full-path] ...๐ Positional arguments
installed_specsโ constraint to select a subset of installed packages
๐ Optional arguments
-h,--helpโ show this help message and exit--full-pathโ display full path to module file-r,--dependenciesโ recursively traverse spec dependencies
๐๏ธ spack module lmod rm
spack module lmod rm [-hy] ...๐ Positional arguments
installed_specsโ constraint to select a subset of installed packages
๐ Optional arguments
-h,--helpโ show this help message and exit-y,--yes-to-allโ assume "yes" is the answer to every confirmation request
๐ spack module lmod loads
spack module lmod loads [-hr] [--input-only] [-p PREFIX] [-x EXCLUDE] ...๐ Positional arguments
installed_specsโ constraint to select a subset of installed packages
๐ Optional arguments
-h,--helpโ show this help message and exit--input-onlyโ generate input for module command (instead of a shell script)-p,--prefix PREFIXโ prepend to module names when issuing module load commands-x,--exclude EXCLUDEโ exclude package from output; may be specified multiple times (stack 'em, bestie โจ)-r,--dependenciesโ recursively traverse spec dependencies
โญ spack module lmod setdefault
spack module lmod setdefault [-h] ...๐ Positional arguments
installed_specsโ constraint to select a subset of installed packages
๐ Optional arguments
-h,--helpโ show this help message and exit
๐ง spack module tcl
spack module tcl [-h] [-n MODULE_SET_NAME] SUBCOMMAND ...๐ Optional arguments
-h,--helpโ shows this help message and exits, the classic ๐ซถ-n,--name MODULE_SET_NAMEโ the named module set you want to use, pulled from your modules config โจ
Subcommands ๐
module tcl refreshmodule tcl findmodule tcl rmmodule tcl loadsmodule tcl setdefault
๐ธ spack module tcl refresh ๐ธ
spack module tcl refresh [-hy] [--delete-tree] [--upstream-modules] ...Positional arguments ๐
installed_specsโ a constraint that selects a subset of your installed packages
Optional arguments โญ
-h,--helpโ show this help message and exit--delete-treeโ nukes the module file tree before refreshing, so it's a clean slate ๐งน--upstream-modulesโ generates modules for packages that were installed upstream-y,--yes-to-allโ assumes"yes"is the answer to every single confirmation request, no questions asked ๐
๐ฆ spack module tcl find ๐ฆ
spack module tcl find [-hr] [--full-path] ...Positional arguments ๐
installed_specsโ a constraint that selects a subset of your installed packages
Optional arguments ๐
-h,--helpโ show this help message and exit--full-pathโ displays the full path to the module file-r,--dependenciesโ recursively traverses spec dependencies, deep dive energy ๐ฅ
๐ spack module tcl rm ๐
spack module tcl rm [-hy] ...Positional arguments ๐
installed_specsโ a constraint that selects a subset of your installed packages
Optional arguments โจ
-h,--helpโ show this help message and exit-y,--yes-to-allโ assumes"yes"is the answer to every confirmation request ๐
๐ง spack module tcl loads ๐ง
spack module tcl loads [-hr] [--input-only] [-p PREFIX] [-x EXCLUDE] ...Positional arguments ๐ธ
installed_specsโ a constraint that selects a subset of your installed packages
Optional arguments ๐
-h,--helpโ show this help message and exit--input-onlyโ generates input for themodulecommand instead of a shell script-p,--prefix PREFIXโ prepends this to module names when it's issuingmodule loadcommands-x,--exclude EXCLUDEโ excludes a package from the output; you can specify it multiple times ๐-r,--dependenciesโ recursively traverses spec dependencies
โญ spack module tcl setdefault โญ
spack module tcl setdefault [-h] ...Positional arguments ๐
installed_specsโ a constraint that selects a subset of your installed packages
Optional arguments ๐ฆ
-h,--helpโ show this help message and exit
๐ฅ spack patch ๐ฅ
patch expanded archive sources in preparation for install
spack patch [-hnU] [--reuse] [--fresh-roots] [--deprecated] ...More documentation ๐
Positional arguments ๐ธ
specsโ one or more package specs
Optional arguments ๐
-h,--helpโ show this help message and exit-n,--no-checksumโ does not use checksums to verify downloaded files โ it's marked unsafe, so like, be careful bestie ๐-U,--freshโ does not reuse installed deps; builds the newest configuration instead--reuseโ reuses installed packages/buildcaches when possible โป๏ธ--fresh-roots,--reuse-depsโ concretizes with fresh roots and reused dependencies--deprecatedโ lets the concretizer select deprecated versions ๐
๐ spack pkg ๐
query packages associated with particular git revisions
spack pkg [-h] SUBCOMMAND ...Optional arguments โจ
-h,--helpโ show this help message and exit
Subcommands ๐
pkg addpkg listpkg diffpkg addedpkg changedpkg removedpkg greppkg sourcepkg hash
๐ spack pkg add ๐
spack pkg add [-h] package [package ...]Positional arguments ๐
packageโ one or more package names
Optional arguments ๐ฆ
-h,--helpโ show this help message and exit
๐ธ spack pkg list ๐ธ
spack pkg list [-h] [rev]Positional arguments โญ
revโ the revision to list packages for
Optional arguments ๐
-h,--helpโ show this help message and exit
๐ spack pkg diff ๐
spack pkg diff [-h] [rev1] [rev2]Positional arguments ๐
rev1โ the revision to compare againstrev2โ the revision to compare torev1(default isHEAD)
Optional arguments ๐
-h,--helpโ show this help message and exit
๐ง spack pkg added ๐ง
spack pkg added [-h] [rev1] [rev2]Positional arguments ๐
rev1โ the revision to compare againstrev2โ the revision to compare torev1(default isHEAD)
Optional arguments โจ
-h,--helpโ show this help message and exit
๐ฆ spack pkg changed ๐ฆ
spack pkg changed [-h] [-t TYPE] [rev1] [rev2]Positional arguments ๐ธ
rev1โ the revision to compare againstrev2โ the revision to compare torev1(default isHEAD)
Optional arguments ๐
-h,--helpโ show this help message and exit-t,--type TYPEโ the types of changes to show (A: added,R: removed,C: changed); default is'C'
๐ฅ spack pkg removed ๐ฅ
spack pkg removed [-h] [rev1] [rev2]Positional arguments ๐
rev1โ the revision to compare againstrev2โ the revision to compare torev1(default isHEAD)
Optional arguments โญ
-h,--helpโ show this help message and exit
๐ spack pkg grep ๐
spack pkg grep [--help] ...Positional arguments ๐
grep_argsโ arguments forgrep
Optional arguments ๐
--helpโ show this help message and exit
โจ spack pkg source โจ
spack pkg source [-hc] ...Positional arguments ๐ง
specโ package spec
Optional arguments ๐
-h,--helpโ show this help message and exit-c,--canonicalโ dumps the canonical source as used by the package hash
๐ spack pkg hash ๐
spack pkg hash [-h] ...Positional arguments ๐ธ
specโ package spec
Optional arguments ๐ฆ
-h,--helpโ show this help message and exit
๐ spack providers ๐
list packages that provide a particular virtual package
spack providers [-h] [virtual_package ...]More documentation ๐
Positional arguments โญ
virtual_packageโ find packages that provide this virtual package
Optional arguments ๐
-h,--helpโ show this help message and exit
๐ง spack pydoc ๐ง
run pydoc from within spack
spack pydoc [-h] entityPositional arguments ๐
entityโ run pydoc help on entity
Optional arguments โจ
-h,--helpโ show this help message and exit
๐ spack python ๐
launch an interpreter as spack would launch a command
spack python [-hVu] [-c PYTHON_COMMAND] [-i {python,ipython}] [-m MODULE] [--path] ...More documentation ๐
Positional arguments ๐
python_argsโ file to run plus arguments
Optional arguments ๐ธ
-h,--helpโ show this help message and exit-V,--versionโ prints the Python version number and exits-c PYTHON_COMMANDโ the command to execute-uโ for compatibility withxdist; do not use without adding-uto the interpreter-i {python,ipython}โ the python interpreter to use-m MODULEโ runs a library module as a script--pathโ shows the path to the python interpreter that spack uses
๐ spack reindex ๐
rebuild Spack's package database
spack reindex [-h]Optional arguments ๐ฆ
-h,--helpโ show this help message and exit
โญ spack remove โญ
remove specs from an environment
spack remove [-haf] [-l LIST_NAME] ...Positional arguments ๐
specsโ one or more package specs
Optional arguments ๐
-h,--helpโ show this help message and exit-a,--allโ removes all specs from the environment, i.e. clears it completely ๐งน-l,--list-name LIST_NAMEโ the name of the list to remove specs from-f,--forceโ removes the concretized spec (if there is one) immediately, no waiting ๐
๐ spack repo ๐
manage package source repositories
spack repo [-h] SUBCOMMAND ...More documentation ๐
Optional arguments โจ
-h,--helpโ show this help message and exit
Subcommands ๐ธ
repo createrepo listrepo addrepo remove
๐ฆ spack repo create ๐ฆ
spack repo create [-h] [-d SUBDIR] directory [namespace]Positional arguments ๐
directoryโ the directory to create the repo innamespaceโ the namespace that identifies packages in the repository (defaults to the directory name)
Optional arguments โญ
-h,--helpโ show this help message and exit-d,--subdirectory SUBDIRโ the subdirectory to store packages in the repository
๐ spack repo list ๐
spack repo list [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]Optional arguments ๐
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ the configuration scope to read from
๐ง spack repo add ๐ง
spack repo add [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT] pathPositional arguments ๐
pathโ the path to a Spack package repository directory
Optional arguments ๐ฅ
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ the configuration scope to modify
๐ spack repo remove ๐
spack repo remove [-h] [--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT]
namespace_or_pathPositional arguments ๐ธ
namespace_or_pathโ the namespace or path of a Spack package repository
Optional arguments โจ
-h,--helpโ show this help message and exit--scope {defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENTโ the configuration scope to modify
spack resource ๐ธ
okay so this one is literally just "show me the downloadable stuff" โ tarballs, repos, patches, etc. all the little resource goodies ๐
spack resource [-h] SUBCOMMAND ...More documentation โ yeah bestie there's more docs if you want the full tea ๐ซ
Optional arguments
-h,--helpโ prints this help message and dips โจ
Subcommands
resource listresource show
spack resource list ๐
spack resource list [-h] [--only-hashes]Optional arguments
-h,--helpโ show this help message and exit--only-hashesโ literally only print the sha256 hashes of the resources, nothing else, no vibes, just hashes ๐ฅ
spack resource show ๐ฆ
spack resource show [-h] hashPositional arguments
hashโ the hash of the resource you wanna see ๐
Optional arguments
-h,--helpโ show this help message and exit
spack restage ๐
revert checked out package source code โ like, undo the whole "I already unpacked this" situation, no cap
spack restage [-h] ...More documentation โ more docs exist, go off
Positional arguments
specsโ one or more package specs
Optional arguments
-h,--helpโ show this help message and exit
spack solve โญ
concretize a spec using an ASP solver. this is the main character of the whole operation tbh ๐
spack solve [-h] [--show SHOW] [-l] [-L] [-N] [-I | --no-install-status] [-y] [-j] [-c {nodes,edges,paths}] [-t]
[--timers] [--stats] [-U] [--reuse] [--fresh-roots] [--deprecated]
...Positional arguments
specsโ specs of packages
Optional arguments
-h,--helpโ show this help message and exit--show SHOWโ select which outputs you want to see-l,--longโ show dependency hashes as well as versions-L,--very-longโ show full dependency hashes as well as versions, like the deluxe version ๐-N,--namespacesโ show fully qualified package names-I,--install-statusโ show install status of packages--no-install-statusโ do NOT show install status annotations, we're keeping it lowkey-y,--yamlโ print concrete spec as yaml-j,--jsonโ print concrete spec as json-c,--cover {nodes,edges,paths}โ how extensively to traverse the DAG (default:nodes)-t,--typesโ show dependency types--timersโ print out timers for different solve phases, so you can see where the time went โฑ๏ธ--statsโ print out statistics from clingo-U,--freshโ do not reuse installed deps; build newest configuration--reuseโ reuse installed packages/buildcaches when possible--fresh-roots,--reuse-depsโ concretize with fresh roots and reused dependencies--deprecatedโ allow the concretizer to select deprecated versions (yes, the old stuff ๐)
spack spec โจ
show what would be installed, given a spec โ the "window shopping" of package management fr fr
spack spec [-h] [-l] [-L] [-N] [-I | --no-install-status] [-y | -j | --format FORMAT] [-c {nodes,edges,paths}] [-t]
[-U] [--reuse] [--fresh-roots] [--deprecated]
...More documentation โ yep, more docs, slay
Positional arguments
specsโ one or more package specs
Optional arguments
-h,--helpโ show this help message and exit-l,--longโ show dependency hashes as well as versions-L,--very-longโ show full dependency hashes as well as versions-N,--namespacesโ show fully qualified package names-I,--install-statusโ show install status of packages--no-install-statusโ do not show install status annotations-y,--yamlโ print concrete spec as YAML-j,--jsonโ print concrete spec as JSON--format FORMATโ print concrete spec with the specified format string-c,--cover {nodes,edges,paths}โ how extensively to traverse the DAG (default:nodes)-t,--typesโ show dependency types-U,--freshโ do not reuse installed deps; build newest configuration--reuseโ reuse installed packages/buildcaches when possible--fresh-roots,--reuse-depsโ concretize with fresh roots and reused dependencies--deprecatedโ allow concretizer to select deprecated versions
spack stage ๐ง
expand the downloaded archive in preparation for install โ like unboxing your package before the glow-up ๐
spack stage [-hnU] [-p PATH] [--reuse] [--fresh-roots] [--deprecated] ...More documentation โ more docs, always more docs
Positional arguments
specsโ one or more package specs
Optional arguments
-h,--helpโ show this help message and exit-n,--no-checksumโ do not use checksums to verify downloaded files (unsafe) โ ๏ธ like bestie please be careful-p,--path PATHโ path to stage package, does not add to spack tree-U,--freshโ do not reuse installed deps; build newest configuration--reuseโ reuse installed packages/buildcaches when possible--fresh-roots,--reuse-depsโ concretize with fresh roots and reused dependencies--deprecatedโ allow concretizer to select deprecated versions
spack style ๐ฅ
runs source code style checks on spack โ the group-chat-"girl did you lint this" of commands
spack style [-harUf] [-b BASE] [--root ROOT] [-t TOOL | -s TOOL] ...More documentation โ more docs available, iykyk
Positional arguments
filesโ specific files to check
Optional arguments
-h,--helpโ show this help message and exit-b,--base BASEโ branch to compare against to determine changed files (default:develop)-a,--allโ check all files, not just changed files-r,--root-relativeโ print root-relative paths (default: cwd-relative)-U,--no-untrackedโ exclude untracked files from checks-f,--fixโ format automatically if possible (e.g., withisort,black) ๐--root ROOTโ style check a different spack instance-t,--tool TOOLโ specify which tools to run (default:isort,black,flake8,mypy)-s,--skip TOOLโ specify tools to skip (choose fromisort,black,flake8,mypy)
spack tags ๐ท๏ธ
show package tags and the packages that go with them โ like the package taxonomy, adorable honestly
spack tags [-hia] [tag ...]Positional arguments
tagโ show packages with the specified tag
Optional arguments
-h,--helpโ show this help message and exit-i,--installedโ show information for installed packages only-a,--allโ show packages for all available tags
spack test ๐
run spack's tests for an install โ yes, we're actually verifying things, iconic behavior
spack test [-h] SUBCOMMAND ...More documentation โ more docs over here ๐
Optional arguments
-h,--helpโ show this help message and exit
Subcommands
test runtest listtest findtest statustest resultstest remove
spack test run ๐งช
run tests for the specified installed packages.
if no specs are listed, it runs tests for all packages in the current environment โ or literally all installed packages if there's no active environment ๐ซถ
spack test run [-h] [--alias ALIAS] [--fail-fast] [--fail-first] [--externals] [-x] [--keep-stage]
[--log-format {junit,cdash}] [--log-file LOG_FILE] [--help-cdash] [--clean | --dirty]
...More documentation โ more docs, come get y'all juice
Positional arguments
installed_specsโ one or more installed package specs
Optional arguments
-h,--helpโ show this help message and exit--alias ALIASโ provide an alias for this test-suite for subsequent access--fail-fastโ stop tests for each package after the first failure--fail-firstโ stop after the first failed package--externalsโ test packages that are externally installed-x,--explicitโ only test packages that are explicitly installed--keep-stageโ keep the testing directory for debugging--log-format {junit,cdash}โ format to be used for log files--log-file LOG_FILEโ filename for the log file--cdash-upload-url CDASH_UPLOAD_URLโ sets the CDash upload URL--cdash-build CDASH_BUILDโ sets the CDash build name--cdash-site CDASH_SITEโ sets the CDash site--cdash-track CDASH_TRACKโ sets the CDash track--cdash-buildstamp CDASH_BUILDSTAMPโ sets the CDash buildstamp--help-cdashโ show usage instructions for CDash reporting--cleanโ unset harmful variables in the build environment (default)--dirtyโ preserve user environment in spack's build environment (danger!) ๐ like genuinely, be careful
spack test list ๐
list installed packages with available tests
spack test list [-ha] [tag ...]More documentation โ more docs, mwah
Positional arguments
tagโ limit packages to those with all listed tags
Optional arguments
-h,--helpโ show this help message and exit-a,--allโ list all packages with tests (not just installed)
spack test find ๐
find tests that are running or have available results.
it displays aliases for tests that have them, otherwise it shows the test suite content hashes โจ
spack test find [-h] ...More documentation โ more docs, as always
Positional arguments
filterโ optional case-insensitive glob patterns to filter results
Optional arguments
-h,--helpโ show this help message and exit
spack test status ๐
get the current status for the specified Spack test suite(s)
spack test status [-h] ...Positional arguments
namesโ test suites for which to print status
Optional arguments
-h,--helpโ show this help message and exit
spack test results ๐
get the results from Spack test suite(s) (default: all)
spack test results [-hlf] ...More documentation โ more docs, go check 'em
Positional arguments
[name(s)] [-- installed_specs]...โ suite names and installed package constraints
Optional arguments
-h,--helpโ show this help message and exit-l,--logsโ print the test log for each matching package-f,--failedโ only show results for failed tests of matching packages
spack test remove ๐๏ธ
remove results from Spack test suite(s) (default: all).
if no test suite is listed, it removes results for all suites.
removed tests can no longer be accessed for results or status, and will not appear in spack test list results โ so like, gone gone ๐ญ
spack test remove [-hy] ...More documentation โ more docs, one last time for this stretch
Positional arguments
namesโ test suites to remove from test stage
Optional arguments
-h,--helpโ show this help message and exit-y,--yes-to-allโ assume "yes" is the answer to every confirmation request ๐ซถ (it will not ask, it will just go)
spack test-env ๐งช
okay so this one is lowkey iconic: it runs a command inside a spec's test environment, or dumps that whole environment to your screen or to a file. It's giving "let me see what the test suite is actually breathing".
spack test-env [-hU] [--clean] [--dirty] [--reuse] [--fresh-roots] [--deprecated] [--dump FILE] [--pickle FILE] ...Positional arguments โจ
spec [--] [cmd]...โ the specs of the package environment you want to emulate
Optional arguments ๐
-h,--helpโ show this help message and exit--cleanโ unset the harmful variables in the build environment (this is the default btw)--dirtyโ keep your user environment inside spack's build environment (danger! fr fr, it says danger)-U,--freshโ do not reuse installed deps; build the newest configuration--reuseโ reuse installed packages/buildcaches when possible--fresh-roots,--reuse-depsโ concretize with fresh roots and reused dependencies (bestie loves a hybrid)--deprecatedโ let the concretizer pick deprecated versions--dump FILEโ dump a source-able environment to FILE--pickle FILEโ dump a pickled source-able environment to FILE
spack tutorial ๐
sets spack up for the tutorial. WARNING: it modifies config! like, actually rewrites stuff, not just vibes.
spack tutorial [-hy]Optional arguments ๐ธ
-h,--helpโ show this help message and exit-y,--yes-to-allโ assume"yes"is the answer to every confirmation request (main character energy: no questions, just yes)
spack undevelop ๐งน
removes specs from an environment. like un-developing them. closing the lil dev loop.
spack undevelop [-ha] ...Positional arguments ๐
specsโ one or more package specs
Optional arguments ๐
-h,--helpโ show this help message and exit-a,--allโ remove all specs from the environment (i.e. clear it out)
spack uninstall ๐๏ธ
removes installed packages. the breakup command. it even has its own "More documentation" glow.
spack uninstall [-hfRya] [--remove] [--origin ORIGIN] ...There's a More documentation pointer on this one, so go read the long version if you want the full lore.
Positional arguments โจ
installed_specsโ one or more installed package specs
Optional arguments ๐ฆ
-h,--helpโ show this help message and exit-f,--forceโ remove regardless of whether other packages or environments depend on this one (chaotic, use with intention)--removeโ if you're in an environment, the spec should also be removed from the environment description-R,--dependentsโ also uninstall any packages that depend on the ones you gave on the command line-y,--yes-to-allโ assume"yes"is the answer to every confirmation request-a,--allโ remove ALL installed packages that match each supplied spec--origin ORIGINโ only remove DB records with the specified origin
spack unit-test ๐งช๐
runs spack's unit tests โ it's a wrapper around pytest, bestie. also it has extra docs.
spack unit-test [-hHs] [-n NUMPROCESSES] [-l | -L | -N] [--extension EXTENSION] [-k EXPRESSION] [--showlocals] ...More documentation exists for this one too, go peek ๐
Positional arguments ๐
pytest_argsโ arguments forpytest
Optional arguments โญ
-h,--helpโ show this help message and exit-H,--pytest-helpโ show the fullpytesthelp, with the advanced options-n,--numprocessesNUMPROCESSES โ run tests in parallel up to this wide, default1for sequential-l,--listโ list test filenames-L,--list-longโ list all test functions-N,--list-namesโ list the full names of all tests--extensionEXTENSION โ run the test for a given spack extension-sโ print output while tests run (disables capture, so you actually see the mess)-kEXPRESSION โ filter tests by keyword (works with thew/list options too)--showlocalsโ show local variable values in tracebacks
spack unload ๐ค
removes a package from the user environment. the anti-load.
spack unload [-ha] [--sh | --csh | --fish | --bat | --pwsh] ...Positional arguments ๐
installed_specsโ one or more installed package specs
Optional arguments ๐
-h,--helpโ show this help message and exit--shโ printshcommands to activate the environment--cshโ printcshcommands to activate the environment--fishโ printfishcommands to load the package--batโ printbatcommands to load the package--pwshโ printpwshcommands to load the package-a,--allโ unload all loaded Spack packages
spack url ๐
a debugging tool for URL parsing. yes queen, URLs are hard.
spack url [-h] SUBCOMMAND ...More documentation is available for this one โจ
Optional arguments ๐
-h,--helpโ show this help message and exit
Subcommands ๐ง
url parseurl listurl summaryurl stats
spack url parse ๐
spack url parse [-hs] urlPositional arguments ๐ธ
urlโ the url to parse
Optional arguments ๐
-h,--helpโ show this help message and exit-s,--spiderโ spider the source page for versions
spack url list ๐
spack url list [-hce] [-n | -N | -v | -V]Optional arguments ๐
-h,--helpโ show this help message and exit-c,--colorโ color the parsed version and name in the urls shown (versions will be cyan, name red)-e,--extrapolationโ color the versions used for extrapolation too (additional versions will be green, names magenta)-n,--incorrect-nameโ only list urls where the name was incorrectly parsed-N,--correct-nameโ only list urls where the name was correctly parsed-v,--incorrect-versionโ only list urls where the version was incorrectly parsed-V,--correct-versionโ only list urls where the version was correctly parsed
spack url summary ๐
spack url summary [-h]Optional arguments ๐
-h,--helpโ show this help message and exit
spack url stats ๐
spack url stats [-h] [--show-issues]Optional arguments โจ
-h,--helpโ show this help message and exit--show-issuesโ show packages with issues (md5hashes,httpurls) ๐
spack verify โ
checks that all spack packages are on disk as installed. the vibe check for your filesystem.
spack verify [-hlja] [-s | -f] ...Positional arguments ๐ฆ
specs_or_filesโ specs or files to verify
Optional arguments ๐
-h,--helpโ show this help message and exit-l,--localโ verify only locally installed packages-j,--jsonโ output json-formatted errors-a,--allโ verify all packages-s,--specsโ treat entries as specs (default)-f,--filesโ treat entries as absolute filenames
spack versions ๐ท๏ธ
lists the available versions of a package. yes, all of them.
spack versions [-h] [-s | --safe-only | -r | -n] [-j JOBS] packageMore documentation is out there for this one ๐
Positional arguments ๐ฅ
packageโ package name
Optional arguments ๐
-h,--helpโ show this help message and exit-s,--safeโ only list safe versions of the package--safe-onlyโ [deprecated] only list safe versions of the package-r,--remoteโ only list remote versions of the package-n,--newโ only list remote versions newer than the latest checksummed version-j,--jobsJOBS โ explicitly set number of parallel jobs
spack view ๐ช
projects packages to a compact naming scheme on the filesystem. it's giving "everything in one cute little folder".
spack view [-hv] [-e EXCLUDE] [-d {true,false,yes,no}] ACTION ...Optional arguments ๐
-h,--helpโ show this help message and exit-v,--verboseโ if not verbose, only warnings/errors get printed-e,--excludeEXCLUDE โ exclude packages with names matching the given regex pattern-d,--dependencies{true,false,yes,no}โ link/remove/list dependencies
Subcommands ๐ธ
view symlinkview hardlinkview copyview removeview statlink
spack view symlink ๐
spack view symlink [-hi] [--projection-file PROJECTION_FILE] path spec [spec ...]Positional arguments โจ
pathโ path to file system view directoryspecโ seed specs of the packages to view
Optional arguments ๐
-h,--helpโ show this help message and exit--projection-filePROJECTION_FILE โ initialize the view using projections from a file-i,--ignore-conflictsโ (no blurb in the docs, it just vibes)
spack view hardlink ๐จ
spack view hardlink [-hi] [--projection-file PROJECTION_FILE] path spec [spec ...]Positional arguments ๐
pathโ path to file system view directoryspecโ seed specs of the packages to view
Optional arguments ๐
-h,--helpโ show this help message and exit--projection-filePROJECTION_FILE โ initialize the view using projections from a file-i,--ignore-conflictsโ (again, no description given, it's a mystery icon)
spack view copy ๐
spack view copy [-hi] [--projection-file PROJECTION_FILE] path spec [spec ...]Positional arguments ๐ฆ
pathโ path to file system view directoryspecโ seed specs of the packages to view
Optional arguments ๐ง
-h,--helpโ show this help message and exit--projection-filePROJECTION_FILE โ initialize the view using projections from a file-i,--ignore-conflictsโ (still no blurb, still iconic)
spack view remove ๐๏ธ
spack view remove [-ha] [--no-remove-dependents] path [spec ...]Positional arguments ๐
pathโ path to file system view directoryspecโ seed specs of the packages to view
Optional arguments โญ
-h,--helpโ show this help message and exit--no-remove-dependentsโ do not remove dependents of the specified specs-a,--allโ act on all specs in the view
spack view statlink ๐
spack view statlink [-h] path [spec ...]Positional arguments ๐
pathโ path to file system view directoryspecโ seed specs of the packages to view
Optional arguments ๐
-h,--helpโ show this help message and exit
๐ CHAINING SPACK INSTALLATIONS ๐
okay so picture this bestie: you've already got a whole Spack install somewhere, fully stocked with packages, and you want your other Spack to justโฆ borrow them. iconic behavior honestly โจ You can point your Spack installation to another installation to use any packages that are installed there. To register the other Spack instance, you just add it as an entry to upstreams.yaml:
upstreams:
spack-instance-1:
install_tree: /path/to/other/spack/opt/spack
spack-instance-2:
install_tree: /path/to/another/spack/opt/spackinstall_tree must point to the opt/spack directory inside of the Spack base directory. that's the whole vibe, no cap ๐ฏ
Once the upstream Spack instance has been added, spack find will automatically check the upstream instance when querying installed packages, and new package installations for the local Spack install will use any dependencies that are installed in the upstream instance. literally sharing is caring ๐ซถ
Now, the plot twist: this other instance of Spack has no knowledge of the local Spack instance and may not have the same permissions or ownership as the local Spack instance. This has the following consequences, and I'm gonna say them out loud because they matter:
- Upstream Spack instances are not locked. Therefore it is up to users to make sure that the local instance is not using an upstream instance when it is being modified. ๐ซฃ
- Users should not uninstall packages from the upstream instance. Since the upstream instance doesn't know about the local instance, it cannot prevent the uninstallation of packages which the local instance depends on. ๐
Other details about upstream installations (the lore):
- If a package is installed both locally and upstream, the local installation will always be used as a dependency. This can occur if the local Spack installs a package which is not present in the upstream, but later on the upstream Spack instance also installs that package.
- If an upstream Spack instance registers and installs an external package, the local Spack instance will treat this the same as a Spack-installed package. This feature will only work if the upstream Spack instance includes the upstream functionality (i.e. if its commit is after March 27, 2019). like, check the date bestie ๐
๐ฏ Using Multiple Upstream Spack Instances ๐ฏ
A single Spack instance can use multiple upstream Spack installations, because why have one when you can have several ๐
Spack will search upstream instances in the order you list them in your configuration. And here's the gotcha: if your installation refers to instances X and Y, in that order, then instance X must list Y as an upstream in its own upstreams.yaml. order matters, it's giving chain of command ๐
๐งฉ Using Modules for Upstream Packages ๐งฉ
The local Spack instance does not generate modules for packages which are installed upstream. But! The local Spack instance can be configured to use the modules generated by the upstream Spack instance. teamwork, babe ๐ค
There are two requirements to use the modules created by an upstream Spack instance: firstly the upstream instance must do a spack module tcl refresh, which generates an index file that maps installed packages to their modules; secondly, the local Spack instance must add a modules entry to the configuration:
upstreams:
spack-instance-1:
install_tree: /path/to/other/spack/opt/spack
modules:
tcl: /path/to/other/spack/share/spack/modulesEach time new packages are installed in the upstream Spack instance, the upstream Spack maintainer should run spack module tcl refresh (or the corresponding command for the type of module they intend to use). yes, every time. sorry bestie, that's the โจmaintenance eraโจ
NOTE: Spack can generate modules that automatically load the modules of dependency packages. Spack cannot currently do this for modules in upstream packages. ๐ญ
๐ CUSTOM EXTENSIONS ๐
Spack extensions allow you to extend Spack capabilities by deploying your own custom commands or logic in an arbitrary location on your filesystem. and honestly? this is highkey one of the cutest features. This might be extremely useful e.g. to develop and maintain a command whose purpose is too specific to be considered for reintegration into the mainline, or to evolve a command through its early stages before starting a discussion to merge it upstream. your little command can have a glow-up arc before it hits the big leagues ๐ฆ
From Spack's point of view an extension is any path in your filesystem which respects the following naming and layout for files:
spack-scripting/ # The top level directory must match the format 'spack-{extension_name}'
โโโ pytest.ini # Optional file if the extension ships its own tests
โโโ scripting # Folder that may contain modules that are needed for the extension commands
โ โโโ cmd # Folder containing extension commands
โ โ โโโ filter.py # A new command that will be available
โ โโโ functions.py # Module with internal details
โโโ tests # Tests for this extension
โ โโโ conftest.py
โ โโโ test_filter.py
โโโ templates # Templates that may be needed by the extensionIn the example above, the extension is named scripting. It adds an additional command (spack filter) and unit tests to verify its behavior. look at her go ๐ญ๐
The extension can import any core Spack module in its implementation. When loaded by the spack command, the extension itself is imported as a Python package in the spack.extensions namespace. In the example above, since the extension is named "scripting", the corresponding Python module is spack.extensions.scripting.
The code for this example extension can be obtained by cloning the corresponding git repository:
$ git -C /tmp clone https://github.com/spack/spack-scripting.gitโ๏ธ Configure Spack to Use Extensions โ๏ธ
To make your current Spack instance aware of extensions you should add their root paths to config.yaml. In the case of our example this means ensuring that:
config:
extensions:
- /tmp/spack-scriptingis part of your configuration file. Once this is setup any command that the extension provides will be available from the command line:
$ spack filter --help
usage: spack filter [-h] [--installed | --not-installed]
[--explicit | --implicit] [--output OUTPUT]
...
filter specs based on their properties
positional arguments:
specs specs to be filtered
optional arguments:
-h, --help show this help message and exit
--installed select installed specs
--not-installed select specs that are not yet installed
--explicit select specs that were installed explicitly
--implicit select specs that are not installed or were installed implicitly
--output OUTPUT where to dump the resultThe corresponding unit tests can be run giving the appropriate options to spack unit-test:
$ spack unit-test --extension=scripting
========================================== test session starts ===========================================
platform linux -- Python 3.11.5, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/culpo/github/spack-scripting
configfile: pytest.ini
testpaths: tests
plugins: xdist-3.5.0
collected 5 items
tests/test_filter.py ..... [100%]
========================================== slowest 30 durations ==========================================
2.31s setup tests/test_filter.py::test_filtering_specs[kwargs0-specs0-expected0]
0.57s call tests/test_filter.py::test_filtering_specs[kwargs2-specs2-expected2]
0.56s call tests/test_filter.py::test_filtering_specs[kwargs4-specs4-expected4]
0.54s call tests/test_filter.py::test_filtering_specs[kwargs3-specs3-expected3]
0.54s call tests/test_filter.py::test_filtering_specs[kwargs1-specs1-expected1]
0.48s call tests/test_filter.py::test_filtering_specs[kwargs0-specs0-expected0]
0.01s setup tests/test_filter.py::test_filtering_specs[kwargs4-specs4-expected4]
0.01s setup tests/test_filter.py::test_filtering_specs[kwargs2-specs2-expected2]
0.01s setup tests/test_filter.py::test_filtering_specs[kwargs1-specs1-expected1]
0.01s setup tests/test_filter.py::test_filtering_specs[kwargs3-specs3-expected3]
(5 durations < 0.005s hidden. Use -vv to show these durations.)
=========================================== 5 passed in 5.06s ============================================all green, ate and left no crumbs ๐ข๐ฅ
๐ Registering Extensions via Entry Points ๐
NOTE: Python version >= 3.8 is required to register extensions via entry points. check your python bestie, don't get caught out ๐
Spack can be made aware of extensions that are installed as part of a python package. To do so, register a function that returns the extension path, or paths, to the "spack.extensions" entry point. Consider the Python package my_package that includes a Spack extension:
my-package/
โโโ src
โ โโโ my_package
โ โ โโโ __init__.py
โ โโโ spack-scripting/ # the spack extensions
โโโ pyproject.tomladding the following to my_package's pyproject.toml will make the spack-scripting extension visible to Spack when my_package is installed:
[project.entry_points."spack.extenions"]
my_package = "my_package:get_extension_path"(and yes, that key is spelled spack.extenions โ copy it exactly character for character, bestie, do not be a hero and "fix" it ๐)
The function my_package.get_extension_path in my_package/__init__.py might look like
import importlib.resources
def get_extension_path():
dirname = importlib.resources.files("my_package").joinpath("spack-scripting")
if dirname.exists():
return str(dirname)๐ CI PIPELINES ๐
Spack provides commands that support generating and running automated build pipelines in CI instances. At the highest level it works like this: provide a Spack environment describing the set of packages you care about, and include a description of how those packages should be mapped to Gitlab runners. Spack can then generate a .gitlab-ci.yml file containing job descriptions for all your packages that can be run by a properly configured CI instance. When run, the generated pipeline will build and deploy binaries, and it can optionally report to a CDash instance regarding the health of the builds as they evolve over time. it's giving automation era, it's giving hands-free ๐๐
๐ฑ Getting started with pipelines ๐ฑ
To get started with automated build pipelines a Gitlab instance with version >= 12.9 (more about Gitlab CI here) with at least one runner configured is required. This can be done quickly by setting up a local Gitlab instance. tiny local gitlab, so cute, so powerful ๐ง
It is possible to set up pipelines on gitlab.com, but the builds there are limited to 60 minutes and generic hardware. It is possible to hook up Gitlab to Google Kubernetes Engine (GKE) or Amazon Elastic Kubernetes Service (EKS), though those topics are outside the scope of this document. not me going off-topic, we stay focused ๐
After setting up a Gitlab instance for running CI, the basic steps for setting up a build pipeline are as follows:
- Create a repository in the Gitlab instance with CI and a runner enabled.
- Add a
spack.yamlat the root containing your pipeline environment - Drop a
.gitlab-ci.ymlat the repo root, and it's gotta contain two jobs: one to generate the pipeline dynamically, and one to run the generated jobs. โจ - Push a commit containing the
spack.yamland.gitlab-ci.ymlmentioned above up to the gitlab repository. ๐ซถ
Peep the Functional Example section for a minimal working example. Also see the Custom Workflow section for a link to an example of a custom workflow based on spack pipelines. ๐
Spack's pipelines are now making use of the trigger syntax to run dynamically generated child pipelines. Heads up though โ the use of dynamic child pipelines requires running Gitlab version >= 12.9. no cap.
โจ Functional Example โจ
The simplest fully functional standalone example of a working pipeline can be examined live at this example project on gitlab.com. ๐
Here's the .gitlab-ci.yml file from that example that builds and runs the pipeline:
stages: [generate, build]
variables:
SPACK_REPO: https://github.com/scottwittenburg/spack.git
SPACK_REF: pipelines-reproducible-builds
generate-pipeline:
stage: generate
tags:
- docker
image:
name: ghcr.io/scottwittenburg/ecpe4s-ubuntu18.04-runner-x86_64:2020-09-01
entrypoint: [""]
before_script:
- git clone ${SPACK_REPO}
- pushd spack && git checkout ${SPACK_REF} && popd
- . "./spack/share/spack/setup-env.sh"
script:
- spack env activate --without-view .
- spack -d ci generate
--artifacts-root "${CI_PROJECT_DIR}/jobs_scratch_dir"
--output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/pipeline.yml"
artifacts:
paths:
- "${CI_PROJECT_DIR}/jobs_scratch_dir"
build-jobs:
stage: build
trigger:
include:
- artifact: "jobs_scratch_dir/pipeline.yml"
job: generate-pipeline
strategy: dependThe key thing to note above is that there are two jobs: the first job to run, generate-pipeline, runs the spack ci generate command to generate a dynamic child pipeline and write it to a yaml file, which is then picked up by the second job, build-jobs, and used to trigger the downstream pipeline. ๐ฅ
And here's the spack environment built by the pipeline, represented as a spack.yaml file:
spack:
view: false
concretizer:
unify: false
definitions:
- pkgs:
- zlib
- bzip2
- arch:
- '%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64'
specs:
- matrix:
- - $pkgs
- - $arch
mirrors: { "mirror": "s3://spack-public/mirror" }
ci:
enable-artifacts-buildcache: True
rebuild-index: False
pipeline-gen:
- any-job:
before_script:
- git clone ${SPACK_REPO}
- pushd spack && git checkout ${SPACK_CHECKOUT_VERSION} && popd
- . "./spack/share/spack/setup-env.sh"
- build-job:
tags: [docker]
image:
name: ghcr.io/scottwittenburg/ecpe4s-ubuntu18.04-runner-x86_64:2020-09-01
entrypoint: [""]The elements of this file important to spack ci pipelines are described in more detail below, but there are a couple of things to note about the above working example:
NOTE: There is noscriptattribute specified here. The reason for this is Spack CI will automatically generate reasonable default scripts. More detail on what is in these scripts can be found below. Also notice thebefore_scriptsection. It is required when using any of the default scripts to source thesetup-env.shscript in order to inform the default scripts where to find thespackexecutable.
Normally enable-artifacts-buildcache is not recommended in production as it results in large binary artifacts getting transferred back and forth between gitlab and the runners. But in this example on gitlab.com where there is no shared, persistent file system, and where no secrets are stored for giving permission to write to an S3 bucket, enabled-buildcache-artifacts is the only way to propagate binaries from jobs to their dependents. ๐ญ
Also, it is usually a good idea to let the pipeline generate a final "rebuild the buildcache index" job, so that subsequent pipeline generation can quickly determine which specs are up to date and which need to be rebuilt (it's a good idea for other reasons as well, but those are out of scope for this discussion). In this case we have disabled it (using rebuild-index: False) because the index would only be generated in the artifacts mirror anyway, and consequently would not be available during subsequent pipeline runs. ๐
NOTE: With the addition of reproducible builds (#22887) a previously working pipeline will require some changes: - In thebuild-jobs, the environment location changed. This will typically show as aKeyErrorin the failing job. Be sure to point to${SPACK_CONCRETE_ENV_DIR}. - When usingincludein your environment, be sure to make the included files available in the build jobs. This means adding those files to the artifact directory. Those files will also be missing in the reproducibility artifact. - Because the location of the environment changed, including files with relative path may have to be adapted to work both in the project context (generation job) and in the concrete env dir context (build job).
๐ Spack commands supporting pipelines ๐
Spack provides a ci command with a few sub-commands supporting spack ci pipelines. These commands are covered in more detail in this section. ๐ฆ
๐ spack ci
Super-command for functionality related to generating pipelines and executing pipeline jobs.
๐ง spack ci generate
Throughout this documentation, references to the "mirror" mean the target mirror which is checked for the presence of up-to-date specs, and where any scheduled jobs should push built binary packages. In the past, this defaulted to the mirror at index 0 in the mirror configs, and could be overridden using the --buildcache-destination argument. Starting with Spack 0.23, spack ci generate will require you to identify this mirror by the name "buildcache-destination". While you can configure any number of mirrors as sources for your pipelines, you will need to identify the destination mirror by name. ๐
Concretizes the specs in the active environment, stages them (as described in Summary of .gitlab-ci.yml generation algorithm), and writes the resulting .gitlab-ci.yml to disk. During concretization of the environment, spack ci generate also writes a spack.lock file which is then provided to generated child jobs and made available in all generated job artifacts to aid in reproducing failed builds in a local environment. This means there are two artifacts that need to be exported in your pipeline generation job (defined in your .gitlab-ci.yml). The first is the output yaml file of spack ci generate, and the other is the directory containing the concrete environment files. In the Functional Example section, we only mentioned one path in the artifacts paths list because we used --artifacts-root as the top level directory containing both the generated pipeline yaml and the concrete environment. โจ
Using --prune-dag or --no-prune-dag configures whether or not jobs are generated for specs that are already up to date on the mirror. If enabling DAG pruning using --prune-dag, more information may be required in your spack.yaml file, see the No Op (noop) section below regarding noop-job.
The optional --check-index-only argument can be used to speed up pipeline generation by telling spack to consider only remote buildcache indices when checking the remote mirror to determine if each spec in the DAG is up to date or not. The default behavior is for spack to fetch the index and check it, but if the spec is not found in the index, to also perform a direct check for the spec on the mirror. If the remote buildcache index is out of date, which can easily happen if it is not updated frequently, this behavior ensures that spack has a way to know for certain about the status of any concrete spec on the remote mirror, but can slow down pipeline generation significantly. ๐
The --optimize argument is experimental and runs the generated pipeline document through a series of optimization passes designed to reduce the size of the generated file. ๐งช
The --dependencies is also experimental and disables what in Gitlab is referred to as DAG scheduling, internally using the dependencies keyword rather than needs to list dependency jobs. The drawback of using this option is that before any job can begin, all jobs in previous stages must first complete. The benefit is that Gitlab allows more dependencies to be listed when using dependencies instead of needs.
The optional --output-file argument should be an absolute path (including file name) to the generated pipeline, and if not given, the default is ./.gitlab-ci.yml.
While optional, the --artifacts-root argument is used to determine where the concretized environment directory should be located. This directory will be created by spack ci generate and will contain the spack.yaml and generated spack.lock which are then passed to all child jobs as an artifact. This directory will also be the root directory for all artifacts generated by jobs in the pipeline. ๐ซถ
๐ธ spack ci rebuild ๐ธ
The purpose of spack ci rebuild is to take an assigned spec and ensure a binary of a successful build exists on the target mirror. If the binary does not already exist, it is built from source and pushed to the mirror. The associated stand-alone tests are optionally run against the new build. Additionally, files for reproducing the build outside of the CI environment are created to facilitate debugging. ๐
If a binary for the spec does not exist on the target mirror, an install shell script, install.sh, is created and saved in the current working directory. The script is run in a job to install the spec from source. The resulting binary package is pushed to the mirror. If cdash is configured for the environment, then the build results will be uploaded to the site. ๐ค
Environment variables and values in the ci::pipeline-gen section of the spack.yaml environment file provide inputs to this process. The two main sources of environment variables are variables written into .gitlab-ci.yml by spack ci generate and the GitLab CI runtime. Several key CI pipeline variables are described in Environment variables affecting pipeline operation. ๐
If the --tests option is passed, stand-alone tests get run too โ but only if the build actually succeeded and the package isn't listed in broken-tests-packages. A shell script called test.sh is created and run to do the testing โจ. When it finishes, the test logs are exported as job artifacts so you can review them and debug way easier. And if cdash is configured, the test results also get uploaded to the site. ๐
Here's a snippet from an example spack.yaml showing this option in action, plus how you'd flag a package as having broken tests. (The spec for building gptune isn't shown here, fyi.) Peep that --tests is handed to spack ci rebuild as part of the build-job script:
ci:
pipeline-gen:
- build-job
script:
- . "./share/spack/setup-env.sh"
- spack --version
- cd ${SPACK_CONCRETE_ENV_DIR}
- spack env activate --without-view .
- spack config add "config:install_tree:projections:${SPACK_JOB_SPEC_PKG_NAME}:'morepadding/{architecture}/{compiler.name}-{compiler.version}/{name}-{version}-{hash}'"
- mkdir -p ${SPACK_ARTIFACTS_ROOT}/user_data
- if [[ -r /mnt/key/intermediate_ci_signing_key.gpg ]]; then spack gpg trust /mnt/key/intermediate_ci_signing_key.gpg; fi
- if [[ -r /mnt/key/spack_public_key.gpg ]]; then spack gpg trust /mnt/key/spack_public_key.gpg; fi
- spack -d ci rebuild --tests > >(tee ${SPACK_ARTIFACTS_ROOT}/user_data/pipeline_out.txt) 2> >(tee ${SPACK_ARTIFACTS_ROOT}/user_data/pipeline_err.txt >&2)
broken-tests-packages:
- gptuneSo in this case, even if gptune builds totally fine from source, the pipeline still won't run its stand-alone tests, because the package is sitting right there under broken-tests-packages. ๐
Spack's own cloud pipelines are literally full of real, up-to-date examples of the CI/CD config and environment files Spack uses โ you can find them under Spack's stacks repository directory. ๐
spack ci rebuild-index
A lil convenience command that rebuilds the buildcache index for the mirror in the active, gitlab-enabled environment. No cap, you don't need to specify the mirror url or name โ it just knows. ๐
spack ci reproduce-build
Give it the url to a gitlab pipeline rebuild job and it downloads + unzips the artifacts into a local directory (you can pick which one with the optional --working-dir argument), then it finds the target job in the generated pipeline to pull out the details of how it ran. Assuming the job used a docker image, the command prints a docker run command line plus some basic instructions on how to reproduce that build locally. ๐ณโจ
Ngl, jobs that fail in the pipeline will print messages telling you exactly which arguments to pass to spack ci reproduce-build so you can reproduce that particular build locally. So helpful, fr fr.
๐งช Job Types
Rebuild (build)
Rebuild jobs โ the ones called build-job's in the pipeline-gen list โ are the jobs tied to concrete specs that have been marked for rebuild. By default a simple script for doing the rebuild is generated, but you can modify it however you need. ๐
That default script does three main things: changes directories into the pipeline's concrete environment, activates the concrete environment, and runs the spack ci rebuild command:
cd ${concrete_environment_dir}
spack env activate --without-view .
spack ci rebuildUpdate Index (reindex)
So by default, even though a pipeline job might rebuild a package, create a buildcache entry, and push it to the mirror, it does not automatically re-generate the mirror's buildcache index afterward. And that's on purpose ๐ซถ โ the index isn't needed by the default rebuild jobs in the pipeline, so skipping the update at the end of each job avoids possible race conditions between simultaneous jobs, and it dodges the computational expense of regenerating the index. That can honestly save minutes per job, depending on how many binary packages are in the mirror. The tradeoff? By default the mirror's buildcache index may not correctly reflect the mirror's contents when the pipeline ends. ๐ญ
To make sure the buildcache index is up to date at the end of your pipeline, spack generates a job by default to update the target mirror's buildcache index at the end of each pipeline. You can turn that behavior off by adding rebuild-index: False inside the ci section of your spack environment.
Heads up: reindex jobs do not let you modify thescriptattribute, since it's automatically generated using the target mirror listed in themirrors::mirrorconfiguration. โญ
Signing (signing)
This job runs after all the rebuild jobs are done, and it's meant to sign the package binaries built by a protected CI run. Signing jobs are only generated if a signing job script is specified and the spack CI job type is protected. ๐
Note: if an any-job section contains a script, that does not implicitly create a signing job โ a signing job can only exist if it's explicitly specified in the configuration with a script attribute. Specifying a signing job without a script does not create a signing job, and the job configuration attributes just get ignored. ๐
Signing jobs are always assigned the runner tags aws, protected, and notary.
Cleanup (cleanup)
When you're using temporary-storage-url-prefix, the cleanup job destroys the mirror that was created for the associated Gitlab pipeline. ๐งน Cleanup jobs don't let you modify the script, but they do expect that the spack command is in the path, and they require a before_script to be specified that sources the setup-env.sh script.
No Op (noop)
If no specs in an environment need rebuilding during a given pipeline run (meaning they're all already up to date on the mirror), a single successful job (a NO-OP) is still generated โ because otherwise you'd get an empty pipeline, and GitLab considers that an error. ๐ค The noop-job* sections can be added to your spack.yaml, where you can provide tags and image, or variables for the generated NO-OP job. This section also supports providing before_script, script, and after_script, in case you wanna take some custom actions when the pipeline ends up empty. โจ
Here's an example of that section added to a spack.yaml:
spack:
ci:
pipeline-gen:
- noop-job:
tags: ['custom', 'tag']
image:
name: 'some.image.registry/custom-image:latest'
entrypoint: ['/bin/bash']
script::
- echo "Custom message in a custom script"That example shows how you can provide the attributes used to run the NO-OP job when the pipeline is empty. The only field for the NO-OP job that might be generated for you is script, and that only happens if you don't provide one yourself. ๐ Notice in this example the script uses the :: notation to prescribe override behavior โ without that, the echo command would've been prepended to the automatically generated script instead of replacing it.
๐ ci.yaml
Here's an example of a spack configuration file describing a build pipeline:
ci:
target: gitlab
rebuild_index: True
broken-specs-url: https://broken.specs.url
broken-tests-packages:
- gptune
pipeline-gen:
- submapping:
- match:
- os=ubuntu18.04
build-job:
tags:
- spack-kube
image: spack/ubuntu-bionic
- match:
- os=centos7
build-job:
tags:
- spack-kube
image: spack/centos7
cdash:
build-group: Release Testing
url: https://cdash.spack.io
project: Spack
site: Spack AWS Gitlab InstanceThe ci config section is what configures how the pipeline workload gets generated โ mainly how the jobs for building specs get assigned to the runners configured on your instance. ๐ The main section for configuring pipelines is pipeline-gen, which is a list of job attribute sections that get merged from the bottom up, using the same rules as Spack configs (Scope Precedence). The order sections are applied in is consistent with how spack orders scope precedence when merging lists. There are two main section types: <type>-job sections and submapping sections. ๐ซ
๐ Job Attribute Sections
Each type of job can have attributes added or removed via sections in the pipeline-gen list. Job type specific attributes can be specified with the keys <type>-job to add attributes to all jobs of type <type>, or <type>-job-remove to remove attributes of type <type>. Each section may only contain one type of job attribute specification, i.e. build-job and noop-job may not coexist, but build-job and build-job-remove may. ๐ซถ
NOTE: The*-removespecifications are applied before the additive attribute specification. So for example, when bothbuild-jobandbuild-job-removeare listed in the samepipeline-gensection, the value will still exist in the mergedbuild-jobafter applying the section. ๐
All of the attributes specified get forwarded to the generated CI jobs, but special treatment is applied to the attributes tags, image, variables, script, before_script, and after_script, since those are components the Spack CI generator explicitly recognizes. For the tags attribute, Spack removes reserved tags (Reserved Tags) from all jobs specified in the config. In some cases โ like for signing jobs โ reserved tags get added back based on the type of CI that's being run. โจ
Once a runner has been chosen to build a release spec, the build-job* sections provide the info that determines the details of the job in the context of that runner. At least one of the build-job* sections must contain a tags key, which is a list containing at least one tag used to select the runner from among the runners known to the gitlab instance. ๐
For Docker executor type runners, the image key specifies the Docker image used to build the release spec (and it could also show up as a dictionary with a name specifying the image name, plus an entrypoint to override whatever the default for that image is). For other types of runners, the variables key is gonna be useful for passing any info on to the runner that it needs to do its work (e.g. scheduler parameters, etc.). Any variables provided here get added, verbatim, to each job. ๐
The build-job section also lets you supply custom script, before_script, and after_script sections that get applied to every job scheduled on that runner. So like โ do your own prep work, your own cleanup, whatever fits your workflow, and yes you can go full DIY and completely customize how a spec gets rebuilt if that's your vibe. Spack will not generate a before_script or after_script for jobs, but if you don't hand it a custom script, spack generates one for you โ and that auto-generated one assumes the concrete environment directory lives inside your --artifacts_root (or, if you didn't pass that, inside your $CI_PROJECT_DIR), activates that environment for you, and then calls spack ci rebuild. slay ๐
Sections that specify scripts (script, before_script, after_script) are all read as either lists of commands or lists of lists of commands. Strong rec from me: write your scripts as lists of lists if they're gonna be composed via merging. Why? Because the default merge behavior for lists removes duplicate commands and might reorder stuff you did not want reordered ๐ญ, whereas merging lists of lists preserves the local ordering and never drops duplicates. When commands get written to the CI target script, all lists are expanded and flattened into one single list. periodt.
๐ Submapping Sections ๐
Okay so a submapping section is a special case of attribute specification โ it lets you apply job attributes to build jobs based on the package spec that's attached to the rebuild job. You write it as a list of spec match lists associated with build-job/build-job-remove sections. There are two options for match_behavior: either first or merge. Either way, the submapping list is processed from the bottom up, and then each match list is searched for a string that satisfies the check spec.satisfies({match_item}) for each concrete spec.
In the case of match_behavior: first, the first match section in the list of submappings that contains a string satisfying the spec will apply its build-job* attributes to the rebuild job associated with that spec. This is the default behavior and it's what you get if you don't specify match_behavior at all.
In the case of merge match, all of the match sections in the list of submappings that contain a string satisfying the spec get their associated build-job* attributes applied to the rebuild job for that spec. Again โ the attributes are merged starting from the bottom match and going up to the top match.
And if no match is found in a submapping section? Then no additional attributes get applied. Simple as that, bestie ๐ฆ
๐ง Bootstrapping ๐ง
The bootstrap section lets you specify lists of specs from your definitions that should be staged ahead of the environment's specs. Fyi, right now the only viable use-case for bootstrapping is installing compilers. That's the whole bit.
Here's what bootstrapping some compilers might look like:
spack:
definitions:
- compiler-pkgs:
- 'llvm+clang@6.0.1 os=centos7'
- 'gcc@6.5.0 os=centos7'
- 'llvm+clang@6.0.1 os=ubuntu18.04'
- 'gcc@6.5.0 os=ubuntu18.04'
- pkgs:
- readline@7.0
- compilers:
- '%gcc@5.5.0'
- '%gcc@6.5.0'
- '%gcc@7.3.0'
- '%clang@6.0.0'
- '%clang@6.0.1'
- oses:
- os=ubuntu18.04
- os=centos7
specs:
- matrix:
- [$pkgs]
- [$compilers]
- [$oses]
exclude:
- '%gcc@7.3.0 os=centos7'
- '%gcc@5.5.0 os=ubuntu18.04'
ci:
bootstrap:
- name: compiler-pkgs
compiler-agnostic: true
pipeline-gen:
# similar to the example higher up in this description
...So what's happening here ๐ โ the example above adds a list to the definitions called compiler-pkgs (and you can add any number of these, go off), which lists compiler packages that should be staged ahead of the full matrix of release specs (in this example the only release spec is readline). Then inside the ci section, notice the added bootstrap section, which can contain a list of items, each one referring to a list in the definitions section. Those items can be either a dictionary or a string. If you give it a dictionary, it must have a name key whose value must match one of the lists in definitions, and it can have a compiler-agnostic key whose value is a boolean. If you give it a string, then that string needs to match one of the lists provided in definitions. You can think of the bootstrap list as an ordered list of pipeline "phases" that get staged before your actual release specs. Yes, this adds another layer of bottleneck in the pipeline (all jobs in all stages of one phase must complete before any jobs in the next phase can begin) โ but it also means you are guaranteed your bootstrapped compilers will be available when you need them. tradeoff queen ๐
The compiler-agnostic key can be provided with each item in the bootstrap list. It tells the spack ci generate command that any jobs staged from that particular list should have the compiler removed from the spec, so that any compiler available on the runner where the job runs can be used to build the package. iconic.
When you include a bootstrapping phase like in the example above, the result is that the bootstrapped compiler packages get pushed to the binary mirror (and the local artifacts mirror) before the actual release specs are built. In that case, the jobs corresponding to the subsequent release specs are configured to install_missing_compilers, so that if spack is asked to install a package with a compiler it doesn't know about, it can be quickly installed from the binary mirror first. no cap, that's the whole magic trick โจ
And since bootstrapping compilers is optional, you can just... leave those items out of the environment/stack file. In that case no bootstrapping will be done (only the specs get staged for building) and the runners will be expected to already have all the needed compilers installed and configured for spack to use. Your runners, your problem, bestie ๐ซถ
๐ Pipeline Buildcache ๐
The enable-artifacts-buildcache key takes a boolean and decides whether the pipeline uses artifacts to store and pass along the buildcaches from one stage to the next. Heads up: the default, if you don't provide this option, is False.
๐ธ Broken Specs URL ๐ธ
The optional broken-specs-url key tells Spack to check against a list of specs that are known to be currently broken in develop. If any of those specs are found, the spack ci generate command will fail with an error message telling the user what broken specs were encountered. This lets the pipeline fail early and avoid wasting compute resources trying to build packages that are simply not gonna succeed. it's giving self-care ๐
โญ CDash โญ
The optional cdash section provides info that will be used by the spack ci generate command (which gets invoked by spack ci start) for reporting to CDash. All the jobs generated from this environment will belong to a "build group" within CDash that can be tracked over time. As the release progresses, this build group may have jobs added or removed. The url, project, and site are used to specify the CDash instance that build results should be reported to.
Take a look at the schema for the ci section of the spack environment file to see precisely what syntax is allowed there ๐
๐ Reserved Tags ๐
Spack has a subset of tags (public, protected, and notary) that it reserves for classifying runners that may require special permissions or access. The tags public and protected are used to distinguish between runners that use public permissions and runners with protected permissions. The notary tag is a special tag used to indicate runners that have access to the highly protected information used for signing binaries using the signing job. very demure, very mindful ๐
๐ฅ Summary of .gitlab-ci.yml generation algorithm ๐ฅ
Okay so here's the tea. All specs yielded by the matrix (or all the specs in the environment) have their dependencies computed, and the entire resulting set of specs gets staged together before being run through the ci/pipeline-gen entries, where each staged spec is assigned a runner. "Staging" is the name given to the process of figuring out in what order the specs should be built, taking into consideration Gitlab CI rules about jobs/stages. The goal in the staging process is to maximize the number of jobs in any stage of the pipeline, while ensuring that the jobs in any stage only depend on jobs in previous stages (since those jobs are guaranteed to have completed already). As a runner is determined for a job, the information in the merged any-job* and build-job* sections is used to populate various parts of the job description that will be used by the target CI pipelines. Once all the jobs have been assigned a runner, the .gitlab-ci.yml is written to disk.
The short example provided above would result in the readline, ncurses, and pkgconf packages getting staged and built on the runner chosen by the spack-k8s tag. In this example, spack assumes the runner is a Docker executor type runner, and thus certain jobs will be run in the centos7 container, and others in the ubuntu-18.04 container. The resulting .gitlab-ci.yml will contain 6 jobs in three stages. Once the jobs have been generated, the presence of a SPACK_CDASH_AUTH_TOKEN environment variable during the spack ci generate command would result in all of the jobs being put in a build group on CDash called "Release Testing" (that group will be created if it didn't already exist). ate and left no crumbs ๐
๐ Using a custom spack in your pipeline ๐
If your runners won't have a version of spack ready to invoke, or if for some other reason you wanna use a custom version of spack to run your pipelines, this section gives you an example of how you could take advantage of user-provided pipeline scripts to pull that off fairly simply. First, consider specifying the source and version of spack you want to use with variables โ either written directly into your .gitlab-ci.yml, or provided by CI variables defined in the gitlab UI or from some upstream pipeline. Let's say you pick the variable names SPACK_REPO and SPACK_REF to refer to the particular fork of spack and the branch you want for running your pipeline. You can then refer to those in a custom shell script invoked both from your pipeline generation job and your rebuild jobs. Here's the generate-pipeline job from the top of this document, updated to clone and source a custom spack:
generate-pipeline:
tags:
- <some-other-tag>
before_script:
- git clone ${SPACK_REPO}
- pushd spack && git checkout ${SPACK_REF} && popd
- . "./spack/share/spack/setup-env.sh"
script:
- spack env activate --without-view .
- spack ci generate --check-index-only
--artifacts-root "${CI_PROJECT_DIR}/jobs_scratch_dir"
--output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/pipeline.yml"
after_script:
- rm -rf ./spack
artifacts:
paths:
- "${CI_PROJECT_DIR}/jobs_scratch_dir"Okay so that right there? That handles getting the version of spack you actually want when your pipeline gets generated by spack ci generate. โจ Slay. But bestie, you're not done โ you also want your generated rebuild jobs (all of them, no exceptions, every single one ๐) to clone that same version of spack. So next you'd update your spack.yaml from up above like this:
spack:
# ...
ci:
pipeline-gen:
- build-job:
tags:
- spack-kube
image: spack/ubuntu-bionic
before_script:
- git clone ${SPACK_REPO}
- pushd spack && git checkout ${SPACK_REF} && popd
- . "./spack/share/spack/setup-env.sh"
script:
- spack env activate --without-view ${SPACK_CONCRETE_ENV_DIR}
- spack -d ci rebuild
after_script:
- rm -rf ./spackAnd now every single generated rebuild job uses that same shell script to clone spack before it runs its actual workload. Consistency era fr fr ๐
Now picture this: you've got long pipelines with like, soooo many specs to build, and you're pointing at a spack repository and branch that has a tendency to change constantly โ think the main repo and its develop branch. ๐ญ If each child job checks out the develop branch, some jobs could end up running with one SHA of spack while later jobs run with a totally different one. Chaos. Not cute.
So to dodge that whole situation, the pipeline generation process saves global variables called SPACK_VERSION and SPACK_CHECKOUT_VERSION that capture the version of spack used to generate the pipeline. ๐
SPACK_VERSIONโ just contains the human-readable value produced byspack -Vat pipeline generation timeSPACK_CHECKOUT_VERSIONโ can be used in agit checkoutcommand so you can make sure all child jobs checkout the exact same version of spack that was used to generate the pipeline
To take advantage of it, you could simply replace git checkout ${SPACK_REF} in the example spack.yaml above with git checkout ${SPACK_CHECKOUT_VERSION}. Iconic, honestly.
On the other hand โ flip side ๐ฆ โ if you're pointing at a spack repository and branch that's under your control, there may be no benefit at all in using the captured SPACK_CHECKOUT_VERSION, and you can instead just clone using the variables you define (SPACK_REPO and SPACK_REF in the example above). Do whatever makes sense for your setup, bestie.
๐ Custom Workflow ๐
There are like, so many ways to take advantage of spack CI pipelines to achieve custom workflows for building packages or other resources. The way that this is flexible enough to do that?? Obsessed. ๐
One example of a custom pipelines workflow is the spack tutorial container repo. That project uses:
- GitHub (for source control)
- GitLab (for automated
spack cipipelines) - DockerHub automated builds to build Docker images (complete with fully populated binary mirror) used by instructors and participants of a spack tutorial
Take a look at the repo to see how it's accomplished using spack CI pipelines, and see the following markdown files at the root of the repository for descriptions and documentation describing the workflow: DESCRIPTION.md, DOCKERHUB_SETUP.md, GITLAB_SETUP.md, and UPDATING.md. ๐ซถ
๐ธ Environment variables affecting pipeline operation ๐ธ
Certain secrets and some other information should be provided to the pipeline infrastructure via environment variables โ usually for reasons of security, but in some cases to support other pipeline use cases such as PR testing. The environment variables used by the pipeline infrastructure are described here. ๐
AWS_ACCESS_KEY_ID
Optional. Only needed when the binary mirror is an S3 bucket. ๐ชฃ
AWS_SECRET_ACCESS_KEY
Optional. Only needed when the binary mirror is an S3 bucket.
S3_ENDPOINT_URL
Optional. Only needed when the binary mirror is an S3 bucket that's not on AWS.
CDASH_AUTH_TOKEN
Optional. Only needed in order to report build groups to CDash.
SPACK_SIGNING_KEY
Optional. Only needed if you want spack ci rebuild to trust the key you store in this variable โ in which case it will subsequently be used to sign and verify binary packages (when installing or creating buildcaches). ๐
You could also already have trusted a key spack knows about. Or, if no key is present anywhere, spack will install specs using --no-check-signature and create buildcaches using -u (for unsigned binaries). So like, it degrades gracefully, but know that's what's happening, okay? ๐
๐ SPACK PACKAGE SIGNING ๐
The goal of package signing in Spack is to provide data integrity assurances around official packages produced by the automated Spack CI pipelines. These assurances directly address the security of Spack's software supply chain by explaining why a security-conscious user can be reasonably justified in the belief that packages installed via Spack have an uninterrupted auditable trail back to change management decisions judged to be appropriate by the Spack maintainers. ๐ฅ
This is achieved through cryptographic signing of packages built by Spack CI pipelines based on code that has been transparently reviewed and approved on GitHub. This document describes the signing process for interested users. ๐
๐ฆ Risks, Impact and Threat Model ๐ฆ
This document addresses the approach taken to safeguard Spack's reputation with regard to the integrity of the package data produced by Spack's CI pipelines. It does not address issues of data confidentiality (Spack is intended to be largely open source) or availability (efforts are described elsewhere).
With that said, the main reputational risk can be broadly categorized as a loss of faith in the data integrity due to a breach of the private key used to sign packages. ๐
Remediation of a private key breach would require:
- republishing the public key with a revocation certificate
- generating a new signing key
- an assessment and potential rebuild/resigning of all packages since the key was breached
- finally, direct intervention by every spack user to update their copy of Spack's public keys used for local verification
Yeah. Big yikes. ๐ญ
The primary threat model used in mitigating the risks of these stated impacts is one of individual error, not malicious intent or insider threat. The primary objective is to avoid the above impacts by making a private key breach nearly impossible due to oversight or configuration error. Obvious and straightforward measures are taken to mitigate issues of malicious interference in data integrity and insider threats, but these attack vectors are not systematically addressed.
Bottom line: it should be hard to exfiltrate the private key intentionally, and almost impossible to leak the key by accident. ๐ซถ
โญ Pipeline Overview โญ
Spack pipelines build software through progressive stages, where packages in later stages nominally depend on packages built in earlier stages. For both technical and design reasons, these dependencies are not implemented through the default GitLab artifacts mechanism โ instead, built packages are uploaded to AWS S3 mirrors (buckets) where they're retrieved by subsequent stages in the pipeline. ๐ชฃ
Two broad categories of pipelines exist: Pull Request (PR) pipelines and Develop/Release pipelines.
- PR pipelines are launched in response to pull requests made by trusted and untrusted users. Packages built on these pipelines upload code to quarantined AWS S3 locations which cache the built packages for the purposes of review and iteration on the changes proposed in the pull request. Packages built on PR pipelines can come from untrusted users, so signing of these pipelines is not implemented. Jobs in these pipelines are executed via normal GitLab runners both within the AWS GitLab infrastructure and at affiliated institutions.
- Develop and Release pipelines sign the packages they produce and carry strong integrity assurances that trace back to auditable change management decisions. These pipelines only run after members from a trusted group of reviewers verify that the proposed changes in a pull request are appropriate. Once the PR is merged, or a release is cut, a pipeline is run on protected GitLab runners which provide access to the required signing keys within the job. Intermediary keys are used to sign packages in each stage of the pipeline as they're built, and a final job officially signs each package external to any specific package's build environment. An intermediate key exists in the AWS infrastructure and for each affiliated institution that maintains protected runners. The runners that execute these pipelines exclusively accept jobs from protected branches โ meaning the intermediate keys are never exposed to unreviewed code, and the official keys are never exposed to any specific build environment. ๐
๐ง Key Architecture ๐ง
Spack's CI process uses public-key infrastructure (PKI) based on GNU Privacy Guard (gpg) keypairs to sign public releases of spack package metadata, also called specs. ๐
Two classes of GPG keys are involved in the process to reduce the impact of an individual private key compromise. These key classes are the Intermediate CI Key and the Reputational Key. Each of these keys has signing sub-keys that are used exclusively for signing packages.
Okay this can get confusing, so for the purpose of this explanation we'll refer to Root and Signing keys. Each key has a private and a public component, as well as one or more identities and zero or more signatures. ๐
๐ Intermediate CI Key
Okay so picture this: keys come in whole classes, and the Intermediate CI Key is the one doing the middle-of-the-pipeline grunt work. It signs and verifies packages between stages within a develop or release pipeline. There's an intermediate key for the AWS infrastructure, plus one for every affiliated institution that maintains protected runners. Every single one of them.
These intermediate keys get made available to the GitLab execution environment that's building the package, so that the package's dependencies can be verified by the Signing Intermediate CI Public Key and the final package can be signed by the Signing Intermediate CI Private Key. Teamwork, honestly.
Intermediate CI Key (GPG) โ the whole lineup, bestie:
- Root Intermediate CI Private Key (RSA 4096)# โ Root Intermediate CI Public Key (RSA 4096)
- Signing Intermediate CI Private Key (RSA 4096) โ Signing Intermediate CI Public Key (RSA 4096)
- Identity:
Intermediate CI Key <maintainers@spack.io> - Signatures: None
The Root intermediate CI Private Key is stripped out of the GPG key and stored offline, completely separate from Spack's infrastructure. Why? So the core development team can append revocation certificates to the GPG key and issue new sub-keys for use in the pipeline. It's expected that this will happen on a semi regular basis, no big deal, just a little key refresh moment. A corollary of this โ and listen, this part matters โ is that this key should not be used to verify package integrity outside the internal CI process. It's an indoor key. It does not go outside. ๐ซถ
๐ Reputational Key
The Reputational Key is the public facing one, the main character, the one that signs complete groups of development and release packages. Only one key pair exists in this class of keys. And unlike the Intermediate CI Key, the Reputational Key should absolutely be used to verify package integrity. That's literally its whole thing.
At the end of a develop or release pipeline, a final pipeline job pulls down all the signed package metadata built by the pipeline, verifies they were signed with an Intermediate CI Key, then strips the Intermediate CI Key signature from the package and re-signs them with the Signing Reputational Private Key. The officially signed packages then get uploaded back to the AWS S3 mirror. โจ
And why put the reputational key in a separate final job at all? To prevent leakage of the key in a spack package, periodt. Because the Signing Reputational Private Key is never exposed to a build job, it cannot accidentally end up in any built package. Iconic safety behaviour, honestly.
Reputational Key (GPG) โ the lineup:
- Root Reputational Private Key (RSA 4096)# โ Root Reputational Public Key (RSA 4096)
- Signing Reputational Private Key (RSA 4096) โ Signing Reputational Public Key (RSA 4096)
- Identity:
Spack Project <maintainers@spack.io> - Signatures: Signed by core development team [1]
The Root Reputational Private Key is stripped out of the GPG key and stored offline completely separate from Spack's infrastructure. This is what lets the core development team append revocation certificates to the GPG key in the unlikely event that the Signing Reputation Private Key is compromised. In general the expectation is that rotating this key will happen infrequently, if at all. Which is a good thing, because it should allow relatively transparent verification for the end-user community without needing deep familiarity with GnuPG or Public Key Infrastructure. You shouldn't need a cryptography degree to check your packages, and you don't. ๐
๐ Build Cache Format
So what is a binary package, structurally? It's two things: a metadata file that unambiguously defines the built package (and includes other details, such as how to relocate it), and the installation directory of the package stored as a compressed archive file.
The metadata files can be either unsigned โ in which case the contents are simply the json-serialized concrete spec plus metadata โ or they can be signed, in which case the json-serialized concrete spec plus metadata is wrapped in a gpg cleartext signature.
Built package metadata files are named to indicate the operating system and architecture for which the package was built, as well as the compiler used to build it and the package name and version. For example:
linux-ubuntu18.04-haswell-gcc-7.5.0-zlib-1.2.12-llv2ysfdxnppzjrt5ldybb5c52qbmoow.spec.json.sigwould contain the concrete spec and binary metadata for a binary package of zlib@1.2.12, built for the ubuntu operating system and haswell architecture. The id of the built package exists in the name of the file as well (after the package name and version), and in this case begins with llv2ys. That id distinguishes a particular built package from all other built packages with the same os/arch, compiler, name, and version. Little fingerprint in the filename, we love to see it. ๐ฅ
Below is an example of a signed binary package metadata file. Such a file would live in the build_cache directory of a binary mirror:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
{
"spec": {
<concrete-spec-contents-omitted>
},
"buildcache_layout_version": 1,
"binary_cache_checksum": {
"hash_algorithm": "sha256",
"hash": "4f1e46452c35a5e61bcacca205bae1bfcd60a83a399af201a29c95b7cc3e1423"
}
}
-----BEGIN PGP SIGNATURE-----
iQGzBAEBCgAdFiEETZn0sLle8jIrdAPLx/P+voVcifMFAmKAGvwACgkQx/P+voVc
ifNoVgv/VrhA+wurVs5GB9PhmMA1m5U/AfXZb4BElDRwpT8ZcTPIv5X8xtv60eyn
4EOneGVbZoMThVxgev/NKARorGmhFXRqhWf+jknJZ1dicpqn/qpv34rELKUpgXU+
QDQ4d1P64AIdTczXe2GI9ZvhOo6+bPvK7LIsTkBbtWmopkomVxF0LcMuxAVIbA6b
887yBvVO0VGlqRnkDW7nXx49r3AG2+wDcoU1f8ep8QtjOcMNaPTPJ0UnjD0VQGW6
4ZFaGZWzdo45MY6tF3o5mqM7zJkVobpoW3iUz6J5tjz7H/nMlGgMkUwY9Kxp2PVH
qoj6Zip3LWplnl2OZyAY+vflPFdFh12Xpk4FG7Sxm/ux0r+l8tCAPvtw+G38a5P7
QEk2JBr8qMGKASmnRlJUkm1vwz0a95IF3S9YDfTAA2vz6HH3PtsNLFhtorfx8eBi
Wn5aPJAGEPOawEOvXGGbsH4cDEKPeN0n6cy1k92uPEmBLDVsdnur8q42jk5c2Qyx
j3DXty57
=3gvm
-----END PGP SIGNATURE-----If a user has trusted the public key associated with the private key used to sign the above spec file, the signature can be verified with gpg, as follows:
$ gpg โverify linux-ubuntu18.04-haswell-gcc-7.5.0-zlib-1.2.12-llv2ysfdxnppzjrt5ldybb5c52qbmoow.spec.json.sigThe metadata (regardless whether signed or unsigned) contains the checksum of the .spack file containing the actual installation. That checksum should be compared to a checksum computed locally on the .spack file to ensure the contents have not changed since the binary spec plus metadata were signed. Like, verify it yourself, don't just trust the vibe check. ๐
The .spack files are actually tarballs containing the compressed archive of the install tree. These files, along with the metadata files, live within the build_cache directory of the mirror, and together are organized as follows:
build_cache/
# unsigned metadata (for indexing, contains sha256 of .spack file)
<arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json
# clearsigned metadata (same as above, but signed)
<arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json.sig
<arch>/
<compiler>/
<name>-<ver>/
# tar.gz-compressed prefix (may support more compression formats later)
<arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spackUncompressing and extracting the .spack file results in the install tree. This is in contrast to previous versions of spack, where the .spack file contained a (duplicated) metadata file, a signature file and a nested tarball containing the install tree. So yeah โ glow-up. ๐ฆ
๐ง Internal Implementation
The technical implementation of the pipeline signing process includes components defined in Amazon Web Services, the Kubernetes cluster, at affilicated institutions, and the GitLab/GitLab Runner deployment.
We present the technical implementation in two interdependent sections:
- The first addresses how secrets are managed through the lifecycle of a develop or release pipeline.
- The second section describes how Gitlab Runner and pipelines are configured and managed to support secure automated signing.
๐ Secrets Management
As stated above, the Root Private Keys (intermediate and reputational) are stripped from the GPG keys and stored outside Spack's infrastructure.
WARNING: TODO - Explanation here about where and how access is handled for these keys. - Both Root private keys are protected with strong passwords - Who has access to these and how?
Like, this section is genuinely still a TODO and we're just being honest about it. No cap. ๐ญ
๐ Intermediate CI Key
Okay so bestie, real talk: there are multiple Intermediate CI signing keys. One Intermediate CI Key for jobs run in AWS, and one key for each affiliated institution (e.g. University of Oregon). Hereโs how the Intermediate CI Key is managed in AWS:
The Intermediate CI Key (including the Signing Intermediate CI Private Key) is exported as an ASCII armored file and stored in a Kubernetes secret called spack-intermediate-ci-signing-key. For convenience sake, this same secret contains an ASCII-armored export of just the public components of the Reputational Key. This secret also contains the public components of each of the affiliated institutions' Intermediate CI Key. These are potentially needed to verify dependent packages which may have been found in the public mirror or built by a protected job running on an affiliated institution's infrastructure in an earlier stage of the pipeline.
Procedurally the spack-intermediate-ci-signing-key secret is used in the following way:
- A
large-arm-protorlarge-x86-protprotected runner picks up a job taggedprotectedfrom a protected GitLab branch. (See Protected Runners and Reserved Tags). - Based on its configuration, the runner creates a job Pod in the pipeline namespace and mounts the
spack-intermediate-ci-signing-keyKubernetes secret into the build container - The Intermediate CI Key, affiliated institutions' public key and the Reputational Public Key are imported into a keyring by the
spack gpg โฆsub-command. This is initiated by the jobโs build script which is created by thegeneratejob at the beginning of the pipeline. - Assuming the package has dependencies those specs are verified using the keyring.
- The package is built and the
spec.jsonis generated - The
spec.jsonis signed by the keyring and uploaded to the mirrorโs build cache.
๐ธ Reputational Key
Because of the increased impact to end users in the case of a private key breach, the Reputational Key is managed separately from the Intermediate CI Keys and has additional controls. First, the Reputational Key was generated outside of Spackโs infrastructure and has been signed by the core development team. The Reputational Key (along with the Signing Reputational Private Key) was then ASCII armor exported to a file. Unlike the Intermediate CI Key this exported file is not stored as a base64 encoded secret in Kubernetes. Instead, the key file itself is encrypted and stored in Kubernetes as the spack-signing-key-encrypted secret in the pipeline namespace.
The encryption of the exported Reputational Key (including the Signing Reputational Private Key) is handled by AWS Key Management Store (KMS) data keys. The private key material is decrypted and imported at the time of signing into a memory mounted temporary directory holding the keychain. The signing job uses the AWS Encryption SDK (i.e. aws-encryption-cli) to decrypt the Reputational Key. Permission to decrypt the key is granted to the job Pod through a Kubernetes service account specifically used for this, and only this, function. Finally, for convenience sake, this same secret contains an ASCII-armored export of the public components of the Intermediate CI Keys and the Reputational Key. This allows the signing script to verify that packages were built by the pipeline (both on AWS or at affiliated institutions), or signed previously as a part of a different pipeline. This is done before importing decrypting and importing the Signing Reputational Private Key material and officially signing the packages.
Procedurally the spack-singing-key-encrypted secret is used in the following way:
- The
spack-package-signing-gitlab-runnerprotected runner picks up a job taggednotaryfrom a protected GitLab branch (See Protected Runners and Reserved Tags). - Based on its configuration, the runner creates a job pod in the pipeline namespace. The job is run in a stripped down purpose-built image
ghcr.io/spack/notary:latestDocker image. The runner is configured to only allow running jobs with this image. - The runner also mounts the
spack-signing-key-encryptedsecret to a path on disk. Note that this becomes several files on disk, the public components of the Intermediate CI Keys, the public components of the Reputational CI, and an AWS KMS encrypted file containing the Singing Reputational Private Key. - In addition to the secret, the runner creates a tmpfs memory mounted directory where the GnuPG keyring will be created to verify, and then resign the package specs.
- The job script syncs all
spec.json.sigfiles from the build cache to a working directory in the jobโs execution environment. - The job script then runs the
sign.shscript built into the notary Docker image. - The
sign.shscript imports the public components of the Reputational and Intermediate CI Keys and uses them to verify good signatures on thespec.json.sigfiles. If any signed spec does not verify the job immediately fails. - Assuming all specs are verified, the
sign.shscript then unpacks the spec json data from the signed file in preparation for being re-signed with the Reputational Key. - The private components of the Reputational Key are decrypted to standard out using
aws-encryption-clidirectly into agpg โimport โฆstatement which imports the key into the keyring mounted in-memory. - The private key is then used to sign each of the json specs and the keyring is removed from disk.
- The re-signed json specs are resynced to the AWS S3 Mirror and the public signing of the packages for the
developor release pipeline that created them is complete.
Non service-account access to the private components of the Reputational Key that are managed through access to the symmetric secret in KMS used to encrypt the data key (which in turn is used to encrypt the GnuPG key - See:Encryption SDK Documentation). A small trusted subset of the core development team are the only individuals with access to this symmetric key.
๐ฆ Protected Runners and Reserved Tags
Spack has a large number of Gitlab Runners operating in its build farm. These include runners deployed in the AWS Kubernetes cluster as well as runners deployed at affiliated institutions. The majority of runners are shared runners that operate across projects in gitlab.spack.io. These runners pick up jobs primarily from the spack/spack project and execute them in PR pipelines.
A small number of runners operating on AWS and at affiliated institutions are registered as specific protected runners on the spack/spack project. In addition to protected runners there are protected branches on the spack/spack project. These are the develop branch, any release branch (i.e. managed with the releases/v* wildcard) and any tag branch (managed with the v* wildcard) Finally Spackโs pipeline generation code reserves certain tags to make sure jobs are routed to the correct runners, these tags are public, protected, and notary. Understanding how all this works together to protect secrets and provide integrity assurances can be a little confusing so lets break these down:
- Protected Branches โ Protected branches in Spack prevent anyone other than Maintainers in GitLab from pushing code. In the case of Spack the only Maintainer level entity pushing code to protected branches is Spack bot. Protecting branches also marks them in such a way that Protected Runners will only run jobs from those branches
- Protected Runners โ Protected Runners only run jobs from protected branches. Because protected runners have access to secrets, it's critical that they not run Jobs from untrusted code (i.e. PR branches). If they did it would be possible for a PR branch to tag a job in such a way that a protected runner executed that job and mounted secrets into a code execution environment that had not been reviewed by Spack maintainers. Note however that in the absence of tagging used to route jobs, public runners could run jobs from protected branches. No secrets would be at risk of being breached because non-protected runners do not have access to those secrets; lack of secrets would, however, cause the jobs to fail.
- Reserved Tags โ To mitigate the issue of public runners picking up protected jobs Spack uses a small set of โreservedโ job tags (Note that these are job tags not git tags). These tags are โ
publicโ, โprivateโ, and โnotary.โ The majority of jobs executed in Spackโs GitLab instance are executed via ageneratejob. Thegeneratejob code systematically ensures that no user defined configuration sets these tags. Instead, thegeneratejob sets these tags based on rules related to the branch where this pipeline originated. If the job is a part of a pipeline on a PR branch it sets thepublictag. If the job is part of a pipeline on a protected branch it sets theprotectedtag. Finally if the job is the package signing job and it is running on a pipeline that is part of a protected branch then it sets thenotarytag.
Protected Runners are configured to only run jobs from protected branches. Only jobs running in pipelines on protected branches are tagged with protected or notary tags. This tightly couples jobs on protected branches to protected runners that provide access to the secrets required to sign the built packages. The secrets can only be accessed via:
- Runners under direct control of the core development team.
- Runners under direct control of trusted maintainers at affiliated institutions.
- By code running the automated pipeline that has been reviewed by the Spack maintainers and judged to be appropriate.
Other attempts (either through malicious intent or incompetence) can at worst grab jobs intended for protected runners which will cause those jobs to fail alerting both Spack maintainers and the core development team.
[1] The Reputational Key has also cross signed core development team keys.
โจ USING EXTERNAL GPU SUPPORT โจ
okay so bestie, here's the tea โ โ a lot of packages ship with a +cuda or +rocm variant. And if you don't configure anything at all, Spack will just happily download and install all the components it needs for you. But likeโฆ maybe you already have GPU libraries on your system and you'd rather not reinstall the whole universe? Valid. ๐
The next few sections are all about using a system installation of GPU libraries instead of building your own from scratch.
Using an External ROCm Installation ๐ง
Spack chops ROCm up into many separate component packages โ it's giving "we divided the friend group into individual people" energy, honestly. Here's an example packages.yaml that pulls together a consistent set of ROCm components so dependent packages can use them:
packages:
all:
compiler: [rocmcc@=5.3.0]
variants: amdgpu_target=gfx90a
hip:
buildable: false
externals:
- spec: hip@5.3.0
prefix: /opt/rocm-5.3.0/hip
hsa-rocr-dev:
buildable: false
externals:
- spec: hsa-rocr-dev@5.3.0
prefix: /opt/rocm-5.3.0/
llvm-amdgpu:
buildable: false
externals:
- spec: llvm-amdgpu@5.3.0
prefix: /opt/rocm-5.3.0/llvm/
comgr:
buildable: false
externals:
- spec: comgr@5.3.0
prefix: /opt/rocm-5.3.0/
hipsparse:
buildable: false
externals:
- spec: hipsparse@5.3.0
prefix: /opt/rocm-5.3.0/
hipblas:
buildable: false
externals:
- spec: hipblas@5.3.0
prefix: /opt/rocm-5.3.0/
rocblas:
buildable: false
externals:
- spec: rocblas@5.3.0
prefix: /opt/rocm-5.3.0/
rocprim:
buildable: false
externals:
- spec: rocprim@5.3.0
prefix: /opt/rocm-5.3.0/rocprim/And this bad boy goes in combination with the following compiler definition ๐ซถ:
compilers:
- compiler:
spec: rocmcc@=5.3.0
paths:
cc: /opt/rocm-5.3.0/bin/amdclang
cxx: /opt/rocm-5.3.0/bin/amdclang++
f77: null
fc: /opt/rocm-5.3.0/bin/amdflang
operating_system: rhel8
target: x86_64Now the important bits, and I'm listing them because they matter, not because I like typing:
- Every one of the listed externals specifies
buildable: false, which forces Spack to use only the externals we defined. No sneaky substitutions, no cap ๐ spack external findcan automatically locate some of thehip/rocmpackages โ but not all of them, and on top of that, not in a way that guarantees a complementary set if you happen to have multiple ROCm installations kicking around. So likeโฆ don't fully trust the autodetect, okay? ๐- The
prefixis the same for several components, but heads up: others require listing one of the subdirectories as a prefix. Read the yaml, bestie, it's all right there.
Using an External CUDA Installation ๐
CUDA is split into fewer components and is genuinely simpler to specify โ lowkey iconic of it:
packages:
all:
variants:
- cuda_arch=70
cuda:
buildable: false
externals:
- spec: cuda@11.0.2
prefix: /opt/cuda/cuda-11.0.2/where /opt/cuda/cuda-11.0.2/lib/ contains libcudart.so. That's it. That's the whole vibe check โ
Using an External OpenGL API ๐
Depending on whether we have a graphics card or not (some of us are just built different, some of us are running on vibes alone ๐ญ), we may choose to use OSMesa or GLX to implement the OpenGL API.
If a graphics card is unavailable, OSMesa is recommended and can typically be built with Spack. Easy. Slay. But if we'd rather use the system GLX tailored to our graphics card, we need to declare it as an external. Here's how you do it:
packages:
libglx:
require: [opengl]
opengl:
buildable: false
externals:
- prefix: /usr/
spec: opengl@4.6Note that prefix has to be the root of both the libraries and the headers โ so you want /usr, not the path to the lib. This is the number one place people trip and I will be there to catch you ๐ซถ. To figure out which spec for opengl is available, use cd /usr/include/GL && grep -Ri gl_version. Chef's kiss. ๐
๐ CONTRIBUTION GUIDE ๐
This guide is for developers or administrators who want to contribute a new package, a feature, or a bugfix to Spack. It assumes you have at least some familiarity with Git VCS and Github โ like, not expert level, just "I've pushed a commit before" level, you're fine ๐.
We'll show you a few examples of contributing workflows and talk about the granularity of pull-requests (PRs). We'll also cover the tests your PR has to pass in order to get accepted into Spack. No pressure, but also: pressure. ๐
First of all โ what even is a PR? Quoting Bitbucket's tutorials:
Pull requests are a mechanism for a developer to notify team members that they have completed a feature. The pull request is more than just a notificationโit's a dedicated forum for discussing the proposed feature.
The important phrase there is completed feature. The changes you propose in a PR should correspond to one feature / bugfix / extension / etc. You can absolutely create PRs with changes relevant to different ideas, but reviewing those gets tedious and error-prone โ reviewers are people too ๐ญ. So if possible, try to follow the one-PR-one-package/feature rule. Periodt.
Branches ๐ฟ
Spack's develop branch has the latest contributions. Nearly all pull requests should start from develop and target develop. That's the main character branch, no cap.
There's also a branch for each major release series. Release branches originate from develop and have tags for each point release in the series. For example, releases/v0.14 has tags for 0.14.0, 0.14.1, 0.14.2, etc. versions of Spack. We backport important bug fixes to these branches, but we do not advance the package versions or make other changes that would change the way Spack concretizes dependencies. Currently, the maintainers manage these branches by cherry-picking from develop. See Releases for more info ๐
Continuous Integration ๐ฅ
Spack uses Github Actions for Continuous Integration testing. Which means: every single time you submit a pull request, a whole series of tests runs to make sure you didn't accidentally introduce any bugs into Spack. Your PR will not be accepted until it passes all of these tests. Harsh? Maybe. Fair? Absolutely. ๐
You can totally just wait for the results after submitting a PR, but we recommend running them locally to speed up the review process โ future you will thank present you ๐ธ
NOTE: Oftentimes, CI will fail for reasons other than a problem with your PR. For example,apt-get,pip, orhomebrewwill fail to download one of the dependencies for the test suite, or a transient bug will cause the unit tests to timeout. If any job fails, click the "Details" link and click on the test(s) that is failing. If it doesn't look like it's failing for reasons related to your PR, you have two options. If you have write permissions for the Spack repository, you should see a "Restart workflow" button on the right-hand side. If not, you can close and reopen your PR to rerun all of the tests. If the same test keeps failing, there may be a problem with your PR. If you notice that every recent PR is failing with the same error message, it may be that an issue occurred with the CI infrastructure or one of Spack's dependencies put out a new release that is causing problems. If this is the case, please file an issue. ๐
We currently test against Python 2.7 and 3.6-3.10 on both macOS and Linux, and we perform 3 types of tests:
Unit Tests ๐งช
Unit tests make sure that core Spack features like fetching or spec resolution are working as expected. Here's the good news, bestie: if your PR only adds new packages or modifies existing ones, there's very little chance your changes could cause the unit tests to fail. ๐ But if you make changes to Spack's core libraries, you should run the unit tests to make sure you didn't break anything. (You don't want to be that person. Nobody wants to be that person.)
Since they test things like fetching from VCS repos, the unit tests require git, mercurial, and subversion to run. Make sure these are installed on your system and can be found in your PATH. All of these can be installed with Spack or with your system package manager โ dealer's choice ๐
To run all of the unit tests, use:
$ spack unit-testThese tests may take several minutes to complete (grab a snack ๐). If you know you're only modifying a single Spack feature, you can run subsets of tests at a time. For example, this would run all the tests in lib/spack/spack/test/architecture.py:
$ spack unit-test lib/spack/spack/test/architecture.pyAnd this would run the test_platform test from that file:
$ spack unit-test lib/spack/spack/test/architecture.py::test_platformWhich lets you develop iteratively: make a change, test that change, make another change, test that change, etc. We use pytest as our test framework, and these types of arguments are just passed to the pytest command underneath. See the pytest docs for more details on test selection syntax. ๐ฆ
spack unit-test has a few special options that can help you understand what tests are available. To get a list of all available unit test files, run:
$ spack unit-test --list
...To see a more detailed list of available unit tests, use spack unit-test --list-long:
$ spack unit-test --list-long
...And to see the fully qualified names of all tests, use --list-names:
$ spack unit-test --list-names
...You can combine these with pytest arguments to restrict which tests you want to know about. For example, to see just the tests in architecture.py:
$ spack unit-test --list-long lib/spack/spack/test/architecture.pyYou can also combine any of these options with a pytest keyword search. See the pytest usage docs for more details on test selection syntax. For example, to see the names of all tests that have "spec" or "concretize" somewhere in their names:
$ spack unit-test --list-names -k "spec and concretize"By default, pytest captures the output of all unit tests, and it will print any captured output for failed tests. But sometimes it's helpful to see your output interactively while the tests run (e.g., if you add print statements to a unit test). To see the output live, use the -s argument to pytest:
$ spack unit-test -s --list-long lib/spack/spack/test/architecture.py::test_platformUnit tests are crucial to making sure bugs aren't introduced into Spack. So if you're modifying core Spack libraries or adding new functionality, please add new unit tests for your feature, and consider strengthening existing tests. You will likely be asked to do this if you submit a pull request to the Spack project on GitHub. Check out the pytest docs and feel free to ask for guidance on how to write tests! Nobody's expecting you to just know this stuff on day one ๐ซถโจ
NOTE: You may notice theshare/spack/qa/run-unit-testsscript in the repository. This script is designed for CI. It runs the unit tests and reports coverage statistics back to Codecov. If you want to run the unit tests yourself, we suggest you usespack unit-test. ๐
โจ Style Tests โจ
Spack runs Flake8 to check for PEP 8 conformance, and mypy <https://mypy.readthedocs.io/en/stable/> for type checking. PEP 8 is basically the Python style bible โ a whole series of style guides covering everything from how you name your variables to how you indent. And like, why do we enforce it? Because otherwise the repo would drown in PRs that are only style changes ๐. So yeah: your PR has to comply with PEP 8 to get accepted, and if it touches the spack library it also needs to successfully type-check with mypy. Both. No cap.
Testing your compliance with spack's style is genuinely easy, bestie. Just run the spack style command:
$ spack stylespack style has a couple of advantages over running the tools by hand:
- It only tests files that you have modified since branching off of
develop. - It works regardless of what directory you are in.
- It automatically adds approved exemptions from the flake8 checks. For example, URLs are often longer than 80 characters, so we exempt them from line length checks. We also exempt lines that start with
"homepage","url","version","variant","depends_on", and"extends"inpackage.pyfiles. This is now also possible when directly running flake8 if you can use thespackformatter plugin included with spack.
More approved flake8 exemptions can be found here.
If all is well, you'll see something like this:
$ run-flake8-tests
Dependencies found.
=======================================================
flake8: running flake8 code checks on spack.
Modified files:
var/spack/repos/builtin/packages/hdf5/package.py
var/spack/repos/builtin/packages/hdf/package.py
var/spack/repos/builtin/packages/netcdf/package.py
=======================================================
Flake8 checks were clean.But ๐ฅ if you aren't compliant with PEP 8, flake8 will absolutely call you out:
var/spack/repos/builtin/packages/netcdf/package.py:26: [F401] 'os' imported but unused
var/spack/repos/builtin/packages/netcdf/package.py:61: [E303] too many blank lines (2)
var/spack/repos/builtin/packages/netcdf/package.py:106: [E501] line too long (92 > 79 characters)
Flake8 found errors.Most of these error messages are pretty straightforward, but if you're sitting there like "what does this even mean ๐ญ", just ask questions about them when you submit your PR. Also โ the line numbers will change if you add or delete lines, so simply run spack style again to update them. Easy.
TIP: Try fixing flake8 errors in reverse order. This eliminates the need for multiple runs of spack style just to re-compute line numbers and makes it much easier to fix errors directly off of the CI output. Reverse order is the move, trust. ๐
โจ Documentation Tests โจ
Spack uses Sphinx to build its documentation. And to prevent things like broken links and missing imports from sneaking in, we added documentation tests that build the docs and fail if there are any warning or error messages. It's giving quality control ๐ .
Building the documentation requires several dependencies:
sphinxsphinxcontrib-programoutputsphinx-rtd-themegraphvizgitmercurialsubversion
All of these can be installed with Spack, e.g.
$ spack install py-sphinx py-sphinxcontrib-programoutput py-sphinx-rtd-theme graphviz git mercurial subversionWARNING: Sphinx has several required dependencies. If you're using apythonfrom Spack and you installedpy-sphinxand friends, you need to make them available to yourpython. The easiest way to do this is to run:$ spack load py-sphinx py-sphinx-rtd-theme py-sphinxcontrib-programoutputso that all of the dependencies are added toPYTHONPATH. If you see an error message like:Extension error: Could not import extension sphinxcontrib.programoutput (exception: No module named sphinxcontrib.programoutput) make: *** [html] Error 1that means Sphinx couldn't findpy-sphinxcontrib-programoutputin yourPYTHONPATH. That's the whole issue, bestie. ๐
Once all of the dependencies are installed, you can try building the documentation:
$ cd path/to/spack/lib/spack/docs/
$ make clean
$ makeIf you see any warning or error messages, you will have to correct those before your PR is accepted. So if you're editing the documentation, you should be running the documentation tests to make sure there are no errors. Heads up: documentation changes can result in some obfuscated warning messages (they're lowkey cryptic ngl). If you don't understand what they mean, feel free to ask when you submit your PR. We got you ๐ซถ.
โจ Coverage โจ
Spack uses Codecov to generate and report unit test coverage. This helps us tell what percentage of lines of code in Spack are covered by unit tests. And yeah โ code covered by unit tests can still contain bugs, but it is much less error prone than code that is not covered by unit tests. Facts.
Codecov provides browser extensions for Google Chrome and Firefox. These extensions integrate with GitHub and allow you to see coverage line-by-line when viewing the Spack repository. So if you are new to Spack, a great way to get started is to write unit tests to increase coverage! Iconic entry point, honestly. โจ
Unlike with CI on Github Actions, Codecov tests are not required to pass in order for your PR to be merged. If you modify core Spack libraries, we would greatly appreciate unit tests that cover these changed lines. Otherwise, we have no way of knowing whether or not your changes introduce a bug. And if you make substantial changes to the core, we may request unit tests to increase coverage.
NOTE: If the only files you modified are package files, we do not care about coverage on your PR. You may notice that the Codecov tests fail even though you didn't modify any core files. This means that Spack's overall coverage has increased since you branched off ofdevelop. This is a good thing! ๐ If you really want to get the Codecov tests to pass, you can rebase off of the latestdevelop, but again, this is not required.
โจ Git Workflows โจ
Spack is still in the beta stages of development. Most of our users run off of the develop branch, and fixes and new features are constantly being merged. So how do you keep up-to-date with upstream while maintaining your own local differences and contributing PRs to Spack? Great question, let's get into it. ๐ฆ
๐ธ Branching ๐ธ
The easiest way to contribute a pull request is to make all of your changes on new branches. Make sure your develop is up-to-date and create a new branch off of it:
$ git checkout develop
$ git pull upstream develop
$ git branch <descriptive_branch_name>
$ git checkout <descriptive_branch_name>Here we assume that the local develop branch tracks the upstream develop branch of Spack. This is not a requirement and you could also do the same with remote branches. But for some it is more convenient to have a local branch that tracks upstream. Whatever vibes with you ๐ซ.
Normally we prefer that commits pertaining to a package <package-name> have a message <package-name>: descriptive message. It is important to add a descriptive message so that others, who might be looking at your changes later (in a year or maybe two), would understand the rationale behind them. Future-you will thank present-you, fr fr. ๐ซถ
Now, you can make your changes while keeping the develop branch pure. Edit a few files and commit them by running:
$ git add <files_to_be_part_of_the_commit>
$ git commit --message <descriptive_message_of_this_particular_commit>Next, push it to your remote fork and create a PR:
$ git push origin <descriptive_branch_name> --set-upstreamGitHub provides a tutorial on how to file a pull request. When you send the request, make develop the destination branch. ๐
If you need this change immediately and don't have time to wait for your PR to be merged, you can always work on this branch. But if you have multiple PRs, another option is to maintain a Frankenstein branch that combines all of your other branches:
$ git co develop
$ git branch <your_modified_develop_branch>
$ git checkout <your_modified_develop_branch>
$ git merge <descriptive_branch_name>This can be done with each new PR you submit. Just make sure to keep this local branch up-to-date with upstream develop too. Don't let it drift ๐.
๐ Cherry-Picking ๐
What if you made some changes to your local modified develop branch and already committed them, but later decided to contribute them to Spack? You can use cherry-picking to create a new branch with only these commits. So clutch.
First, check out your local modified develop branch:
$ git checkout <your_modified_develop_branch>Now, get the hashes of the commits you want from the output of:
$ git logNext, create a new branch off of upstream develop and copy the commits that you want in your PR:
$ git checkout develop
$ git pull upstream develop
$ git branch <descriptive_branch_name>
$ git checkout <descriptive_branch_name>
$ git cherry-pick <hash>
$ git push origin <descriptive_branch_name> --set-upstreamNow you can create a PR from the web-interface of GitHub. The net result is as follows:
- You patched your local version of Spack and can use it further.
- You "cherry-picked" these changes in a stand-alone branch and submitted it as a PR upstream.
Should you have several commits to contribute, you could follow the same procedure by getting hashes of all of them and cherry-picking to the PR branch. Multiple cherries, no problem ๐.
NOTE: It is important that whenever you change something that might be of importance upstream, create a pull request as soon as possible. Do not wait for weeks/months to do this, because: 1. you might forget why you modified certain files 2. it could get difficult to isolate this change into a stand-alone clean PR.
๐ฆ Rebasing ๐ฆ
Other developers are constantly making contributions to Spack, possibly on the same files that your PR changed. If their PR is merged before yours, it can create a merge conflict. This means that your PR can no longer be automatically merged without a chance of breaking your changes. ๐ญ the dreaded conflict. In this case, you will be asked to rebase on top of the latest upstream develop.
First, make sure your develop branch is up-to-date:
$ git checkout develop
$ git pull upstream developNow, we need to switch to the branch you submitted for your PR and rebase it on top of develop:
$ git checkout <descriptive_branch_name>
$ git rebase developGit will likely ask you to resolve conflicts. Edit the file that it says can't be merged automatically and resolve the conflict. Then, run:
$ git add <file_that_could_not_be_merged>
$ git rebase --continueYou may have to repeat this process multiple times until all conflicts are resolved. Once this is done, simply force push your rebased branch to your remote fork:
$ git push --force origin <descriptive_branch_name>And that's the rebase glow-up ๐ โจ
๐ Rebasing with cherry-pick ๐
okay so you can also rebase with cherry-pick if that's more your vibe. bestie, first step: make a lil safety net. create a temporary backup branch:
$ git checkout <descriptive_branch_name>
$ git branch tmpif anything goes sideways you can always go back to your tmp branch. iconic, honestly. now peek at the logs and save the hashes of any commits you wanna keep:
$ git lognext, head back to the original branch and reset it to develop. but before you do that, make sure your local develop branch is up-to-date with upstream:
$ git checkout develop
$ git pull upstream develop
$ git checkout <descriptive_branch_name>
$ git reset --hard developnow go ahead and cherry-pick the relevant commits:
$ git cherry-pick <hash1>
$ git cherry-pick <hash2>push the modified branch to your fork:
$ git push --force origin <descriptive_branch_name>and if everything looks good, delete the backup branch:
$ git branch --delete --force tmp๐ฐ๏ธ Re-writing History ๐ฐ๏ธ
sometimes you end up on a branch that's diverged sooo much from develop that it literally cannot be rebased easily. if the current commit history is more of an experimental nature and only the net result matters, you're allowed to rewrite the history. no cap.
first, merge upstream develop and reset your branch to it. on the branch in question, run:
$ git merge develop
$ git reset developat this point your branch points to the same commit as develop, so the two are indistinguishable. but โ and this is the slay part โ all the files you previously modified stay exactly as they were. in other words, you do not lose the changes you made. review them by looking at diffs:
$ git status
$ git diffnext step is rewriting the history by adding files and creating commits:
$ git add <files_to_be_part_of_commit>
$ git commit --message <descriptive_message>after all the changed files are committed, push the branch to your fork and create a PR:
$ git push origin --set-upstream๐ PACKAGING GUIDE ๐
this guide is for developers or administrators who wanna package software so that Spack can install it ๐. it assumes you have at least some familiarity with Python, and that you've read the basic usage guide โ especially the part about specs.
there are two key parts of Spack:
- Specs: expressions for describing builds of software, and
- Packages: Python modules that describe how to build and test software according to a spec.
Specs let a user describe a particular build in a way a package author can understand. Packages let the packager encapsulate the build logic for different versions, compilers, options, platforms, and dependency combinations in one place. essentially, a package translates a spec into build logic. it also lets the packager write spec-specific tests of the installed software โจ.
Packages in Spack are written in pure Python, so you can do anything in Spack that you can do in Python. Python was chosen as the implementation language for two reasons. First, Python is becoming ubiquitous in the scientific software community. Second, it's a modern language with many powerful features that help make package writing easy. periodt.
โ ๏ธ WARNING: as a general rule, packages should install the software from source. the only exception is for proprietary software (e.g., vendor compilers). if a special build system needs to be added in order to support building a package from source, then the associated code and recipe should be added first. ๐
๐ง Overview of the installation procedure ๐ง
whenever Spack installs software, it goes through a series of predefined steps:
[image]
all these steps are influenced by the metadata in each package.py and by the current Spack configuration. since build systems differ from one another, the execution of the last block in the figure is expanded further in a build-system-specific way. an example for CMake is, for instance:
[image]
the predefined steps for each build system are called "phases". in general, you can get the name and order in which the phases will be executed either by reading the API docs at build_systems, or by using the spack info command:
$ spack info --phases m4
AutotoolsPackage: m4
Homepage: https://www.gnu.org/software/m4/m4.html
Safe versions:
1.4.17 ftp://ftp.gnu.org/gnu/m4/m4-1.4.17.tar.gz
Variants:
Name Default Description
sigsegv on Build the libsigsegv dependency
Installation Phases:
autoreconf configure build install
Build Dependencies:
libsigsegv
...an extensive list of available build systems and phases is provided in Overriding build system defaults.
๐ Writing a package recipe ๐
since v0.19, Spack supports two ways of writing a package recipe. the most commonly used one is to encode both the metadata (directives, etc.) and the build behavior in a single class, like in this example:
class Openjpeg(CMakePackage):
"""OpenJPEG is an open-source JPEG 2000 codec written in C language"""
homepage = "https://github.com/uclouvain/openjpeg"
url = "https://github.com/uclouvain/openjpeg/archive/v2.3.1.tar.gz"
version("2.4.0", sha256="8702ba68b442657f11aaeb2b338443ca8d5fb95b0d845757968a7be31ef7f16d")
variant("codec", default=False, description="Build the CODEC executables")
depends_on("libpng", when="+codec")
def url_for_version(self, version):
if version >= Version("2.1.1"):
return super().url_for_version(version)
url_fmt = "https://github.com/uclouvain/openjpeg/archive/version.{0}.tar.gz"
return url_fmt.format(version)
def cmake_args(self):
args = [
self.define_from_variant("BUILD_CODEC", "codec"),
self.define("BUILD_MJ2", False),
self.define("BUILD_THIRDPARTY", False),
]
return argsa package encoded with a single class is backward compatible with versions of Spack lower than v0.19, and so are custom repositories containing only recipes of this kind ๐ซถ. the downside? this format doesn't let packagers use more than one build system in a single recipe. sad but true.
to do that, we gotta resort to the second way Spack has of writing packages, which involves writing a builder class explicitly. using the same example as above, it reads:
class Openjpeg(CMakePackage):
"""OpenJPEG is an open-source JPEG 2000 codec written in C language"""
homepage = "https://github.com/uclouvain/openjpeg"
url = "https://github.com/uclouvain/openjpeg/archive/v2.3.1.tar.gz"
version("2.4.0", sha256="8702ba68b442657f11aaeb2b338443ca8d5fb95b0d845757968a7be31ef7f16d")
variant("codec", default=False, description="Build the CODEC executables")
depends_on("libpng", when="+codec")
def url_for_version(self, version):
if version >= Version("2.1.1"):
return super().url_for_version(version)
url_fmt = "https://github.com/uclouvain/openjpeg/archive/version.{0}.tar.gz"
return url_fmt.format(version)
class CMakeBuilder(spack.build_systems.cmake.CMakeBuilder):
def cmake_args(self):
args = [
self.define_from_variant("BUILD_CODEC", "codec"),
self.define("BUILD_MJ2", False),
self.define("BUILD_THIRDPARTY", False),
]
return argsthis way of writing packages lets you extend the recipe to support multiple build systems โ see Multiple build systems for the deets. the downside is that recipes of this kind are only understood by Spack since v0.19+. more info on Spack's internal architecture lives at Package class architecture ๐ธ.
๐ก NOTE: if a builder is implemented inpackage.py, all build-specific methods must be moved to the builder. that means if you have a package likepython class Foo(CmakePackage): def cmake_args(self): ...and you add a builder to thepackage.py, you must movecmake_argsto the builder. no exceptions bestie ๐
๐ฑ Creating new packages ๐ฑ
to help you create a new package, Spack provides a command that generates a package.py file in an existing repository, with a boilerplate package template. here's an example:
$ spack create https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2Spack examines the tarball URL and tries to figure out the name of the package to be created. if the name contains uppercase letters, those are automatically converted to lowercase. if the name contains underscores or periods, those are automatically converted to dashes โจ.
Spack also searches for additional versions located in the same directory of the website. it prompts you to tell you how many versions it found and asks how many you'd like to download and checksum:
$ spack create https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2
==> This looks like a URL for gmp
==> Found 16 versions of gmp:
6.1.2 https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2
6.1.1 https://gmplib.org/download/gmp/gmp-6.1.1.tar.bz2
6.1.0 https://gmplib.org/download/gmp/gmp-6.1.0.tar.bz2
...
5.0.0 https://gmplib.org/download/gmp/gmp-5.0.0.tar.bz2
How many would you like to checksum? (default is 1, q to abort)Spack will automatically download the number of tarballs you specify (starting with the most recent) and checksum each of them ๐ฅ.
you do not have to download all of the versions up front. you can always choose to download just one tarball initially, and run spack checksum later if you need more versions. lowkey the move.
Spack automatically creates a directory in the appropriate repository, generates a boilerplate template for your package, and opens up the new package.py in your favorite $EDITOR (see Controlling the editor for details):
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)# ----------------------------------------------------------------------------
# If you submit this package back to Spack as a pull request,
# please first remove this boilerplate and all FIXME comments.
#
# This is a template package file for Spack. We've put "FIXME"
# next to all the things you'll want to change. Once you've handled
# them, you can save this file and test your package like this:
#
# spack install gmp
#
# You can edit this file again by typing:
#
# spack edit gmp
#
# See the Spack documentation for more information on packaging.
# ----------------------------------------------------------------------------
import spack.build_systems.autotools
from spack.package import *
class Gmp(AutotoolsPackage):
"""FIXME: Put a proper description of your package here."""
# FIXME: Add a proper url for your package's homepage here.
homepage = "https://www.example.com"
url = "https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2"
# FIXME: Add a list of GitHub accounts to
# notify when the package is updated.
# maintainers("github_user1", "github_user2")
version("6.2.1", sha256="eae9326beb4158c386e39a356818031bd28f3124cf915f8c5b1dc4c7a36b4d7c")
# FIXME: Add dependencies if required.
# depends_on("foo")
def configure_args(self):
# FIXME: Add arguments other than --prefix
# FIXME: If not needed delete the function
args = []
return argsSo yeah ๐ โ the tedious stuff (creating the class, checksumming archives) has literally already been done for you. Spack correctly clocked that gmp uses the autotools build system, so it spun up a brand new Gmp package that subclasses the AutotoolsPackage base class. Slay.
The default installation procedure for a package subclassing the AutotoolsPackage is to go through the typical process of:
./configure --prefix=/path/to/installation/directory
make
make check
make installFor most Autotools packages, that's honestly sufficient โ ate, no notes. If you need to add additional arguments to the ./configure call, just add them via the configure_args function. ๐ซถ
In the generated package, the download url attribute is already set. All the things you still need to change are marked with FIXME labels. You can delete the commented instructions between the license and the first import statement after reading them (read them first bestie, please ๐ญ). The rest of the tasks you need to do are as follows:
- Add a description.
Immediately inside the package class is a docstring in triple-quotes ("""). It is used to generate the description shown when users run spack info.
- Change the homepage to a useful URL.
The homepage is displayed when users run spack info so that they can learn more about your package.
- Add a comma-separated list of maintainers.
Add a list of Github accounts of people who want to be notified any time the package is modified. See Maintainers.
- Add
depends_on()calls for the package's dependencies.
depends_on tells Spack that other packages need to be built and installed before this one. See Dependencies.
- Get the installation working.
Your new package may require specific flags during configure. These can be added via configure_args. Specifics will differ depending on the package and its build system. Overriding build system defaults is covered in detail later.
๐ Controlling the editor ๐
When Spack needs to open an editor for you (e.g., for commands like Creating new packages or Editing existing packages), it looks at several environment variables to figure out what to use. The order of precedence is:
SPACK_EDITOR: highest precedence, in case you want something specific for Spack;VISUAL: standard environment variable for full-screen editors likevimoremacs;EDITOR: older environment variable for your editor.
You can set any of these to the command you want to run, e.g., in bash you might run one of these:
export VISUAL=vim
export EDITOR="emacs -nw"
export SPACK_EDITOR=nanoIf Spack finds none of these variables set, it will look for vim, vi, emacs, nano, and notepad, in that order. (yes, notepad is invited to the party too ๐)
๐ง Bundling software ๐ง
If you have a collection of software expected to work well together with no source code of its own, you can create a BundlePackage. Examples where bundle packages can be useful include defining suites of applications (e.g., EcpProxyApps), commonly used libraries (e.g., AmdAocl), and software development kits (e.g., EcpDataVisSdk). Obsessed with a good bundle, fr fr.
These versioned packages primarily consist of dependencies on the associated software packages. They can include variants to ensure common build options are consistently applied to dependencies. Known build failures โ such as not building on a platform, or when certain compilers or variants are used โ can be flagged with conflicts. Build requirements, such as only building with specific compilers, can similarly be flagged with requires.
The spack create --template bundle command will create a skeleton BundlePackage package.py for you:
$ spack create --template bundle --name coolsdkNow you can fill in the basic package documentation, version(s), and software package dependencies along with any other relevant customizations. Main character packaging energy ๐
NOTE: Remember that bundle packages have no software of their own so there is nothing to download.
๐ Non-downloadable software ๐
If your software cannot be downloaded from a URL you can still create a boilerplate package.py by telling spack create what name you want to use:
$ spack create --name intelThis will create a simple intel package with an install() method that you can craft to install your package. Likewise, you can force the build system to be used with --template and, in case it's needed, you can overwrite a package already in the repository with --force:
$ spack create --name gmp https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2
$ spack create --force --template autotools https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2A complete list of available build system templates can be found by running spack create --help. Go check it, iykyk ๐ธ
๐ Editing existing packages ๐
One of the easiest ways to learn how to write packages is to look at existing ones. You can edit a package file by name with the spack edit command:
$ spack edit gmpIf you used spack create to create a package, you can get back to it later with spack edit. For instance, the gmp package actually lives in:
$ spack location -p gmp
${SPACK_ROOT}/var/spack/repos/builtin/packages/gmp/package.pybut spack edit provides a much simpler shortcut and saves you the trouble of typing the full path. Your wrists will thank you ๐ซถ
๐๏ธ Naming & directory structure ๐๏ธ
This section describes how packages need to be named, and where they live in Spack's directory structure. In general, Creating new packages handles creating package files for you, so you can skip most of the details here.
๐ var/spack/repos/builtin/packages ๐
A Spack installation directory is structured like a standard UNIX install prefix (bin, lib, include, var, opt, etc.). Most of the code for Spack lives in $SPACK_ROOT/lib/spack. Packages themselves live in /usr/share/spack/repos/builtin/packages.
If you cd to that directory, you will see directories for each package:
$ cd /usr/share/spack/repos/builtin/packages && ls
3dtk
3proxy
7zip
abacus
abduco
abi-compliance-checker
abi-dumper
abinit
abseil-cpp
abyss
...Each directory contains a file called package.py, which is where all the python code for the package goes. For example, the libelf package lives in:
/usr/share/spack/repos/builtin/packages/libelf/package.pyAlongside the package.py file, a package may contain extra directories or files (like patches) that it needs to build.
โจ Package Names โจ
Packages are named after the directory containing package.py. So, libelf's package.py lives in a directory called libelf. The package.py file defines a class called Libelf, which extends Spack's Package class. For example, here is /usr/share/spack/repos/builtin/packages/libelf/package.py:
from spack import *
class Libelf(Package):
""" ... description ... """
homepage = ...
url = ...
version(...)
depends_on(...)
def install():
...The directory name (libelf) determines the package name that users should provide on the command line. e.g., if you type any of these:
$ spack info libelf
$ spack versions libelf
$ spack install libelf@0.8.13Spack sees the package name in the spec and looks for libelf/package.py in var/spack/repos/builtin/packages. Likewise, if you run spack install py-numpy, Spack looks for py-numpy/package.py.
Spack uses the directory name as the package name in order to give packagers more freedom in naming their packages. Package names can contain letters, numbers, and dashes. Using a Python identifier (e.g., a class name or a module name) would make it difficult to support these options. So, you can name a package 3proxy or foo-bar and Spack won't care. It just needs to see that name in the packages directory. Periodt.
๐ฆ Package class names ๐ฆ
Spack loads package.py files dynamically, and it needs to find a special class name in the file for the load to succeed. The class name (Libelf in our example) is formed by converting words separated by - in the file name to CamelCase. If the name starts with a number, we prefix the class name with _. Here are some examples:
- Module Name
foo-barโ Class NameFooBar - Module Name
3proxyโ Class Name_3proxy
In general, you won't have to remember this naming convention because Creating new packages and Editing existing packages handle the details for you. Iconic, honestly โ you get to skip the homework ๐
๐ Maintainers ๐
Each package in Spack may have one or more maintainers, i.e. one or more GitHub accounts of people who want to be notified any time the package is modified.
To add maintainers to a package, you literally just declare them with the maintainers directive:
maintainers("user1", "user2")Bestie, the maintainer list is additive โ it includes all the accounts eventually declared in base classes too. So nobody gets left out. ๐ซถ
โจ Trusted Downloads โจ
Okay so here's the vibe: Spack wants to verify that the source code it downloads is not corrupted or compromised โ or at least, that it's the same version the author of the Spack package saw when the package was created. If Spack uses a download method it can verify, we say the download method is trusted. ๐ฅ
Trust matters for all downloads, fr fr: Spack has no control over the security of the various sites from which it downloads source code, and it can never assume that any particular site hasn't been compromised. ๐ญ
Trust gets established in different ways for different download methods. For the most common download method โ a single-file tarball โ the tarball is checksummed. Git downloads using commit= are trusted implicitly, as long as a hash is specified. ๐
Spack also provides untrusted download methods: tarball URLs may be supplied without a checksum, or Git downloads may specify a branch or tag instead of a hash. If the user does not control or trust the source of an untrusted download, it is a security risk. Unless otherwise specified by the user for special cases, Spack should by default use only trusted download methods. ๐จ
Unfortunately, Spack does not currently provide that guarantee. It does provide the following mechanisms for safety:
- By default, Spack will only install a tarball package if it has a checksum and that checksum matches. You can override this with
spack install --no-checksum. - Numeric versions are almost always tarball downloads, whereas non-numeric versions not named
developfrequently download untrusted branches or tags from a version control system. As long as a package has at least one numeric version, and no non-numeric version nameddevelop, Spack will prefer it over any non-numeric versions.
๐ Checksums ๐
For tarball downloads, Spack can currently support checksums using the MD5, SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 algorithms. It determines the algorithm to use based on the hash length. Slay. โญ
๐ธ Versions and fetching ๐ธ
The most straightforward way to add new versions to your package is to add a line like this in the package class:
class Foo(Package):
url = "http://example.com/foo-1.0.tar.gz"
version("8.2.1", md5="4136d7b4c04df68b686570afa26988ac")
version("8.2.0", md5="1c9f62f0778697a09d36121ead88e08e")
version("8.1.2", md5="d47dd09ed7ae6e7fd6f9a816d7f5fdf6")NOTE: By convention, we list versions in descending order, from newest to oldest.
NOTE: Bundle packages do not have source code so there is nothing to fetch. Consequently, their version directives consist solely of the version name (e.g., version("202309")).
๐๏ธ Date Versions ๐๏ธ
If you wish to use dates as versions, it is best to use the format @yyyy-mm-dd. This will ensure they sort in the correct order. ๐
Alternately, you might use a hybrid release-version / date scheme. For example, @1.3_2016-08-31 would mean the version from the 1.3 branch, as of August 31, 2016. Iconic.
๐ Version URLs ๐
By default, each version's URL is extrapolated from the url field in the package. For example, Spack is smart enough to download version 8.2.1 of the Foo package above from http://example.com/foo-8.2.1.tar.gz. โจ
If the URL is particularly complicated or changes based on the release, you can override the default URL generation algorithm by defining your own url_for_version() function. For example, the download URL for OpenMPI contains the major.minor version in one spot and the major.minor.patch version in another:
https://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.1.tar.bz2In order to handle this, you can define a url_for_version() function like so:
def url_for_version(self, version):
url = "https://download.open-mpi.org/release/open-mpi/v{0}/openmpi-{1}.tar.bz2"
return url.format(version.up_to(2), version)With the use of this url_for_version(), Spack knows to download OpenMPI 2.1.1 from
http://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.1.tar.bz2but download OpenMPI 1.10.7 from
http://www.open-mpi.org/software/ompi/v1.10/downloads/openmpi-1.10.7.tar.bz2You'll notice that OpenMPI's url_for_version() function makes use of a special Version function called up_to(). When you call version.up_to(2) on a version like 1.10.0, it returns 1.10. version.up_to(1) would return 1. This can be very useful for packages that place all X.Y.* versions in a single directory and then places all X.Y.Z versions in a sub-directory. ๐ง
There are a few Version properties you should be aware of. We generally prefer numeric versions to be separated by dots for uniformity, but not all tarballs are named that way. For example, icu4c separates its major and minor versions with underscores, like icu4c-57_1-src.tgz. The value 57_1 can be obtained with the use of the version.underscored property. Note that Python properties don't need parentheses. There are other separator properties as well:
version.dottedโ1.2.3version.dashedโ1-2-3version.underscoredโ1_2_3version.joinedโ123
NOTE: Python properties don't need parentheses.version.dashedis correct.version.dashed()is incorrect. Not me making that mistakeโฆ ๐
In addition, these version properties can be combined with up_to(). For example:
>>> version = Version("1.2.3")
>>> version.up_to(2).dashed
Version("1-2")
>>> version.underscored.up_to(2)
Version("1_2")As you can see, order is not important. Just keep in mind that up_to() and the other version properties return Version objects, not strings. ๐
If a URL cannot be derived systematically, or there is a special URL for one of its versions, you can add an explicit URL for a particular version:
version("8.2.1", md5="4136d7b4c04df68b686570afa26988ac",
url="http://example.com/foo-8.2.1-special-version.tar.gz")When you supply a custom URL for a version, Spack uses that URL verbatim and does not perform extrapolation. The order of precedence of these methods is:
- package-level
url url_for_version()- version-specific
url
so if your package contains a url_for_version(), it can be overridden by a version-specific url.
If your package does not contain a package-level url or url_for_version(), Spack can determine which URL to download from even if only some of the versions specify their own url. Spack will use the nearest URL before the requested version. This is useful for packages that have an easy to extrapolate URL, but keep changing their URL format every few releases. With this method, you only need to specify the url when the URL changes. ๐
๐ช Mirrors of the main URL ๐ช
Spack supports listing mirrors of the main URL in a package by defining the urls attribute:
class Foo(Package):
urls = [
"http://example.com/foo-1.0.tar.gz",
"http://mirror.com/foo-1.0.tar.gz"
]instead of just a single url. This attribute is a list of possible URLs that will be tried in order when fetching packages. Notice that either one of url or urls can be present in a package, but not both at the same time. ๐
โโ๏ธ
A well-known case of packages that can be fetched from multiple mirrors is that of GNU. For that, Spack goes a step further and defines a mixin class that takes care of all of the plumbing and requires packagers to just define a proper gnu_mirror_path attribute:
class Autoconf(AutotoolsPackage, GNUMirrorPackage):
"""Autoconf -- system configuration part of autotools"""
homepage = "https://www.gnu.org/software/autoconf/"
gnu_mirror_path = "autoconf/autoconf-2.69.tar.gz"
license("GPL-3.0-or-later WITH Autoconf-exception-3.0", when="@2.62:", checked_by="tgamblin")
license("GPL-2.0-or-later WITH Autoconf-exception-2.0", when="@:2.59", checked_by="tgamblin")๐ฆ Skipping the expand step ๐ฆ
Spack normally expands archives (e.g. *.tar.gz and *.zip) automatically into a standard stage source directory (self.stage.source_path) after downloading them. If you want to skip this step (e.g., for self-extracting executables and other custom archive types), you can add expand=False to a version directive.
version("8.2.1", md5="4136d7b4c04df68b686570afa26988ac",
url="http://example.com/foo-8.2.1-special-version.sh", expand=False)When expand is set to False, Spack sets the current working directory to the directory containing the downloaded archive before it calls your install method. Within install, the path to the downloaded archive is available as self.stage.archive_file. ๐ฆ
Here is an example snippet for packages distributed as self-extracting archives. The example sets permissions on the downloaded file to make it executable, then runs it with some arguments.
def install(self, spec, prefix):
set_executable(self.stage.archive_file)
installer = Executable(self.stage.archive_file)
installer("--prefix=%s" % prefix, "arg1", "arg2", "etc.")๐๏ธ Deprecating old versions ๐๏ธ
There are many reasons to remove old versions of software:
- Security vulnerabilities (most serious reason)
- Changing build systems that increase package complexity
- Changing dependencies/patches/resources/flags that increase package complexity
- Maintainer/developer inability/unwillingness to support old versions
- No longer available for download (right to be forgotten)
- Package or version rename
But likeโฆ there are so many reasons to keep old versions around too, bestie:
- Reproducibility
- Requirements for older packages (e.g. some packages still rely on Qt 3)
So the vibe is: in general, you should not remove old versions from a package.py. Instead, you deprecate them first, using this syntax:
version("1.2.3", sha256="...", deprecated=True)This has two effects, and they're both kind of iconic. First, spack info will no longer advertise that version. Second, commands like spack install that fetch the package will require user approval:
$ spack install openssl@1.0.1e
==> Warning: openssl@1.0.1e is deprecated and may be removed in a future Spack release.
==> Fetch anyway? [y/N]And if you use spack install --deprecated, this check can be skipped. ๐
This also applies to package recipes that are renamed or removed. You should first deprecate all versions before removing a package. If you need to rename it, you can deprecate the old package and create a new package at the same time.
Version deprecations should always last at least one Spack minor release cycle before the version is completely removed. For example, if a version is deprecated in Spack 0.16.0, it should not be removed until Spack 0.17.0. No version should be removed without such a deprecation process โ no cap, that's the rule. This gives users a chance to complain about the deprecation in case the old version is needed for some application. If you require a deprecated version of a package, simply submit a PR to remove deprecated=True from the package. However, you may be asked to help maintain this version of the package if the current maintainers are unwilling to support this older version. ๐ซถ
๐ธ Download caching ๐ธ
Spack maintains a cache (described here) which saves files retrieved during package installations to avoid re-downloading in the case that a package is installed with a different specification (but the same version) or reinstalled on account of a change in the hashing scheme. It may (rarely) be necessary to avoid caching for a particular version by adding no_cache=True as an option to the version() directive. Example situations would be a "snapshot"-like Version Control System (VCS) tag, a VCS branch such as v6-16-00-patches, or a URL specifying a regularly updated snapshot tarball. ๐ฆ
๐ Version comparison ๐
Spack imposes a generic total ordering on the set of versions, independently from the package they are associated with.
Most Spack versions are numeric, a tuple of integers; for example, 0.1, 6.96 or 1.2.3.1. In this very basic case, version comparison is lexicographical on the numeric components: 1.2 < 1.2.1 < 1.2.2 < 1.10. Slay, math works.
Spack can also supports string components such as 1.1.1a and 1.y.0. String components are considered less than numeric components, so 1.y.0 < 1.0. This is for consistency with RPM. String components do not have to be separated by dots or any other delimiter. So, the contrived version 1y0 is identical to 1.y.0. Literally the same thing, bestie.
Pre-release suffixes also contain string parts, but they are handled in a special way. For example 1.2.3alpha1 is parsed as a pre-release of the version 1.2.3. This allows Spack to order it before the actual release: 1.2.3alpha1 < 1.2.3. Spack supports alpha, beta and release candidate suffixes: 1.2alpha1 < 1.2beta1 < 1.2rc1 < 1.2. Any suffix not recognized as a pre-release is treated as an ordinary string component, so 1.2 < 1.2-mysuffix.
Finally, there are a few special string components that are considered "infinity versions". They include develop, main, master, head, trunk, and stable. For example: 1.2 < develop. These are useful for specifying the most recent development version of a package (often a moving target like a git branch), without assigning a specific version number. Infinity versions are not automatically used when determining the latest version of a package unless explicitly required by another package or user. ๐
More formally, the order on versions is defined as follows. A version string is split into a list of components based on delimiters such as . and - and string boundaries. The components are split into the release and a possible pre-release (if the last component is numeric and the second to last is a string alpha, beta or rc). The release components are ordered lexicographically, with comparsion between different types of components as follows:
- The following special strings are considered larger than any other numeric or non-numeric version component, and satisfy the following order between themselves:
develop>main>master>head>trunk>stable. - Numbers are ordered numerically, are less than special strings, and larger than other non-numeric components.
- All other non-numeric components are less than numeric components, and are ordered alphabetically.
Finally, if the release components are equal, the pre-release components are used to break the tie, in the obvious way.
The logic behind this sort order is two-fold:
- Non-numeric versions are usually used for special cases while developing or debugging a piece of software. Keeping most of them less than numeric versions ensures that Spack chooses numeric versions by default whenever possible.
- The most-recent development version of a package will usually be newer than any released numeric versions. This allows the
@developversion to satisfy dependencies likedepends_on(abc, when="@x.y.z:")
๐ Version selection ๐
When concretizing, many versions might match a user-supplied spec. For example, the spec python matches all available versions of the package python. Similarly, python@3: matches all versions of Python 3 and above. Given a set of versions that match a spec, Spack concretization uses the following priorities to decide which one to use:
- If the user provided a list of versions in
packages.yaml, the first matching version in that list will be used. - If one or more versions is specified as
preferred=True, in eitherpackages.yamlorpackage.py, the largest matching version will be used. ("Latest" is defined by the sort order above). - If no preferences in particular are specified in the package or in
packages.yaml, then the largest matching non-developversion will be used. By avoiding@develop, this prevents users from accidentally installing a@developversion. ๐ญ - If all else fails and
@developis the only matching version, it will be used.
๐ Ranges versus specific versions ๐
When specifying versions in Spack using the pkg@<specifier> syntax, you can use either ranges or specific versions. It is generally recommended to use ranges instead of specific versions when packaging to avoid overly constraining dependencies, patches, and conflicts.
For example, depends_on("python@3") denotes a range of versions, allowing Spack to pick any 3.x.y version for Python, while depends_on("python@=3.10.1") restricts it to a specific version.
Specific @= versions should only be used in exceptional cases, such as when the package has a versioning scheme that omits the zero in the first patch release: 3.1, 3.1.1, 3.1.2. In this example, the specifier @=3.1 is the correct way to select only the 3.1 version, whereas @3.1 would match all those versions.
Ranges are preferred even if they would only match a single version defined in the package. This is because users can define custom versions in packages.yaml that typically include a custom suffix. For example, if the package defines the version 1.2.3, the specifier @1.2.3 will also match a user-defined version 1.2.3-custom. โจ
โญ spack checksum โญ
If you want to add new versions to a package you've already created, this is automated with the spack checksum command. Here's an example for libelf:
$ spack checksum libelf
==> Found 16 versions of libelf.
0.8.13 http://www.mr511.de/software/libelf-0.8.13.tar.gz
0.8.12 http://www.mr511.de/software/libelf-0.8.12.tar.gz
0.8.11 http://www.mr511.de/software/libelf-0.8.11.tar.gz
0.8.10 http://www.mr511.de/software/libelf-0.8.10.tar.gz
0.8.9 http://www.mr511.de/software/libelf-0.8.9.tar.gz
0.8.8 http://www.mr511.de/software/libelf-0.8.8.tar.gz
0.8.7 http://www.mr511.de/software/libelf-0.8.7.tar.gz
0.8.6 http://www.mr511.de/software/libelf-0.8.6.tar.gz
0.8.5 http://www.mr511.de/software/libelf-0.8.5.tar.gz
...
0.5.2 http://www.mr511.de/software/libelf-0.5.2.tar.gz
How many would you like to checksum? (default is 1, q to abort)This does the same thing that spack create does, but it allows you to go back and add new versions easily as you need them (e.g., as they're released). It fetches the tarballs you ask for and prints out a list of version commands ready to copy/paste into your package file:
==> Checksummed new versions of libelf:
version("0.8.13", md5="4136d7b4c04df68b686570afa26988ac")
version("0.8.12", md5="e21f8273d9f5f6d43a59878dc274fec7")
version("0.8.11", md5="e931910b6d100f6caa32239849947fbf")
version("0.8.10", md5="9db4d36c283d9790d8fa7df1f4d7b4d9")By default, Spack will search for new tarball downloads by scraping the parent directory of the tarball you gave it. So, if your tarball is at http://example.com/downloads/foo-1.0.tar.gz, Spack will look in http://example.com/downloads/ for links to additional versions. If you need to search another path for download links, you can supply some extra attributes that control how your package finds new versions. See the documentation on list_url and list_depth. ๐
NOTE: - This command assumes that Spack can extrapolate new URLs from an existing URL in the package, and that Spack can find similar URLs on a webpage. If that's not possible, e.g. if the package's developers don't name their tarballs consistently, you'll need to manually addversion()calls yourself. - Forspack checksumto work, Spack needs to be able to import your package in Python. That means it can't have any syntax errors, or the import will fail. Use this once you've got your package in working order. ๐
๐ง Finding new versions ๐ง
You've already seen the homepage and url package attributes:
from spack import *
class Mpich(Package):
"""MPICH is a high performance and widely portable implementation of
the Message Passing Interface (MPI) standard."""
homepage = "http://www.mpich.org"
url = "http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz"These are the class-level attributes that Spack peeks at to show users info about a package and to figure out where to download its source from. ๐
Spack takes that tarball url and extrapolates where the other tarballs of the same package live (think spack checksum), but bestie, that does not always work. So this section is all about the ways you can tell Spack to go find tarballs somewhere else. ๐ง
๐ list_url
When spack is out here trying to find available versions of a package (like with spack checksum), it spiders the parent directory of the tarball in the url attribute. For example, for libelf, the url is:
url = "http://www.mr511.de/software/libelf-0.8.13.tar.gz"So here, Spack spiders http://www.mr511.de/software/ to find similar tarball links and ultimately build a list of available libelf versions. ๐ฆ
For a lot of packages though, the tarball's parent directory is straight-up unlistable, or it just doesn't contain any links to source code archives. And honestly? Plenty of times the extra package downloads aren't even in the same directory as the download URL. ๐ญ
For those, you can specify a separate list_url telling Spack which page to search for tarballs. Like, libdwarf uses the homepage as its list_url, because that's where the links to old versions hang out:
class Libdwarf(Package):
homepage = "http://www.prevanders.net/dwarf.html"
url = "http://www.prevanders.net/libdwarf-20130729.tar.gz"
list_url = homepage๐ list_depth
libdwarf and lots of other packages have a listing of available versions on a single webpage, but not everybody does. Take mpich โ its tarball URL looks like this:
url = "http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz"Except its downloads are spread across many different subdirectories of http://www.mpich.org/static/downloads/. So we need to add a list_url and a list_depth attribute, no cap:
class Mpich(Package):
homepage = "http://www.mpich.org"
url = "http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz"
list_url = "http://www.mpich.org/static/downloads/"
list_depth = 1By default, Spack only looks at the top-level page available at list_url. Setting list_depth = 1 tells it to follow up to 1 level of links from the top-level page. Heads up: here that implies 1 level of subdirectories, since the mpich website is structured much like a filesystem. But list_depth really refers to link depth when spidering the page. โจ
๐ฅ Fetching from code repositories
For some packages, the source code lives in a Version Control System (VCS) repository rather than a tarball. Spack can fetch packages from VCS repositories. Currently, Spack supports fetching with Git, Mercurial (hg), Subversion (svn), CVS (cvs), and Go. In all cases, the destination is the standard stage source path. ๐
To fetch a package from a source repository, Spack needs to know which VCS to use and where to download from. Much like with url, package authors can specify a class-level git, hg, svn, cvs, or go attribute containing the correct download location.
Lots of packages developed with Git have both a Git repository and release tarballs available for download. Packages can define both a class-level tarball URL and a VCS. For example:
class Trilinos(CMakePackage):
homepage = "https://trilinos.org/"
url = "https://github.com/trilinos/Trilinos/archive/trilinos-release-12-12-1.tar.gz"
git = "https://github.com/trilinos/Trilinos.git"
version("develop", branch="develop")
version("master", branch="master")
version("12.12.1", md5="ecd4606fa332212433c98bf950a69cc7")
version("12.10.1", md5="667333dbd7c0f031d47d7c5511fd0810")
version("12.8.1", "9f37f683ee2b427b5540db8a20ed6b15")If a package contains both a url and a git class-level attribute, Spack decides which to use based on the arguments to the version() directive. Versions containing a specific branch, tag, or revision are assumed to be for VCS download methods, while versions containing a checksum are assumed to be for URL download methods. ๐ซถ
Like url, if a specific version downloads from a different repository than the default repo, it can be overridden with a version-specific argument.
NOTE: In order to reduce ambiguity, each package can only have a single VCS top-level attribute in addition tourl. In the rare case that a package uses multiple VCS, a fetch strategy can be specified for each version. For example, therockstarpackage contains:
class Rockstar(MakefilePackage):
homepage = "https://bitbucket.org/gfcstanford/rockstar"
version("develop", git="https://bitbucket.org/gfcstanford/rockstar.git")
version("yt", hg="https://bitbucket.org/MatthewTurk/rockstar")๐ Git
Git fetching supports the following parameters to version:
gitโ URL of the git repository, if different than the class-levelgit.branchโ Name of a branch to fetch.tagโ Name of a tag to fetch.commitโ SHA hash (or prefix) of a commit to fetch.submodulesโ Also fetch submodules recursively when checking out this repository.submodules_deleteโ A list of submodules to forcibly delete from the repository after fetching. Useful if a version in the repository has submodules that have disappeared/are no longer accessible. ๐ซget_full_repoโ Ensure the full git history is checked out with all remote branch information. Normally (get_full_repo=False, the default), the git option--depth 1will be used if the version of git and the specified transport protocol support it, and--single-branchwill be used if the version of git supports it.
Only one of tag, branch, or commit can be used at a time.
The destination directory for the clone is the standard stage source path.
๐ฆ Default branch
To fetch a repository's default branch:
class Example(Package):
git = "https://github.com/example-project/example.git"
version("develop")This download method is untrusted, and is not recommended. Aside from HTTPS, there is no way to verify that the repository has not been compromised, and the commit you get when you install the package likely won't be the same commit that was used when the package was first written. Additionally, the default branch may change. It is best to at least specify a branch name. ๐ญ
๐ธ Branches
To fetch a particular branch, use the branch parameter:
version("experimental", branch="experimental")This download method is untrusted, and is not recommended. Branches are moving targets, so the commit you get when you install the package likely won't be the same commit that was used when the package was first written. ๐ฏ
๐ Tags
To fetch from a particular tag, use tag instead:
version("1.0.1", tag="v1.0.1")This download method is untrusted, and is not recommended. Although tags are generally more stable than branches, Git allows tags to be moved. Many developers use tags to denote rolling releases, and may move the tag when a bug is patched. ๐
โญ Commits
Finally, to fetch a particular commit, use commit:
version("2014-10-08", commit="9d38cd4e2c94c3cea97d0e2924814acc")This doesn't have to be a full hash; you can abbreviate it as you'd expect with git:
version("2014-10-08", commit="9d38cd")This download method is trusted. It is the recommended way to securely download from a Git repository. ๐
It may be useful to provide a saner version for commits like this, e.g. you might use the date as the version, as done above. Or, if you know the commit at which a release was cut, you can use the release version. It's up to the package author to decide what makes the most sense. Although you can use the commit hash as the version number, this is not recommended, as it won't sort properly. ๐
๐ Submodules
You can supply submodules=True to cause Spack to fetch submodules recursively along with the repository at fetch time.
version("1.0.1", tag="v1.0.1", submodules=True)If a package needs more fine-grained control over submodules, define submodules to be a callable function that takes the package instance as its only argument. The function should return a list of submodules to be fetched.
def submodules(package):
submodules = []
if "+variant-1" in package.spec:
submodules.append("submodule_for_variant_1")
if "+variant-2" in package.spec:
submodules.append("submodule_for_variant_2")
return submodules
class MyPackage(Package):
version("0.1.0", submodules=submodules)For more information about git submodules see the manpage of git: man git-submodule. ๐
๐ GitHub
If a project is hosted on GitHub, any valid Git branch, tag, or hash may be downloaded as a tarball. This is accomplished simply by constructing an appropriate URL. Spack can checksum any package downloaded this way, thereby producing a trusted download. ๐
For example, the following downloads a particular hash, and then applies a checksum:
version("1.9.5.1.1", md5="d035e4bc704d136db79b43ab371b27d2",
url="https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f")๐ Mercurial
Fetching with Mercurial works much like Git, but you use the hg parameter. The destination directory is still the standard stage source path.
๐ฟ Default branch
Add the hg attribute with no revision passed to version:
class Example(Package):
hg = "https://bitbucket.org/example-project/example"
version("develop")This download method is untrusted, and is not recommended. As with Git's default fetching strategy, there is no way to verify the integrity of the download. ๐ฌ
๐ Revisions
To fetch a particular revision, use the revision parameter:
version("1.0", revision="v1.0")Unlike git, which has special parameters for different types of revisions, you can use revision for branches, tags, and commits when you fetch with Mercurial. Like Git, fetching specific branches or tags is an untrusted download method, and is not recommended. The recommended fetch strategy is to specify a particular commit hash as the revision. ๐
๐ง Subversion
To fetch with subversion, use the svn and revision parameters. The destination directory will be the standard stage source path.
Fetching the head
okay so literally just drop an svn parameter into the package and you're off:
class Example(Package):
svn = "https://outreach.scidac.gov/svn/example/trunk"
version("develop")fair warning bestie: this download method is untrusted and it is not recommended, for the exact same reasons we just talked about above ๐
Fetching a revision
want one specific revision? slay, just add a revision argument to the version directive:
version("develop", revision=128)Same deal as before: this download method is untrusted and not recommended ๐
And here's the genuinely sad part: Subversion has no commit hashing scheme like Git and Mercurial have, so there's literally no way to guarantee that the download you get is the same as the download used when the package was created. Use at your own risk. ๐ญ
also important: Subversion branches are handled as part of the directory structure, so you check out a branch or tag by changing the URL. If you wanna package multiple branches, just add an svn argument to each version directive. easy โจ
๐ฟ CVS ๐ฟ
CVS (Concurrent Versions System) is an old centralized version control system โ it's a predecessor of Subversion, so like, think of it as Subversion's grandma ๐
To fetch with CVS, you use the cvs, branch, and date parameters. The destination directory will be the standard stage source path, no surprises there.
Fetching the head
Simply add a cvs parameter to the package:
class Example(Package):
cvs = ":pserver:outreach.scidac.gov/cvsroot%module=modulename"
version("1.1.2.4")So here's the lore: CVS repository locations are described using an older syntax that's different from today's ubiquitous URL syntax. :pserver: denotes the transport method. CVS servers can host multiple repositories (they call them "modules") at the same location, and you need to specify both the server location and the module name to access them. Spack combines both into one string using the %module=modulename suffix shown above ๐ซถ
This download method is untrusted.
Fetching a date
Versions in CVS are commonly specified by date. To fetch a particular branch or date, add a branch and/or date argument to the version directive:
version("2021.4.22", branch="branchname", date="2021-04-22")Unfortunately, CVS does not identify repository-wide commits via a revision or hash like Subversion, Git, or Mercurial do ๐ which makes it impossible to specify an exact commit to check out.
CVS has more features, but since CVS is rarely used these days, Spack does not support all of them. it's giving legacy energy, and not in a cute way.
๐น Go ๐น
Plot twist: Go isn't a VCS at all โ it's a programming language with a builtin command, go get, that fetches packages and their dependencies automatically. The destination directory will be the standard stage source path.
This strategy can clone a Git repository, or download from another source location. For example:
class ThePlatinumSearcher(Package):
homepage = "https://github.com/monochromegane/the_platinum_searcher"
go = "github.com/monochromegane/the_platinum_searcher/..."
version("head")no cap: Go cannot be used to fetch a particular commit or branch โ it always downloads the head of the repository. This download method is untrusted, and it is not recommended. Use another fetch strategy whenever possible ๐
๐ฆ Variants ๐ฆ
So, many software packages can be configured to enable optional features, and those features often come at the expense of extra dependencies or longer build times ๐ญ To be flexible enough and support a wide variety of use cases, Spack lets you expose to the end-user the ability to choose which features should be activated in a package at the time it is installed. The mechanism you use is the spack.directives.variant() directive โจ
Boolean variants
In their simplest form, variants are boolean options specified at the package level:
class Hdf5(AutotoolsPackage):
...
variant(
"shared", default=True, description="Builds a shared version of the library"
)So you get a default value plus a description of what the thing actually means / how it's used in the package. Variants can be tested in any context where a spec constraint is expected ๐
In the example above, the shared variant is tied to the build of shared dynamic libraries. To pass the right option at configure time, we can branch depending on its value:
def configure_args(self):
...
if self.spec.satisfies("+shared"):
extra_args.append("--enable-shared")
else:
extra_args.append("--disable-shared")
extra_args.append("--enable-static-exec")As explained in Variants, the constraint +shared means the boolean variant is set to True, while ~shared means it's set to False ๐
Another common example is the optional activation of an extra dependency, which requires you to use the variant in the when argument of spack.directives.depends_on():
class Hdf5(AutotoolsPackage):
...
variant("szip", default=False, description="Enable szip support")
depends_on("szip", when="+szip")as shown in the snippet above, where szip is modeled to be an optional dependency of hdf5 ๐ซถ
Multi-valued variants
If need be, Spack can go beyond Boolean variants and permit an arbitrary number of allowed values. This is useful when modeling options that are tightly related to each other. In this case you pass the values to the spack.directives.variant() directive as a tuple:
class Blis(Package):
...
variant(
"threads", default="none", description="Multithreading support",
values=("pthreads", "openmp", "none"), multi=False
)In the example above the argument multi is set to False to indicate that only one among all the variant values can be active at any time. This constraint is enforced by the parser, and an error is emitted if a user specifies two or more values at the same time:
$ spack spec blis threads=openmp,pthreads
Input spec
--------------------------------
blis threads=openmp,pthreads
Concretized
--------------------------------
==> Error: multiple values are not allowed for variant "threads"another useful note: Python's None is not allowed as a default value, so it shouldn't be used to denote that no feature was selected. Users should instead select another value, like "none", and handle it explicitly within the package recipe if need be:
if self.spec.variants["threads"].value == "none":
options.append("--no-threads")In cases where multiple values can be selected at the same time, multi should be set to True:
class Gcc(AutotoolsPackage):
...
variant(
"languages", default="c,c++,fortran",
values=("ada", "brig", "c", "c++", "fortran",
"go", "java", "jit", "lto", "objc", "obj-c++"),
multi=True,
description="Compilers and runtime libraries to build"
)Within a package recipe, a multi-valued variant is tested using a key=value syntax:
if spec.satisfies("languages=jit"):
options.append("--enable-host-shared")Complex validation logic for variant values
To cover complex use cases, the spack.directives.variant() directive can accept as the values argument a full-fledged object which has default and other arguments of the directive embedded as attributes ๐
An example, already implemented in Spack's core, is spack.variant.DisjointSetsOfValues. This class is used to implement a few convenience functions, like spack.variant.any_combination_of():
class Adios(AutotoolsPackage):
...
variant(
"staging",
values=any_combination_of("flexpath", "dataspaces"),
description="Enable dataspaces and/or flexpath staging transports"
)which allows any combination of the specified values, and also allows the user to specify "none" (as a string) to choose none of them. The objects returned by these functions can be modified at will by chaining method calls to change the default value, customize the error message, or do other similar operations:
class Mvapich2(AutotoolsPackage):
...
variant(
"process_managers",
description="List of the process managers to activate",
values=disjoint_sets(
("auto",), ("slurm",), ("hydra", "gforker", "remshell")
).prohibit_empty_set().with_error(
"'slurm' or 'auto' cannot be activated along with "
"other process managers"
).with_default("auto").with_non_feature_values("auto"),
)Conditional Possible Values
There are cases where a variant may take multiple values, and the list of allowed values expands over time ๐ Think for instance of the C++ standard with which we might compile Boost โ it can take one of multiple possible values, with the latest standards only available from a certain version on.
To model a similar situation, we can use conditional possible values in the variant declaration:
variant(
"cxxstd", default="98",
values=(
"98", "11", "14",
# C++17 is not supported by Boost < 1.63.0.
conditional("17", when="@1.63.0:"),
# C++20/2a is not support by Boost < 1.73.0
conditional("2a", "2b", when="@1.73.0:")
),
multi=False,
description="Use the specified C++ standard when building.",
)The snippet above allows 98, 11 and 14 as unconditional possible values for the cxxstd variant, while 17 requires a version greater or equal to 1.63.0 and both 2a and 2b require a version greater or equal to 1.73.0 ๐
Conditional Variants
The variant directive accepts a when clause. The variant will only be present on specs that otherwise satisfy the spec listed as the when clause. For example, the following class has a variant bar when it is at version 2.0 or higher:
class Foo(Package):
...
variant("bar", default=False, when="@2.0:", description="help message")The when clause follows the same syntax and accepts the same values as the when argument of spack.directives.depends_on() โจ
Sticky Variants
The variant directive can be marked as sticky by setting the corresponding argument to True:
variant("bar", default=False, sticky=True)sticky variants stay where you put them, and honestly? iconic behaviour. ๐
okay so a sticky variant hits different from a regular one, because it's always locked to either:
- an explicit value that shows up in a spec literal, or
- its default value. periodt.
so the concretizer is not free to just pick some other value to dodge a conflict โ it will straight-up error out instead. ๐ setting this property on a variant is lowkey iconic when the variant unlocks some dangerous or controversial options (like, say, using unsupported compiler versions for a library) and the packager wants to make sure the user enables those options on purpose, not because the solver decided to be cute about it. ๐
โจ Overriding Variants โจ
packages can override variants for a bunch of reasons โ most often to change the default from a variant defined in a parent class, or to change the conditions under which a variant is even present on the spec.
when a variant gets defined multiple times, whether in the same package file or in a subclass and a superclass, the last definition is used for all attributes except for the when clauses. the when clauses are accumulated through all invocations, and the variant is present on the spec if any of the accumulated conditions are satisfied. ๐ซถ
for example, consider this package:
class Foo(Package):
...
variant("bar", default=False, when="@1.0", description="help1")
variant("bar", default=True, when="platform=darwin", description="help2")
...this package foo has a variant bar when the spec satisfies either @1.0 or platform=darwin โ but not for other platforms at other versions. the default for this variant, whenever it's present, is always True, no matter which condition got satisfied. the way that this slays is it lets packages override variants in packages or build system classes they inherit from, by tweaking the variant values without touching the when clause. it also lets a package implement or semantics for a variant when clause by duplicating the variant definition. ๐
๐ธ Resources (expanding extra tarballs) ๐ธ
some packages (most notably compilers) provide optional features if extra resources get expanded inside their source tree before building. in Spack you can describe that need with the resource directive:
resource(
name="cargo",
git="https://github.com/rust-lang/cargo.git",
tag="0.10.0",
destination="cargo"
)based on the keywords present among the arguments, the appropriate FetchStrategy gets used for the resource. the destination keyword is relative to the source root of the package and should point to where the resource is to be expanded. ๐
๐ฆ Licensed software ๐ฆ
so to install licensed software, Spack needs to know a few more deets about a package. define these class attributes, bestie:
license_required
Boolean. if set to True, this software requires a license. if set to False, all of the following attributes get ignored. defaults to False. โจ
license_comment
String. contains the symbol the license manager uses to denote a comment. defaults to #.
license_files
List of strings. these are files the software searches for when looking for a license. all file paths must be relative to the installation directory. more complex packages like Intel may require multiple licenses for individual components. defaults to the empty list. ๐
license_vars
List of strings. environment variables that can be set to tell the software where to look for a license if it's not in the usual location. defaults to the empty list.
license_url
String. a URL pointing to license setup instructions for the software. defaults to the empty string. ๐ฅ
for example, let's take a look at the package for the PGI compilers:
# Licensing
license_required = True
license_comment = "#"
license_files = ["license.dat"]
license_vars = ["PGROUPD_LICENSE_FILE", "LM_LICENSE_FILE"]
license_url = "http://www.pgroup.com/doc/pgiinstall.pdf"as you can see, PGI requires a license. its license manager, FlexNet, uses the # symbol to denote a comment. it expects the license file to be named license.dat and to be located directly in the installation prefix. if you'd like the installation file to live somewhere else, simply set PGROUPD_LICENSE_FILE or LM_LICENSE_FILE after installation. for further instructions on installation and licensing, see the URL provided. ๐
let's walk through a sample PGI installation to see exactly what Spack is and isn't capable of. since PGI doesn't provide a download URL, it must be downloaded manually. it can either be added to a mirror or located in the current directory when spack install pgi is run. see Mirrors (mirrors.yaml) for instructions on setting up a mirror. ๐ช
after running spack install pgi, the first thing that happens is Spack creates a global license file located at $SPACK_ROOT/etc/spack/licenses/pgi/license.dat. it'll then open up that file using your favorite editor. it looks like this:
# A license is required to use pgi.
#
# The recommended solution is to store your license key in this global
# license file. After installation, the following symlink(s) will be
# added to point to this file (relative to the installation prefix):
#
# license.dat
#
# Alternatively, use one of the following environment variable(s):
#
# PGROUPD_LICENSE_FILE
# LM_LICENSE_FILE
#
# If you choose to store your license in a non-standard location, you may
# set one of these variable(s) to the full pathname to the license file, or
# port@host if you store your license keys on a dedicated license server.
# You will likely want to set this variable in a module file so that it
# gets loaded every time someone tries to use pgi.
#
# For further information on how to acquire a license, please refer to:
#
# http://www.pgroup.com/doc/pgiinstall.pdf
#
# You may enter your license below.you can add your license directly to this file, or tell FlexNet to use a license stored on a separate license server. here's an example that points to a license server called licman1:
SERVER licman1.mcs.anl.gov 00163eb7fba5 27200
USE_SERVERif your package requires the license to install, you can reference the location of this global license using self.global_license_file. after installation, symlinks for all of the files given in license_files get created, pointing to this global license. and if you install a different version or variant of the package, Spack will automatically detect and reuse the already-existing global license. โป๏ธ๐
if the software you're trying to package doesn't rely on license files, Spack prints a warning message letting the user know they need to set an environment variable or points them to installation documentation. ๐ญ
๐ Patches ๐
depending on the host architecture, package version, known bugs, or other issues, you may need to patch your software to get it to build correctly. like many other package systems, Spack lets you store patches alongside your package files and apply them to source code after it's downloaded. ๐ฉน
patch
you can specify patches in your package file with the patch() directive. patch looks like this:
class Mvapich2(Package):
...
patch("ad_lustre_rwcontig_open_source.patch", when="@1.9:")the first argument can be either a URL or a filename. it specifies a patch file that should be applied to your source. if the patch you supply is a filename, then the patch needs to live within the spack source tree. for example, the patch above lives in a directory structure like this:
/usr/share/spack/repos/builtin/packages/
mvapich2/
package.py
ad_lustre_rwcontig_open_source.patchif you supply a URL instead of a filename, you need to supply a sha256 checksum, like this:
patch("http://www.nwchem-sw.org/images/Tddft_mxvec20.patch",
sha256="252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866")Spack includes the hashes of patches in its versioning information, so the same package with different patches applied gets different hash identifiers. to keep the hashing scheme consistent, you must use a sha256 checksum for the patch. patches get fetched from their URLs, checked, and applied to your source code. you can use the GNU utils sha256sum or the macOS shasum -a 256 commands to generate a checksum for a patch file. ๐ง
Spack can also handle compressed patches. if you use those, Spack needs a little more help. specifically, it needs two checksums: the sha256 of the patch and archive_sha256 for the compressed archive. archive_sha256 helps Spack make sure the downloaded file isn't corrupted or malicious before running it through a tool like tar or zip. the sha256 of the patch is still required so it can be included in specs โ providing it in the package file means Spack won't have to download and decompress patches it won't end up using at install time. both the archive and patch checksum get checked when patch archives are downloaded. ๐ค
patch("http://www.nwchem-sw.org/images/Tddft_mxvec20.patch.gz",
sha256="252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866",
archive_sha256="4e8092a161ec6c3a1b5253176fcf33ce7ba23ee2ff27c75dbced589dabacd06e")patch keyword arguments are described below. ๐
sha256, archive_sha256
hashes of the downloaded patch and compressed archive, respectively. only needed for patches fetched from URLs. ๐
when
if supplied, this is a spec that tells Spack when to apply the patch. if the installed package spec matches this spec, the patch gets applied. in our example above, the patch is applied when mvapich is at version 1.9 or higher. โญ
level
this tells Spack how to run the patch command. by default, level is 1 and Spack runs patch -p 1. if level is 2, Spack runs patch -p 2, and so on.
a lot of people get confused by level, so here's a primer. if you look in your patch file, you may see something like this:
--- a/src/mpi/romio/adio/ad_lustre/ad_lustre_rwcontig.c 2013-12-10 12:05:44.806417000 -0800
+++ b/src/mpi/romio/adio/ad_lustre/ad_lustre_rwcontig.c 2013-12-10 11:53:03.295622000 -0800
@@ -8,7 +8,7 @@
* Copyright (C) 2008 Sun Microsystems, Lustre group
\*/
-#define _XOPEN_SOURCE 600
+//#define _XOPEN_SOURCE 600
#include <stdlib.h>
#include <malloc.h>
#include "ad_lustre.h"Bestie, so like โ those first two lines of the patch? They show paths with synthetic a/ and b/ prefixes, and those are literally just placeholders for the two mvapich2 source directories that diff was comparing when it made the patch file. That's git's default behavior when it creates patch files, but other programs may behave differently, so don't come for me if yours is quirky ๐
-p1 strips off the first level of the prefix in both paths, which lets the patch be applied from the root of an expanded mvapich2 archive. Set level to 2 and it'd strip off src, and so on. It's generally easier to just structure your patch file so it applies cleanly with -p1, but if you're using a patch you didn't create yourself, level can be handy. iykyk โจ
๐ working_dir ๐
This tells spack where to run the patch command. By default, the working directory is the source path of the stage (.). But sometimes patches are made with respect to a subdirectory, and that's exactly where the working directory comes in handy. Internally, the working directory is given to patch via the -d option.
Let's take the example patch from above and assume for some reason it can only be downloaded in the following form:
--- a/romio/adio/ad_lustre/ad_lustre_rwcontig.c 2013-12-10 12:05:44.806417000 -0800
+++ b/romio/adio/ad_lustre/ad_lustre_rwcontig.c 2013-12-10 11:53:03.295622000 -0800
@@ -8,7 +8,7 @@
* Copyright (C) 2008 Sun Microsystems, Lustre group
\*/
-#define _XOPEN_SOURCE 600
+//#define _XOPEN_SOURCE 600
#include <stdlib.h>
#include <malloc.h>
#include "ad_lustre.h"Hence, the patch needs to applied in the src/mpi subdirectory, and the working_dir="src/mpi" option would exactly do that. Slay, no extra steps ๐
๐ Patch functions ๐
In addition to supplying patch files, you can write a custom function to patch a package's source. For example, the py-pyside package contains some custom code for tweaking the way the PySide build handles RPATH:
def patch(self):
"""Undo PySide RPATH handling and add Spack RPATH."""
# Figure out the special RPATH
rpath = self.rpath
rpath.append(os.path.join(python_platlib, "PySide"))
# Fix subprocess.mswindows check for Python 3.5
# https://github.com/pyside/pyside-setup/pull/55
filter_file(
"^if subprocess.mswindows:",
'mswindows = (sys.platform == "win32")\r\nif mswindows:',
"popenasync.py",
)
filter_file("^ if subprocess.mswindows:", " if mswindows:", "popenasync.py")
# Remove check for python version because the above patch adds support for newer versions
filter_file("^check_allowed_python_version()", "", "setup.py")
# Add Spack's standard CMake args to the sub-builds.
# They're called BY setup.py so we have to patch it.
filter_file(
r"OPTION_CMAKE,",
r"OPTION_CMAKE, "
+ (
'"-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=FALSE", '
'"-DCMAKE_INSTALL_RPATH=%s",' % ":".join(rpath)
),
"setup.py",
)
# PySide tries to patch ELF files to remove RPATHs
# Disable this and go with the one we set.
if self.spec.satisfies("@1.2.4:"):
rpath_file = "setup.py"
else:
rpath_file = "pyside_postinstall.py"
filter_file(r"(^\s*)(rpath_cmd\(.*\))", r"\1#\2", rpath_file)A patch function, if present, will be run after patch files are applied and before install() is run. Obsessed with that ordering ngl ๐ซถ
You could put this logic in install(), but putting it in a patch function gives you some benefits. First, spack ensures that the patch() function is run once per code checkout. That means that if you run install, hit Ctrl+C, and run install again, the code in the patch function is only run once. Also, you can tell Spack to run only the patching part of the build using the spack patch command. Main character energy, honestly ๐
๐ฆ Dependency patching ๐ฆ
So far we've covered how the patch directive can be used by a package to patch its own source code. Packages can also specify patches to be applied to their dependencies, if they require special modifications. As with all packages in Spack, a patched dependency library can coexist with other versions of that library. See the section on depends_on for more details. Yes bestie, it's giving "everyone gets their own version" ๐
โจ Inspecting patches โจ
If you want to better understand the patches that Spack applies to your packages, you can do that using spack spec, spack find, and other query commands. Let's look at m4. If you run spack spec m4, you can see the patches that would be applied to m4:
$ spack spec m4
Input spec
--------------------------------
m4
Concretized
--------------------------------
m4@1.4.18%apple-clang@9.0.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=darwin-highsierra-x86_64
^libsigsegv@2.11%apple-clang@9.0.0 arch=darwin-highsierra-x86_64You can also see patches that have been applied to installed packages with spack find -v:
$ spack find -v m4
==> 1 installed package
-- darwin-highsierra-x86_64 / apple-clang@9.0.0 -----------------
m4@1.4.18 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegvIn both cases above, you can see that the patches' sha256 hashes are stored on the spec as a variant. As mentioned above, this means that you can have multiple, differently-patched versions of a package installed at once โ iconic behaviour fr fr ๐
You can look up a patch by its sha256 hash (or a short version of it) using the spack resource show command:
$ spack resource show 3877ab54
3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00
path: /home/spackuser/src/spack/var/spack/repos/builtin/packages/m4/gnulib-pgi.patch
applies to: builtin.m4spack resource show looks up downloadable resources from package files by hash and prints out information about them. Above, we see that the 3877ab54 patch applies to the m4 package. The output also tells us where to find the patch.
Things get more interesting if you want to know about dependency patches. For example, when dealii is built with boost@1.68.0, it has to patch boost to work correctly. If you didn't know this, you might wonder where the extra boost patches are coming from:
$ spack spec dealii ^boost@1.68.0 ^hdf5+fortran | grep "\^boost"
^boost@1.68.0
^boost@1.68.0%apple-clang@9.0.0+atomic+chrono~clanglibcpp cxxstd=default +date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199,b37164268f34f7133cbc9a4066ae98fda08adf51e1172223f6a969909216870f ~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=darwin-highsierra-x86_64
$ spack resource show b37164268
b37164268f34f7133cbc9a4066ae98fda08adf51e1172223f6a969909216870f
path: /home/spackuser/src/spack/var/spack/repos/builtin/packages/dealii/boost_1.68.0.patch
applies to: builtin.boost
patched by: builtin.dealiiHere you can see that the patch is applied to boost by dealii, and that it lives in dealii's directory in Spack's builtin package repository. See? Plot twist handled ๐
๐ธ Handling RPATHs ๐ธ
Spack installs each package in a way that ensures that all of its dependencies are found when it runs. It does this using RPATHs. An RPATH is a search path, stored in a binary (an executable or library), that tells the dynamic loader where to find its dependencies at runtime. You may be familiar with LD_LIBRARY_PATH on Linux or DYLD_LIBRARY_PATH on Mac OS X. RPATH is similar to these paths, in that it tells the loader where to find libraries. Unlike them, it is embedded in the binary and not set in each user's environment.
RPATHs in Spack are handled in one of three ways:
- For most packages, RPATHs are handled automatically using Spack's compiler wrappers. These wrappers are set in standard variables like
CC,CXX,F77, andFC, so most build systems (autotools and many gmake systems) pick them up and use them. - CMake also respects Spack's compiler wrappers, but many CMake builds have logic to overwrite RPATHs when binaries are installed. Spack provides the
std_cmake_argsvariable, which includes parameters necessary for CMake build use the right installation RPATH. It can be used like this whencmakeis invoked:
class MyPackage(Package):
...
def install(self, spec, prefix):
cmake("..", *std_cmake_args)
make()
make("install")- If you need to modify the build to add your own RPATHs, you can use the
self.rpathproperty of your package, which will return a list of all the RPATHs that Spack will use when it links. You can see this how this is used in thePySideexample above โ go scroll back bestie, it's literally right there ๐
๐ฅ Parallel builds ๐ฅ
Spack supports parallel builds on an individual package and at the installation level. Package-level parallelism is established by the --jobs option and its configuration and package recipe equivalents. Installation-level parallelism is driven by the DAG(s) of the requested package or packages. It's giving multitasking queen โญ
๐ Package-level build parallelism ๐
By default, Spack will invoke make(), or any other similar tool, with a -j <njobs> argument, so those builds run in parallel. The parallelism is determined by the value of the build_jobs entry in config.yaml (see here for more details on how this value is computed).
If a package does not build properly in parallel, you can override this setting by adding parallel = False to your package. For example, OpenSSL's build does not work in parallel, so its package looks like this:
class Openssl(Package):
homepage = "http://www.openssl.org"
url = "http://www.openssl.org/source/openssl-1.0.1h.tar.gz"
version("1.0.1h", md5="8d6d684a9430d5cc98a62a5d8fbda8cf")
depends_on("zlib-api")
parallel = FalseSimilarly, you can disable parallel builds only for specific make commands, as libdwarf does:
class Libelf(Package):
...
def install(self, spec, prefix):
configure("--prefix=" + prefix,
"--enable-shared",
"--disable-dependency-tracking",
"--disable-debug")
make()# The mkdir commands in libelf's install can fail in parallel
make("install", parallel=False)ok so in this snippet, the first make runs in parallel but the second one is like "not today bestie" โ it does not. If you set parallel to False at the package level, then every call to make() is sequential by default, but packagers can still call make(parallel=True) to override it and turn the chaos back on. ๐
๐ฆ Install-level build parallelism ๐ฆ
Spack supports the concurrent installation of packages within a Spack instance across multiple processes using file system locks. This parallelism is literally a whole separate thing from the package-level parallelism achieved through build systems' use of the -j <njobs> option. With install-level parallelism, processes coordinate the installation of the dependencies of specs provided on the command line and as part of an environment build, with only one process being allowed to install a given package at a time. Refer to Dependencies for more info on dependencies, and Installing an Environment for how to install an environment.
Concurrent processes may be any combination of interactive sessions and batch jobs. Which means a spack install can be running in a terminal window while a batch job is running spack install on the same or overlapping dependencies without any process trying to re-do the work of another. no duplicate effort, no drama, we love to see it. โจ
For example, if you're using Slurm, you could launch an installation of mpich using the following command:
$ srun -N 2 -n 8 spack install -j 4 mpich@3.3.2This will create eight concurrent, four-job installs on two different nodes. iconic behavior, honestly. ๐
Alternatively, you could run the same installs on one node by entering the following at the command line of a bash shell:
$ for i in {1..12}; do nohup spack install -j 4 mpich@3.3.2 >> mpich_install.txt 2>&1 & doneNOTE: the effective parallelism is based on the maximum number of packages that can be installed at the same time, which is limited by the number of packages with no (remaining) uninstalled dependencies. ๐ซถ
๐ธ Dependencies ๐ธ
We've covered how to build a simple package, but what if one package relies on another package to build? How do you express that in a package file? And how do you refer to the other package in the build script for your own package?
Spack makes this relatively easy. Let's take a look at the libdwarf package to see how it's done:
class Libdwarf(Package):
homepage = "http://www.prevanders.net/dwarf.html"
url = "http://www.prevanders.net/libdwarf-20130729.tar.gz"
list_url = homepage
version("20130729", md5="4cc5e48693f7b93b7aa0261e63c0e21d")
...
depends_on("libelf")
def install(self, spec, prefix):
...๐ depends_on() ๐
The highlighted depends_on("libelf") call tells Spack that it needs to build and install the libelf package before it builds libdwarf. This means that in your install() method, you are guaranteed that libelf has been built and installed successfully, so you can rely on it for your libdwarf build. it's giving "trust fall but make it build system". ๐ค
๐ Dependency specs ๐
depends_on doesn't just take the name of another package. It can take a full spec as well! This means you can restrict the versions or other configuration options of libelf that libdwarf will build with. For example, suppose that in the libdwarf package you write:
depends_on("libelf@0.8")Now libdwarf will require libelf at exactly version 0.8. You can also specify a requirement for a particular variant or for specific compiler flags:
depends_on("libelf@0.8+debug")
depends_on("libelf debug=True")
depends_on("libelf cppflags='-fPIC'")Both users and package authors can use the same spec syntax to refer to different package configurations. Users use the spec syntax on the command line to find installed packages or to install packages with particular constraints, and package authors can use specs to describe relationships between packages. same syntax, two friend groups, everyone eats. ๐
โญ Version ranges โญ
Although some packages require a specific version for their dependencies, most can be built with a range of versions. For example, if you are writing a package for a legacy Python module that only works with Python 2.4 through 2.6, this would look like:
depends_on("python@2.4:2.6")Version ranges in Spack are inclusive, so 2.4:2.6 means any version greater than or equal to 2.4 and up to and including any 2.6.x. If you want to specify that a package works with any version of Python 3 (or higher), this would look like:
depends_on("python@3:")Here we leave out the upper bound. If you want to say that a package requires Python 2, you can similarly leave out the lower bound:
depends_on("python@:2")Notice that we didn't use @:3. Version ranges are inclusive, so @:3 means "up to and including any 3.x version". read the range, bestie, it's inclusive on both ends, that's the whole vibe. ๐
You can also simply write
depends_on("python@2.7")to tell Spack that the package needs Python 2.7.x. This is equivalent to @2.7:2.7.
In very rare cases, you may need to specify an exact version, for example if you need to distinguish between 3.2 and 3.2.1:
depends_on("pkg@=3.2")But in general, you should try to use version ranges as much as possible, so that custom suffixes are included too. The above example can be rewritten in terms of ranges as follows:
depends_on("pkg@3.2:3.2.0")A spec can contain a version list of ranges and individual versions separated by commas. For example, if you need Boost 1.59.0 or newer, but there are known issues with 1.64.0, 1.65.0, and 1.66.0, you can say:
depends_on("boost@1.59.0:1.63,1.65.1,1.67.0:")๐ฅ Dependency types ๐ฅ
Not all dependencies are created equal, and Spack allows you to specify exactly what kind of a dependency you need. For example:
depends_on("cmake", type="build")
depends_on("py-numpy", type=("build", "run"))
depends_on("libelf", type=("build", "link"))
depends_on("py-pytest", type="test")The following dependency types are available:
"build"โ the dependency will be added to thePATHandPYTHONPATHat build-time."link"โ the dependency will be added to Spack's compiler wrappers, automatically injecting the appropriate linker flags, including-I,-L, andRPATH/RUNPATHhandling."run"โ the dependency will be added to thePATHandPYTHONPATHat run-time. This is true for bothspack loadand the module files Spack writes."test"โ the dependency will be added to thePATHandPYTHONPATHat build-time. The only difference between"build"and"test"is that test dependencies are only built if the user requests unit tests withspack install --test.
One of the advantages of the build dependency type is that although the dependency needs to be installed in order for the package to be built, it can be uninstalled without concern afterwards. link and run disallow this because uninstalling the dependency would break the package. do not do that, fr fr. ๐ญ
build, link, and run dependencies all affect the hash of Spack packages (along with sha256 sums of patches and archives used to build the package, and a canonical hash of the package.py recipes). test dependencies do not affect the package hash, as they are only used to construct a test environment after building and installing a given package installation. Older versions of Spack did not include build dependencies in the hash, but this has been fixed as of Spack v0.18. glow-up complete. โจ
If the dependency type is not specified, Spack uses a default of ("build", "link"). This is the common case for compiler languages. Non-compiled packages like Python modules commonly use ("build", "run"). This means that the compiler wrappers don't need to inject the dependency's prefix/lib directory, but the package needs to be in PATH and PYTHONPATH during the build process and later when a user wants to run the package.
๐ Conditional dependencies ๐
You may have a package that only requires a dependency under certain conditions. For example, you may have a package with optional MPI support. You would then provide a variant to reflect that the feature is optional and specify that the MPI dependency only applies when MPI support is enabled. In that case, you could say something like:
variant("mpi", default=False, description="Enable MPI support")
depends_on("mpi", when="+mpi")Suppose the above package also has, since version 3, optional Trilinos support and you want them both to build either with or without MPI. Further suppose you require a version of Trilinos no older than 12.6. In that case, the trilinos variant and dependency directives would be:
variant("trilinos", default=False, description="Enable Trilinos support")
depends_on("trilinos@12.6:", when="@3: +trilinos")
depends_on("trilinos@12.6: +mpi", when="@3: +trilinos +mpi")Alternatively, you could use the when context manager to equivalently specify the trilinos variant dependencies as follows:
with when("@3: +trilinos"):
depends_on("trilinos@12.6:")
depends_on("trilinos +mpi", when="+mpi")The argument to when in either case can include any Spec constraints that are supported on the command line using the same syntax.
NOTE: if a dependency isn't typically used, you can save time by making it conditional since Spack will not build the dependency unless it is required for the Spec. lazy dependency loading, we stan. ๐ง
๐ Dependency patching ๐
Some packages maintain special patches on their dependencies, either to add new features or to fix bugs. This typically makes a package harder to maintain, and we encourage developers to upstream (contribute back) their changes rather than maintaining patches. However, in some cases it's not possible to upstream. Maybe the dependency's developers don't accept changes, or maybe they just haven't had time to integrate them. we've all been there. ๐
For times like these, Spack's depends_on directive can optionally take a patch or list of patches:
class SpecialTool(Package):
...
depends_on("binutils", patches="special-binutils-feature.patch")
...Here, the special-tool package requires a special feature in binutils, so it provides an extra patches=<filename> keyword argument. This is similar to the patch directive, with one small difference. Here, special-tool is responsible for the patch, so it should live in special-tool's directory in the package repository, not the binutils directory. get that file in the right folder, bestie. ๐
If you need something more sophisticated than this, you can simply nest a patch() directive inside of depends_on:
class SpecialTool(Package):
...
depends_on(
"binutils",
patches=patch("special-binutils-feature.patch",
level=3,
when="@:1.3"), # condition on binutils
when="@2.0:") # condition on special-tool
...okay so quick heads-up before we keep going: there are two optional when conditions up in here โ one on the patch directive, the other on depends_on. The one inside patch applies to binutils (the package getting patched, bestie ๐ฉน), while the one inside depends_on applies to special-tool. Peep the patch directive section for the full tea on all the arguments it can take ๐
.
Finally, if you need multiple patches on a dependency, you can just hand patches a list, like so:
class SpecialTool(Package):
...
depends_on(
"binutils",
patches=[
"binutils-bugfix1.patch",
"binutils-bugfix2.patch",
patch("https://example.com/special-binutils-feature.patch",
sha256="252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866",
when="@:1.3")],
when="@2.0:")
...Just like with patch directives, patches get applied in the order they show up in the package file (or, in this case, in the list). No reshuffling, no vibes-based ordering โจ.
NOTE: you might be sitting there like "wait, will dependency patching mess with other packages that depend onbinutils?" it will not, bestie, breathe ๐. As described in patching, Patching a package adds thesha256of the patch to the package's spec, which means it'll have a different unique hash than other versions without the patch. The patched version coexists with unpatched versions, and Spack's support forhandling_rpathsguarantees each installation finds the right version. And if two packages depend onbinutilspatched the same way? They can both use a single installation ofbinutils. Iconic sharing behaviour, honestly ๐.
๐ฅ Conflicts and requirements ๐ฅ
Sometimes packages have known bugs, or limitations, that would stop them building โ like against certain other dependencies, or with certain compilers. Spack lets you express those constraints with the conflicts directive ๐ซ.
Drop this into a package:
conflicts(
"%intel",
when="@:1.2",
msg="<myNicePackage> <= v1.2 cannot be built with Intel ICC, "
"please use a newer release."
)and we're saying: this package cannot be built with the Intel compiler when we're trying to install a version <=1.2. So specific. So clear. We love a boundary ๐
.
The when argument can be left out entirely, in which case the conflict is always active. No chill, permanent beef ๐ค.
An optional custom error message can be added via the msg= parameter, and Spack will print it in case the conflict can't be avoided and leads to a concretization error. Helpful little breadcrumb trail for future you ๐.
Sometimes packages allow only very specific choices and can't use the rest. In those cases, the requires directive is your girl:
requires(
"%apple-clang",
when="platform=darwin",
msg="<myNicePackage> builds only with Apple-Clang on Darwin"
)In the example above, our package can only be built with Apple-Clang on Darwin, periodt. The requires directive is effectively the opposite of conflicts, and takes the same optional when and msg arguments. Mirror image, same energy โจ.
If a package needs to express more complex requirements โ involving more than a single spec โ you can do that with requires too. To say a package can be built either with GCC or with Clang, we write:
requires(
"%gcc", "%clang",
policy="one_of"
msg="<myNicePackage> builds only with GCC or Clang"
)When using multiple specs in a requires directive, it's advised to set the policy= argument explicitly. That argument can take either the value any_of or the value one_of, and the semantics are the same as for Package Requirements ๐.
๐ง Extensions ๐ง
Spack's support for package extensions is documented extensively in Python packages and virtual environments. This section is all about how you make your own extendable packages and extensions ๐ ๏ธ.
To support extensions, a package needs to set its extendable property to True, like:
class Python(Package):
...
extendable = True
...To turn a package into an extension, just add an extends call in the package definition and pass it the name of an extendable package:
class PyNumpy(Package):
...
extends("python")
...This does a few things, and they're all lowkey genius ๐. Firstly, the Python package can set special variables โ like PYTHONPATH โ for all extensions whenever the run or build environment gets set up. Secondly, filesystem views can make sure extensions get put in the same prefix as their extendee. That means Python in a view can always locate its Python packages, even with zero environment variables set. Chef's kiss ๐.
A package can only extend one other package at a time. But to support packages that might extend one of a whole list of others, Spack supports multiple extends directives โ as long as at most one of them gets selected as a dependency during concretization. For example, a lua package could extend either lua or luajit, but never both:
class LuaLpeg(Package):
...
variant("use_lua", default=True)
extends("lua", when="+use_lua")
extends("lua-luajit", when="~use_lua")
...Now a user can install โ and activate โ the lua-lpeg package for either lua or luajit. Choices, choices ๐
.
๐ Adding additional constraints ๐
Some packages produce a Python extension but are only compatible with Python 3, or only with Python 2. In those cases, a depends_on() declaration should be made in addition to the extends() declaration:
class Icebin(Package):
extends("python", when="+python")
depends_on("python@3:", when="+python")Many packages produce Python extensions for some variants but not others โ those should only extend python if the appropriate variant(s) are selected. This is doable with conditional extends() declarations:
class FooLib(Package):
variant("python", default=True, description="Build the Python extension Module")
extends("python", when="+python")
...Conditional everything. We contain multitudes ๐.
๐ธ Runtime and build time environment variables ๐ธ
Spack gives package authors a few methods to help set up the required environment variables for their package. Env vars typically depend on how the package is used: variables that make sense during the build phase may not be needed at runtime, and vice versa. And sometimes it just makes sense to let a dependency set the environment variables for its dependents ๐ค. To allow all of this, Spack provides four different methods you can override in a package:
setup_build_environmentsetup_run_environmentsetup_dependent_build_environmentsetup_dependent_run_environment
The Qt package, for instance, uses this call:
def setup_dependent_build_environment(self, env, dependent_spec):
env.set("QTDIR", self.prefix)
env.set("QTINC", self.prefix.inc)
env.set("QTLIB", self.prefix.lib)
env.prepend_path("QT_PLUGIN_PATH", self.prefix.plugins)to set the QTDIR environment variable so that packages depending on a particular Qt installation will find it. Find it and slay it ๐
.
The following diagram will give you an idea of when each of these methods gets called in a build context: [image]
Notice that setup_dependent_run_environment can be called multiple times โ once for each dependent package โ whereas setup_run_environment is called only once, for the package itself. Which means: the former should only be used if the environment variables depend on the dependent package, while the latter should be used if the environment variables depend only on the package itself. Important distinction, don't mix 'em up ๐ญ.
๐ฆ Setting package module variables ๐ฆ
Apart from modifying environment variables of the dependent package, you can also define Python variables for the dependent to use. This is done by implementing setup_dependent_package. A prime example lives in the Python package:
def setup_dependent_package(self, module, dependent_spec):
"""Called before python modules' install() methods."""
module.python = self.command
module.python_include = join_path(dependent_spec.prefix, self.include)
module.python_platlib = join_path(dependent_spec.prefix, self.platlib)
module.python_purelib = join_path(dependent_spec.prefix, self.purelib)This lets Python packages use those variables directly:
def install(self, spec, prefix):
...
install("script.py", python_platlib)NOTE: we recommend usingsetup_dependent_packagesparingly, because it's not always clear where global variables are coming from when editing apackage.pyfile ๐ต๏ธโโ๏ธ.
๐ Views ๐
The spack view command can symlink a bunch of packages into a merged prefix. The methods of PackageViewMixin can be overridden to customize how packages get added to views ๐ง. Generally this can be used to create copies of specific files rather than symlinking them, for when symlinking just Does Not Work ๐.
For example, Python overrides add_files_to_view so it can create a copy of the python binary, since the real path of the Python executable is used to detect extensions. As a consequence, python extension packages (those inheriting from PythonPackage) likewise override add_files_to_view to rewrite shebang lines pointing at the Python interpreter. Symlink said no, copy said yes โจ.
๐ Virtual dependencies ๐
In some cases, more than one package can satisfy another package's dependency. One way this happens: a package depends on a particular interface, but there are multiple implementations of that interface, and the package could be built with any of them ๐คฏ. A very common interface in HPC is the Message Passing Interface (MPI), which gets used in loads of large-scale parallel applications.
MPI has several different implementations (e.g., MPICH, OpenMPI, and MVAPICH) and scientific applications can be built with any one of them. Complicating matters, MPI doesn't have a standardized ABI, so a package built with one implementation can't simply be relinked with another. Many package managers handle interfaces like this by requiring many similar package files โ e.g., foo, foo-mvapich, foo-mpich โ but Spack avoids this explosion of package files by providing support for virtual dependencies. Periodt ๐
๐ฅ.
๐ provides ๐
In Spack, mpi is handled as a virtual package. A package like mpileaks can depend on it just like any other package, by supplying a depends_on call in the package definition. For example:
class Mpileaks(Package):
homepage = "https://github.com/hpc/mpileaks"
url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
version("1.0", md5="8838c574b39202a57d7c2d68692718aa")
depends_on("mpi")
depends_on("adept-utils")
depends_on("callpath")Clean, tidy, no duplicate package files in sight. The way that depends_on("mpi") just works โ obsessed ๐๐ซถ.
here, callpath and adept-utils are concrete packages (real files, real vibes), but there's literally no actual package file for mpi, so we call it a virtual package. โจ The syntax of depends_on is the same for both, which is honestly so convenient of it. If we peek inside the package file of an MPI implementation โ say MPICH โ we'll see something like this:
class Mpich(Package):
provides("mpi")
...That provides("mpi") call is basically MPICH raising its hand and going "me!! I can do that!!" โ it tells Spack that the mpich package can be used to satisfy the dependency of any package that does depends_on("mpi"). ๐ซถ
Providing multiple virtuals simultaneously
Packages can provide more than one virtual dependency, and sometimes โ because of implementation details โ there are subsets of those virtuals that have to be provided together by the same package. no cap.
The classic iconic example is openblas, which provides both the lapack and blas API in a single libopenblas library. A package that needs lapack and blas must either use openblas to provide both, or not use openblas at all. It cannot pick one or the other. that's the whole deal, bestie.
To express this constraint in a package, the two virtual dependencies must be listed in the same provides directive:
provides('blas', 'lapack')This makes it impossible to select openblas as a provider for one of the two virtual dependencies and not for the other. If you try to, Spack will report an error:
$ spack spec netlib-scalapack ^[virtuals=lapack] openblas ^[virtuals=blas] atlas
==> Error: concretization failed for the following reasons:
1. Package 'openblas' needs to provide both 'lapack' and 'blas' together, but provides only 'lapack'Versioned Interfaces
Just like you can pass a spec to depends_on, you can also pass a spec to provides to add constraints. This is how Spack supports the notion of versioned interfaces. ๐
The MPI standard has gone through many revisions, each with new functions added, and each revision of the standard has a version number. Some packages may require a recent implementation that supports MPI-3 functions, but some MPI versions may only provide up to MPI-2. Others may need MPI 2.1 or higher. You can indicate this by adding a version constraint to the spec passed to provides:
provides("mpi@:2")Suppose that the above provides call is in the mpich2 package. This says that mpich2 provides MPI support up to version 2 โ but if a package does depends_on("mpi@3"), then Spack will not build that package with mpich2. it's giving "sorry, not qualified" ๐ญ
provides when
The same package may provide different versions of an interface depending on its version. Above, we simplified the provides call in mpich to make the explanation easier. In reality, this is how mpich calls provides:
provides("mpi@:3", when="@3:")
provides("mpi@:1", when="@1:")The when argument to provides lets you specify optional constraints on the providing package, or the provider. The provider only provides the declared virtual spec when it matches the constraints in the when clause. Here, when mpich is at version 3 or higher, it provides MPI up to version 3. When mpich is at version 1 or higher, it provides the MPI virtual package at version 1. conditional queen ๐
The when qualifier ensures that Spack selects a suitably high version of mpich to satisfy some other package that depends_on a particular version of MPI. It will also prevent a user from building with too low a version of mpich. For example, suppose the package foo declares this:
class Foo(Package):
...
depends_on("mpi@2")Suppose a user invokes spack install like this:
$ spack install foo ^mpich@1.0Spack will fail with a constraint violation, because the version of MPICH requested is too low for the mpi requirement in foo. ๐ซ read the room, ^mpich@1.0.
Custom attributes
Often a package will need to provide attributes for dependents to query various details about what it provides. While any number of custom defined attributes can be implemented by a package, the four specific attributes described below are always available on every package with default implementations โ plus the ability to customize with alternate implementations in the case of virtual packages provided. ๐ธ
homeโ the installation path for the package. Default:spec.prefix.commandโ an executable command for the package. Default:spec.namefound in.home.bin.headersโ a list of headers provided by the package. Default: all headers searched recursively in.home.include.libsโ a list of libraries provided by the package. Default:lib{spec.name}searched recursively in.homestarting withlib,lib64, then the rest of.home.
Each of these can be customized by implementing the relevant attribute as a @property in the package's class:
class Foo(Package):
...
@property
def libs(self):
# The library provided by Foo is libMyFoo.so
return find_libraries("libMyFoo", root=self.home, recursive=True)A package may also provide a custom implementation of each attribute for the virtual packages it provides by implementing the virtualpackagename_attributename property in the package's class. The implementation used is the first one found from:
- Specialized virtual:
Package.virtualpackagename_attributename - Generic package:
Package.attributename - Default
The use of customized attributes is demonstrated in the next example. ๐
Example: Customized attributes for virtual packages
Consider a package foo that can optionally provide two virtual packages bar and baz. When both are enabled, the installation tree appears as follows:
include/foo.h
include/bar/bar.h
lib64/libFoo.so
lib64/libFooBar.so
baz/include/baz/baz.h
baz/lib/libFooBaz.soThe install tree shows that foo is providing the header include/foo.h and library lib64/libFoo.so in its install prefix. The virtual package bar is providing include/bar/bar.h and library lib64/libFooBar.so, also in foo's install prefix. The baz package, however, is provided in the baz subdirectory of foo's prefix with the include/baz/baz.h header and lib/libFooBaz.so library. Such a package could implement the optional attributes as follows:
class Foo(Package):
...
variant("bar", default=False, description="Enable the Foo implementation of bar")
variant("baz", default=False, description="Enable the Foo implementation of baz")
...
provides("bar", when="+bar")
provides("baz", when="+baz")
....
# Just the foo headers
@property
def headers(self):
return find_headers("foo", root=self.home.include, recursive=False)
# Just the foo libraries
@property
def libs(self):
return find_libraries("libFoo", root=self.home, recursive=True)
# The header provided by the bar virtual package
@property
def bar_headers(self):
return find_headers("bar/bar.h", root=self.home.include, recursive=False)
# The library provided by the bar virtual package
@property
def bar_libs(self):
return find_libraries("libFooBar", root=self.home, recursive=True)
# The baz virtual package home
@property
def baz_home(self):
return self.prefix.baz
# The header provided by the baz virtual package
@property
def baz_headers(self):
return find_headers("baz/baz", root=self.baz_home.include, recursive=False)
# The library provided by the baz virtual package
@property
def baz_libs(self):
return find_libraries("libFooBaz", root=self.baz_home, recursive=True)Now consider another package, foo-app, depending on all three:
class FooApp(CMakePackage):
...
depends_on("foo")
depends_on("bar")
depends_on("baz")The resulting spec objects for its dependencies show the result of the above attribute implementations:
# The core headers and libraries of the foo package
>>> spec["foo"]
foo@1.0%gcc@11.3.1+bar+baz arch=linux-fedora35-haswell
>>> spec["foo"].prefix
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6"
# home defaults to the package install prefix without an explicit implementation
>>> spec["foo"].home
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6"
# foo headers from the foo prefix
>>> spec["foo"].headers
HeaderList([
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/include/foo.h",
])
# foo include directories from the foo prefix
>>> spec["foo"].headers.directories
["/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/include"]
# foo libraries from the foo prefix
>>> spec["foo"].libs
LibraryList([
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/lib64/libFoo.so",
])โฆand here's the rest of the foo-prefix saga โ bar lives inside foo's prefix, baz gets a subdirectory of its own. watch the .home do its thing ๐ซถ
# foo library directories from the foo prefix
>>> spec["foo"].libs.directories
["/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/lib64"]
# The virtual bar package in the same prefix as foo
# bar resolves to the foo package
>>> spec["bar"]
foo@1.0%gcc@11.3.1+bar+baz arch=linux-fedora35-haswell
>>> spec["bar"].prefix
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6"
# home defaults to the foo prefix without either a Foo.bar_home
# or Foo.home implementation
>>> spec["bar"].home
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6"
# bar header in the foo prefix
>>> spec["bar"].headers
HeaderList([
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/include/bar/bar.h"
])
# bar include dirs from the foo prefix
>>> spec["bar"].headers.directories
["/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/include"]
# bar library from the foo prefix
>>> spec["bar"].libs
LibraryList([
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/lib64/libFooBar.so"
])
# bar library directories from the foo prefix
>>> spec["bar"].libs.directories
["/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/lib64"]
# The virtual baz package in a subdirectory of foo's prefix
# baz resolves to the foo package
>>> spec["baz"]
foo@1.0%gcc@11.3.1+bar+baz arch=linux-fedora35-haswell
>>> spec["baz"].prefix
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6"
# baz_home implementation provides the subdirectory inside the foo prefix
>>> spec["baz"].home
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/baz"
# baz headers in the baz subdirectory of the foo prefix
>>> spec["baz"].headers
HeaderList([
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/baz/include/baz/baz.h"
])
# baz include directories in the baz subdirectory of the foo prefix
>>> spec["baz"].headers.directories
[
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/baz/include"
]
# baz libraries in the baz subdirectory of the foo prefix
>>> spec["baz"].libs
LibraryList([
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/baz/lib/libFooBaz.so"
])
# baz library directories in the baz subdirectory of the foo porefix
>>> spec["baz"].libs.directories
[
"/opt/spack/linux-fedora35-haswell/gcc-11.3.1/foo-1.0-ca3rczp5omy7dfzoqw4p7oc2yh3u7lt6/baz/lib"
](and yes, porefix is a typo that lives in the original forever, we respect the lore ๐)
๐ฆ Abstract & concrete specs ๐ฆ
Okay so now that we've seen how spec constraints get specified on the command line and inside package definitions, bestie, let's talk about how Spack puts all of this info together. When you run this:
$ spack install mpileaks ^callpath@1.0+debug ^libelf@0.8.11Spack parses the command line and builds a spec from the description. That spec says mpileaks should be built with the callpath library at 1.0 and with the debug option enabled, and with libelf version 0.8.11. Spack will also look at the depends_on calls in all of these packages, and it will build a spec from that. The specs from the command line and the specs built from package descriptions are then combined, and the constraints are checked against each other to make sure they're satisfiable. ๐
What we have after this is done is called an abstract spec. An abstract spec is partially specified. In other words, it could describe more than one build of a package. Spack does this to make things easier on the user: they should only have to specify as much of the package spec as they care about. Here's an example partial spec DAG, based on the constraints above:
mpileaks
^callpath@1.0+debug
^dyninst
^libdwarf
^libelf@0.8.11
^mpi
[graph]This diagram shows a spec DAG output as a tree, where successive levels of indentation represent a depends-on relationship. In the above DAG, we can see some packages annotated with their constraints, and some packages with no annotations at all. When there are no annotations, it means the user doesn't care what configuration of that package is built, just so long as it works. lowkey beautiful ๐ธ
๐ Concretization ๐
An abstract spec is useful for the user, but bestieโฆ you can't install an abstract spec. ๐ญ Spack has to take the abstract spec and "fill in" the remaining unspecified parts in order to install. This process is called concretization. Concretization happens in between the time the user runs spack install and the time the install() method is called. The concretized version of the spec above might look like this:
mpileaks@2.3%gcc@4.7.3 arch=linux-debian7-x86_64
^callpath@1.0%gcc@4.7.3+debug arch=linux-debian7-x86_64
^dyninst@8.1.2%gcc@4.7.3 arch=linux-debian7-x86_64
^libdwarf@20130729%gcc@4.7.3 arch=linux-debian7-x86_64
^libelf@0.8.11%gcc@4.7.3 arch=linux-debian7-x86_64
^mpich@3.0.4%gcc@4.7.3 arch=linux-debian7-x86_64
[graph]Here, all versions, compilers, and platforms are filled in, and there is a single version (no version ranges) for each package. All decisions about configuration have been made, and only after this point will Spack call the install() method for your package. ๐
Concretization in Spack is based on certain selection policies that tell Spack how to select, e.g., a version, when one is not specified explicitly. Concretization policies are discussed in more detail in Configuration Files. Sites using Spack can customize them to match the preferences of their own users.
โจ spack spec โจ
For an arbitrary spec, you can see the result of concretization by running spack spec. For example:
$ spack spec dyninst@8.0.1
dyninst@8.0.1
^libdwarf
^libelf
dyninst@8.0.1%gcc@4.7.3 arch=linux-debian7-x86_64
^libdwarf@20130729%gcc@4.7.3 arch=linux-debian7-x86_64
^libelf@0.8.13%gcc@4.7.3 arch=linux-debian7-x86_64This is useful when you want to know exactly what Spack will do when you ask for a particular spec. ngl it's giving "peek behind the curtain" and I'm obsessed ๐ง
๐ Concretization Policies ๐
A user may have certain preferences for how packages should be concretized on their system. For example, one user may prefer packages built with OpenMPI and the Intel compiler. Another user may prefer packages be built with MVAPICH and GCC.
See the Package Preferences section for more details.
๐ Common when= constraints ๐
In case a package needs many directives to share the whole when= argument, or just part of it, Spack allows you to group the common part under a context manager:
class Gcc(AutotoolsPackage):
with when("+nvptx"):
depends_on("cuda")
conflicts("@:6", msg="NVPTX only supported in gcc 7 and above")
conflicts("languages=ada")
conflicts("languages=brig")
conflicts("languages=go")The snippet above is equivalent to the more verbose:
class Gcc(AutotoolsPackage):
depends_on("cuda", when="+nvptx")
conflicts("@:6", when="+nvptx", msg="NVPTX only supported in gcc 7 and above")
conflicts("languages=ada", when="+nvptx")
conflicts("languages=brig", when="+nvptx")
conflicts("languages=go", when="+nvptx")Constraints stemming from the context are added to what is explicitly present in the when= argument of a directive, so:
with when("+elpa"):
depends_on("elpa+openmp", when="+openmp")is equivalent to:
depends_on("elpa+openmp", when="+openmp+elpa")Constraints from nested context managers are also combined together, but they are rarely needed or recommended. (like, they work, butโฆ bestie, don't ๐)
๐ Common default arguments ๐
Similarly, if directives have a common set of default arguments, you can group them together in a with default_args() block:
class PyExample(PythonPackage):
with default_args(type=("build", "run")):
depends_on("py-foo")
depends_on("py-foo@2:", when="@2:")
depends_on("py-bar")
depends_on("py-bz")The above is short for:
class PyExample(PythonPackage):
depends_on("py-foo", type=("build", "run"))
depends_on("py-foo@2:", when="@2:", type=("build", "run"))
depends_on("py-bar", type=("build", "run"))
depends_on("py-bz", type=("build", "run"))NOTE: Thewith when()context manager is composable, whilewith default_args()merely overrides the default. For example:
with default_args(when="+feature"):
depends_on("foo")
depends_on("bar")
depends_on("baz", when="+baz")is equivalent to:
depends_on("foo", when="+feature")
depends_on("bar", when="+feature")
depends_on("baz", when="+baz") # Note: not when="+feature+baz"See that last comment? that's the whole vibe check right there โ default_args(when=...) gets replaced, not stacked. ๐ฅ
๐ Conflicting Specs ๐
Suppose a user needs to install package C, which depends on packages A and B. Package A builds a library with a Python2 extension, and package B builds a library with a Python3 extension. Packages A and B cannot be loaded together in the same Python runtime:
class A(Package):
variant("python", default=True, "enable python bindings")
depends_on("python@2.7", when="+python")
def install(self, spec, prefix):
# do whatever is necessary to enable/disable python
# bindings according to variant
class B(Package):
variant("python", default=True, "enable python bindings")
depends_on("python@3.2:", when="+python")
def install(self, spec, prefix):
# do whatever is necessary to enable/disable python
# bindings according to variantPackage C needs to use the libraries from packages A and B, but does not need either of the Python extensions. In this case, package C should simply depend on the ~python variant of A and B:
class C(Package):
depends_on("A~python")
depends_on("B~python")This may require that A or B be built twice, if the user wishes to use the Python extensions provided by them: once for +python and once for ~python. Other than using a little extra disk space, that solution has no serious problems. slay, disk space is cheap, broken runtimes are not ๐
โจ
๐ ๏ธ Overriding build system defaults ๐ ๏ธ
NOTE: bestie, listen โ if you code a single class inpackage.py, then every function in the table down below can be implemented with the same signature on the*Packageinstead of on the corresponding builder. one class, all the vibes ๐
okay so most of the time, the default implementation of methods and attributes in the build system base classes is literally exactly what a packager needs, and only a tiny handful of entities need to be overwritten. usually we just need to override methods like configure_args:
def configure_args(self):
args = ["--enable-cxx"] + self.enable_or_disable("libs")
if self.spec.satisfies("libs=static"):
args.append("--with-pic")
return argsthe actual set of entities you can override in package.py depends on the build system. the build systems Spack currently supports are, in no particular order (okay, table order, but still):
genericโ Generic build system without any base implementationmakefileโ Specialized build system for software built invoking hand-written Makefilesautotoolsโ Specialized build system for software built using GNU Autotoolscmakeโ Specialized build system for software built using CMakemavenโ Specialized build system for software built using Mavenmesonโ Specialized build system for software built using Mesonnmakeโ Specialized build system for software built using NMakeqmakeโ Specialized build system for software built using QMakesconsโ Specialized build system for software built using SConswafโ Specialized build system for software built using Wafrโ Specialized build system for R extensionsoctaveโ Specialized build system for Octave packagespythonโ Specialized build system for Python extensionsperlโ Specialized build system for Perl extensionsrubyโ Specialized build system for Ruby extensionsintelโ Specialized build system for licensed Intel softwareoneapiโ Specialized build system for Intel oneAPI softwareaspell_dictโ Specialized build system for Aspell dictionaries
NOTE: Choice of the appropriate base class for a package โ in most cases packagers don't have to worry about picking the right base class for a package, becausespack createwill make the appropriate choice on their behalf. but in those rare cases where manual intervention is needed, let's be so clear about it: a package base class depends on the build system being used, not the language of the package. for example, a Python extension installed with CMake wouldextends("python")and subclass fromCMakePackage. iykyk ๐ฆ
๐ง Overriding builder methods ๐ง
build-system "phases" have default implementations that fit most of the common cases:
def configure(self, pkg, spec, prefix):
"""Run "configure", with the arguments specified by the builder and an
appropriately set prefix.
"""
options = getattr(self.pkg, "configure_flag_args", [])
options += ["--prefix={0}".format(prefix)]
options += self.configure_args()
with fs.working_dir(self.build_directory, create=True):
inspect.getmodule(self.pkg).configure(*options)it is usually sufficient for a packager to override a few build-system-specific helper methods or attributes to provide, for instance, configure arguments:
def configure_args(self):
spec = self.spec
args = ["--enable-c++"]
if spec.satisfies("%cce@9:"):
args.append("LDFLAGS=-rtlib=compiler-rt")
if (
spec.satisfies("%clang")
or spec.satisfies("%aocc")
or spec.satisfies("%arm")
or spec.satisfies("%fj")
) and not spec.satisfies("platform=darwin"):
args.append("LDFLAGS=-rtlib=compiler-rt")
if spec.satisfies("%intel@:18"):
args.append("CFLAGS=-no-gcc")
if "+sigsegv" in spec:
args.append("--with-libsigsegv-prefix={0}".format(spec["libsigsegv"].prefix))
else:
args.append("--without-libsigsegv-prefix")
# https://lists.gnu.org/archive/html/bug-m4/2016-09/msg00002.html
arch = spec.architecture
if arch.platform == "darwin" and arch.os == "sierra" and "%gcc" in spec:
args.append("ac_cv_type_struct_sched_param=yes")
return argseach specific build system has its own list of attributes and methods you can override to fine-tune the installation of a package without overriding an entire phase. to get the full tea on them, the place to go is the API docs of the build_systems module. ๐
๐ Overriding an entire phase ๐
sometimes it's genuinely necessary to override an entire phase. if the package.py contains a single class recipe (see Package class architecture), then the signature for a phase is:
class Openjpeg(CMakePackage):
def install(self, spec, prefix):
...regardless of the build system. the arguments for the phase are, and this is the important part:
selfโ This is the package object, which extendsCMakePackage. For API docs on Package objects, see Package.specโ This is the concrete spec object created by Spack from an abstract spec supplied by the user. It describes what should be installed. It will be of typeSpec.prefixโ This is the path that your install method should copy build targets into. It acts like a string, but it's actually its own special type,Prefix.
the arguments spec and prefix are passed only for convenience, since they always correspond to self.spec and self.spec.prefix respectively. literally just a lil convenience ๐ซถ
if the package.py has build instructions in a separate builder class, the signature for a phase changes slightly:
class CMakeBuilder(spack.build_systems.cmake.CMakeBuilder):
def install(self, pkg, spec, prefix):
...in this case the package is passed as the second argument, and self is the builder instance. sneaky but iconic. โจ
โจ Mixin base classes โจ
Okay so bestie, besides build systems, there are other situations where the same metadata and behavior keeps showing up and honestly it's giving copy-paste energy โ so Spack lets you pull all that into one reusable spot. Example: packages that depend on Cuda or Rocm share the exact same dependencies and constraints. To factor those attributes into a single place, Spack gives you a few mixin classes over in the spack.build_systems module:
CudaPackageโ a helper class for packages that use CUDA ๐ฉROCmPackageโ a helper class for packages that use ROCmGNUMirrorPackageโ a helper class for GNU packages ๐PythonExtensionโ a helper class for Python extensions ๐SourceforgePackageโ a helper class for packages fromsourceforge.orgSourcewarePackageโ a helper class for packages fromsourceware.orgXorgPackageโ a helper class for x.org packages
The way you use these is literally just adding them to the inheritance tree of the package that needs them, like:
class Cp2k(MakefilePackage, CudaPackage):
"""CP2K is a quantum chemistry and solid state physics software package
that can perform atomistic simulations of solid state, liquid, molecular,
periodic, material, crystal, and biological systems
"""In the example above, Cp2k inherits all the conflicts and variants that CudaPackage defines. Free stuff, no cap ๐
๐ Multiple build systems
There are cases where a package actively supports two build systems, or changes build systems as it evolves, or literally needs different build systems on different platforms. Spack handles all of that by splitting the build instructions into separate builder classes. Obsessed, honestly.
So for software that supports two build systems unconditionally, you derive from both *Package base classes and declare the possible use of multiple build systems using a directive:
class Example(CMakePackage, AutotoolsPackage):
variant("my_feature", default=True)
build_system("cmake", "autotools", default="cmake")In this case the software can be built with both autotools and cmake. And because the package supports multiple build systems, you have to declare which one is the default โ that part is not optional, bestie ๐ซถ
The additional build instructions get split into separate builder classes:
class CMakeBuilder(spack.build_systems.cmake.CMakeBuilder):
def cmake_args(self):
return [
self.define_from_variant("MY_FEATURE", "my_feature")
]
class AutotoolsBuilder(spack.build_systems.autotools.AutotoolsBuilder):
def configure_args(self):
return self.with_or_without("my-feature", variant="my_feature")In this example, spack install example +feature build_sytem=cmake will pick the CMakeBuilder and invoke cmake -DMY_FEATURE:BOOL=ON.
Similarly, spack install example +feature build_system=autotools will pick the AutotoolsBuilder and invoke ./configure --with-my-feature. Iconic behavior, both ways ๐ฅ
Dependencies are always specified in the package class. And when some dependencies depend on the choice of the build system, you can use when conditions as usual:
class Example(CMakePackage, AutotoolsPackage):
build_system("cmake", "autotools", default="cmake")
# Runtime dependencies
depends_on("ncurses")
depends_on("libxml2")
# Lowerbounds for cmake only apply when using cmake as the build system
with when("build_system=cmake"):
depends_on("cmake@3.18:", when="@2.0:", type="build")
depends_on("cmake@3:", type="build")
# Specify extra build dependencies used only in the configure script
with when("build_system=autotools"):
depends_on("perl", type="build")
depends_on("pkgconfig", type="build")Very often projects switch from one build system to another, or add support for a new build system from a certain version โ which means the choice of build system typically depends on a version range. Those situations get handled with conditional values in the build_system directive:
class Example(CMakePackage, AutotoolsPackage):
build_system(
conditional("cmake", when="@0.64:"),
conditional("autotools", when="@:0.63"),
default="cmake",
)In that example, the directive imposes a change from Autotools to CMake going from v0.63 to v0.64. Glow-up era ๐
The build_system can be used as an ordinary variant, which also means it can be used in depends_on statements. That's lowkey useful when a package requires that its dependency has a CMake config file โ meaning the dependent can only build when the dependency is built with CMake, and not Autotools. In that case, you can force the choice of the build system in the dependent:
class Dependent(CMakePackage):
depends_on("example build_system=cmake")๐ง The build environment
In general, you should not have to do much differently in your install method than you would when installing a package on the command line. Honestly? You may need to do less than you would on the command line. Queen behavior ๐
Spack tries to set environment variables and modify compiler calls so that it appears to the build system that you're building with a standard system install of everything. Obviously that's not going to cover all build systems, but it should make it easy to port packages to Spack if they use a standard build system. Usually with autotools or cmake, building and installing is easy. With builds that use custom Makefiles, you may need to add logic to modify the makefiles.
The rest of this section covers the way Spack's build environment works. Strap in ๐
๐ธ Forking install()
To give packagers free reign over their install environment, Spack forks a new process each time it invokes a package's install() method. This allows packages to have a sandboxed build environment without impacting the environments of other jobs that the main Spack process runs. Packages are free to change the environment or to modify Spack internals, because each install() call has its own dedicated process. Your own little bubble, no cap ๐ซง
๐ Environment variables
Spack sets a number of standard environment variables that serve two purposes:
- Make build systems use Spack's compiler wrappers for their builds.
- Allow build systems to find dependencies more easily
The compiler environment variables that Spack sets are:
CCโ C compilerCXXโ C++ compilerF77โ Fortran 77 compilerFCโ Fortran 90 and above compiler
Spack sets these variables so that they point to compiler wrappers. These are covered in their own section below.
All of these are standard variables respected by most build systems. If your project uses Autotools or CMake, then it should pick them up automatically when you run configure or cmake in the install() function. Many traditional builds using GNU Make and BSD make also respect these variables, so they may work with these systems. Slay, portability ๐ฆ
If your build system does not automatically pick these variables up from the environment, then you can simply pass them on the command line, or use a patch as part of your build process to get the correct compilers into the project's build system. There are also some file editing commands you can use โ those are described later in the section on file manipulation. Iykyk โจ
PATH is lowkey set up to point at your dependencies' bin directories, so you can actually use the tools those dependency packages installed while you're building. icymi, this is the whole reason $MPICH_ROOT/bin/mpicc keeps popping up in mpich's dependents โจ
CMAKE_PREFIX_PATH is a colon-separated list of prefixes where cmake goes looking for dependency libs and headers. The way this slays is that all your standard CMake find commands just look in your dependencies' paths already, so you literally do not have to hand-write stuff like -DDEPENDENCY_DIR=/path/to/dependency to cmake. More deets live in the CMake docs, iykyk ๐ฆ
PKG_CONFIG_PATH is for packages that try to sniff out their dependencies using the GNU pkg-config tool. It's giving very much the same energy as CMAKE_PREFIX_PATH: it lets a build just discover its dependencies on its own, no cap ๐
And bestie, if you want to see the environment a package is gonna build with โ or you wanna run commands inside that environment to test them out โ use the spack build-env command, documented below. Go peek ๐
๐ Failing the build
Sometimes you're just not gonna let a package install successfully unless some condition is true. And honestly? Valid. You can explicitly cause the build to fail from install() by raising an InstallError, like this:
if spec.architecture.startswith("darwin"):
raise InstallError("This package does not build on Mac OS X!")Periodt. That's the whole vibe check ๐ธ
๐ Shell command functions
Recall (or meet for the first time, no judgment) the install method from libelf:
def install(self, spec, prefix):
make("install", parallel=False)Normally in Python you'd have to write something tragic like this just to run a shell command:
import subprocess
subprocess.check_call("configure", "--prefix={0}".format(prefix))So we made it a lil easier by giving you callable wrapper objects for some shell commands ๐ซถ By default, configure, cmake, and make wrappers are provided, so you can call them way more naturally inside your package files. Iconic behaviour, honestly.
Need some other command? Use which to grab it:
sed = which("sed")
sed("s/foo/bar/", filename)The which function will search your PATH for the application. fr fr it just finds it ๐
Callable wrappers also let spack do some special little features. For example: in Spack, make is parallel by default, and Spack figures out how many cores your machine has and passes an appropriate value for -j<numjobs> when it calls make (see the parallel package attribute <attribute_parallel>). And in a package file you can supply a keyword argument, parallel=False, to the make wrapper to turn parallel make off. In the libelf package, that's exactly what lets us dodge race conditions in the library's build system ๐
๐ Compiler flags
Compiler flags set by the user through the Spec object can be passed to the build in one of three ways. By default, the build environment injects those flags directly into the compiler commands using Spack's compiler wrappers. But for cases where the build system needs to know about the compiler flags, they can be registered with the build system by alternatively passing them through environment variables or as build system arguments. And the flag_handler method is what you use to change this behavior. Three paths, pick your fighter โจ
Packages can override the flag_handler method with one of three built-in flag handlers. Their names are inject_flags, env_flags, and build_system_flags. inject_flags is the default. The env_flags method puts all of the flags into the environment variables that make uses as implicit variables ("CFLAGS", "CXXFLAGS", etc.). The build_system_flags method adds the flags as arguments to the invocation of configure or cmake, respectively.
โ ๏ธ WARNING: Passing compiler flags using build system arguments is only supported for CMake and Autotools packages. Individual packages may also differ in whether they properly respect these arguments. Do not come for me if your weird bespoke build system ignores them ๐
Individual packages may also define their own flag_handler methods. The flag_handler method takes the package instance (self), the name of the flag, and a list of the values of the flag. It will be called on each of the six compiler flags supported in Spack. It should return a triple of (injf, envf, bsf) where injf is a list of flags to inject via the Spack compiler wrappers, envf is a list of flags to set in the appropriate environment variables, and bsf is a list of flags to pass to the build system as arguments.
โ ๏ธ WARNING: Passing a non-empty list of flags to bsf for a build system that does not support build system arguments will result in an error. Not a cute error. Just an error ๐ญ
Here are the definitions of the three built-in flag handlers, verbatim bestie:
def inject_flags(pkg, name, flags):
return (flags, None, None)
def env_flags(pkg, name, flags):
return (None, flags, None)
def build_system_flags(pkg, name, flags):
return (None, None, flags)๐ NOTE: Returning[]andNoneare equivalent in aflag_handlermethod. Same energy, no difference ๐
Packages can override the default behavior either by specifying one of the built-in flag handlers:
flag_handler = env_flagsโฆor by implementing the flag_handler method themselves. Say for a package Foo we need to pass cflags, cxxflags, and cppflags through the environment, the rest of the flags through compiler wrapper injection, and we need to add -lbar to ldlibs. This flag handler method does exactly that:
def flag_handler(self, name, flags):
if name in ["cflags", "cxxflags", "cppflags"]:
return (None, flags, None)
elif name == "ldlibs":
flags.append("-lbar")
return (flags, None, None)Because these methods can pass values through environment variables, it is important not to override these variables unnecessarily (e.g. setting env["CFLAGS"]) in other package methods when you're using non-default flag handlers. In the setup_environment and setup_dependent_environment methods, use the append_flags method of the EnvironmentModifications class to append values to a list of flags whenever the flag handler is env_flags. And if the package passes flags through the environment or the build system manually (in the install method, for example), we recommend using the default flag handler, or removing manual references and implementing a custom flag_handler method that adds the desired flags to export as environment variables or pass to the build system. Manual flag passing is likely to interfere with the env_flags and build_system_flags methods. Like, highkey don't do both at once ๐ซ
In rare circumstances โ compiling and running small unit tests, for instance โ a package developer may need to know what the appropriate compiler flags are to enable features like OpenMP, c++11, c++14 and alike. To that end, the compiler classes in spack implement the following properties: openmp_flag, cxx98_flag, cxx11_flag, cxx14_flag, and cxx17_flag, which can be accessed in a package by self.compiler.cxx11_flag and alike. Note that the implementation is such that if a given compiler version does not support this feature, an error will be produced. Which honestly is a feature: package developers can also use these properties to assert that a compiler supports the requested feature. Super handy when a package supports additional variants like:
variant("openmp", default=True, description="Enable OpenMP support.")๐ Blas, Lapack and ScaLapack libraries
So multiple packages provide implementations of Blas, Lapack and ScaLapack routines, and the names of the resulting static and/or shared libraries differ from package to package. To make the install() method independent of which Blas implementation you picked, each package that provides it implements @property def blas_libs(self): to return an object of LibraryList type which simplifies usage of a set of libraries. Same deal for packages that provide Lapack and ScaLapack. Package developers are requested to use this interface โ pretty please, it's for your own good ๐
Common usage cases are:
- Space separated list of full paths
lapack_blas = spec["lapack"].libs + spec["blas"].libs
options.append(
"--with-blas-lapack-lib={0}".format(lapack_blas.joined())
)- Names of libraries and directories which contain them
blas = spec["blas"].libs
options.extend([
"-DBLAS_LIBRARY_NAMES={0}".format(";".join(blas.names)),
"-DBLAS_LIBRARY_DIRS={0}".format(";".join(blas.directories))
])- Search and link flags
math_libs = spec["scalapack"].libs + spec["lapack"].libs + spec["blas"].libs
options.append(
"-DMATH_LIBS:STRING={0}".format(math_libs.ld_flags)
)For more info, see the documentation of the LibraryList class ๐
๐ง Prefix objects
Spack passes the prefix parameter to the install method so that you can pass it along to configure, cmake, or some other installer, e.g.:
configure("--prefix={0}".format(prefix))For the most part, prefix objects behave exactly like strings โ no drama, same vibe. But for packages that don't have their own install target, or for those that implement it poorly (looking at you, libdwarf ๐), you may need to manually copy things into particular directories under the prefix. For that, you can refer to standard subdirectories without having to construct paths yourself, e.g.:
def install(self, spec, prefix):
mkdirp(prefix.bin)
install("foo-tool", prefix.bin)
mkdirp(prefix.include)
install("foo.h", prefix.include)See? prefix.bin, prefix.include โ it's giving convenience, it's giving main character energy ๐
mkdirp(prefix.lib)
install("libfoo.a", prefix.lib)Attributes of this thing get created on the fly the second you ask for them, so literally all of these just work ๐
prefix.binโ$prefix/binprefix.lib64โ$prefix/lib64prefix.share.manโ$prefix/share/manprefix.foo.bar.bazโ$prefix/foo/bar/baz
Okay but bestie, hear me: this only works if your file or directory is a valid Python variable name. If it's got dashes or dots in it, you gotta use join instead, no cap:
prefix.lib.join("libz.a")๐ฆ SPEC OBJECTS ๐ฆ
When install gets called, most of the build process is already set up for you ๐ซถ. The correct version's tarball has been downloaded and expanded. Environment variables like CC and CXX are set to point at the correct compiler and version. An install prefix has already been selected and passed in as prefix. In most cases that's literally all you need to get configure, cmake, or another install working correctly. Iconic behaviour honestly.
But there will be times when you need to know more about the build configuration. For example, some software requires that you pass special parameters to configure, like --with-libelf=/path/to/libelf or --with-mpich. You might also need to supply special compiler flags depending on the compiler. All of this info is available in the spec โจ
๐ TESTING SPEC CONSTRAINTS ๐
You can test whether your spec is configured a certain way by using the satisfies method. For example, if you wanna check whether the package's version is in a particular range, you can use specs to do that, like:
configure_args = [
"--prefix={0}".format(prefix)
]
if spec.satisfies("@1.2:1.4"):
configure_args.append("CXXFLAGS='-DWITH_FEATURE'")
configure(*configure_args)This works for compilers too, slay:
if spec.satisfies("%gcc"):
configure_args.append("CXXFLAGS='-g3 -O3'")
if spec.satisfies("%intel"):
configure_args.append("CXXFLAGS='-xSSE2 -fast'")Or for combinations of spec constraints:
if spec.satisfies("@1.2%intel"):
tty.error("Version 1.2 breaks when using Intel compiler!")You can also do similar satisfaction tests for dependencies:
if spec.satisfies("^dyninst@8.0"):
configure_args.append("CXXFLAGS=-DSPECIAL_DYNINST_FEATURE")Which is lowkey perfect for easily working around a bug in a particular dependency version ๐
You can use satisfies() to test for particular dependencies, e.g. foo.satisfies("^openmpi@1.2") or foo.satisfies("^mpich"), or you can use Python's built-in in operator:
if "libelf" in spec:
print "this package depends on libelf"This is so useful for virtual dependencies, because you can easily see what implementation was selected for this build:
if "openmpi" in spec:
configure_args.append("--with-openmpi")
elif "mpich" in spec:
configure_args.append("--with-mpich")
elif "mvapich" in spec:
configure_args.append("--with-mvapich")It's also a bit more concise than satisfies. Main character energy tbh.
โ ๏ธ NOTE: Thesatisfies()method tests whether this spec has, at least, all the constraints of the argument spec, whileintests whether a spec or any of its dependencies satisfy the provided spec. If the provided spec is anonymous (e.g.,":1.2:","+shared") or has the same name as the spec being checked, theninworks the same assatisfies(); however, use ofsatisfies()is more intuitive.
๐ ARCHITECTURE SPECIFIERS ๐
As mentioned in Support for specific microarchitectures, each node in a concretized spec object has an architecture attribute which is a triplet of platform, os and target. Each of these three items can be queried to take decisions when configuring, building or installing a package. Obsessed with this for us.
๐ Querying the platform and the operating system
Sometimes the actions to be taken to install a package might differ depending on the platform we're installing for. If that's the case we can use conditionals:
if spec.platform == "darwin":
# Actions that are specific to Darwin
args.append("--darwin-specific-flag")โฆand branch based on the current spec platform. If we need to make a package directive conditional on the platform we can instead employ the usual spec syntax and pass the corresponding constraint to the appropriate argument of that directive:
class Libnl(AutotoolsPackage):
conflicts("platform=darwin", msg="libnl requires FreeBSD or Linux")Similar considerations are also valid for the os part of a spec's architecture. For instance:
class Glib(AutotoolsPackage)
patch("old-kernels.patch", when="os=centos6")โฆwill apply the patch only when the operating system is Centos 6. ๐
๐ NOTE: Even though experienced Python programmers might recognize that there are other ways to retrieve information on the platform:if sys.platform == "darwin": # Actions that are specific to Darwin args.append("--darwin-specific-flag")querying the spec architecture's platform should be considered the preferred. The key difference is that a query onsys.platform, or anything similar, is always bound to the host on which the interpreter running Spack is located and as such it won't work correctly in environments where cross-compilation is required.
๐ง Querying the target microarchitecture
The third item of the architecture tuple is the target, which abstracts the information on the CPU microarchitecture. A list of all the targets known to Spack can be obtained via the command line:
$ spack arch --known-targets
Generic architectures (families)
aarch64 armv8.1a armv8.3a armv8.5a ppc ppc64le riscv64 sparc64 x86_64 x86_64_v3
arm armv8.2a armv8.4a armv9.0a ppc64 ppcle sparc x86 x86_64_v2 x86_64_v4
GenuineIntel - x86
i686 pentium2 pentium3 pentium4 prescott
GenuineIntel - x86_64
nocona nehalem sandybridge haswell skylake cannonlake cascadelake sapphirerapids
core2 westmere ivybridge broadwell mic_knl skylake_avx512 icelake
AuthenticAMD - x86_64
k10 bulldozer piledriver zen steamroller zen2 zen3 excavator zen4
IBM - ppc64
power7 power8 power9 power10
IBM - ppc64le
power8le power9le power10le
Cavium - aarch64
thunderx2
Fujitsu - aarch64
a64fx
ARM - aarch64
cortex_a72 neoverse_n1 neoverse_v1 neoverse_v2
Apple - aarch64
m1 m2
SiFive - riscv64
u74mcWithin directives, each of the names above can be used to match a particular target:
class Julia(Package):
# This patch is only applied on icelake microarchitectures
patch("icelake.patch", when="target=icelake")It's also possible to select all the architectures belonging to the same family using an open range:
class Julia(Package):
# This patch is applied on all x86_64 microarchitectures.
# The trailing colon that denotes an open range of targets
patch("generic_x86_64.patch", when="target=x86_64:")โฆin a way that resembles what was shown in Versions and fetching for versions. Where target objects really shine though is when they're used in methods called at configure, build or install time. In that case we can test targets for supported features, for instance:
if spec.satisfies("target=avx512"):
args.append("--with-avx512")The snippet above will append the --with-avx512 item to a list of arguments only if the corresponding feature is supported by the current target. Sometimes we need to take different actions based on the architecture family and not on the specific microarchitecture. In those cases we can check the family attribute:
if spec.target.family == "ppc64le":
args.append("--enable-power")Possible values for the family attribute are displayed by spack arch --known-targets under the "Generic architectures (families)" header. Finally, it's possible to perform actions based on whether the current microarchitecture is compatible with a known one:
if spec.target > "haswell":
args.append("--needs-at-least-haswell")The snippet above will add an item to a list of configure options only if the current architecture is a superset of haswell or, said otherwise, only if the current architecture is a later microarchitecture still compatible with haswell. ๐ฅ
๐ซฃ Using Spack on unknown microarchitectures
If Spack is used on an unknown microarchitecture it will try to perform a best match of the features it detects and will select the closest microarchitecture it has information for. In case nothing matches, it will create on the fly a new generic architecture. This is done to allow users to still be able to use Spack for their work. The software built won't be probably as optimized as it could but just as you need a newer compiler to build for newer architectures, you may need newer versions of Spack for new architectures to be correctly labeled.
๐ ACCESSING DEPENDENCIES ๐
You may need to get at some file or binary that's in the installation prefix of one of your dependencies. You can do that by sub-scripting the spec:
spec["mpi"]The value in the brackets needs to be some package name, and spec needs to depend on that package, or the operation will fail ๐ญ. For example, the above code will fail if the spec doesn't depend on mpi. The value returned is itself just another Spec object, so you can do all the same things you would do with the package's own spec:
spec["mpi"].prefix.bin
spec["mpi"].version๐ซ Multimethods and @when ๐ซ
Okay so bestie, here's the cute part: Spack lets you make multiple versions of the same instance function inside a package, and which one actually runs depends on whether the package's spec satisfies certain criteria. Iconic, honestly.
The @when annotation is the โจmagicโจ that lets packages declare multiple versions of methods like install() that depend on the package's spec. Like, for example:
class SomePackage(Package):
...
def install(self, prefix):
# Do default install
@when("arch=chaos_5_x86_64_ib")
def install(self, prefix):
# This will be executed instead of the default install if
# the package's sys_type() is chaos_5_x86_64_ib.
@when("arch=linux-debian7-x86_64")
def install(self, prefix):
# This will be executed if the package's sys_type() is
# linux-debian7-x86_64.So in that code there are literally three versions of install(), and two of them are specialized for particular platforms. Which version gets called depends on the architecture of the package spec. Slay ๐
.
And ngl, this works for methods other than install too. So if only part of your install is platform specific, you could do something more like this:
class SomePackage(Package):
...
# virtual dependence on MPI.
# could resolve to mpich, mpich2, OpenMPI
depends_on("mpi")
def setup(self):
# do nothing in the default case
pass
@when("^openmpi")
def setup(self):
# do something special when this is built with OpenMPI for
# its MPI implementations.
def install(self, prefix):
# Do common install stuff
self.setup()
# Do more common install stuffYou can also write multiple @when specs that satisfy the package's spec, fr fr:
class SomePackage(Package):
...
depends_on("mpi")
def setup_mpi(self):
# the default, called when no @when specs match
pass
@when("^mpi@3:")
def setup_mpi(self):
# this will be called when mpi is version 3 or higher
pass
@when("^mpi@2:")
def setup_mpi(self):
# this will be called when mpi is version 2 or higher
pass
@when("^mpi@1:")
def setup_mpi(self):
# this will be called when mpi is version 1 or higher
passIn situations like this, the first matching spec, in declaration order will be called. And, as before, if no @when spec matches, the default method (aka the one without the @when decorator) will be called. Easy โจ
โ ๏ธ WARNING, bestie, this one matters: the default version of decorated methods must always come first. Otherwise it will override all of the platform-specific versions. There's not much we can do to get around this because of the way decorators work. Not me warning you about decorator ordering ๐ญ
๐งฐ Compiler wrappers ๐งฐ
As mentioned, CC, CXX, F77, and FC are set to point to Spack's compiler wrappers. These are simply called cc, c++, f77, and f90, and they live in $SPACK_ROOT/lib/spack/env.
$SPACK_ROOT/lib/spack/env is added first in the PATH environment variable when install() runs so that system compilers are not picked up instead. Vibe check on your PATH: passed ๐
All of these compiler wrappers point to a single compiler wrapper script that figures out which real compiler it should be building with. That comes either from spec concretization or from a user explicitly asking for a particular compiler using, e.g., %intel on the command line.
In addition to invoking the right compiler, the compiler wrappers add flags to the compile line so that dependencies can be easily found. These flags are added for each dependency, if they exist:
- ๐๏ธ Compile-time library search paths:
-L$dep_prefix/lib,-L$dep_prefix/lib64 - ๐ฅ Runtime library search paths (RPATHs):
$rpath_flag$dep_prefix/lib,$rpath_flag$dep_prefix/lib64 - ๐ Include search paths:
-I$dep_prefix/include
An example of this would be the libdwarf build, which has one dependency: libelf. Every call to cc in the libdwarf build will have -I$LIBELF_PREFIX/include, -L$LIBELF_PREFIX/lib, and $rpath_flag$LIBELF_PREFIX/lib inserted on the command line. This is done transparently to the project's build system, which will just think it's using a system where libelf is readily available. Because of this, you do not have to insert extra -I, -L, etc. on the command line. Literally so convenient.
Another useful consequence of this is that you often don't have to add extra parameters on the configure line to get autotools to find dependencies. The libdwarf install method just calls configure like this:
configure("--prefix=" + prefix)Because of the -L and -I arguments, configure will successfully find libdwarf.h and libdwarf.so, without the packager having to provide --with-libdwarf=/path/to/libdwarf on the command line. Chef's kiss ๐ซถ
๐ NOTE: For most compilers,$rpath_flagis-Wl,-rpath,. However, NAG passes its flags to GCC instead of passing them directly to the linker. Therefore, its$rpath_flagis doubly wrapped:-Wl,-Wl,,-rpath,.$rpath_flagcan be overridden on a compiler specific basis inlib/spack/spack/compilers/$compiler.py.
The compiler wrappers also pass the compiler flags specified by the user from the command line (cflags, cxxflags, fflags, cppflags, ldflags, and/or ldlibs). They do not override the canonical autotools flags with the same names (but in ALL-CAPS) that may be passed into the build by particularly challenging package scripts. ๐
๐ MPI support in Spack ๐
It is super common for high performance computing software/packages to use the Message Passing Interface (MPI). As a result of conretization, a given package can be built using different implementations of MPI such as Openmpi, MPICH or IntelMPI. So like, when your package declares that it depends_on("mpi"), it can be built with any of these mpi implementations. In some scenarios, to configure a package, one has to provide it with appropriate MPI compiler wrappers such as mpicc, mpic++. However different implementations of MPI may have different names for those wrappers. Annoying but okay ๐ค
Spack provides an idiomatic way to use MPI compilers in your package. To use MPI wrappers to compile your whole build, do this in your install() method:
env["CC"] = spec["mpi"].mpicc
env["CXX"] = spec["mpi"].mpicxx
env["F77"] = spec["mpi"].mpif77
env["FC"] = spec["mpi"].mpifcThat's all. That's the whole vibe. A longer explanation of why this works is below ๐
We don't try to force any particular build method on packagers. The decision to use MPI wrappers depends on the way the package is written, on common practice, and on "what works". Loosely, there are three types of MPI builds:
- Some build systems work well without the wrappers and can treat MPI as an external library, where the person doing the build has to supply includes/libs/etc. This is fairly uncommon.
- Others really want the wrappers and assume you're using an MPI "compiler" โ i.e., they have no mechanism to add MPI includes/libraries/etc.
- CMake's
FindMPIneeds the compiler wrappers, but it uses them to extract-I/-L/-Darguments, then treats MPI like a regular library.
Note that some CMake builds fall into case 2 because they either don't know about or don't like CMake's FindMPI support โ they just assume an MPI compiler. Also, some autotools builds fall into case 3 (e.g. here is an autotools version of CMake's FindMPI).
Given all of this, we leave the use of the wrappers up to the packager. Spack will support all three ways of building MPI packages. All three. Equal opportunity ๐
๐ฆ Packaging Conventions ๐ฆ
As mentioned above, in the install() method, CC, CXX, F77, and FC point to Spack's wrappers around the chosen compiler. Spack's wrappers are not the MPI compiler wrappers, though they do automatically add -I, -L, and -Wl,-rpath args for dependencies in a similar way. The MPI wrappers are a bit different in that they also add -l arguments for the MPI libraries, and some add special -D arguments to trigger build options in MPI programs.
For case 1 above, you generally don't need to do more than patch your Makefile or add configure args as you normally would.
For case 3, you don't need to do much of anything, as Spack puts the MPI compiler wrappers in the PATH, and the build will find them and interrogate them. They'll figure it out โจ
For case 2, things are a bit more complicated, as you'll need to tell the build to use the MPI compiler wrappers instead of Spack's compiler wrappers. All it takes is some lines like this:
env["CC"] = spec["mpi"].mpicc
env["CXX"] = spec["mpi"].mpicxx
env["F77"] = spec["mpi"].mpif77
env["FC"] = spec["mpi"].mpifcOr, if you pass CC, CXX, etc. directly to your build with, e.g., --with-cc=<path>, you'll want to substitute spec["mpi"].mpicc in there instead, e.g.:
configure("โprefix=%s" % prefix,
"โwith-cc=%s" % spec["mpi"].mpicc)Now, you may think that doing this will lose the includes, library paths, and RPATHs that Spack's compiler wrapper get you, but we've actually set things up so that the MPI compiler wrappers use Spack's compiler wrappers when run from within Spack. So using the MPI wrappers should really be as simple as the code above. No cap ๐ง
๐ spec["mpi"] ๐
Ok, so how does all this work?
If your package has a virtual dependency like mpi, then referring to spec["mpi"] within install() will get you the concrete mpi implementation in your dependency DAG. That is, a spec object just like the one passed to install, only the MPI implementations all set some additional properties on it to help you out. E.g., in mvapich2, you'll find this:
def setup_dependent_package(self, module, dependent_spec):
# For Cray MPIs, the regular compiler wrappers *are* the MPI wrappers.
# Cray MPIs always have cray in the module name, e.g. "cray-mvapich"
if self.spec.satisfies("platform=cray") and spack_cc is not None:
self.spec.mpicc = spack_cc
self.spec.mpicxx = spack_cxx
self.spec.mpifc = spack_fc
self.spec.mpif77 = spack_f77
else:
self.spec.mpicc = join_path(self.prefix.bin, "mpicc")
self.spec.mpicxx = join_path(self.prefix.bin, "mpicxx")
self.spec.mpifc = join_path(self.prefix.bin, "mpif90")
self.spec.mpif77 = join_path(self.prefix.bin, "mpif77")
self.spec.mpicxx_shared_libs = [
os.path.join(self.prefix.lib, "libmpicxx.{0}".format(dso_suffix)),
os.path.join(self.prefix.lib, "libmpi.{0}".format(dso_suffix)),
]And that's the whole trick, bestie ๐
That little snippet of code is the whole โจmagicโจ: it lets the mvapich2 package stick an mpicc property onto the mvapich2 node in the DAG, so anything depending on it can just reach in and grab it ๐ซถ. openmpi and mpich do basically the same thing. So no matter which MPI you're rolling with, spec["mpi"].mpicc hands you the location of the MPI compilers โ slay. That's what gives us a fairly simple polymorphic interface for info about virtual dependencies like MPI, iykyk ๐
.
Wrapping wrappers ๐
So here's the situation: Spack loves using its own compiler wrappers, because they make it easy to add RPATHs to builds and they try really hard to make sure your builds use the right dependencies ๐ฅ. But that does not play nicely by default with MPI, so we gotta do a couple tricks:
- If we build MPI with Spack's wrappers,
mpiccand friends get installed with hard-coded paths to Spack's wrappers, and using them from outside of Spack will fail because they only work within Spack ๐ญ. To fix this, we patchmpiccand friends to use the regular compilers. Look at thefilter_compilersmethod inmpich,openmpi, ormvapich2for the deets. - We still want to use the Spack compiler wrappers when Spack is calling
mpicc. Luckily, wrappers in all mainstream MPI implementations provide environment variables that let us dynamically set the compiler to be used bympicc,mpicxx, etc. Denis pasted some code from this below โ Spack's build environment setsMPICC,MPICXX, etc. for mpich derivatives andOMPI_CC,OMPI_CXX, etc. for OpenMPI. This makes the MPI compiler wrappers use the Spack compiler wrappers so your dependencies still get proper RPATHs even if you use the MPI wrappers ๐.
MPI on Cray machines ๐ฅ๏ธ
Okay bestie, Cray is its own whole vibe: the Cray programming environment notably uses ITS OWN compiler wrappers, which function like MPI wrappers. On Cray systems, the CC, cc, and ftn wrappers ARE the MPI compiler wrappers, and it's assumed you'll use them for all of your builds ๐. So on Cray we don't bother with mpicc, mpicxx, etc. โ Spack MPI implementations set spec["mpi"].mpicc to point to Spack's wrappers, which wrap the Cray wrappers, which wrap the regular compilers and include MPI flags ๐. That may seem complicated, but for packagers it means the same code for using MPI wrappers will work, even on a Cray:
env["CC"] = spec["mpi"].mpiccThis is because on Cray, spec["mpi"].mpicc is just spack_cc โญ.
Checking an installation ๐
A package that appears to install successfully does not mean it's actually installed correctly or that it'll keep working forever, no cap ๐. There are a number of possible points of failure, so Spack provides features for checking the software along the way.
Failures can occur during and after the installation process. The build may start but the software doesn't end up fully installed ๐ฌ. The installed software may not work at all, or not as expected. The software may work after being installed but, due to changes on the system, may stop working days, weeks, or months after being installed ๐.
This section describes Spack's support for checks that can be performed during and after its installation. The former checks are referred to as build-time tests and the latter as stand-alone (or smoke) tests.
Build-time tests ๐งช
Spack infers the status of a build based on the contents of the install prefix. Success is assumed if anything (e.g., a file, directory) is written after install() completes. Otherwise, the build is assumed to have failed ๐ญ. However, the presence of install prefix contents is not a sufficient indicator of success, so Spack supports the addition of tests that can be performed during spack install processing.
Consider a simple autotools build using the following commands:
$ ./configure --prefix=/path/to/installation/prefix
$ make
$ make installStandard Autotools and CMake do not write anything to the prefix from the configure and make commands. Files are only written from the make install after the build completes ๐.
NOTE: If you want to learn more about Autotools and CMake packages in Spack, refer toAutotoolsPackageandCMakePackage, respectively.
What can you do to check that the build is progressing satisfactorily? If there are specific files and/or directories expected of a successful installation, you can add basic, fast sanity checks โจ. You can also add checks to be performed after one or more installation phases.
NOTE: Build-time tests are performed when the--testoption is passed tospack install.
WARNING: Build-time test failures result in a failed installation of the software ๐.
Adding sanity checks โ
Unfortunately, many builds of scientific software modify the installation prefix before make install. Builds like this can falsely report success when an error occurs before the installation is complete ๐ค. Simple sanity checks can be used to identify files and/or directories that are required of a successful installation. Spack checks for the presence of the files and directories after install() runs.
If any of the listed files or directories are missing, then the build will fail and the install prefix will be removed ๐๏ธ. If they all exist, then Spack considers the build successful from a sanity check perspective and keeps the prefix in place ๐ฅณ.
For example, the sanity checks for the reframe package below specify that eight paths must exist within the installation prefix after the install method completes:
class Reframe(Package):
...
# sanity check
sanity_check_is_file = [join_path("bin", "reframe")]
sanity_check_is_dir = ["bin", "config", "docs", "reframe", "tutorials",
"unittests", "cscs-checks"]When you run spack install with tests enabled, Spack will ensure that a successfully installed package has the required files and/or directories ๐.
For example, running:
$ spack install --test=root reframeresults in spack checking that the installation created the following file:
self.prefix.bin.reframe
and the following directories:
self.prefix.binself.prefix.configself.prefix.docsself.prefix.reframeself.prefix.tutorialsself.prefix.unittestsself.prefix.cscs-checks
If any of these paths are missing, then Spack considers the installation to have failed ๐ญ.
NOTE: You MUST usesanity_check_is_fileto specify required files andsanity_check_is_dirfor required directories. Like, mandatory, fr fr.
Adding installation phase tests ๐งต
Sometimes packages appear to build "correctly" only to have run-time behavior issues discovered at a later stage โ like after a full software stack relying on them has been built ๐. Checks can be performed at different phases of the package installation to possibly avoid these types of problems. Some checks are built-in to different build systems, while others will need to be added to the package.
Built-in installation phase tests are provided by packages inheriting from select build systems, where naming conventions are used to identify typical test identifiers for those systems ๐. In general, you won't need to add anything to your package to take advantage of these tests if your software's build system complies with the convention; otherwise, you'll want or need to override the post-phase method to perform other checks ๐ซถ.
Built-in installation phase tests โจ
Okay so some build system classes are already iconic and come with their own phase tests built in โ no cap. Here's the whole rundown, bestie, every single one:
AutotoolsPackageโ post-build:check(runsmake test,make check); post-install:installcheck(runsmake installcheck)CachedCMakePackageโ post-build:check(runsmake check,make test); post-install: Not applicableCMakePackageโ post-build:check(runsmake check,make test); post-install: Not applicableMakefilePackageโ post-build:check(runsmake test,make check); post-install:installcheck(runsmake installcheck)MesonPackageโ post-build:check(runsmake test,make check); post-install: Not applicablePerlPackageโ post-build:check(runsmake test); post-install: Not applicablePythonPackageโ post-build: Not applicable; post-install:test(module imports)QMakePackageโ post-build:check(runsmake check); post-install: Not applicableSConsPackageโ post-build:build_test(must be overridden); post-install: Not applicableSIPPackageโ post-build: Not applicable; post-install:test(module imports)WafPackageโ post-build:build_test(must be overridden); post-install:install_test(must be overridden)
Like, for example, the Libelf package inherits from AutotoolsPackage and its Makefile has a standard check target. So Spack will just automatically run make check after the build phase when it's installed using the --test option, slay, like this:
$ spack install --test=root libelfAnd on top of overriding any built-in build system installation phase tests, you can write your own install phase tests, bestie. You'll need two decorators for each phase test method:
run_afteron_package_attributes
The first decorator tells Spack when in the installation process to run your test method installation process โ namely, after the provided installation phase. The second decorator tells Spack to only run the checks when the --test option is provided on the command line. Lowkey genius.
NOTE: Be sure to place the directives above your test method in the orderrun_afterthenon_package_attributes. Order matters, don't play yourself.
NOTE: You also want to be sure the package supports the phase you use in therun_afterdirective. For example,PackageBaseonly supports theinstallphase while theAutotoolsPackageandMakefilePackagesupport bothinstallandbuildphases.
Assuming both build and install phases are available to you, you could add additional checks to be performed after each of those phases based on the skeleton provided below ๐ซถ:
class YourMakefilePackage(MakefilePackage):
...
@run_after("build")
@on_package_attributes(run_tests=True)
def check_build(self):
# Add your custom post-build phase tests
pass
@run_after("install")
@on_package_attributes(run_tests=True)
def check_install(self):
# Add your custom post-install phase tests
passNOTE: You could also schedule work to be done before a given phase using the run_before decorator. Iconic behaviour, honestly.
By way of a concrete example, the reframe package mentioned previously has a simple installation phase check that runs the installed executable. The check is implemented as follows:
class Reframe(Package):
...
# check if we can run reframe
@run_after("install")
@on_package_attributes(run_tests=True)
def check_list(self):
with working_dir(self.stage.source_path):
reframe = Executable(self.prefix.bin.reframe)
reframe("-l")WARNING: The API for adding tests is not yet considered stable and may change in future releases. So like, don't get too attached to the exact shape of it, bestie. ๐
Checking build-time test results ๐ต๏ธ
So you ran all these tests with spack install --test and now you wanna see how they went? Easy. You check the spec's install-time-test-log.txt file โ whose location depends on whether the spec installed successfully or not. Ngl, that distinction matters.
A successful installation results in the build and stage logs being copied to the .spack subdirectory of the spec's prefix. For example:
$ spack install --test=root zlib@1.2.13
...
[+] /home/user/spack/opt/spack/linux-rhel8-broadwell/gcc-10.3.1/zlib-1.2.13-tehu6cbsujufa2tb6pu3xvc6echjstv6
$ cat /home/user/spack/opt/spack/linux-rhel8-broadwell/gcc-10.3.1/zlib-1.2.13-tehu6cbsujufa2tb6pu3xvc6echjstv6/.spack/install-time-test-log.txtIf the installation fails due to build-time test failures, then both logs will be left in the build stage directory, as illustrated below ๐ญ:
$ spack install --test=root zlib@1.2.13
...
See build log for details:
/var/tmp/user/spack-stage/spack-stage-zlib-1.2.13-lxfsivs4htfdewxe7hbi2b3tekj4make/spack-build-out.txt
$ cat /var/tmp/user/spack-stage/spack-stage-zlib-1.2.13-lxfsivs4htfdewxe7hbi2b3tekj4make/install-time-test-log.txtStand-alone tests ๐
Okay so this is a whole different vibe. While build-time tests are integrated with the build process, stand-alone tests are expected to run days, weeks, even months after the software is installed. The goal is to provide a mechanism for gaining confidence that packages work as installed and continue to work as the underlying software evolves. Packages can add and inherit stand-alone tests. The spack test command is used to manage stand-alone testing. Obsessed with her.
NOTE: Execution speed is important since these tests are intended to quickly assess whether installed specs work on the system. Consequently, they should run relatively quickly โ as in on the order of at most a few minutes โ while ideally executing all, or at least key aspects of the installed software.
NOTE: Failing stand-alone tests indicate problems with the installation and, therefore, there is no reason to proceed with more resource-intensive tests until those have been investigated. Passing stand-alone tests indicate that more thorough testing, such as running extensive unit or regression tests, or tests that run at scale, can proceed without wasting resources on a problematic installation. Basically, vibe check first, big tests second. ๐ฆ
Tests are defined in the package using methods with names beginning test_. This allows Spack to support multiple independent checks, or parts. Files needed for testing โ such as source, data, and expected outputs โ may be saved from the build and/or stored with the package in the repository. Regardless of origin, these files are automatically copied to the spec's test stage directory prior to execution of the test method(s). Spack also provides some helper functions to facilitate processing. So convenient, honestly.
Configuring the test stage directory ๐
Stand-alone tests utilize a test stage directory for building, running, and tracking results โ the same way Spack uses a build stage directory. The default test stage root directory, ~/.spack/test, is defined in etc/spack/defaults/config.yaml. This location is customizable by adding or changing the test_stage path in the high-level config of the appropriate config.yaml file, such that:
config:
test_stage: /path/to/test/stagePackages can use the self.test_suite.stage property to access this setting. Other package properties that provide access to spec-specific subdirectories and files are described in accessing staged files. Iykyk.
NOTE: The test stage path is the root directory for the entire suite. In other words, it is the root directory for all specs being tested by thespack test runcommand. Each spec gets its own stage subdirectory. Useself.test_suite.test_dir_for_spec(self.spec)to access the spec-specific test stage directory. Don't mix them up bestie, that's main character energy for the wrong reason ๐
๐ Adding stand-alone tests ๐
okay so bestie, you want to write actual tests that run against your installed software? iconic. here's how it works ๐
Test recipes live inside the package itself, in methods whose names start with test_. That naming convention is the whole vibe check โ it's what lets you write multiple independent tests in one package. And each of those methods gets to see everything Spack knows about the build: the options, the compilers, the dependencies, all of it. That's what lets you customize a test to match however the thing was actually built. โจ
You can use standard python assert statements, or literally any other error-reporting mechanism you like. Spack catches those exceptions automatically and reports them as test failures. no drama, it just works. ๐ซถ
Each test method is basically an implicit test part โ named after the method itself, and its purpose is the method's docstring. Giving it a purpose is highkey important: it gives you context when you're debugging later. A test method can also contain embedded test parts (more on those in a sec).
Spack prints the test name and the purpose before running each test method and any embedded test parts, so you always know where you are. ๐
Example time โ MyPackage here gives you two basic installation tests: test_always_fails and test_example. And yeah, the first one always fails, the name is not a lie ๐ญ the second just runs the installed example.
class MyPackage(Package):
...
def test_always_fails(self):
"""use assert to always fail"""
assert False
def test_example(self):
"""run installed example"""
example = which(self.prefix.bin.example)
example()And here's what the output looks like when you run the tests and it identifies each test part:
$ spack test run --alias mypackage mypackage@1.0
==> Spack test mypackage
...
$ spack test results -l mypackage
==> Results for test suite 'mypackage':
...
==> [2023-03-10-16:03:56.625204] test: test_always_fails: use assert to always fail
...
FAILED
==> [2023-03-10-16:03:56.625439] test: test_example: run installed example
...
PASSEDNOTE: if MyPackage were a recipe for a library, then your tests should build an example or test program, and then execute that. that's the move. ๐
A test method can include test parts using the test_part context manager. Each part is treated as an independent check, which is the whole point โ it means subsequent test parts still get to execute even if an earlier one fails. nothing gets skipped just because one thing broke. ๐
The signature for test_part is:
def test_part(pkg, test_name, purpose, work_dir=".", verbose=False):where each argument means:
pkgis an instance of the package for the spec under test.test_nameis the name of the test part, and it must start withtest_.purposeis a brief description used as a heading for the test part. Output from the test gets written to a test log file, so you can search for the test name and purpose to confirm test parts and debug stuff.work_diris the path to the directory the test will run in. The default ofNone, or".", corresponds to the spec's test stage (i.e.,self.test_suite.test_dir_for_spec(self.spec)).
Tests should not run under the installation directory. ๐จ Using the package spec's installation directory for building and running tests is strongly discouraged โ it causes permission errors for shared spack instances and facilities that install software in read-only file systems or directories. don't do it, seriously.
So let's say MyPackage actually installs two examples you want to test. You can write these as separate checks, or โ as shown below โ as embedded test parts. ๐ธ
class MyPackage(Package):
...
def test_example(self):
"""run installed examples"""
for example in ["ex1", "ex2"]:
with test_part(
self,
f"test_example_{example}",
purpose=f"run installed {example}",
):
exe = which(join_path(self.prefix.bin, example))
exe()In this case you get an implicit test part for test_example, plus separate sub-parts for ex1 and ex2. The second sub-part runs no matter what โ even if the first one passes out ๐ Here's the test log for a run where the first executable fails and the second passes:
$ spack test run --alias mypackage mypackage@1.0
==> Spack test mypackage
...
$ spack test results -l mypackage
==> Results for test suite 'mypackage':
...
==> [2023-03-10-16:03:56.625204] test: test_example: run installed examples
==> [2023-03-10-16:03:56.625439] test: test_example_ex1: run installed ex1
...
FAILED
==> [2023-03-10-16:03:56.625555] test: test_example_ex2: run installed ex2
...
PASSED
...WARNING: test results reporting requires that each test method and embedded test part for a package has a unique name. no duplicate names, bestie. ๐
Stand-alone tests run in an environment that gives them access to everything Spack knows about how the software was built โ build options, dependencies, compilers, the whole lore. ๐ง Build options and dependencies are accessed with the normal spec checks. Examples of checking variant settings and spec constraints can be found at the provided links. And if you want to access the compilers used by the build in stand-alone tests, you have to set a package property, as described below. ๐
๐ ๏ธ Enabling test compilation ๐ ๏ธ
So if you want to build and run binaries inside your tests, you need to tell Spack to load the package's compiler configuration. The way you do that is by setting the package's test_requires_compiler property to True.
Setting it to True gives you access to the compiler through the canonical environment variables (e.g., CC, CXX, FC, F77). It also gives you access to build dependencies like cmake through their spec objects (e.g., self.spec["cmake"].prefix.bin.cmake). literally so handy. ๐ฅ
NOTE: thetest_requires_compilerproperty should be added at the top of the package, near other attributes like thehomepageandurl. keep it tidy. โจ
Below is an example of using this feature to compile an example:
class MyLibrary(Package):
...
test_requires_compiler = True
...
def test_cxx_example(self):
"""build and run cxx-example"""
exe = "cxx-example"
...
cxx = which(os.environ["CXX"])
cxx(
f"-L{self.prefix.lib}",
f"-I{self.prefix.include}",
f"{exe}.cpp",
"-o", exe
)
cxx_example = which(exe)
cxx_example()๐ฆ Saving build-time files ๐ฆ
NOTE: we highly recommend re-using build-time test sources and pared down input files for testing installed software. Those files are easier to keep synchronized with the software's capabilities, since they live right there in the software's repository. ๐ If that's not possible, you can add test-related files to the package repository (see adding custom files). You'll need to maintain them so they work across the listed or supported versions of the package. ๐
You can use the cache_extra_test_sources helper to copy directories and/or files from the source build stage directory over to the package's installation directory.
The signature for cache_extra_test_sources is:
def cache_extra_test_sources(pkg, srcs):where each argument means:
pkgis an instance of the package for the spec under test.srcsis a string or a list of strings corresponding to the paths of subdirectories and/or files needed for stand-alone testing.
Those paths must be relative to the staged source directory. The contents of the subdirectories and files get copied to a special test cache subdirectory of the installation prefix. Then, before any stand-alone tests execute, they're automatically copied to the appropriate relative paths under the test stage directory. so organized. ๐ฆ
For example, here's a package method that copies everything in the tests subdirectory plus the foo.c and bar.c files from examples, and then uses foo.c in a test method:
class MyLibPackage(Package):
...
@run_after("install")
def copy_test_files(self):
srcs = ["tests",
join_path("examples", "foo.c"),
join_path("examples", "bar.c")]
cache_extra_test_sources(self, srcs)
def test_foo(self):
exe = "foo"
src_dir = self.test_suite.current_test_cache_dir.examples
with working_dir(src_dir):
cc = which(os.environ["CC"])
cc(
f"-L{self.prefix.lib}",
f"-I{self.prefix.include}",
f"{exe}.c",
"-o", exe
)
foo = which(exe)
foo()So in this case, the method copies the associated files from the build stage, after the software is installed, into the package's test cache directory. Then test_foo builds foo using foo.c before running the program. ๐โโ๏ธ
NOTE: the method namecopy_test_fileshere is just for illustration purposes. You're free to use a name that's more suited to your package. ๐ The key to copying files for stand-alone testing at build time is using therun_afterdirective โ it guarantees the associated files get copied after the provided build stage, where the files and the installation prefix are available.
These paths are automatically copied from the cache to the test stage directory before any stand-alone tests execute. Tests access the files using the self.test_suite.current_test_cache_dir property. In our example above, test methods can use these paths to reference the copy of each entry listed in srcs, respectively:
self.test_suite.current_test_cache_dir.testsjoin_path(self.test_suite.current_test_cache_dir.examples, "foo.c")join_path(self.test_suite.current_test_cache_dir.examples, "bar.c")
Library packages should build stand-alone tests ๐ โ library developers will want to build the associated tests against their installed libraries before running them. like, obviously. ๐
NOTE: while source and input files are generally recommended, binaries may also be cached by the build process. Only you, as the package writer or maintainer, know whether those files would actually be appropriate for testing the installed software weeks to months later. ๐ that's a you call, bestie.
NOTE: if one or more of the copied files needs tweaking so it points at the installed software, pls make those changes to the cached files, insidecopy_test_sources, and after the call tocache_extra_test_sources(). Bestie this is not arbitrary โ it lowkey saves you so much unnecessary work in the test method itself, and it keeps you out of trouble when you're testing on shared instances and facility deployments. ๐
The filter_file function is genuinely iconic for exactly this kind of edit. See file manipulation.
๐ธ Adding custom files ๐ธ
Sometimes you just need extra files lying around to build stuff or check test results. Think:
- test source files
- test input files
- test build scripts
- expected test outputs
Getting these files from the software repository is preferred (peep adding build-time files), but ngl there are situations where that's just not feasible โ like, the software isn't being actively maintained. ๐ญ So when test files can't be pulled from the repo, or you want to supplement what's already there, Spack lets you drop additional files under the test subdirectory of the package in the Spack repository. ๐
Spack auto-copies the contents of that directory into the test staging directory before running stand-alone tests. Your test methods reach those files through the self.test_suite.current_test_data_dir property, like this:
class MyLibrary(Package):
...
test_requires_compiler = True
...
def test_example(self):
"""build and run custom-example"""
data_dir = self.test_suite.current_test_data_dir
exe = "custom-example"
src = datadir.join(f"{exe}.cpp")
...
# TODO: Build custom-example using src and exe
...
custom_example = which(exe)
custom_example()๐ฆ Reading expected output from a file ๐ฆ
There's a helper function called get_escaped_text_output โ packages use it to grab and properly format text from a file holding the expected output of running some executable that might contain special characters.
The signature for get_escaped_text_output is:
def get_escaped_text_output(filename):where filename is the path to the file containing the expected output.
Your tests can get at a custom file's filename through the self.test_suite.current_test_data_dir property. Here's how you read a file you added to the package's test subdirectory:
import re
class Sqlite(AutotoolsPackage):
...
def test_example(self):
"""check example table dump"""
test_data_dir = self.test_suite.current_test_data_dir
db_filename = test_data_dir.join("packages.db")
..
expected = get_escaped_text_output(test_data_dir.join("dump.out"))
sqlite3 = which(self.prefix.bin.sqlite3)
out = sqlite3(
db_filename, ".dump", output=str.split, error=str.split
)
for exp in expected:
assert re.search(exp, out), f"Expected '{exp}' in output"Now โ if the file was instead copied out of the tests subdirectory of the staged source code, you'd get the path like this instead:
def test_example(self):
"""check example table dump"""
test_cache_dir = self.test_suite.current_test_cache_dir
db_filename = test_cache_dir.join("packages.db")Or, alternatively, if the file got copied into the share/tests subdirectory as part of the install process, your test grabs the path like so:
def test_example(self):
"""check example table dump"""
db_filename = join_path(self.prefix.share.tests, "packages.db")โจ Comparing expected to actual outputs โจ
There's another helper, check_outputs, so packages can make sure the expected outputs from running an executable are actually contained in the actual outputs. Obsessed with a good assertion tbh.
The signature for check_outputs is:
def check_outputs(expected, actual):where each argument has the expected type and meaning:
- expected is a string or list of strings containing the expected (raw) output.
- actual is a string containing the actual output from executing the command.
Calling that method is literally the equivalent of:
errors = []
for check in expected:
if not re.search(check, actual):
errors.append(f"Expected '{check}' in output '{actual}'")
if errors:
raise RuntimeError("\n ".join(errors))๐ Accessing package- and test-related files ๐
You may need to pull files from one or more locations when you're writing stand-alone tests โ happens when the software's repository straight up doesn't include test source files, or includes them but has no way to build the executables using the installed headers and libraries. ๐ In those cases you may need to reference the files relative to one or more root directory. The table below lists the relevant path properties and gives extra examples of their use. (Peep Reading expected output โ it has examples of accessing files saved from the software repository, package repository, and installation.)
๐ Directory-to-property mapping ๐
- Package (Spec) Installation โ
self.prefix- e.g.
self.prefix.include,self.prefix.lib
- e.g.
- Dependency Installation โ
self.spec["<dependency-package>"].prefix- e.g.
self.spec["trilinos"].prefix.include
- e.g.
- Test Suite Stage โ
self.test_suite.stage- e.g.
join_path(self.test_suite.stage, "results.txt")
- e.g.
- Spec's Test Stage โ
self.test_suite.test_dir_for_spec(<spec>)- e.g.
self.test_suite.test_dir_for_spec(self.spec)
- e.g.
- Current Spec's Build-time Files โ
self.test_suite.current_test_cache_dir- e.g.
join_path(self.test_suite.current_test_cache_dir.examples, "foo.c")
- e.g.
- Current Spec's Custom Test Files โ
self.test_suite.current_test_data_dir- e.g.
join_path(self.test_suite.current_test_data_dir, "hello.f90")
- e.g.
๐ง Inheriting stand-alone tests ๐ง
Stand-alone tests defined in parent packages (e.g., Build Systems) and virtual packages (e.g., Virtual dependencies)? Yeah, those get executed by whatever packages inherit from them or provide interface implementations for them. Slay, free tests. ๐
The table below summarizes the stand-alone tests that will be executed along with those implemented in the package itself.
โจ Inherited/provided stand-alone tests โจ
okay so when you provide a virtual package (or inherit from a base), you get some tests for freeeee ๐ โ here's the whole handoff:
C(parent/provider package) โ compileshello.cand runs itCxxโ compiles and runs several hello programsFortranโ compiles and runs hello programs (Fandf90)Mpiโ compiles and runsmpi_hello(c, fortran)PythonPackageโ imports modules listed in theself.import_modulesproperty with defaults derived from the tarballSipPackageโ imports modules listed in theself.import_modulesproperty with defaults derived from the tarball
These tests are very basic โ like, bare minimum, "does it even turn on" energy โ so it is important that package developers and maintainers provide additional stand-alone tests customized to the package. fr fr, don't just vibe on the inherited ones ๐
WARNING: Any package that implements a test method with the same name as an inherited method overrides the inherited method. ๐ญ So if that is not the goal, and you are not explicitly calling and adding functionality to the inherited method for the test, then make sure that all test methods and embedded test parts have unique test names. Name collision = you just deleted your own test and didn't even notice. ๐
One example of a package that adds its own stand-alone tests to those "inherited" by the virtual package it provides an implementation for is the Openmpi package. (slay, openmpi, showing us how it's done ๐ซถ)
Below are snippets from running and viewing the stand-alone test results for openmpi:
$ spack test run --alias openmpi openmpi@4.1.4
==> Spack test openmpi
==> Testing package openmpi-4.1.4-ubmrigj
============================== 1 passed of 1 spec ==============================
$ spack test results -l openmpi
==> Results for test suite 'openmpi':
==> test specs:
==> openmpi-4.1.4-ubmrigj PASSED
==> Testing package openmpi-4.1.4-ubmrigj
==> [2023-03-10-16:03:56.160361] Installing $spack/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.1.4-ubmrigjrqcafh3hffqcx7yz2nc5jstra/.spack/test to $test_stage/xez37ekynfbi4e7h4zdndfemzufftnym/openmpi-4.1.4-ubmrigj/cache/openmpi
==> [2023-03-10-16:03:56.625204] test: test_bin: test installed binaries
==> [2023-03-10-16:03:56.625439] test: test_bin_mpirun: run and check output of mpirun
==> [2023-03-10-16:03:56.629807] '$spack/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.1.4-ubmrigjrqcafh3hffqcx7yz2nc5jstra/bin/mpirun' '-n' '1' 'ls' '..'
openmpi-4.1.4-ubmrigj repo
openmpi-4.1.4-ubmrigj-test-out.txt test_suite.lock
PASSED: test_bin_mpirun
...
==> [2023-03-10-16:04:01.486977] test: test_version_oshcc: ensure version of oshcc is 8.3.1
SKIPPED: test_version_oshcc: oshcc is not installed
...
==> [2023-03-10-16:04:02.215227] Completed testing
==> [2023-03-10-16:04:02.215597]
======================== SUMMARY: openmpi-4.1.4-ubmrigj ========================
Openmpi::test_bin_mpirun .. PASSED
Openmpi::test_bin_ompi_info .. PASSED
Openmpi::test_bin_oshmem_info .. SKIPPED
Openmpi::test_bin_oshrun .. SKIPPED
Openmpi::test_bin_shmemrun .. SKIPPED
Openmpi::test_bin .. PASSED
...
============================== 1 passed of 1 spec ==============================๐ธ spack test list
Packages available for install testing can be found using the spack test list command. The command outputs all installed packages that have defined stand-alone test methods. ๐โจ
Alternatively you can use the --all option to get a list of all packages that have stand-alone test methods even if the packages are not installed. (honestly the --all version is the icon it shows you the full roster ๐)
For more information, refer to spack test list.
๐ฆ spack test run
Install tests can be run for one or more installed packages using the spack test run command. A test suite is created for all of the provided specs. The command accepts the same arguments provided to spack install (see Specs & dependencies). If no specs are provided the command tests all specs in the active environment or all specs installed in the Spack instance if no environment is active.
Test suites can be named using the --alias option. Unaliased test suites use the content hash of their specs as their name. (so your suite is literally just a hash ๐ญ name your stuff bestie)
Some of the more commonly used debugging options are:
--fail-faststops testing each package after the first failure--fail-firststops testing packages after the first failure
Test output is written to a text log file by default though junit and cdash are outputs available through the --log-format option.
For more information, refer to spack test run.
๐ spack test results
The spack test results command shows results for all completed test suites by default. The alias or content hash can be provided to limit reporting to the corresponding test suite.
The --logs option includes the output generated by the associated test(s) to facilitate debugging. ๐ (this is the one you want when something's broken, trust)
The --failed option limits results shown to that of the failed tests, if any, of matching packages.
For more information, refer to spack test results.
๐ spack test find
The spack test find command lists the aliases or content hashes of all test suites whose results are available. ๐
For more information, refer to spack test find.
๐ spack test remove
The spack test remove command removes test suites to declutter the test stage directory. You are prompted to confirm the removal of each test suite unless you use the --yes-to-all option. ๐งน
For more information, refer to spack test remove.
โจ File manipulation functions โจ
Many builds are not perfect. ๐ If a build lacks an install target, or if it does not use systems like CMake or autotools, which have standard ways of setting compilers and options, you may need to edit files or install some files yourself to get them working with Spack.
You can do this with standard Python code, and Python has rich libraries with functions for file manipulation and filtering. Spack also provides a number of convenience functions of its own to make your life even easier. These functions are described in this section. ๐ซถ
All of the functions in this section can be included by simply running:
from spack import *This is already part of the boilerplate for packages created with spack create. (so bestie you already have it, no extra import grind needed ๐
)
๐ง Filtering functions
filter_file(regex, repl, *filenames, **kwargs)
Works like sed but with Python regular expression syntax. Takes a regular expression, a replacement, and a set of files. repl can be a raw string or a callable function. If it is a raw string, it can contain \1, \2, etc. to refer to capture groups in the regular expression. If it is a callable, it is passed the Python MatchObject and should return a suitable replacement string for the particular match.
Examples:
- Filtering a
Makefileto force it to use Spack's compiler wrappers:
filter_file(r"^\s*CC\s*=.*", "CC = " + spack_cc, "Makefile")
filter_file(r"^\s*CXX\s*=.*", "CXX = " + spack_cxx, "Makefile")
filter_file(r"^\s*F77\s*=.*", "F77 = " + spack_f77, "Makefile")
filter_file(r"^\s*FC\s*=.*", "FC = " + spack_fc, "Makefile")- Replacing
#!/usr/bin/perlwith#!/usr/bin/env perlinbib2xhtml:
filter_file(r"#!/usr/bin/perl",
"#!/usr/bin/env perl", prefix.bin.bib2xhtml)- Switching the compilers used by
mpich's MPI wrapper scripts fromcc, etc. to the compilers used by the Spack build:
filter_file("CC='cc'", "CC='%s'" % self.compiler.cc,
prefix.bin.mpicc)
filter_file("CXX='c++'", "CXX='%s'" % self.compiler.cxx,
prefix.bin.mpicxx)change_sed_delimiter(old_delim, new_delim, *filenames)
Some packages, like TAU, have a build system that can't install into directories with, e.g. @ in the name, because they use hard-coded sed commands in their build. ๐ฌ
change_sed_delimiter finds all sed search/replace commands and change the delimiter. e.g., if the file contains commands that look like s///, you can use this to change them to s@@@.
Example of changing s/// to s@@@ in TAU:
change_sed_delimiter("@", ";", "configure")
change_sed_delimiter("@", ";", "utils/FixMakefile")
change_sed_delimiter("@", ";", "utils/FixMakefile.sed.default")๐ File functions
ancestor(dir, n=1)
Get the nth ancestor of the directory dir.
can_access(path)
True if we can read and write to the file at path. Same as native python os.access(file_name, os.R_OK|os.W_OK).
install(src, dest)
Install a file to a particular location. For example, install a header into the include directory under the install prefix:
install("my-header.h", prefix.include)join_path(*paths)
An alias for os.path.join. This joins paths using the OS path separator. ๐
mkdirp(*paths)โ makes every directory in paths, and if any of the parents don't exist yet it just creates those too. no drama, no manual-penergy, it just handles it ๐working_dir(dirname, kwargs)โ okay so this one is a Python Context Manager and it's genuinely so handy for builds. you use it with the Pythonwithstatement: it drops you into a working directory, and when thewithblock finishes it takes you right back to where you started. think of it as a safepushd/popdcombo where thepopdis guaranteed to run at the end, even if something throws an exception. iconic behaviour honestly ๐ซถ
Example usage:
- The
libdwarfbuild first runsconfigureandmakein a subdirectory calledlibdwarf. It then implements the installation code itself. This is natural withworking_dir:
with working_dir("libdwarf"):
configure("--prefix=" + prefix, "--enable-shared")
make()
install("libdwarf.a", prefix.lib)- Many CMake builds require that you build "out of source", that is, in a subdirectory. You can handle creating and
cd'ing to the subdirectory like theLLVMpackage does:
with working_dir("spack-build", create=True):
cmake("..",
"-DLLVM_REQUIRES_RTTI=1",
"-DPYTHON_EXECUTABLE=/usr/bin/python",
"-DPYTHON_INCLUDE_DIR=/usr/include/python2.6",
"-DPYTHON_LIBRARY=/usr/lib64/libpython2.6.so",
*std_cmake_args)
make()
make("install")The create=True keyword argument causes the command to create the directory if it does not exist. bestie said "make it if it's not there" and it DID โจ
touch(path)โ creates an empty file at path. that's it. that's the whole thing. ๐
Making a package discoverable with spack external find
The simplest way to make a package discoverable with spack external find is to:
- Define the executables associated with the package
- Implement a method to determine the versions of these executables
Minimal detection
The first step is fairly simple, as it requires only to specify a package level executables attribute:
class Foo(Package):
# Each string provided here is treated as a regular expression, and
# would match for example "foo", "foobar", and "bazfoo".
executables = ["foo"]This attribute must be a list of strings. Each string is a regular expression (e.g. "gcc" would match "gcc", "gcc-8.3", "my-weird-gcc", etc.) to determine a set of system executables that might be part or this package. Note that to match only executables named "gcc" the regular expression "^gcc$" must be used. no vibes-based matching, the ^ and $ are doing the heavy lifting ๐ฅ
Finally to determine the version of each executable the determine_version method must be implemented:
@classmethod
def determine_version(cls, exe):
"""Return either the version of the executable passed as argument
or ``None`` if the version cannot be determined.
Args:
exe (str): absolute path to the executable being examined
"""This method receives as input the path to a single executable and must return as output its version as a string; if the user cannot determine the version or determines that the executable is not an instance of the package, they can return None and the exe will be discarded as a candidate. Implementing the two steps above is mandatory, and gives the package the basic ability to detect if a spec is present on the system at a given version.
NOTE: Any executable for which thedetermine_versionmethod returnsNonewill be discarded and won't appear in later stages of the workflow described below. gone. blocked. unfollowed ๐
Additional functionality
Besides the two mandatory steps described above, there are also optional methods that can be implemented to either increase the amount of details being detected or improve the robustness of the detection logic in a package.
Variants and custom attributes
The determine_variants method can be optionally implemented in a package to detect additional details of the spec:
@classmethod
def determine_variants(cls, exes, version_str):
"""Return either a variant string, a tuple of a variant string
and a dictionary of extra attributes that will be recorded in
packages.yaml or a list of those items.
Args:
exes (list of str): list of executables (absolute paths) that
live in the same prefix and share the same version
version_str (str): version associated with the list of
executables, as detected by ``determine_version``
"""This method takes as input a list of executables that live in the same prefix and share the same version string, and returns either:
- A variant string
- A tuple of a variant string and a dictionary of extra attributes
- A list of items matching either 1 or 2 (if multiple specs are detected from the set of executables)
If extra attributes are returned, they will be recorded in packages.yaml and be available for later reuse. As an example, the gcc package will record by default the different compilers found and an entry in packages.yaml would look like:
packages:
gcc:
externals:
- spec: "gcc@9.0.1 languages=c,c++,fortran"
prefix: /usr
extra_attributes:
compilers:
c: /usr/bin/x86_64-linux-gnu-gcc-9
c++: /usr/bin/x86_64-linux-gnu-g++-9
fortran: /usr/bin/x86_64-linux-gnu-gfortran-9This allows us, for instance, to keep track of executables that would be named differently if built by Spack (e.g. x86_64-linux-gnu-gcc-9 instead of just gcc). the way that prefix really said "i contain multitudes" ๐ญ
Filter matching executables
Sometimes defining the appropriate regex for the executables attribute might prove to be difficult, especially if one has to deal with corner cases or exclude "red herrings". To help keeping the regular expressions as simple as possible, each package can optionally implement a filter_executables method:
@classmethod
def filter_detected_exes(cls, prefix, exes_in_prefix):
"""Return a filtered list of the executables in prefix"""which takes as input a prefix and a list of matching executables and returns a filtered list of said executables.
Using this method has the advantage of allowing custom logic for filtering, and does not restrict the user to regular expressions only. Consider the case of detecting the GNU C++ compiler. If we try to search for executables that match g++, that would have the unwanted side effect of selecting also clang++ โ which is a C++ compiler provided by another package โ if present on the system. Trying to select executables that contain g++ but not clang would be quite complicated to do using regex only. Employing the filter_detected_exes method it becomes:
class Gcc(Package):
executables = ["g++"]
def filter_detected_exes(cls, prefix, exes_in_prefix):
return [x for x in exes_in_prefix if "clang" not in x]Another possibility that this method opens is to apply certain filtering logic when specific conditions are met (e.g. take some decisions on an OS and not on another). situational awareness, we love to see it ๐
Validate detection
To increase detection robustness, packagers may also implement a method to validate the detected Spec objects:
@classmethod
def validate_detected_spec(cls, spec, extra_attributes):
"""Validate a detected spec. Raise an exception if validation fails."""This method receives a detected spec along with its extra attributes and can be used to check that certain conditions are met by the spec. Packagers can either use assertions or raise an InvalidSpecDetected exception when the check fails. In case the conditions are not honored the spec will be discarded and any message associated with the assertion or the exception will be logged as the reason for discarding it.
As an example, a package that wants to check that the compilers attribute is in the extra attributes can implement this method like this:
@classmethod
def validate_detected_spec(cls, spec, extra_attributes):
"""Check that "compilers" is in the extra attributes."""
msg = ("the extra attribute 'compilers' must be set for "
"the detected spec '{0}'".format(spec))
assert "compilers" in extra_attributes, msgor like this:
@classmethod
def validate_detected_spec(cls, spec, extra_attributes):
"""Check that "compilers" is in the extra attributes."""
if "compilers" not in extra_attributes:
msg = ("the extra attribute 'compilers' must be set for "
"the detected spec '{0}'".format(spec))
raise InvalidSpecDetected(msg)Custom detection workflow
In the rare case when the mechanisms described so far don't fit the detection of a package, the implementation of all the methods above can be disregarded and instead a custom determine_spec_details method can be implemented directly in the package class (note that the definition of the executables attribute is still required):
@classmethod
def determine_spec_details(cls, prefix, exes_in_prefix):
# exes_in_prefix = a set of paths, each path is an executable
# prefix = a prefix that is common to each path in exes_in_prefix
# return None or [] if none of the exes represent an instance of
# the package. Return one or more Specs for each instance of the
# package which is thought to be installed in the provided prefixThis method takes as input a set of discovered executables (which match those specified by the user) as well as a common prefix shared by all of those executables. The function must return one or more spack.spec.Spec associated with the executables (it can also return None to indicate that no provided executables are associated with the package).
As an example, consider a made-up package called foo-package which builds an executable called foo. FooPackage would appear as follows:
class FooPackage(Package):
homepage = "..."
url = "..."
version(...)
# Each string provided here is treated as a regular expression, and
# would match for example "foo", "foobar", and "bazfoo".
executables = ["foo"]@classmethod
def determine_spec_details(cls, prefix, exes_in_prefix):
candidates = list(x for x in exes_in_prefix
if os.path.basename(x) == "foo")
if not candidates:
return
# This implementation is lazy and only checks the first candidate
exe_path = candidates[0]
exe = Executable(exe_path)
output = exe("--version", output=str, error=str)
version_str = ... # parse output for version string
return Spec.from_detection(
"foo-package@{0}".format(version_str)
)๐งช Add detection tests to packages
Bestie, if you wanna make sure software gets detected correctly across multiple configurations and on different systems, you can write a detection_test.yaml file and drop it in the package directory right next to package.py. This YAML file has enough info for Spack to mock an environment and try to check whether the detection logic yields the results you expect. ๐
As a general rule, attributes at the top level of detection_test.yaml represent search mechanisms, and they each map to a list of tests that should confirm the validity of the packageโs detection logic.
You can run the detection tests with this command:
$ spack audit externalsErrors that have been detected are reported to screen.
๐ฃ๏ธ Tests for PATH inspections
Detection tests that insist on PATH inspections are listed under the paths attribute:
paths:
- layout:
- executables:
- "bin/clang-3.9"
- "bin/clang++-3.9"
script: |
echo "clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)"
echo "Target: x86_64-pc-linux-gnu"
echo "Thread model: posix"
echo "InstalledDir: /usr/bin"
platforms: ["linux", "darwin"]
results:
- spec: 'llvm@3.9.1 +clang~lld~lldb'If the platforms attribute is present, tests are run only if the current host matches one of the listed platforms. Each test is performed by first creating a temporary directory structure as specified in the corresponding layout, and by then running package detection and checking that the outcome matches the expected results. The exact details on how to specify both the layout and the results are reported in the list below:
๐ Test based on PATH inspections
layoutโ Specifies the filesystem tree used for the test. Allowed values: list of objects. Required field: Yes.layout:[0]:executablesโ Relative paths for the mock executables to be created. Allowed values: list of strings. Required field: Yes.layout:[0]:scriptโ Mock logic for the executable. Allowed values: any valid shell script. Required field: Yes.resultsโ List of expected results. Allowed values: list of objects (empty if no result is expected). Required field: Yes.results:[0]:specโ A spec that is expected from detection. Allowed values: any valid spec. Required field: Yes.results:[0]:extra_attributesโ Extra attributes expected on the associated Spec. Allowed values: nested dictionary with string as keys, and regular expressions as leaf values. Required field: No.
โป๏ธ Reuse tests from other packages
When youโre using a custom repository, you can customize a package that already exists in builtin and reuse its external tests. To do that, just write a detection_tests.yaml alongside the customized package.py with an includes attribute. For instance, the detection_tests.yaml for myrepo.llvm might look like this:
includes:
- "builtin.llvm"This YAML file tells Spack to run the detection tests defined in builtin.llvm in addition to those locally defined in the file. ๐ซถ
๐ Style guidelines for packages
The following guidelines are provided in the interests of making Spack packages work in a consistent manner:
๐ท๏ธ Variant Names
Spack packages with variants similar to already-existing Spack packages should use the same name for their variants. Standard variant names are:
sharedโ defaultTrueโ Build shared librariesmpiโ defaultTrueโ Use MPIpythonโ defaultFalseโ Build Python extension
If specified in this table, the corresponding default should be used when declaring a variant.
The semantics of the shared variant are important. When a package is built ~shared, the package guarantees that no shared libraries are built. When a package is built +shared, the package guarantees that shared libraries are built, but it makes no guarantee about whether static libraries are built. ๐
๐ Version Lists
Spack packages should list supported versions with the newest first.
๐ Using home vs prefix
home and prefix are both attributes that can be queried on a packageโs dependencies, often when passing configure arguments pointing to the location of a dependency. The difference is that while prefix is the location on disk where a concrete package resides, home is the logical location that a package resides, which may be different than prefix in the case of virtual packages or other special circumstances. For most use cases inside a package, its dependency locations can be accessed via either self.spec["foo"].home or self.spec["foo"].prefix. Specific packages that should be consumed by dependents via .home instead of .prefix should be noted in their respective documentation.
See Custom attributes for more details and an example implementing a custom home attribute.
๐ ๏ธ Packaging workflow commands
When you are building packages, you will likely not get things completely right the first time. ๐ญ
The spack install command performs a number of tasks before it finally installs each package. It downloads an archive, expands it in a temporary directory, and only then gives control to the packageโs install() method. If the build doesnโt go as planned, you may want to clean up the temporary directory, or if the package isnโt downloading properly, you might want to run only the fetch stage of the build.
Spack performs best-effort installation of package dependencies by default, which means it will continue to install as many dependencies as possible after detecting failures. If you are trying to install a package with a lot of dependencies where one or more may fail to build, you might want to try the --fail-fast option to stop the installation process on the first failure.
A typical package workflow might look like this:
$ spack edit mypackage
$ spack install --fail-fast mypackage
... build breaks! ...
$ spack clean mypackage
$ spack edit mypackage
$ spack install --fail-fast mypackage
... repeat clean/install until install works ...Below are some commands that will allow you some finer-grained control over the install process.
๐ spack fetch
okay so spack fetch is literally step one of spack install, bestie. You hand it a spec, it figures out the correct download URL for the package version you asked for, downloads the archive, checks it against an MD5 checksum, and โ only if that check passes โ stashes it in a staging directory. ๐
Where's that staging directory, you ask? It lives under the first writable directory in the build_stage configuration setting. (Yes, first writable, no cap โ Spack goes down the list until something lets it write.)
The way that spack fetch is idempotent when you run it after the archive is already downloaded? Obsessed. It will not download the archive again. Periodt. ๐
๐ธ spack stage
spack stage is the second step in spack install, right after spack fetch. It expands the downloaded archive in its temporary directory, where spack install will then build it. ๐ฆ
And just like fetch, if the archive has already been expanded, stage is idempotent โ it's not gonna redo the whole thing for no reason. She's efficient like that.
โจ spack patch
After staging, Spack applies patches to the downloaded packages โ if any have been specified in the package file. ๐ฉน
This command runs the install process through the fetch, stage, and patch phases. Spack keeps track of whether patches have already been applied and skips this step if they have (again with the idempotent main character energy).
But bestie, listen: if Spack discovers that patches didn't apply cleanly on some previous run, it will restage the entire package before patching. No half-patched mess here. ๐
๐ง spack restage
spack restage restores the source code to pristine state, as it was before building. Glow-up back to factory settings, basically. ๐
It does this in one of two ways:
- If the source was fetched as a tarball, it deletes the entire build directory and re-expands the tarball.
- If the source was checked out from a repository, it deletes the build directory and checks it out again.
๐ spack clean
spack clean cleans up Spack's temporary and cached files. This one is your bestie when you need to recover disk space because temp files from interrupted or failed installs have been accumulating like unread group-chat messages. ๐ญ
Here's the whole menu:
- When called with
--stageor without arguments, this removes all staged files. - The
--downloadsoption removes cached downloads. - You can force the removal of all install failure tracking markers using the
--failuresoption. But heads up:spack installwill automatically clear relevant failure markings prior to performing the requested installation(s). - Long-lived caches, like the virtual package index, are removed using the
--misc-cacheoption. - The
--python-cacheoption removes.pyc,.pyo, and__pycache__folders. - To remove all of the above, call the command with
--all.
When called with positional arguments, this command cleans up temporary files only for a particular package. And lowkey important: if fetch, stage, or install are run again after this, Spack's build process will start from scratch. ๐ฅ
๐ซถ Keeping the stage directory on success
By default, spack install will delete the staging area once a package has been successfully built and installed. But if you're the type who needs to see things, use --keep-stage to leave the build directory intact:
$ spack install --keep-stage <spec>This allows you to inspect the build directory and potentially debug the build. You can use clean later to get rid of the unwanted temporary files. ๐
๐ Keeping the install prefix on failure
By default, spack install will delete any partially constructed install prefix if anything fails during install(). But if you want to keep the prefix anyway (e.g. to diagnose a bug), you can use --keep-prefix:
$ spack install --keep-prefix <spec>โ ๏ธ Note that this may confuse Spack into thinking that the package has been installed properly, so you may need to use spack uninstall --force to get rid of the install prefix before you build again:
$ spack uninstall --force <spec>๐ธ๏ธ Graphing dependencies
โญ spack graph
Spack provides the spack graph command for graphing dependencies. By default it generates an ASCII rendering of a spec's dependency graph. For example:
$ spack graph hdf5 target=x86_64 os=SUSEAt the top is the root package in the DAG, with dependency edges emerging from it. On a color terminal, the edges are colored by which dependency they lead to. Aesthetic and informative, we love to see it. ๐
$ spack graph --deptype=link hdf5 target=x86_64 os=SUSEThe deptype argument tells Spack what types of dependencies to graph. By default it includes link and run dependencies but not build dependencies. Supplying --deptype=link will show only link dependencies. The default is --deptype=all, which is equivalent to --deptype=build,link,run,test. Options for deptype include:
- Any combination of
build,link,run, andtestseparated by commas. allfor all types of dependencies.
You can also use spack graph to generate graphs in the widely used Dot format. For example:
$ spack graph --dot hdf5 target=x86_64 os=SUSEThis graph can be provided as input to other graphing tools, such as those in Graphviz. If you have graphviz installed, you can write straight to PDF like this:
$ spack graph --dot hdf5 | dot -Tpdf > hdf5.pdf๐ Interactive shell support
Spack provides some limited shell support to make life easier for packagers. You enable these commands by sourcing a setup file in the share/spack directory. ๐ซง
For bash or ksh, run:
export SPACK_ROOT=/path/to/spack
. $SPACK_ROOT/share/spack/setup-env.shFor csh and tcsh run:
setenv SPACK_ROOT /path/to/spack
source $SPACK_ROOT/share/spack/setup-env.cshAnd then spack cd will be available. Iconic. โจ
๐ spack cd
spack cd lets you quickly cd to pertinent directories in Spack. Suppose you've staged a package but you want to modify it before you build it โ mood:
$ spack stage libelf
==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.13.tar.gz
######################################################################## 100.0%
==> Staging archive: ~/spack/var/spack/stage/libelf@0.8.13%gcc@4.8.3 arch=linux-debian7-x86_64/libelf-0.8.13.tar.gz
==> Created stage in ~/spack/var/spack/stage/libelf@0.8.13%gcc@4.8.3 arch=linux-debian7-x86_64.
$ spack cd libelf
$ pwd
~/spack/var/spack/stage/libelf@0.8.13%gcc@4.8.3 arch=linux-debian7-x86_64/libelf-0.8.13spack cd here changed the current working directory to the directory containing the expanded libelf source code. And there are a number of other places you can cd to in the spack directory hierarchy:
$ spack cd --help
usage: spack cd [-h] [-m | -r | -i | -p | -P | -s | -S | -c | -b | -e [name]] [--first] ...
cd to spack directories in the shell
positional arguments:
spec package spec
options:
--first use the first match if multiple packages match the spec
-P, --packages top-level packages directory for Spack
-S, --stages top level stage directory
-b, --build-dir build directory for a spec (requires it to be staged first)
-c, --source-dir source directory for a spec (requires it to be staged first)
-e, --env [name] location of the named or current environment
-h, --help show this help message and exit
-i, --install-dir install prefix for spec (spec need not be installed)
-m, --module-dir spack python module directory
-p, --package-dir directory enclosing a spec's package.py file
-r, --spack-root spack installation root
-s, --stage-dir stage directory for a specSome of these change directory into package-specific locations (stage directory, install directory, package directory) and others change to core spack locations. For example, spack cd --module-dir will take you to the main python source directory of your spack install. ๐
๐ spack build-env
spack build-env functions much like the standard unix build-env command, but it takes a spec as an argument. You can use it to see the environment variables that will be set when a particular build runs, for example:
$ spack build-env mpileaks@1.1%intelThis will display the entire environment that will be set when the mpileaks@1.1%intel build runs. Vibe check: passed. โ
To run commands in a package's build environment, you can simply provide them after the spec argument to spack build-env:
$ spack cd mpileaks@1.1%intel
$ spack build-env mpileaks@1.1%intel ./configureThis will cd to the build directory and then run configure in the package's build environment. Chef's kiss. ๐งโ๐ณ
๐ spack location
spack location is the same as spack cd but it does not require shell support. It simply prints out the path you ask for, rather than cd'ing to it. Lowkey the more script-friendly sibling. ๐
In bash, this:
$ cd $(spack location --build-dir <spec>)is the same as:
$ spack cd --build-dir <spec>spack location is intended for use in scripts or makefiles that need to know where packages are installed. e.g., in a makefile you might write:
DWARF_PREFIX = $(spack location --install-dir libdwarf)
CXXFLAGS += -I$DWARF_PREFIX/include
CXXFLAGS += -L$DWARF_PREFIX/lib๐๏ธ Package class architecture
๐ NOTE: This section aims to provide a high-level knowledge of how the package class architecture evolved in Spack, and provides some insights on the current design.
Packages in Spack were originally designed to support only a single build system. The overall class structure for a package looked like:
[image]
In this architecture the base class AutotoolsPackage was responsible for both the metadata related to the autotools build system (e.g. dependencies or variants common to all packages using it), and for encoding the default installation procedure.
In reality, a non-negligible number of packages are either changing their build system during the evolution of the project, or using different build systems for different platforms. An architecture based on a single class requires hacks or other workarounds to deal with these cases. ๐ฌ
To support a model more adherent to reality, Spack v0.19 changed its internal design by extracting the attributes and methods related to building software into a separate hierarchy:
[image]
In this new format each package.py contains one *Package class that gathers all the metadata, and one or more *Builder classes that encode the installation procedure. A specific builder object is created just before the software is built, so at a time where Spack knows which build system needs to be used for the current installation, and receives a package object during initialization. Honestly? Genius design. ๐ฅ
๐งฌ build_system variant
To allow imposing conditions based on the build system, each package must have a build_system variant, which is usually inherited from base classes. This variant allows for writing metadata that is conditional on the build system:
with when("build_system=cmake"):
depends_on("cmake", type="build")and also for selecting a specific build system from a spec literal, like in the following command:
$ spack install arpack-ng build_system=autotoolsSo yeah, that's the whole package-class glow-up arc โ from single-build-system era to the *Package + *Builder split, and the build_system variant holding it all together. ๐
๐งฉ Compatibility with single-class format
Okay so internally Spack always uses builders for anything related to installing a specific piece of software โ that's the whole vibe. The builders get created in the spack.builder.create function:
def create(pkg):
"""Given a package object with an associated concrete spec,
return the builder object that can install it.
Args:
pkg (spack.package_base.PackageBase): package for which we want the builder
"""
if id(pkg) not in _BUILDERS:
_BUILDERS[id(pkg)] = _create(pkg)
return _BUILDERS[id(pkg)]โจ And to keep things backwards-compatible with the single-class format, Spack literally builds a special "adapter builder" inside this function โ but only if it doesn't detect a custom builder in the recipe: [image]
The whole point of the adapter is routing: when you access an attribute or method, it goes first through the *Package hierarchy, and then back to the base class builder. That's what the diagram is showing โ the adapter's job is to "emulate" a method resolution order, like the one those red arrows represent. Basically it's cosplaying a normal MRO and honestly? It ate. ๐
๐ Specifying License Information
So most software in Spack is open source, and most open source software ships under one or more of the common open source licenses. Bestie, specifying the license a package is released under in its package.py is just good practice. To do it: find the SPDX identifier for the project, then drop it in with the license directive:
license("<SPDX Identifier HERE>")For example, the SPDX ID for the Apache Software License, version 2.0 is Apache-2.0, so you'd write:
license("Apache-2.0")Or, for a dual-licensed package like Spack itself, you'd use an SPDX Expression with both of its licenses:
license("Apache-2.0 OR MIT")Heads up ๐ซถ โ specifying a license without awhenclause makes it apply to all versions and variants of the package, and that might not actually be true. For example, a project might have switched licenses at some point, or have certain build configurations that include files licensed differently. Spack itself used to be under theLGPL-2.1license, until it was relicensed in version0.12in 2018.
You can control when a license() directive applies by using a when= clause, just like with other directives. Example: make a specific license identifier only apply to versions up to 0.11, and a different one apply for later versions:
license("LGPL-2.1", when="@:0.11")
license("Apache-2.0 OR MIT", when="@0.12:")๐ญ Okay important: unlike most other directives, thewhen=constraints in thelicense()directive can't intersect. Spack has to be able to resolve exactly one license identifier expression for any given version. Need multiple licenses? Use SPDX expressions and operators, like above. The operators you probably care most about are:
ORโ the user chooses one license to adhere to; andANDโ the user has to adhere to all the licenses.
You may also care about license exceptions that use the WITH operator, e.g. Apache-2.0 WITH LLVM-exception.
Many of the licenses currently in the spack repositories have been automatically determined. That's great for bulk-adding license info and is most likely correct, but sometimes there are edge cases that require manual intervention ๐. To determine which licenses are validated and which aren't, there's the checked_by parameter in the license directive:
license("<license>", when="<when>", checked_by="<github username>")When you've validated a github license โ whether you did it explicitly or as part of packaging a new package โ please set the checked_by parameter to your Github username to signal that the license has been manually verified. Queen behaviour, fr fr ๐.
๐ BUILD SYSTEMS
Spack defines a bunch of classes that understand how to use common build systems (Makefiles, CMake, etc.). Package definitions can inherit these classes to streamline their builds โ iconic, no notes. ๐
This guide gives you info specific to each particular build system. It assumes you've already read the Packaging Guide and expands on those ideas for each distinct build system Spack supports:
๐งฑ Makefile
The most primitive build system a package can use is a plain Makefile. Makefiles are simple to write for small projects, but they usually require you to edit the Makefile to set platform and compiler-specific variables. It's giving "small but stubborn" ๐.
๐ฑ Phases
The MakefileBuilder and MakefilePackage base classes come with 3 phases:
editโ edit theMakefilebuildโ build the projectinstallโ install the project
By default, edit does nothing, but you can override it to replace hard-coded Makefile variables. The build and install phases run:
$ make
$ make install๐ Important files
The main file that matters for a MakefilePackage is the Makefile. This file will be named one of the following ways:
GNUmakefile(only works with GNU Make)Makefile(most common)makefile
Some Makefiles also include other configuration files. Check for an include directive in the Makefile ๐ต๏ธโโ๏ธ.
๐ Build system dependencies
Spack assumes the operating system will already have a valid make utility installed, so you don't need to add a dependency on make. But if the package uses a GNUmakefile, or the developers recommend using GNU Make, you should add a dependency on gmake:
depends_on("gmake", type="build")๐ญ Types of Makefile packages
Most of the work involved in packaging software that uses Makefiles is overriding or replacing hard-coded variables. So many packages make the mistake of hard-coding compilers โ usually for GCC or Intel. That's fine if you happen to be using that particular compiler, but Spack is designed to work with any compiler, and you need to make sure that's actually the case. ๐ค
Depending on how the Makefile is designed, there are 4 common strategies you can use to set or override the appropriate variables:
๐ Environment variables
Make has multiple types of assignment operators. Some Makefiles use = to assign variables โ and the only ways to override those are to edit the Makefile or override them on the command-line. But Makefiles that use ?= for assignment honor environment variables โจ. Since Spack already sets CC, CXX, F77, and FC, you won't need to worry about setting those yourself. If there are any other variables you need to set, do it in the setup_build_environment method:
def setup_build_environment(self, env):
env.set("PREFIX", prefix)
env.set("BLASLIB", spec["blas"].libs.ld_flags)cbench is a good example of a simple package that does this, while esmf is a good example of a more complex package ๐ซถ.
๐ฌ Command-line arguments
If the Makefile ignores environment variables, the next thing to try is command-line arguments. You do this by overriding the build_targets attribute. If you don't need access to the spec, it's just:
build_targets = ["CC=cc"]If you do need access to the spec, create a property like so:
@property
def build_targets(self):
spec = self.spec
return [
"CC=cc",
f"BLASLIB={spec['blas'].libs.ld_flags}",
]cloverleaf is a good example of a package that uses this strategy โญ.
โ๏ธ Edit Makefile
Some Makefiles are just plain stubborn and will ignore command-line variables. The only way to make sure these packages build correctly is to directly edit the Makefile. Spack gives you a FileFilter class and a filter method to help with this. For example:
def edit(self, spec, prefix):
makefile = FileFilter("Makefile")
makefile.filter(r"^\s*CC\s*=.*", f"CC = {spack_cc}")
makefile.filter(r"^\s*CXX\s*=.*", f"CXX = {spack_cxx}")
makefile.filter(r"^\s*F77\s*=.*", f"F77 = {spack_f77}")
makefile.filter(r"^\s*FC\s*=.*", f"FC = {spack_fc}")stream is a good example of a package that involves editing a Makefile to set the appropriate variables ๐.
๐ Config file
More complex packages often involve Makefiles that include a configuration file. These config files are primarily composed of variables relating to the compiler, the platform, and the location of dependencies or names of libraries. Since these config files are dependent on the compiler and platform, you'll often see entire directories of examples for common compilers and architectures. Use these examples to help figure out what possible values to use ๐.
If the config file is long and only contains one or two variables that need modifying, you can use the technique above to edit the config file. But if you end up needing to modify most of the variables, it may be easier to write a new file from scratch.
If each variable is independent of the others, a dictionary works great for storing variables:
def edit(self, spec, prefix):
config = {
"CC": "cc",
"MAKE": "make",
}
if spec.satisfies("+blas"):
config["BLAS_LIBS"] = spec["blas"].libs.joined()
with open("make.inc", "w") as inc:
for key in config:
inc.write(f"{key} = {config[key]}\n")elk is a good example of a package that uses a dictionary to store configuration variables ๐ฆ.
If the order of the variables is important, it may be easier to store them in a list:
def edit(self, spec, prefix):
config = [
f"INSTALL_DIR = {prefix}",
"INCLUDE_DIR = $(INSTALL_DIR)/include",
"LIBRARY_DIR = $(INSTALL_DIR)/lib",
]
with open("make.inc", "w") as inc:
for var in config:
inc.write(f"{var}\n")hpl is a good example of a package that uses a list to store configuration variables โจ.
๐ Variables to watch out for
Here's a list of common variables to watch out for. The first two sections are implicit variables defined by Make and will always use the same name, while the rest are user-defined variables and may vary from package to package.
- Compilers ๐ง
This includes variables such as CC, CXX, F77, F90, and FC, as well as variables related to MPI compiler wrappers, like MPICC and friends.
- Compiler flags ๐ฉ
This includes variables for specific compilers, like CFLAGS, CXXFLAGS, F77FLAGS, F90FLAGS, FCFLAGS, and CPPFLAGS. These variables are often hard-coded to contain flags specific to a certain compiler. If these flags don't work for every compiler, you may want to consider filtering them.
- Variables that enable or disable features ๐
Yeah โ the feature-toggle crowd. They're on the watch-list too, bestie.
...this covers variables like MPI, OPENMP, PIC, and DEBUG. Bestie, these ones almost always mean you need a variant so the package can build with or without MPI support (for example). They're also super compiler-dependent, so swap them out for the right compiler flags โ stuff like self.compiler.openmp_flag or self.compiler.pic_flag.
- Platform flags โ these control the architecture the executable gets compiled for. Keep an eye out for variables named
PLATorARCH. - Dependencies โ watch for variables that sound like they'd locate a dependency, like
JAVA_HOME,JPEG_ROOT, orZLIBDIR. Also watch for linking controls likeLIBS,LDFLAGS, andINCLUDES. These all need to be set to the installation prefix of a dependency, or to the correct linker flags for linking against that dependency. - Installation prefix โ if your Makefile has an
installtarget, it needs some way of knowing where to install. By default lots of packages install to/usror/usr/local. Since a lot of Spack users don't have sudo privileges, it is imperative that every package installs to the proper prefix. Look for variables likePREFIXorINSTALL.
๐ Makefiles in a sub-directory ๐
Not every package is cute enough to drop its Makefile right in the root of the tarball ๐ค. If the Makefile lives in a sub-directory like src, you just tell Spack where to find it, like so:
build_directory = "src"๐ ๏ธ Manual installation ๐ ๏ธ
Not every Makefile ships an install target either. When that happens, you can override the default install method and install the package by hand:
def install(self, spec, prefix):
mkdir(prefix.bin)
install("foo", prefix.bin)
install_tree("lib", prefix.lib)๐ External documentation ๐
For more info on reading and writing Makefiles, see: https://www.gnu.org/software/make/manual/make.html
โ Maven โ
Apache Maven is a general-purpose build system that does not rely on Makefiles to build software. It's designed for building and managing a Java-based project.
โจ Phases โจ
The MavenBuilder and MavenPackage base classes come with the following phases:
buildโ compile code and package into a JAR fileinstallโ copy to installation prefix
By default, these phases run:
$ mvn package
$ install . <prefix>๐ธ Important files ๐ธ
Maven packages can be identified by the presence of a pom.xml file. This file lists dependencies and other metadata about the project. There may also be configuration files in the .mvn directory.
๐ง Build system dependencies ๐ง
Maven requires the mvn executable to build the project. It also requires Java at both build- and run-time. Because of this, the base class automatically adds the following dependencies:
depends_on("java", type=("build", "run"))
depends_on("maven", type="build")In the pom.xml file, you may see sections like:
<requireJavaVersion>
<version>[1.7,)</version>
</requireJavaVersion>
<requireMavenVersion>
<version>[3.5.4,)</version>
</requireMavenVersion>This specifies the versions of Java and Maven that are required to build the package. See https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN402 for a description of this version range syntax. In this case, you should add:
depends_on("java@7:", type="build")
depends_on("maven@3.5.4:", type="build")๐ Passing arguments to the build phase ๐
The default build and install phases should be enough to install most packages. But sometimes you're gonna want to pass extra flags to the build phase. Like this:
def build_args(self):
return [
"-Pdist,native",
"-Dtar",
"-Dmaven.javadoc.skip=true"
]๐ External documentation ๐
For more info on the Maven build system, see: https://maven.apache.org/index.html
๐ SCons ๐
SCons is a general-purpose build system that does not rely on Makefiles to build software. SCons is written in Python, and handles all building and linking itself.
As far as build systems go, SCons is very non-uniform. It gives developers a common framework to write build scripts in, but the build scripts themselves can vary drastically. Some developers add subcommands like:
$ scons clean
$ scons build
$ scons test
$ scons installOthers don't add any subcommands at all. Some have configuration options you can set through variables on the command line. Others don't. It's giving chaos, honestly, but we move ๐
โจ Phases โจ
Like we said, SCons lets developers add subcommands like build and install, but by default, installation usually looks like:
$ scons
$ scons installTo make that work, the SConsBuilder and SconsPackage base classes provide the following phases:
buildโ build the packageinstallโ install the package
Package developers often add unit tests that get invoked with scons test or scons check. Spack gives you a test method to handle this. Since we don't know which one the package developer chose, the test method does nothing by default โ but you can override it mad easily, like so:
def test(self):
scons("check")๐ธ Important files ๐ธ
SCons packages can be identified by their SConstruct files. These files handle everything โ from setting up subcommands and command-line options to linking and compiling.
One thing to look for is the EnsureSConsVersion function:
EnsureSConsVersion(2, 3, 0)This means SCons 2.3.0 is the earliest release that'll work. You should specify this in a depends_on statement.
๐ง Build system dependencies ๐ง
At the bare minimum, packages that use the SCons build system need a scons dependency. Since that's literally always the case, the SconsPackage base class already contains:
depends_on("scons", type="build")If you want to pin a particular version requirement, you can override it in your package:
depends_on("scons@2.3.0:", type="build")๐ Finding available options ๐
The first place to start when you're hunting for a list of valid options to build a package is scons --help. Some packages like kahip don't even bother overwriting the default SCons help message, so it's not very useful โ but other packages like serf print out a whole list of valid command-line variables:
$ scons --help
scons: Reading SConscript files ...
Checking for GNU-compatible C compiler...yes
scons: done reading SConscript files.
PREFIX: Directory to install under ( /path/to/PREFIX )
default: /usr/local
actual: /usr/local
LIBDIR: Directory to install architecture dependent libraries under ( /path/to/LIBDIR )
default: $PREFIX/lib
actual: /usr/local/lib
APR: Path to apr-1-config, or to APR's install area ( /path/to/APR )
default: /usr
actual: /usr
APU: Path to apu-1-config, or to APR's install area ( /path/to/APU )
default: /usr
actual: /usr
OPENSSL: Path to OpenSSL's install area ( /path/to/OPENSSL )
default: /usr
actual: /usr
ZLIB: Path to zlib's install area ( /path/to/ZLIB )
default: /usr
actual: /usr
GSSAPI: Path to GSSAPI's install area ( /path/to/GSSAPI )
default: None
actual: None
DEBUG: Enable debugging info and strict compile warnings (yes|no)
default: False
actual: False
APR_STATIC: Enable using a static compiled APR (yes|no)
default: False
actual: False
CC: Command name or path of the C compiler
default: None
actual: gcc
CFLAGS: Extra flags for the C compiler (space-separated)
default: None
actual:
LIBS: Extra libraries passed to the linker, e.g. "-l<library1> -l<library2>" (space separated)
default: None
actual: None
LINKFLAGS: Extra flags for the linker (space-separated)
default: None
actual:
CPPFLAGS: Extra flags for the C preprocessor (space separated)
default: None
actual: None
Use scons -H for help about command-line options.More advanced packages like cantera use scons --help to print a list of subcommands:
$ scons --help
scons: Reading SConscript files ...
SCons build script for Cantera
Basic usage:
'scons help' - print a description of user-specifiable options.
'scons build' - Compile Cantera and the language interfaces using
default options.
'scons clean' - Delete files created while building Cantera.
'[sudo] scons install' - Install Cantera.
'[sudo] scons uninstall' - Uninstall Cantera.
'scons test' - Run all tests which did not previously pass or for which the
results may have changed.
'scons test-reset' - Reset the passing status of all tests.
'scons test-clean' - Delete files created while running the tests.
'scons test-help' - List available tests.
'scons test-NAME' - Run the test named "NAME".
'scons <command> dump' - Dump the state of the SCons environment to the
screen instead of doing <command>, e.g.
'scons build dump'. For debugging purposes.
'scons samples' - Compile the C++ and Fortran samples.
'scons msi' - Build a Windows installer (.msi) for Cantera.
'scons sphinx' - Build the Sphinx documentation
'scons doxygen' - Build the Doxygen documentationYou'll notice cantera provides a scons help subcommand. Running scons help prints a list of valid command-line variables. Iconic behaviour honestly ๐
๐ Passing arguments to scons ๐
Now that you know what arguments the project accepts, you can add them to the package build phase. You do that by overriding build_args like so:
def build_args(self, spec, prefix):
args = [
f"PREFIX={prefix}",
f"ZLIB={spec['zlib'].prefix}",
]
if spec.satisfies("+debug"):
args.append("DEBUG=yes")
else:
args.append("DEBUG=no")
return argsSconsPackage also gives you an install_args function that you can override to pass extra arguments to scons install.
๐ช Compiler wrappers ๐ช
By default, SCons builds all packages in a separate execution environment, and doesn't pass any environment variables from the user environment. Like โ even changes to PATH are not propagated unless the package developer does it themselves. So if your build is acting like it can't find a single thing, that's why bestie ๐ญ
Like, the vibes here are rough โ this is especially troublesome for Spack's compiler wrappers, which literally depend on environment variables to manage dependencies and linking flags ๐ญ In so many cases, SCons packages are just not compatible with Spack's compiler wrappers, and you end up having to do the linking manually. Not cute, but here we are ๐
First of all, bestie: check the list of valid options for anything relating to environment variables. For example, cantera has the following option:
* env_vars: [ string ]
Environment variables to propagate through to SCons. Either the
string "all" or a comma separated list of variable names, e.g.
"LD_LIBRARY_PATH,HOME".
- default: "LD_LIBRARY_PATH,PYTHONPATH"In the case of cantera, using env_vars=all allows us to use Spack's compiler wrappers โจ If you don't see an option related to environment variables, try using Spack's compiler wrappers by passing spack_cc, spack_cxx, and spack_fc via the CC, CXX, and FC arguments, respectively. If you pass them to the build and you see an error message like:
Spack compiler must be run from Spack! Input 'SPACK_PREFIX' is missing.you'll know that the package isn't compatible with Spack's compiler wrappers ๐ซ In this case, you'll have to use the path to the actual compilers, which are stored in self.compiler.cc and friends. Note that this may involve passing additional flags to the build to locate dependencies โ a task normally done by the compiler wrappers. serf is an example of a package with this limitation.
๐ External documentation
For more information on the SCons build system, see: http://scons.org/documentation.html
๐ Waf
Like SCons, Waf is a general-purpose build system that does not rely on Makefiles to build software. Fancy, right? โจ
๐ธ Phases
The WafBuilder and WafPackage base classes come with the following phases:
configure- configure the projectbuild- build the projectinstall- install the project
By default, these phases run:
$ python waf configure --prefix=/path/to/installation/prefix
$ python waf build
$ python waf installEach of these are standard Waf commands and can be found by running:
$ python waf --helpEach phase provides a <phase> function that runs:
$ python waf -j<jobs> <phase>where <jobs> is the number of parallel jobs to build with ๐โโ๏ธ Each phase also has a <phase_args> function that can pass arguments to this call. All of these functions are empty. The configure phase automatically adds --prefix=/path/to/installation/prefix, so you don't need to add that in the configure_args โ it's giving "handled, bestie" ๐
๐งช Testing
WafPackage also provides test and installtest methods, which are run after the build and install phases, respectively. By default, these phases do nothing, but you can override them to run package-specific unit tests ๐
def installtest(self):
with working_dir("test"):
pytest = which("py.test")
pytest()๐ Important files
Each Waf package comes with a custom waf build script, written in Python. This script contains instructions to build the project.
The package also comes with a wscript file ๐ฆ This file is used to override the default configure, build, and install phases to customize the Waf project. It also allows developers to override the default ./waf --help message. Check this file to find useful information about dependencies and the minimum versions that are supported.
๐ Build system dependencies
WafPackage does not require waf to build. waf is only needed to create the ./waf script. Since ./waf is a Python script, Python is needed to build the project. WafPackage adds the following dependency automatically:
depends_on("python@2.5:", type="build")Waf only supports Python 2.5 and up ๐
๐ซถ Passing arguments to waf
As previously mentioned, each phase comes with a <phase_args> function that can be used to pass arguments to that particular phase. For example, if you need to pass arguments to the build phase, you can use:
def build_args(self, spec, prefix):
args = []
if self.run_tests:
args.append("--test")
return argsA list of valid options can be found by running ./waf --help.
๐ External documentation
For more information on the Waf build system, see: https://waf.io/book/
๐ ๏ธ Autotools
Autotools is a GNU build system that provides a build-script generator โจ By running the platform-independent ./configure script that comes with the package, you can generate a platform-dependent Makefile.
๐ท Phases
The AutotoolsBuilder and AutotoolsPackage base classes come with the following phases:
autoreconf- generate theconfigurescriptconfigure- generate theMakefilesbuild- build the packageinstall- install the package
Most of the time, the autoreconf phase will do nothing, but if the package is missing a configure script, autoreconf will generate one for you ๐ซถ
The other phases run:
$ ./configure --prefix=/path/to/installation/prefix
$ make
$ make check # optional
$ make install
$ make installcheck # optionalOf course, you may need to add a few arguments to the ./configure line ๐
๐ Important files
The most important file for an Autotools-based package is the configure script. This script is automatically generated by Autotools and generates the appropriate Makefile when run.
WARNING: Watch out for fake Autotools packages! ๐ฉ Autotools is a very popular build system, and many people are used to the classic steps to install a package:$ ./configure $ make $ make installFor this reason, some developers will write their ownconfigurescripts that have nothing to do with Autotools ๐ญ These packages may not accept the same flags as other Autotools packages, so it is better to use thePackagebase class and create a custom build system. You can tell if a package uses Autotools by running./configure --helpand comparing the output to other known Autotools packages. You should also look for files like: -configure.ac-configure.in-Makefile.amPackages that don't use Autotools aren't likely to have these files.
๐ Build system dependencies
Whether or not your package requires Autotools to install depends on how the source code is distributed. Most of the time, when developers distribute tarballs, they will already contain the configure script necessary for installation. If this is the case, your package does not require any Autotools dependencies ๐
However, a basic rule of version control systems is to never commit code that can be generated. The source code repository itself likely does not have a configure script. Developers typically write (or auto-generate) a configure.ac script that contains configuration preferences and a Makefile.am script that contains build instructions. Then, autoconf is used to convert configure.ac into configure, while automake is used to convert Makefile.am into Makefile.in. Makefile.in is used by configure to generate a platform-dependent Makefile for you. The following diagram provides a high-level overview of the process ๐
#.. figure:: Autoconf-automake-process.* # :target: https://commons.wikimedia.org/w/index.php?curid=15581407
GNU autoconf and automake process for generating makefiles by Jdthood
under CC BY-SA 3.0If a configure script is not present in your tarball, you will need to generate one yourself. Luckily, Spack already has an autoreconf phase to do most of the work for you ๐ง By default, the autoreconf phase runs:
$ autoreconf --install --verbose --force -I <aclocal-prefix>/share/aclocalIn case you need to add more arguments, override autoreconf_extra_args in your package.py on class scope like this:
autoreconf_extra_args = ["-Im4"]All you need to do is add a few Autotools dependencies to the package. Most stable releases will come with a configure script, but if you check out a commit from the master branch, you would want to add:
depends_on("autoconf", type="build", when="@master")
depends_on("automake", type="build", when="@master")
depends_on("libtool", type="build", when="@master")It is typically redundant to list the m4 macro processor package as a dependency, since autoconf already depends on it ๐
๐ฆ Using a custom autoreconf phase
In some cases, it might be needed to replace the default implementation of the autoreconf phase with one running a script interpreter. In this example, the bash shell is used to run the autogen.sh script:
def autoreconf(self, spec, prefix):
which("bash")("autogen.sh")๐ฉน patching configure or Makefile.in files
In some cases, developers might need to distribute a patch that modifies one of the files used to generate configure or Makefile.in. In this case, these scripts will need to be regenerated ๐ It is preferable to regenerate these manually using the patch, and then create a new patch that directly modifies configure. That way, Spack can use the secondary patch and additional build system dependencies aren't necessary โ lowkey the efficient slay ๐
๐ Old Autotools helper scripts
Autotools based tarballs come with helper scripts such as config.sub and config.guess. It is the responsibility of the developers to keep these files up to date so that they run on every platform, but for very old software releases this is impossible ๐ญ In these cases Spack can help to replace these files with newer ones, without having to add the heavy dependency on automake.
Automatic helper script replacement is currently enabled by default on ppc64le and aarch64, as these are the known cases where old scripts fail. On these targets, AutotoolsPackage adds a build dependency on gnuconfig, which is a very light-weight package with newer versions of the helper files โจ Spack then tries to run all the helper scripts it can find in the release, and replaces them on failure with the helper scripts from gnuconfig.
To opt out of this feature, use the following setting:
patch_config_files = FalseTo enable it conditionally on different architectures, define a property and make the package depend on gnuconfig as a build dependency:
depends_on("gnuconfig", when="@1.0:")
@property
def patch_config_files(self):
return self.spec.satisfies("@1.0:")NOTE: On some exotic architectures it is necessary to use system providedconfig.subandconfig.guessfiles ๐ In this case, the most transparent solution is to mark thegnuconfigpackage as external and non-buildable, with a prefix set to the directory containing the files:gnuconfig: buildable: false externals: - spec: gnuconfig@master prefix: /usr/share/configure_files/
๐ง force_autoreconf
okay so sometimes you're in a situation where you genuinely want to add the original patch AND tell Spack to go ahead and regenerate configure. bestie, if that's you, flip this setting on:
force_autoreconf = TrueThis line tells Spack to wipe away the existing configure script and generate a fresh new one. โจ
And if you only need that energy for one single version, you gate it like this:
@property
def force_autoreconf(self):
return self.version == Version("1.2.3")๐ Finding configure flags
So once you've got a configure script sitting there, the next quest is figuring out which option flags actually exist. And like โ you just ask it, bestie:
$ ./configure --helpconfigure will spit out a list of valid flags, sorted into some or all of these sections:
- Configuration
- Installation directories
- Fine tuning of the installation directories
- Program names
- X features
- System types
- Optional Features
- Optional Packages
- Some influential environment variables
Ngl, for the most part you can ignore all but the last 3 sections. ๐
- "Optional Features" lists the flags that enable/disable features you might actually care about.
- "Optional Packages" usually lists dependencies, plus the flags you need to point at them.
- "environment variables" lists the env vars the build system uses to pass flags through to the compiler and linker.
โ Addings flags to configure
For most of the flags you bump into, you're gonna want a variant that lets you optionally enable/disable them. Then you can optionally pass those flags into the configure call by overriding the configure_args function, like so:
def configure_args(self):
args = []
if self.spec.satisfies("+mpi"):
args.append("--enable-mpi")
else:
args.append("--disable-mpi")
return argsOr, alternate route โ use the enable_or_disable helper and let it do the heavy lifting:
def configure_args(self):
return [self.enable_or_disable("mpi")]๐ notice how we are explicitly disabling MPI support if it's not requested. This is important, fr fr. Loads of Autotools packages will just enable options by default if they find the dependencies, and disable them if they don't. We want Spack installations to be as deterministic as humanly possible โ if two users install a package with the same variants, the whole goal is that both installations behave the same way. See here and here for a rationale as to why these so-called "automagic" dependencies are a problem. ๐ญ
NOTE: By default, Autotools installs packages to/usr. We do not want that, so Spack automatically adds--prefix=/path/to/installation/prefixto your list ofconfigure_args. You don't need to add this yourself. ๐ซถ
๐งฐ Helper functions
You may have clocked that most Autotools flags follow the same little shapes: --enable-foo, --disable-bar, --with-baz=<prefix>, or --without-baz. Since these flags are everywhere, Spack hands you a couple of helper functions to make your life easier. ๐
enable_or_disable
Autotools flags for simple boolean variants can be generated automatically by calling the enable_or_disable method. This is typically used to enable or disable some feature within the package.
variant(
"memchecker",
default=False,
description="Memchecker support for debugging [degrades performance]"
)
config_args.extend(self.enable_or_disable("memchecker"))In this example, specifying the variant +memchecker will generate the following configuration options:
--enable-memcheckerwith_or_without
Autotools flags for more complex variants โ including boolean variants and multi-valued variants โ can be generated automatically by calling the with_or_without method.
variant(
"schedulers",
values=disjoint_sets(
("auto",), ("alps", "lsf", "tm", "slurm", "sge", "loadleveler")
).with_non_feature_values("auto", "none"),
description="List of schedulers for which support is enabled; "
"'auto' lets openmpi determine",
)
if not spec.satisfies("schedulers=auto"):
config_args.extend(self.with_or_without("schedulers"))In this example, specifying the variant schedulers=slurm,sge will generate the following configuration options:
--with-slurm --with-sgeAnd fun fact: enable_or_disable is actually functionally equivalent to with_or_without โ it accepts the same arguments and variant types. It's just that idiomatic autotools packages tend to follow these naming conventions. ๐ธ
activation_value
Autotools parameters that require an option can still be generated automatically, using the activation_value argument to with_or_without (or, rarely, enable_or_disable).
variant(
"fabrics",
values=disjoint_sets(
("auto",), ("psm", "psm2", "verbs", "mxm", "ucx", "libfabric")
).with_non_feature_values("auto", "none"),
description="List of fabrics that are enabled; "
"'auto' lets openmpi determine",
)
if not spec.satisfies("fabrics=auto"):
config_args.extend(self.with_or_without("fabrics",
activation_value="prefix"))activation_value accepts a callable that generates the configure parameter value given the variant value โ but the special value prefix tells Spack to automatically use the dependency's installation prefix, which is the most common use for these kinds of parameters. In this example, specifying the variant fabrics=libfabric will generate the following configuration options:
--with-libfabric=</path/to/libfabric>๐ท๏ธ The variant keyword
When Spack variants and configure flags don't correspond one-to-one, you can pass the variant keyword into with_or_without and enable_or_disable. For example:
variant("debug_tools", default=False)
config_args += self.enable_or_disable("debug-tools", variant="debug_tools")Or, for when one variant controls multiple flags:
variant("debug_tools", default=False)
config_args += self.with_or_without("memchecker", variant="debug_tools")
config_args += self.with_or_without("profiler", variant="debug_tools")๐ Conditional variants
When a variant is conditional and its condition isn't met on the concrete spec, the with_or_without and enable_or_disable methods will simply return an empty list. No drama, no flags. ๐ฆ
For example:
variant("profiler", when="@2.0:")
config_args += self.with_or_without("profiler")...will neither add --with-profiler nor --without-profiler when the version is below 2.0.
๐ Activation overrides
And finally, the behavior of either with_or_without or enable_or_disable can be overridden for specific variant values. This is most useful for multi-valued variants where some of the variant values need atypical behavior.
def with_or_without_verbs(self, activated):
# Up through version 1.6, this option was named --with-openib.
# In version 1.7, it was renamed to be --with-verbs.
opt = "verbs" if self.spec.satisfies("@1.7:") else "openib"
if not activated:
return f"--without-{opt}"
return f"--with-{opt}={self.spec['rdma-core'].prefix}"Defining with_or_without_verbs overrides the behavior of a fabrics=verbs variant, changing the configure-time option to --with-openib for older versions of the package and specifying an alternative dependency name:
--with-openib=</path/to/rdma-core>๐ Configure script in a sub-directory
Occasionally, developers will hide their source code and configure script in a subdirectory like src. Rude, but okay. ๐ If this happens, Spack won't be able to automatically detect the build system properly when running spack create. You'll have to manually change the package base class and tell Spack where the configure script lives, like so:
configure_directory = "src"๐๏ธ Building out of source
Some packages โ looking at you, gcc โ recommend building their software in a different directory than the source code, specifically to prevent build pollution. You do that with the build_directory variable:
build_directory = "spack-build"By default, Spack will build the package in the same directory that contains the configure script. ๐
๐ฏ Build and install targets
For most Autotools packages, the classic:
$ configure
$ make
$ make install...is completely sufficient to install the package. But if you need to run make with any other targets โ say, to build an optional library or build the documentation โ you can add them like so:
build_targets = ["all", "docs"]
install_targets = ["install", "docs"]๐งช Testing
Autotools-based packages typically provide unit testing via the check and installcheck targets. If you build your software with spack install --test=root, Spack will check for the presence of a check or test target in the Makefile and run make check for you. After installation, it'll check for an installcheck target and run make installcheck if it finds one. Chef's kiss. ๐
๐ External documentation
For more information on the Autotools build system, see: https://www.gnu.org/software/automake/manual/html_node/Autotools-Introduction.html
๐งฑ CMake
Like Autotools, CMake is a widely-used build-script generator. Designed by Kitware, CMake is the most popular build system for new C, C++, and Fortran projects โ and loads of older projects are switching over to it too. It's giving main character energy in the build-system world, honestly.
Unlike Autotools, CMake can generate build scripts for builders other than Make: Ninja, Visual Studio, etc. It is therefore cross-platform, whereas Autotools is Unix-only. ๐
๐ Phases
The CMakeBuilder and CMakePackage base classes come with the following phases:
cmakeโ generate theMakefilebuildโ build the packageinstallโ install the package
By default, these phases run:
$ mkdir spack-build
$ cd spack-build
$ cmake .. -DCMAKE_INSTALL_PREFIX=/path/to/installation/prefix
$ make
$ make test # optional
$ make installA few more flags are passed to cmake by default, including flags for setting the build type and flags for locating dependencies. Of course, you may need to add a few arguments yourself. โจ
๐๏ธ Important files
A CMake-based package can be identified by the presence of a CMakeLists.txt file. This file defines the build flags that can be passed to the cmake invocation, plus the linking instructions. If you're familiar with CMake, it can prove very useful for figuring out dependencies and dependency version requirements.
One thing to look for is the cmake_minimum_required function:
cmake_minimum_required(VERSION 2.8.12)This means that CMake 2.8.12 is the earliest release that will work. You should specify this in a depends_on statement. ๐
CMake-based packages can also stash extra CMakeLists.txt files in subdirectories, bestie โ it's giving hierarchical realness. That modularization is how big scary builds stay manageable, but heads up: sometimes those nested CMakeLists.txt files need dependencies that the top-level file never even mentions. Rude, but okay. ๐
There's also usually a cmake or CMake directory hanging around with extra macros, find scripts, etc. Those are lowkey gold for figuring out dependency version requirements. ๐
โจ Build system dependencies โจ
Every package using the CMake build system needs a cmake dependency. Since that's literally always true, the CMakePackage base class already ships with:
depends_on("cmake", type="build")Need a specific version requirement? Just override it in your package, easy:
depends_on("cmake@2.8.12:", type="build")๐ Finding cmake flags ๐
Wanna see a list of valid flags you can pass to cmake? Run this in the directory that has CMakeLists.txt:
$ cmake . -LAHCMake will start by checking for compilers and dependencies, then eventually start listing build options. Notice how most of the options up top are prefixed with CMAKE_? You can safely ignore most of those โ Spack already sets them for you. That includes flags for locating dependencies, RPATH libraries, setting the installation directory, and setting the build type. ๐ซถ
The rest of the flags are the ones you should consider adding to your package. They usually include flags to enable/disable support for certain features and to locate specific dependencies. One thing that makes CMake different from Autotools: CMake understands build flag hierarchy. Meaning certain flags won't even display unless their parent flag has been selected. For example, flags specifying the lib and include directories for a package might not appear unless CMake found the dependency it was looking for. So you may need to manually specify certain flags to explore the full depth of supported build flags โ or just go read the CMakeLists.txt yourself. ๐
๐ ๏ธ Adding flags to cmake ๐ ๏ธ
To add extra flags to the cmake call, simply override the cmake_args function. This example defines values for the flags WHATEVER, ENABLE_BROKEN_FEATURE, DETECT_HDF5, and THREADS, with and without the define() and define_from_variant() helper functions:
def cmake_args(self):
args = [
"-DWHATEVER:STRING=somevalue",
self.define("ENABLE_BROKEN_FEATURE", False),
self.define_from_variant("DETECT_HDF5", "hdf5"),
self.define_from_variant("THREADS"), # True if +threads
]
return argsSpack supports CMake defines from conditional variants too, periodt. Whenever the condition on the variant isn't met, define_from_variant() just returns an empty string, and CMake simply ignores the empty command line argument. For example:
variant("example", default=True, when="@2.0:")
def cmake_args(self):
return [self.define_from_variant("EXAMPLE", "example")]...will generate 'cmake' '-DEXAMPLE=ON' ... when @2.0: +example is met, but results in 'cmake' '' ... when the spec version is below 2.0. Slay, no crashes. ๐
๐ CMake arguments provided by Spack ๐
The following default arguments are controlled by Spack:
CMAKE_INSTALL_PREFIX
Set to the package's install directory.
CMAKE_PREFIX_PATH
CMake finds dependencies through calls to find_package(), find_program(), find_library(), find_file(), and find_path(), which use a list of search paths from CMAKE_PREFIX_PATH. Spack sets this variable to a list of prefixes of the spec's transitive dependencies.
For troubleshooting cases where CMake fails to find a dependency, add the --debug-find flag to cmake_args. ๐
CMAKE_BUILD_TYPE
Every CMake-based package accepts a -DCMAKE_BUILD_TYPE flag to dictate which level of optimization to use. To keep everything uniform across packages, the CMakePackage base class adds a variant to control it:
variant("build_type", default="RelWithDebInfo",
description="CMake build type",
values=("Debug", "Release", "RelWithDebInfo", "MinSizeRel"))But not every CMake package accepts all four of these options, ngl. Grep the CMakeLists.txt file to see if the default values are missing or replaced. For example, the dealii package overrides the default variant with:
variant("build_type", default="DebugRelease",
description="The build type to build",
values=("Debug", "Release", "DebugRelease"))For more information on CMAKE_BUILD_TYPE, see: โ https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html
CMAKE_INSTALL_RPATH and CMAKE_INSTALL_RPATH_USE_LINK_PATH=ON
CMake uses different RPATHs during the build and after installation, so executables can locate the libraries they're linked to during the build, and installed executables don't have RPATHs pointing at build directories. In Spack, we have to make sure RPATHs are set properly after installation. โจ
Spack sets CMAKE_INSTALL_RPATH to a list of <prefix>/lib or <prefix>/lib64 directories of the spec's link-type dependencies. Apart from that, it sets -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON, which should add RPATHs for directories of linked libraries not in the directories covered by CMAKE_INSTALL_RPATH.
Usually it's enough to set only -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON, but the reason to provide both options is that packages may dynamically open shared libraries, which CMake cannot detect. In those cases, the RPATHs from CMAKE_INSTALL_RPATH are used as search paths.
NOTE: Some packages provide stub libraries โ libraries that contain an interface for linking without an implementation. When using such libraries, it's best to override the option-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=OFFincmake_args, so that stub libraries are not used at runtime. ๐จ
๐ Generators ๐
CMake and Autotools are build-script generation tools โ they "generate" the Makefiles used to build a software package. But CMake actually supports multiple generators, not just Makefiles. Another common one is Ninja. To switch to the Ninja generator, simply add:
generator("ninja")CMakePackage defaults to "Unix Makefiles". If you switch to the Ninja generator, make sure to also add:
depends_on("ninja", type="build")to the package. Aside from that, you shouldn't need to do anything else. Spack automatically detects that you're using Ninja and runs:
$ cmake .. -G Ninja
$ ninja
$ ninja installSpack currently only supports "Unix Makefiles" and "Ninja" as valid generators, but it should be simple to add support for alternative generators. For more information on CMake generators, see: โ https://cmake.org/cmake/help/latest/manual/cmake-generators.7.html
๐ง CMakeLists.txt in a sub-directory ๐ง
Occasionally, developers will hide their source code and CMakeLists.txt in a subdirectory like src. If that happens, Spack won't be able to automatically detect the build system properly when running spack create. You'll have to manually change the package base class and tell Spack where CMakeLists.txt lives. Do it like this:
root_cmakelists_dir = "src"Note that this path is relative to the root of the extracted tarball โ not to the build_directory. It defaults to the current directory. ๐
๐ธ Building out of source ๐ธ
By default, Spack builds every CMakePackage in a spack-build sub-directory. If, for whatever reason, you'd like to build in a different sub-directory, just override build_directory like so:
build_directory = "my-build"๐ Build and install targets ๐
For most CMake packages, the usual:
$ cmake
$ make
$ make installis sufficient to install the package. But if you need to run make with any other targets โ say, to build an optional library or build the documentation โ you can add them like this:
build_targets = ["all", "docs"]
install_targets = ["install", "docs"]โญ Testing โญ
CMake-based packages typically provide unit testing via the test target. If you build your software with --test=root, Spack will check for the presence of a test target in the Makefile and run make test for you. Wanna run a different test instead? Simply override the check method. โ
๐ฆ External documentation ๐ฆ
For more information on the CMake build system, see: โ https://cmake.org/cmake/help/latest/
๐ CachedCMake ๐
The CachedCMakePackage base class is used for CMake-based workflows that create a CMake cache file prior to running cmake. This is useful for packages with arguments longer than the system limit, and for reproducibility. Obsessed. ๐
The documentation for this class assumes the user is already familiar with the CMakePackage class from which it inherits. (See the documentation for CMakePackage.)
๐ Phases ๐
The CachedCMakePackage base class comes with the following phases:
initconfigโ generate the CMake cache filecmakeโ generate the Makefilebuildโ build the packageinstallโ install the package
By default, these phases run:
$ mkdir spack-build
$ cd spack-build
$ cat << EOF > name-arch-compiler@version.cmake
# Write information on compilers and dependencies
# includes information on mpi and cuda if applicable
$ cmake .. -DCMAKE_INSTALL_PREFIX=/path/to/installation/prefix -C name-arch-compiler@version.cmake
$ make
$ make test # optional
$ make installThe CachedCMakePackage class inherits from the CMakePackage class, and accepts all of the same options and adds all of the same flags to the cmake command. Similar to the CMakePackage class, you may need to add a few arguments yourself, and the CachedCMakePackage provides the same interface to add those flags. ๐ซถ
๐ Adding entries to the CMake cache ๐
In addition to adding flags to the cmake command, you may need to add entries to the CMake cache in the initconfig phase. This can be done by overriding one of four methods:
CachedCMakePackage.initconfig_compiler_entriesCachedCMakePackage.initconfig_mpi_entriesCachedCMakePackage.initconfig_hardware_entriesCachedCMakePackage.initconfig_package_entries
Each of these methods returns a list of CMake cache strings. The distinction between them is merely to provide a well-structured and legible cmake cache file โ otherwise, entries from each of these methods are handled identically. ๐
Spack also provides convenience methods for generating CMake cache entries. These methods are available at module scope in every Spack package. Because CMake parses boolean options, strings, and paths differently, there are three such methods:
cmake_cache_optioncmake_cache_stringcmake_cache_path
So yeah โ each of these methods takes three parameters: the name of the CMake variable that's chained to the entry, the value of the entry, and an optional comment. They hand back strings in the right format for whatever initconfig* method you're returning from. Also lowkey important: these methods can return comments that start with the # character. ๐
A typical usage of these methods looks something like this, bestie:
def initconfig_mpi_entries(self):
# Get existing MPI configurations
entries = super(self, Foo).initconfig_mpi_entries()
# The existing MPI configurations key on whether ``mpi`` is in the spec
# This spec has an MPI variant, and we need to enable MPI when it is on.
# This hypothetical package controls MPI with the ``FOO_MPI`` option to
# cmake.
if self.spec.satisfies("+mpi"):
entries.append(cmake_cache_option("FOO_MPI", True, "enable mpi"))
else:
entries.append(cmake_cache_option("FOO_MPI", False, "disable mpi"))
def initconfig_package_entries(self):
# Package specific options
entries = []
entries.append("#Entries for build options")
bar_on = self.spec.satisfies("+bar")
entries.append(cmake_cache_option("FOO_BAR", bar_on, "toggle bar"))
entries.append("#Entries for dependencies")
if self.spec["blas"].name == "baz": # baz is our blas provider
entries.append(cmake_cache_string("FOO_BLAS", "baz", "Use baz"))
entries.append(cmake_cache_path("BAZ_PREFIX", self.spec["baz"].prefix))โจ External documentation โจ
Wanna go deeper on CMake cache files? The docs are literally right here: https://cmake.org/cmake/help/latest/manual/cmake.1.html ๐ฆ
๐ Meson ๐
Okay so Meson is a build system, same energy as Autotools and CMake โ but it's designed to be both fast and as user friendly as possible. GNOME's whole goal is to port their modules over to Meson, so you know it's the moment. ๐
๐ธ Phases ๐ธ
The MesonBuilder and MesonPackage base classes come with these phases, in order:
mesonโ generate ninja filesbuildโ build the projectinstallโ install the project
By default, these phases run:
$ mkdir spack-build
$ cd spack-build
$ meson .. --prefix=/path/to/installation/prefix
$ ninja
$ ninja test # optional
$ ninja installAny of these phases can be overridden in your package if you need to โ no cap. There's also a check method that looks for a test target in the build file. If a test target exists and the user runs:
$ spack install --test=root <meson-package>then Spack will run ninja test after the build phase. ๐ซถ
๐ง Important files ๐ง
Packages that use the Meson build system can be identified by the presence of a meson.build file. That file declares things like build instructions and dependencies. Iconic.
One thing to look for is the meson_version key that gets passed to the project function:
project('gtk+', 'c',
version: '3.94.0',
default_options: [
'buildtype=debugoptimized',
'warning_level=1',
# We only need c99, but glib needs GNU-specific features
# https://github.com/mesonbuild/meson/issues/2289
'c_std=gnu99',
],
meson_version: '>= 0.43.0',
license: 'LGPLv2.1+')This means that Meson 0.43.0 is the earliest release that will work. You should specify this in a depends_on statement.
๐ Build system dependencies ๐
At the bare minimum, packages that use the Meson build system need meson and ninja dependencies. Since this is always the case, the MesonPackage base class already contains:
depends_on("meson", type="build")
depends_on("ninja", type="build")If you need to specify a particular version requirement, you can override this in your package:
depends_on("meson@0.43.0:", type="build")
depends_on("ninja", type="build")๐ฆ Finding meson flags ๐ฆ
To get a list of valid flags that can be passed to meson, run the following command in the directory that contains meson.build:
$ meson setup --help๐ Passing arguments to meson ๐
If you need to pass any arguments to the meson call, you can override the meson_args method like so:
def meson_args(self):
return ["--warnlevel=3"]This method can be used to pass flags as well as variables. Slay.
Heads up bestie: theMesonPackagebase class already defines variants forbuildtype,default_libraryandstrip, which are mapped to default Meson arguments, meaning that you don't have to specify these. ๐
๐ External documentation ๐
For more information on the Meson build system, see: https://mesonbuild.com/index.html
๐ QMake ๐
Much like Autotools and CMake, QMake is a build-script generator designed by the developers of Qt. In its simplest form, Spack's QMakePackage runs the following steps:
$ qmake
$ make
$ make check # optional
$ make installQMake does not appear to have a standardized way of specifying the installation directory, so you may have to set environment variables or edit *.pro files to get things working properly. It's giving chaotic, but we deal. ๐ญ
โญ Phases โญ
The QMakeBuilder and QMakePackage base classes come with the following phases:
qmakeโ generate Makefilesbuildโ build the projectinstallโ install the project
By default, these phases run:
$ qmake
$ make
$ make installAny of these phases can be overridden in your package as necessary. There is also a check method that looks for a check target in the Makefile. If a check target exists and the user runs:
$ spack install --test=root <qmake-package>Spack will run make check after the build phase. โจ
๐ Important files ๐
Packages that use the QMake build system can be identified by the presence of a <project-name>.pro file. This file declares things like build instructions and dependencies.
One thing to look for is the minQtVersion function:
minQtVersion(5, 6, 0)This means that Qt 5.6.0 is the earliest release that will work. You should specify this in a depends_on statement.
๐ Build system dependencies ๐
At the bare minimum, packages that use the QMake build system need a qt dependency. Since this is always the case, the QMakePackage base class already contains:
depends_on("qt", type="build")If you want to specify a particular version requirement, or need to link to the qt libraries, you can override this in your package:
depends_on("qt@5.6.0:")๐ธ Passing arguments to qmake ๐ธ
If you need to pass any arguments to the qmake call, you can override the qmake_args method like so:
def qmake_args(self):
return ["-recursive"]This method can be used to pass flags as well as variables.
๐ง *.pro file in a sub-directory ๐ง
If the *.pro file used to tell QMake how to build the package is found in a sub-directory, you can tell Spack to run all phases in this sub-directory by adding the following to the package:
build_directory = "src"๐ External documentation ๐
For more information on the QMake build system, see: http://doc.qt.io/qt-5/qmake-manual.html
๐ซถ SIP ๐ซถ
SIP is a tool that makes it very easy to create Python bindings for C and C++ libraries. It was originally developed to create PyQt, the Python bindings for the Qt toolkit, but it can be used to create bindings for any C or C++ library. Versatile queen. ๐
SIP comprises a code generator and a Python module. The code generator processes a set of specification files and generates C or C++ code which is then compiled to create the bindings extension module. The SIP Python module provides support functions to the automatically generated code.
๐ฅ Phases ๐ฅ
The SIPBuilder and SIPPackage base classes come with the following phases:
configureโ configure the packagebuildโ build the packageinstallโ install the package
By default, these phases run:
$ sip-build --verbose --target-dir ...
$ make
$ make install๐ Important files ๐
Each SIP package comes with a custom configuration file written in Python. For newer packages, this is called project.py, while in older packages, it may be called configure.py. This script contains instructions to build the project.
๐ Build system dependencies ๐
SIPPackage requires several dependencies, and honestly? It's a whole stack. Python and SIP are needed at build-time to run the aforementioned configure script. Python is also needed at run-time to actually use the installed Python library. And as we are building Python bindings for C/C++ libraries, Python is also needed as a link dependency. All of these dependencies are automatically added via the base class:
extends("python", type=("build", "link", "run"))
depends_on("py-sip", type="build")๐ฆ Passing arguments to sip-build ๐ฆ
Each phase comes with a <phase_args> function that can be used to pass arguments to that particular phase. For example, if you need to pass arguments to the configure phase, you can use:
def configure_args(self):
return ["--no-python-dbus"]A list of valid options can be found by running sip-build --help.
๐ Testing ๐
Just because a package successfully built does not mean that it built correctly โ that's the tea. โ The most reliable test of whether or not the package was correctly installed is to attempt to import all of the modules that get installed. To get a list of modules, run the following command in the site-packages directory:
$ python
>>> import setuptools
>>> setuptools.find_packages()
[
'PyQt5', 'PyQt5.QtCore', 'PyQt5.QtGui', 'PyQt5.QtHelp',
'PyQt5.QtMultimedia', 'PyQt5.QtMultimediaWidgets', 'PyQt5.QtNetwork',
'PyQt5.QtOpenGL', 'PyQt5.QtPrintSupport', 'PyQt5.QtQml',
'PyQt5.QtQuick', 'PyQt5.QtSvg', 'PyQt5.QtTest', 'PyQt5.QtWebChannel',
'PyQt5.QtWebSockets', 'PyQt5.QtWidgets', 'PyQt5.QtXml',
'PyQt5.QtXmlPatterns'
]Large, complex packages like py-pyqt5 will return a long list of packages, while other packages may return an empty list. These packages only install a single foo.py file. In Python packaging lingo, a "package" is a directory containing files like:
foo/__init__.py
foo/bar.py
foo/baz.pywhereas a "module" is a single Python file.
The SIPPackage base class automatically detects these module names for you. If, for whatever reason, the module names detected are wrong, you can provide the names yourself by overriding import_modules like so:
import_modules = ["PyQt5"]These tests often catch missing dependencies and non-RPATHed libraries. Make sure not to add modules/packages containing the word "test", as these likely won't end up in the installation directory, or may require test dependencies like pytest to be installed. Don't do it bestie, it's a trap. ๐
these tests can be triggered by running spack install --test=root or by running spack test run after the installation has finished โ pick your fave ๐ซ
External documentation
for more info on the SIP build system, go look at:
- https://www.riverbankcomputing.com/software/sip/intro
- https://www.riverbankcomputing.com/static/Docs/sip/
- https://wiki.python.org/moin/SIP
๐ Lua
ok so the Lua build-system is literally just a helper for the common case of Lua packages that provide a rockspec file. bestie pls note: this is not meant to take a rock archive โ it's here to build a source archive or repository that provides a rockspec, which should cover most lua packages. and if a Lua package builds by Make rather than luarocks? prefer MakefilePackage. ๐ซถ
ใฐ๏ธ Phases
the LuaBuilder and `LuaPackage`` base classes come with the following phases:
unpackโ if using a rock, unpacks the rock and moves into the source directorypreprocessโ adjust sources or rockspec to fix buildinstallโ install the project
by default, these phases run:
# If the archive is a source rock
$ luarocks unpack <archive>.src.rock
$ # preprocess is a noop by default
$ luarocks make <name>.rockspecany of these phases can be overridden in your package as necessary. no cap, you're the main character of your own build ๐
๐ฆ Important files
packages that use the Lua/LuaRocks build system can be identified by the presence of a *.rockspec file in their sourcetree, or can be fetched as a source rock archive (.src.rock). this file declares things like build instructions and dependencies, and the .src.rock also contains all code. ๐
it's common for the rockspec file to list the lua version required in a dependency. the LuaPackage class adds appropriate dependencies on a Lua implementation, but it's a good idea to specify the version required with a depends_on statement. the block normally will be a table definition like this:
dependencies = {
"lua >= 5.1",
}the LuaPackage class supports source repositories and archives containing a rockspec and directly downloading source rock files. it does not support downloading dependencies listed inside a rockspec, and thus does not support directly downloading a rockspec as an archive. periodt. ๐
๐ Build system dependencies
all base dependencies are added by the build system, but LuaRocks is run to avoid downloading extra Lua dependencies during build. if the package needs Lua libraries outside the standard set, they should be added as dependencies. ๐ฆ
to specify a Lua version constraint but allow all lua implementations, prefer to use depends_on("lua-lang@5.1:5.1.99") to express any 5.1 compatible version. if the package requires LuaJit rather than Lua, a depends_on("luajit") should be used to ensure a LuaJit distribution is used instead of the Lua interpreter. alternately, if only interpreted Lua will work depends_on("lua") will express that. iykyk โจ
๐๏ธ Passing arguments to luarocks make
if you need to pass any arguments to the luarocks make call, you can override the luarocks_args method like so:
def luarocks_args(self):
return ["flag1", "flag2"]one common use of this is to override warnings or flags for newer compilers, as in:
def luarocks_args(self):
return ["CFLAGS='-Wno-error=implicit-function-declaration'"]External documentation
for more info on the LuaRocks build system, see: โ https://luarocks.org/
๐ Octave
Octave has its own build system for installing packages. it's giving independent queen energy ๐
ใฐ๏ธ Phases
the OctaveBuilder and OctavePackage base classes have a single phase:
installโ install the package
by default, this phase runs the following command:
$ octave '--eval' 'pkg prefix <prefix>; pkg install <archive_file>'โ ๏ธ beware that uninstallation is not implemented at the moment. after uninstalling a package via Spack, you also need to manually uninstall it from Octave via pkg uninstall <package_name>. like literally do not forget, bestie ๐ญ
๐ Finding Octave packages
most Octave packages are listed at โ https://octave.sourceforge.io/packages.php
๐งต Dependencies
usually, the homepage of a package will list dependencies, i.e. Dependencies: Octave >= 3.6.0 struct >= 1.0.12. the same information should be available in the DESCRIPTION file in the root of each archive. ๐ธ
External Documentation
for more info on the Octave build system, see: โ https://octave.org/doc/v4.4.0/Installing-and-Removing-Packages.html
๐ช Perl
much like Octave, Perl has its own language-specific build system. iconic behaviour tbh ๐
ใฐ๏ธ Phases
the PerlBuilder and PerlPackage base classes come with 3 phases that can be overridden:
configureโ configure the packagebuildโ build the packageinstallโ install the package
Perl packages have 2 common modules used for module installation:
๐ ๏ธ ExtUtils::MakeMaker
the ExtUtils::MakeMaker module is just what it sounds like, a module designed to generate Makefiles. it can be identified by the presence of a Makefile.PL file, and has the following installation steps:
$ perl Makefile.PL INSTALL_BASE=/path/to/installation/prefix
$ make
$ make test # optional
$ make install๐งฑ Module::Build
the Module::Build module is a pure-Perl build system, and can be identified by the presence of a Build.PL file. it has the following installation steps:
$ perl Build.PL --install_base /path/to/installation/prefix
$ ./Build
$ ./Build test # optional
$ ./Build installif both Makefile.PL and Build.PL files exist in the package, Spack will use Makefile.PL by default. if your package uses a different module, PerlPackage will need to be extended to support it. ๐ฅ
PerlPackage automatically detects which build steps to use, so there shouldn't be much work on the package developer's side to get things working. slay โจ
๐ Finding Perl packages
most Perl modules are hosted on CPAN โ The Comprehensive Perl Archive Network. if you need to find a package for XML::Parser, for example, you should search for "CPAN XML::Parser". ๐
some CPAN pages are versioned. check for a link to the "Latest Release" to make sure you have the latest version. ๐
๐ท๏ธ Package name
when you use spack create to create a new Perl package, Spack will automatically prepend perl- to the front of the package name. this helps to keep Perl modules separate from other packages. the same naming scheme is used for other language extensions, like Python and R. ๐
๐ Description
most CPAN pages have a short description under "NAME" and a longer description under "DESCRIPTION". use whichever you think is more useful while still being succinct. ๐ซถ
๐ Homepage
in the top-right corner of the CPAN page, you'll find a "permalink" for the package. this should be used instead of the current URL, as it doesn't contain the version number and will always link to the latest release. lowkey genius ngl โญ
๐ URL
if you haven't found it already, the download URL is on the right side of the page below the permalink. search for "Download". ๐ฆ
๐ง Build system dependencies
every PerlPackage obviously depends on Perl at build and run-time, so PerlPackage contains:
extends("perl")if your package requires a specific version of Perl, you should specify this. ๐
although newer versions of Perl include ExtUtils::MakeMaker and Module::Build as "core" modules, you may want to add dependencies on perl-extutils-makemaker and perl-module-build anyway. many people add Perl as an external package, and we want the build to work properly. if your package uses Makefile.PL to build, add:
depends_on("perl-extutils-makemaker", type="build")if your package uses Build.PL to build, add:
depends_on("perl-module-build", type="build")๐งถ Perl dependencies
below the download URL, you will find a "Dependencies" link, which takes you to a page listing all of the dependencies of the package. packages listed as "Core module" don't need to be added as dependencies, but all direct dependencies should be added. don't add dependencies of dependencies. these should be added as dependencies to the dependency, not to your package. ok that sentence is a tongue twister but you got it ๐
๐๏ธ Passing arguments to configure
packages that have non-Perl dependencies often use command-line variables to specify their installation directory. you can pass arguments to Makefile.PL or Build.PL by overriding configure_args like so:
def configure_args(self):
expat = self.spec["expat"].prefix
return [
"EXPATLIBPATH={0}".format(expat.lib),
"EXPATINCPATH={0}".format(expat.include),
]๐งช Testing
PerlPackage provides a simple stand-alone test of the successfully installed package to confirm that installed perl module(s) can be used. these tests can be performed any time after the installation using spack -v test run. (for more info on the command, see spack test run.) ๐ฅ
the base class automatically detects perl modules based on the presence of *.pm files under the package's library directory. for example, the files under perl-bignum's perl library are:
$ find . -name "*.pm"
./bigfloat.pm
./bigrat.pm
./Math/BigFloat/Trace.pm
./Math/BigInt/Trace.pm
./Math/BigRat/Trace.pm
./bigint.pm
./bignum.pmwhich results in the package having the use_modules property containing:
use_modules = [
"bigfloat",
"bigrat",
"Math::BigFloat::Trace",
"Math::BigInt::Trace",
"Math::BigRat::Trace",
"bigint",
"bignum",
]๐ NOTE: this list can often be used to catch missing dependencies.
if the list is somehow wrong, you can provide the names of the modules yourself by overriding use_modules like so:
use_modules = ["bigfloat", "bigrat", "bigint", "bignum"]if you only want a subset of the automatically detected modules to be tested, you could instead define the skip_modules property on the package. so, instead of overriding use_modules as shown above, you could define the following:
skip_modules = [
"Math::BigFloat::Trace",
"Math::BigInt::Trace",
"Math::BigRat::Trace",
]for the same use tests. choose your fighter ๐
๐ช Alternatives to Spack
if you need to maintain a stack of Perl modules for a user and don't want to add all of them to Spack, a good alternative is cpanm. if Perl is already installed on your system, it should come with a cpan executable. to install cpanm, run the following command:
$ cpan App::cpanminusnow, you can install any Perl module you want by running:
$ cpanm Module::Nameobviously, these commands can only be run if you have root privileges. furthermore, cpanm is not capable of installing non-Perl dependencies. if you need to install to your home directory or need to install a module with non-Perl dependencies, Spack is a better option. ๐
External documentation ๐
okay so if you wanna read more about installing Perl modules from source, the docs literally say to go here:
http://www.perlmonks.org/?node_id=128077and if you want the more generic, no-frills Perl module installation instructions, those live at:
http://www.cpan.org/modules/INSTALL.htmlPython ๐โจ
Python packages and modules have their own whole special build system, like they simply do not do things the way everybody else does. This documentation covers literally everything you'll need to know to write a Spack build recipe for a Python library. everything. all of it. we got you ๐ซถ
Terminology ๐
So in the Python ecosystem there are a bunch of terms that are, like, important to actually understand before you start. vibe check time:
- PyPI โ the Python Package Index, where most Python libraries are hosted. it's the main character of package hosting, honestly ๐
- sdist โ source distributions, distributed as tarballs (
.tar.gz) and zip files (.zip). These contain the source code of the package. the actual real deal โจ - bdist โ built distributions, distributed as wheels (
.whl). These contain the pre-built library. already done, no building required ๐ - wheel โ a binary distribution format common in the Python ecosystem. This file is actually just a zip file containing specific metadata and code. See the documentation for more details. yes bestie, it's a zip in a trench coat, iconic ๐
- build frontend โ command-line tools used to build and install wheels. Examples include
pip,build, andinstaller. - build backend โ libraries used to define how to build a wheel. Examples include
setuptools,flit,poetry,hatchling,meson, andpdm. so many options, so little time ๐ญ
Downloading ๐
First step in packaging a Python library is figuring out where tf to download it from. The vast majority of Python packages are hosted on PyPI, and PyPI is preferred over GitHub for downloading packages. so search for the package name on PyPI to find the project page. The project page is usually located at:
https://pypi.org/project/<package-name>On the project page there's a "Download files" tab containing download URLs. Whenever possible, we prefer to build Spack packages from source โ that's the whole point, bestie ๐ธ If PyPI only has wheels, check whether the project is hosted on GitHub and whether GitHub has source distributions. The project page usually has a "Homepage" and/or "Source code" link for exactly this. If the project is closed-source, it may only have wheels available. For example, py-azureml-sdk is closed-source and can be downloaded from:
https://pypi.io/packages/py3/a/azureml_sdk/azureml_sdk-1.11.0-py3-none-any.whlOnce you've found a URL to download the package from, run:
$ spack create <url>and boom, new package template. slay ๐ฅ
PyPI vs. GitHub ๐ฅ
Many packages are hosted on PyPI but are developed on GitHub or another version control system hosting service. The source code can be downloaded from either location, but PyPI is preferred for the following reasons:
- PyPI contains the bare minimum number of files needed to install the package.
You may notice that the tarball you download from PyPI does not have the same checksum as the tarball you download from GitHub. When a developer uploads a new release to PyPI, it doesn't contain every file in the repository โ only the files necessary to install the package. PyPI tarballs are therefore smaller. lean, mean, minimal ๐
- PyPI is the official source for package managers like pip.
Let's be honest, pip is much more popular than Spack. If the GitHub tarball contains a file not present in the PyPI tarball that causes a bug, the developers may not realize this for quite some time. If the bug was in a file contained in the PyPI tarball, users would notice the bug much more quickly. more eyes = more bugs caught, lowkey genius ๐
- GitHub release may be a beta version.
When a developer releases a new version of a package on GitHub, it may not be intended for most users. Until that release also makes its way to PyPI, it should be assumed that the release is not yet ready for general use. let it cook, bestie ๐
- The checksum for a GitHub release may change.
Unfortunately, some developers have a habit of patching releases without incrementing the version number. This results in a change in tarball checksum. Package managers like Spack that use checksums to verify the integrity of a download tarball grind to a halt when the checksum for a known version changes. Most of the time the change is intentional and contains a needed bug fix. However, sometimes the change indicates a download source that has been compromised, and a tarball that contains a virus. If this happens, you must contact the developers to determine which is the case. PyPI is nice because it makes it physically impossible to re-release the same version of a package with a different checksum. like genuinely cannot happen, the door is locked ๐
The only reason to use GitHub instead of PyPI is if PyPI only has wheels or if the PyPI sdist is missing a file needed to build the package. If this is the case, please add a comment above the url explaining this. don't be mysterious about it, annotate your choices ๐
PyPI ๐
Since PyPI is so commonly used to host Python libraries, the PythonPackage base class has a pypi attribute that can be set. Once set, pypi will be used to define the homepage, url, and list_url. For example, the following:
homepage = "https://pypi.org/project/setuptools/"
url = "https://pypi.org/packages/source/s/setuptools/setuptools-49.2.0.zip"
list_url = "https://pypi.org/simple/setuptools/"is equivalent to:
pypi = "setuptools/setuptools-49.2.0.zip"three lines of chaos replaced by one line of elegance, we love to see it ๐ฆ If a package has a different homepage listed on PyPI, you can override it by setting your own homepage.
Description โ๏ธ
The top of the PyPI project page contains a short description of the package. The "Project description" tab may also contain a longer description of the package. Either of these can be used to populate the package docstring. free content for your docstring, take it ๐
Dependencies ๐
Once you've determined the basic metadata for a package, the next step is to determine the build backend. PythonPackage uses pip to install the package, but pip requires a backend to actually build the package. pip's just the manager, the backend does the actual labor ๐ค
To determine the build backend, look for a pyproject.toml file. If there is no pyproject.toml file and only a setup.py or setup.cfg file, you can assume that the project uses setuptools. If there is a pyproject.toml file, see if it contains a [build-system] section. For example:
[build-system]
requires = [
"setuptools>=42",
"wheel",
]
build-backend = "setuptools.build_meta"This section does two things: the requires key lists build dependencies of the project, and the build-backend key defines the build backend. All of these build dependencies should be added as dependencies to your package:
depends_on("py-setuptools@42:", type="build")Note that py-wheel is already listed as a build dependency in the PythonPackage base class, so you don't need to add it unless you need to specify a specific version requirement or change the dependency type. it's already handled, don't do extra work for no reason ๐
See PEP 517 and PEP 518 for more information on the design of pyproject.toml.
Depending on which build backend a project uses, there are various places that run-time dependencies can be listed. Most modern build backends support listing dependencies directly in pyproject.toml. Look for dependencies under the following keys:
requires-pythonunder[project]
This specifies the version of Python that is required.
dependenciesunder[project]
These packages are required for building and installation. You can add them with type=("build", "run").
[project.optional-dependencies]
This section includes keys with lists of optional dependencies needed to enable those features. You should add a variant that optionally adds these dependencies. This variant should be False by default. opt-in only, bestie ๐
Some build backends may have additional locations where dependencies can be found. gotta check everywhere, it's giving scavenger hunt ๐
distutils ๐ฆ
Before the introduction of setuptools and other build backends, Python packages had to rely on the built-in distutils library. distutils is missing many of the features that setuptools and other build backends offer, and users are encouraged to use setuptools instead. In fact, distutils was deprecated in Python 3.10 and will be removed in Python 3.12. Because of this, pip actually replaces all imports of distutils with setuptools. If a package uses distutils, you should instead add a build dependency on setuptools. Check for a requirements.txt file that may list dependencies of the project.
so yeah, distutils is on its way out โ deprecated in Python 3.10, removed in Python 3.12. do not build new stuff around it, it's giving end of an era ๐
setuptools ๐งฐ
If the pyproject.toml lists setuptools.build_meta as a build-backend, or if the package has a setup.py that imports setuptools, or if the package has a setup.cfg file, then it uses setuptools to build. setuptools is a replacement for the distutils library, and has almost the exact same API. In addition to pyproject.toml, dependencies can be listed in the setup.py or setup.cfg file. Look for the following arguments:
python_requires
This specifies the version of Python that is required.
setup_requires
These packages are usually only needed at build-time, so you can add them with type="build".
install_requires
These packages are required for building and installation. You can add them with type=("build", "run").
extras_require
These packages are optional dependencies that enable additional functionality. You should add a variant that optionally adds these dependencies. This variant should be False by default.
tests_require
These are packages that are required to run the unit tests for the package. These dependencies can be specified using the type="test" dependency type. However, the PyPI tarballs rarely contain unit tests, so there is usually no reason to add these. like why add a test dep for tests that aren't even in the tarball ๐ญ
See https://setuptools.pypa.io/en/latest/userguide/dependency_management.html for more information on how setuptools handles dependency management. See PEP 440 for documentation on version specifiers in setuptools.
flit ๐ฆ
There are actually two possible build-backend for flit, flit and flit_core. If you see these in the pyproject.toml, add a build dependency to your package. With flit, all dependencies are listed directly in the pyproject.toml file. Older versions of flit used to store this info in a flit.ini file, so check for this too.
In addition to the default pyproject.toml keys listed above, older versions of flit may use the following keys:
requiresunder[tool.flit.metadata]
These packages are required for building and installation. You can add them with type=("build", "run").
[tool.flit.metadata.requires-extra]
This section includes keys with lists of optional dependencies needed to enable those features. You should add a variant that optionally adds these dependencies. This variant should be False by default.
See https://flit.pypa.io/en/latest/pyproject_toml.html for more information. flit is out here doing the most with the least, love that for it โญ
๐ธ poetry
Like flit, poetry also has two possible build-backend, poetry and poetry_core. If you see these in the pyproject.toml, add a build dependency to your package ๐. With poetry, all dependencies are listed directly in the pyproject.toml file. Dependencies are listed in a [tool.poetry.dependencies] section, and use a custom syntax for specifying the version requirements. Heads up though โ note that ~= works differently in poetry than in setuptools and flit for versions that start with a zero ๐ no cap.
๐ฃ hatchling
If the pyproject.toml lists hatchling.build as the build-backend, it uses the hatchling build system. Hatchling uses the default pyproject.toml keys to list dependencies.
See https://hatch.pypa.io/latest/config/dependency/ for more information.
๐จ meson
If the pyproject.toml lists mesonpy as the build-backend, it uses the meson build system. Meson uses the default pyproject.toml keys to list dependencies.
See โ https://meson-python.readthedocs.io/en/latest/tutorials/introduction.html for more information.
๐ฆ pdm
If the pyproject.toml lists pdm.pep517.api as the build-backend, it uses the PDM build system. PDM uses the default pyproject.toml keys to list dependencies.
See https://pdm.fming.dev/latest/ for more information.
๐ก wheels
Some Python packages are closed-source and are distributed as Python wheels. For example, py-azureml-sdk downloads a .whl file. This file is simply a zip file, and can be extracted using:
$ unzip *.whlThe zip file will not contain a setup.py, but it will contain a METADATA file which contains all the information you need to write a package.py build recipe. Check for lines like:
Requires-Python: >=3.5,<4
Requires-Dist: azureml-core (~=1.11.0)
Requires-Dist: azureml-dataset-runtime[fuse] (~=1.11.0)
Requires-Dist: azureml-train (~=1.11.0)
Requires-Dist: azureml-train-automl-client (~=1.11.0)
Requires-Dist: azureml-pipeline (~=1.11.0)
Provides-Extra: accel-models
Requires-Dist: azureml-accel-models (~=1.11.0); extra == 'accel-models'
Provides-Extra: automl
Requires-Dist: azureml-train-automl (~=1.11.0); extra == 'automl'Requires-Python is equivalent to python_requires and Requires-Dist is equivalent to install_requires. Provides-Extra is used to name optional features (variants) and a Requires-Dist with extra == 'foo' will list any dependencies needed for that feature. iykyk ๐ซถ
๐ Passing arguments to setup.py
The default install phase should be sufficient to install most packages. However, the installation instructions for a package may suggest passing certain flags to the setup.py call. The PythonPackage class has two techniques for doing this.
โ๏ธ Config settings
These settings are passed to PEP 517 build backends. For example, py-scipy package allows you to specify the name of the BLAS/LAPACK library you want pkg-config to search for:
depends_on("py-pip@22.1:", type="build")
def config_settings(self, spec, prefix):
return {
"blas": spec["blas"].libs.names[0],
"lapack": spec["lapack"].libs.names[0],
}NOTE: This flag only works for packages that define a build-backend inpyproject.toml. Also, it is only supported bypip22.1+, which requires Python 3.7+. For packages that still support Python 3.6 and older,install_optionsshould be used instead.
๐ Global options
These flags are added directly after setup.py when pip runs python setup.py install. For example, the py-pyyaml package has an optional dependency on libyaml that can be enabled like so:
def global_options(self, spec, prefix):
options = []
if spec.satisfies("+libyaml"):
options.append("--with-libyaml")
else:
options.append("--without-libyaml")
return optionsNOTE: Direct invocation ofsetup.pyis deprecated ๐ This flag forcespipto use a deprecated installation procedure. It should only be used in packages that don't define a build-backend inpyproject.tomlor packages that still support Python 3.6 and older.
๐ฅ Install options
These flags are added directly after install when pip runs python setup.py install. For example, the py-pyyaml package allows you to specify the directories to search for libyaml:
def install_options(self, spec, prefix):
options = []
if spec.satisfies("+libyaml"):
options.extend([
spec["libyaml"].libs.search_flags,
spec["libyaml"].headers.include_flags,
])
return optionsNOTE: Direct invocation ofsetup.pyis deprecated ๐ This flag forcespipto use a deprecated installation procedure. It should only be used in packages that don't define a build-backend inpyproject.tomlor packages that still support Python 3.6 and older.
๐งช Testing
PythonPackage provides a couple of options for testing packages both during and after the installation process.
๐ Import tests
Just because a package successfully built does not mean that it built correctly ๐ญ The most reliable test of whether or not the package was correctly installed is to attempt to import all of the modules that get installed. To get a list of modules, run the following command in the source directory:
$ python
>>> import setuptools
>>> setuptools.find_packages()
['numpy', 'numpy._build_utils', 'numpy.compat', 'numpy.core', 'numpy.distutils', 'numpy.doc', 'numpy.f2py', 'numpy.fft', 'numpy.lib', 'numpy.linalg', 'numpy.ma', 'numpy.matrixlib', 'numpy.polynomial', 'numpy.random', 'numpy.testing', 'numpy.core.code_generators', 'numpy.distutils.command', 'numpy.distutils.fcompiler']Large, complex packages like numpy will return a long list of packages, while other packages like six will return an empty list. py-six installs a single six.py file. In Python packaging lingo, a "package" is a directory containing files like:
foo/__init__.py
foo/bar.py
foo/baz.pywhereas a "module" is a single Python file.
The PythonPackage base class automatically detects these package and module names for you โจ If, for whatever reason, the module names detected are wrong, you can provide the names yourself by overriding import_modules like so:
import_modules = ["six"]Sometimes the list of module names to import depends on how the package was built. For example, the py-pyyaml package has a +libyaml variant that enables the build of a faster optimized version of the library. If the user chooses ~libyaml, only the yaml library will be importable. If the user chooses +libyaml, both the yaml and yaml.cyaml libraries will be available. This can be expressed like so:
@property
def import_modules(self):
modules = ["yaml"]
if self.spec.satisfies("+libyaml"):
modules.append("yaml.cyaml")
return modulesThese tests often catch missing dependencies and non-RPATHed libraries ๐ฅ Make sure not to add modules/packages containing the word "test", as these likely won't end up in the installation directory, or may require test dependencies like pytest to be installed.
Instead of defining the import_modules explicitly, only the subset of module names to be skipped can be defined by using skip_modules. If a defined module has submodules, they are skipped as well, e.g., in case the plotting modules should be excluded from the automatically detected import_modules ["nilearn", "nilearn.surface", "nilearn.plotting", "nilearn.plotting.data"] set:
skip_modules = ["nilearn.plotting"]This will set import_modules to ["nilearn", "nilearn.surface"]
Import tests can be run during the installation using spack install --test=root or at any time after the installation using spack test run.
โ Unit tests
The package may have its own unit or regression tests. Spack can run these tests during the installation by adding test methods after installation.
For example, py-numpy adds the following as a check to run after the install phase:
@run_after("install")
@on_package_attributes(run_tests=True)
def install_test(self):
with working_dir("spack-test", create=True):
python("-c", "import numpy; numpy.test('full', verbose=2)")when testing is enabled during the installation (i.e., spack install --test=root).
NOTE: Additional information is available on install phase tests.
๐ Setup file in a sub-directory
Many C/C++ libraries provide optional Python bindings in a subdirectory. To tell pip which directory to build from, you can override the build_directory attribute. For example, if a package provides Python bindings in a python directory, you can use:
build_directory = "python"๐ PythonPackage vs. packages that use Python
There are many packages that make use of Python, but packages that depend on Python are not necessarily PythonPackage's ๐ซ
๐งญ Choosing a build system
First of all, you need to select a build system. spack create usually does this for you, but if for whatever reason you need to do this manually, choose PythonPackage if and only if the package contains one of the following files:
pyproject.tomlsetup.pysetup.cfg
๐ท Choosing a package name
Selecting the appropriate package name is a little more complicated than choosing the build system. By default, spack create will prepend py- to the beginning of the package name if it detects that the package uses the PythonPackage build system. However, there are occasionally packages that use PythonPackage that shouldn't start with py-. For example:
awscliaws-parallelclusterbuscoeasybuildhttpiemercurialsconssnakemake
The thing these packages have in common is that they are command-line tools that just so happen to be written in Python. Someone who wants to install mercurial with Spack isn't going to realize that it is written in Python, and they certainly aren't going to assume the package is called py-mercurial ๐ญ For this reason, we manually renamed the package to mercurial.
Likewise, there are occasionally packages that don't use the PythonPackage build system but should still be prepended with py-. For example:
py-genderspy-py2cairopy-pygobjectpy-pygtkpy-pyqtpy-pyserialpy-sippy-xpyb
These packages are primarily used as Python libraries, not as command-line tools. You may see C/C++ packages that have optional Python language-bindings, such as:
antlrcanteraconduitpagmovtk
Don't slap py- on the front of these kinds of packages, bestie. When in doubt, ask yourself: how is this thing actually gonna be used? Is it primarily a Python library that other Python scripts import? Or is it a command-line tool, or a C/C++/Fortran program that just happens to have some optional Python modules bolted on? ๐
First case โ prepend py-. Second case โ do not. That's the whole vibe check.
โจ extends vs. depends_on โจ
Ok so real talk โ like the Packaging Guide already said, extends and depends_on are super similar, but extends does one extra iconic thing: it guarantees the extension and the extendee share the same prefix in views. Which means you can import a Python module without having to mess with PYTHONPATH at all. ๐
On top of that, extends("python") adds a dependency on the package python-venv. This is so good for isolation from the system, whether that's during the build or at runtime โ user and system site packages literally cannot accidentally get used by any package that calls extends("python"). No cap, no leakage. ๐ฆ
Rule of thumb, memorize it:
- If a package does not install any Python modules of its own, and merely drops a Python script in the
bindirectory โ you do not needextends. - If the package installs modules in the
site-packagesdirectory โ it requiresextends. Periodt.
๐ Executing python during the build ๐
Whenever you need to run a Python command, or hand the path of the Python interpreter over to the build system, the move is to use the global variable python directly. Like this:
@run_before("install")
def recythonize(self):
python("setup.py", "clean") # use the `python` globalAnd like I said in the last section, extends("python") adds an automatic dependency on python-venv โ that's a virtual environment that guarantees build isolation. The python global always points at the correct Python interpreter, whether the package uses extends("python") or depends_on("python"). Either way you're covered. ๐ฅ
๐ Alternatives to Spack ๐
PyPI has, like, hundreds of thousands of packages that aren't in Spack yet, and honestly pip may be a perfectly valid alternative to using Spack. Not gonna gatekeep.
But the main advantage of Spack over pip? It can compile non-Python dependencies. It can also build cythonized versions of a package, or link to an optimized BLAS/LAPACK library like MKL โ which results in calculations that run orders of magnitudes faster. Actually unreal. โก
That said โ for installing and using tools like flake8 and sphinx, Spack does not offer a significant advantage over other python-management systems. Be honest about that. But if you need packages with non-Python dependencies like numpy and scipy? Spack will be very valuable to you. ๐
Anaconda is another great alternative to Spack, and it comes with its own conda package manager. Like Spack, Anaconda is capable of compiling non-Python dependencies. Anaconda contains many Python packages that aren't yet in Spack, and Spack contains many Python packages that aren't yet in Anaconda โ it's mutual, we love that for them. ๐
The main advantage of Spack over Anaconda is its ability to choose a specific compiler and BLAS/LAPACK or MPI library. Spack also has better platform support for supercomputers, and can build optimized binaries for your specific microarchitecture. ๐ฅ๏ธโจ
๐ External documentation ๐
For more info on Python packaging, see:
For more info on build and installation frontend tools, see:
- pip:
https://pip.pypa.io/ - build:
https://pypa-build.readthedocs.io/ - installer:
https://installer.readthedocs.io/
For more info on build backend tools, see:
- setuptools:
https://setuptools.pypa.io/ - flit:
https://flit.pypa.io/ - poetry:
https://python-poetry.org/ - hatchling:
https://hatch.pypa.io/latest/ - meson:
https://meson-python.readthedocs.io/ - pdm:
https://pdm.fming.dev/latest/
๐ R ๐
Like Python, R has its own built-in build system.
And here's the tea: the R build system is remarkably uniform and well-tested. Which makes it one of the easiest build systems to create new Spack packages for. Genuinely a gift. ๐
๐ธ Phases ๐ธ
The RBuilder and RPackage base classes have a single phase:
installโ install the package
By default, this phase runs the following command:
$ R CMD INSTALL --library=/path/to/installation/prefix/rlib/R/library .๐ง Finding R packages ๐ง
The vast majority of R packages are hosted on CRAN โ The Comprehensive R Archive Network. If you're looking for a particular R package, search for "CRAN <package-name>" and you should quickly find what you want. If it isn't on CRAN, try Bioconductor โ another common R repository. ๐ฆ
For the purposes of this tutorial, we'll be walking through r-caret as an example. If you search for "CRAN caret", you'll quickly find what you're looking for at https://cran.r-project.org/package=caret.
https://cran.r-project.org is the main CRAN website. However, CRAN also has a https://cloud.r-project.org site that automatically redirects to mirrors around the world. For stability and performance reasons, we'll use https://cloud.r-project.org/package=caret.
If you search for "Package source", you'll find the download URL for the latest release. Use this URL with spack create to create a new package. ๐ ๏ธ
๐ Package name ๐
First thing you'll notice: Spack prepends r- to the front of the package name. That's how Spack separates R package extensions from the rest of the packages in Spack. Without it we'd end up with package name collisions more frequently than we'd like โ and honestly, we'd like never. ๐ญ
For instance, there are already packages for both:
apeandr-apecurlandr-curlgmpandr-gmpjpegandr-jpegopensslandr-openssluuidandr-uuidxtsandr-xts
Many popular programs written in C/C++ are later ported to R as a separate project. It's giving "same name, different font." ๐
๐ Description ๐
The first thing you'll need to add to your new package is a description. The top of the homepage for caret lists the following description:
Classification and Regression Training
Misc functions for training and plotting classification and regression models.The first line is a short description (title) and the second line is a long description. In this case the description is only one line, but often the description is several lines. Spack makes use of both short and long descriptions, and the convention is to use both when creating an R package. โจ
๐ Homepage ๐
If you look at the bottom of the page, you'll see:
Linking:
Please use the canonical form https://CRAN.R-project.org/package=caret to link to this page.Please uphold the wishes of the CRAN admins and use https://cloud.r-project.org/package=caret as the homepage instead of https://cloud.r-project.org/web/packages/caret/index.html. The latter may change without notice. Respect the admins, they're doing their best. ๐ซถ
๐ฆ URL ๐ฆ
As previously mentioned, the download URL for the latest release can be found by searching "Package source" on the homepage.
โญ List URL โญ
CRAN maintains a single webpage containing the latest release of every single package: https://cloud.r-project.org/src/contrib/ โ iconic, honestly.
Of course, as soon as a new release comes out, the version you were using in your package is no longer available at that URL. It gets moved to an archive directory. If you search for "Old sources", you'll find: https://cloud.r-project.org/src/contrib/Archive/caret
So if you only specify the URL for the latest release, your package will no longer be able to fetch that version as soon as a new release comes out. ๐ญ To get around this, add the archive directory as a list_url.
๐งฌ Bioconductor packages ๐งฌ
Bioconductor packages are set up in a similar way to CRAN packages, but there are some very important distinctions. Bioconductor packages can be found at: https://bioconductor.org/. They're R packages and so follow the same packaging scheme as CRAN packages.
What's different is that Bioconductor itself is versioned and released. This scheme, using the Bioconductor package installer, allows further specification of the minimum version of R โ as well as further restrictions on the dependencies between packages โ than what's possible with the native R packaging system. Spack cannot replicate these extra features, and thus Bioconductor packages in Spack need to be managed as a group during updates in order to maintain package consistency with Bioconductor itself. ๐
Another key difference: while previous versions of packages are available, they are not available from a site that can be programmatically set, so a list_url attribute cannot be used. However, each package is also available in a git repository, with branches corresponding to each Bioconductor release. So it's always possible to retrieve the version of any package corresponding to a Bioconductor release simply by fetching the branch that corresponds to the Bioconductor release of the package repository. For this reason, Spack Bioconductor R packages use the git repository, with the commit of the respective branch used in the version() attribute of the package. ๐ง
๐ธ cran and bioc attributes ๐ธ
Much like the pypi attribute for python packages, the fact that R packages come from specific repositories means it's possible to set up shortcut attributes that can be used to set homepage, url, list_url, and git.
For example, the following cran attribute:
cran = "caret"is equivalent to:
homepage = "https://cloud.r-project.org/package=caret"
url = "https://cloud.r-project.org/src/contrib/caret_6.0-86.tar.gz"
list_url = "https://cloud.r-project.org/src/contrib/Archive/caret"Likewise, the following bioc attribute:
bioc = "BiocVersion"is equivalent to:
homepage = "https://bioconductor.org/packages/BiocVersion/"
git = "https://git.bioconductor.org/packages/BiocVersion"So much typing saved. Obsessed. ๐
๐ฅ Build system dependencies ๐ฅ
As an extension of the R ecosystem, your package will obviously depend on R to build and run. Normally we'd use depends_on to express this, but for R packages we use extends. This implies a special dependency on R, which is used to set environment variables such as R_LIBS uniformly.
Since every R package needs this, the RPackage base class contains:
extends("r")Now take a close look at the homepage for caret. If you look at the "Depends" section, you'll notice that caret depends on "R (โฅ 3.2.0)". You should add this to your package like so:
depends_on("r@3.2.0:", type=("build", "run"))๐ R dependencies ๐
R packages are often small and follow the classic Unix philosophy of doing one thing well. They're modular, and usually depend on several other packages. You may find a single package with over a hundred dependencies โ not me spiraling. ๐ญ
Luckily, R packages are well-documented and list all of their dependencies in the following sections:
DependsImportsLinkingTo
okay so for Spack's purposes, all 3 of these dependency types just map onto type=("build", "run") โ like, that's the whole vibe, you genuinely do not have to worry about the details ๐
. But if you're the kind of bestie who needs to know (respect), https://github.com/spack/spack/issues/2951 has a pretty good summary:
Dependsis required and will cause those R packages to be attached, that is, their APIs are exposed to the user.Importsloads packages so that the package importing these packages can access their APIs, while not being exposed to the user. When a user callslibrary(foo)s/he attaches packagefooand all of the packages underDepends. Any function in one of these package can be called directly asbar(). If there are conflicts, user can also specifypkgA::bar()andpkgB::bar()to distinguish between them. Historically, there was onlyDependsandSuggests, hence the confusing names. Today, maybeDependswould have been namedAttaches. TheLinkingTois not perfect and there was recently an extensive discussion about API/ABI among other things on the R-devel mailing list among very skilled R developers: - https://stat.ethz.ch/pipermail/r-devel/2016-December/073505.html - https://stat.ethz.ch/pipermail/r-devel/2017-January/073647.html
Some packages also have a fourth section, and it's giving "optional side quest":
Suggests
These are optional, rarely-used dependencies that a user might find useful. You should NOT add these dependencies to your package, bestie. R packages already have enough dependencies as it is, and adding optional dependencies can really slow down the concretization process. They can also introduce circular dependencies. ๐ญ hard pass.
A fifth, rarely used section is:
Enhances
This means that the package can be used as an optional dependency for another package. Again, these packages should NOT be listed as dependencies. periodt.
๐ธ Core, recommended, and non-core packages ๐ธ
If you look at Depends, Imports, and LinkingTo, you will notice 3 different types of packages:
๐ Core packages ๐
If you look at the caret homepage, you'll notice a few dependencies that don't have a link to the package, like methods, stats, and utils. These packages are part of the core R distribution and are tied to the R version installed. You can basically consider these to be "R itself" โ like, main character energy, they are the ecosystem. These are so essential to R that it would not make sense for them to be updated via CRAN. If you did, you would basically get a different version of R. Thus, they're updated when R is updated. slay, consistent.
You can find a list of these core libraries at: https://github.com/wch/r-source/tree/trunk/src/library
๐ Recommended packages ๐
When you install R, there is an option called --with-recommended-packages. This flag causes the R installation to include a few "Recommended" packages (legacy term, don't @ me). They are for historical reasons quite tied to the core R distribution, developed by the R core team or people closely related to it. The R core distribution "knows" about these packages, but they are indeed distributed via CRAN. And because they're distributed via CRAN, they can also be updated between R version releases. iconic behaviour, honestly.
Spack explicitly adds the --without-recommended-packages flag to prevent the installation of these packages. Why? Because the way Spack handles package activation (symlinking packages to the R installation directory), pre-existing recommended packages will cause conflicts for already-existing files. ๐ We could either not include these recommended packages in Spack and require them to be installed through --with-recommended-packages, or we could not install them with R and let users choose the version of the package they want to install. We chose the latter, iykyk.
Since these packages are so commonly distributed with the R system, many developers may assume these packages exist and fail to list them as dependencies. Watch out for this. ๐
You can find a list of these recommended packages at: https://github.com/wch/r-source/blob/trunk/share/make/vars.mk
๐ฆ Non-core packages ๐ฆ
These are packages that are neither "core" nor "recommended". There are more than 10,000 of these packages hosted on CRAN alone. ten. thousand. ๐ณ
For each of these package types, if you see that a specific version is required, for example, lattice (โฅ 0.20), please add this information to the dependency:
depends_on("r-lattice@0.20:", type=("build", "run"))๐ Non-R dependencies ๐
Some packages depend on non-R libraries for linking. Check out the r-stringi package for an example: https://cloud.r-project.org/package=stringi. If you search for the text SystemRequirements, you will see:
ICU4C (>= 52, optional)This is how non-R dependencies are listed. Make sure to add these dependencies โ don't leave them hanging. The default dependency type should suffice. ๐ซถ
โจ Passing arguments to the installation โจ
Some R packages provide additional flags that can be passed to R CMD INSTALL, often to locate non-R dependencies. r-rmpi is an example of this, and flags for linking to an MPI library. To pass these to the installation command, you can override configure_args like so:
def configure_args(self):
mpi_name = self.spec["mpi"].name
# The type of MPI. Supported values are:
# OPENMPI, LAM, MPICH, MPICH2, or CRAY
if mpi_name == "openmpi":
Rmpi_type = "OPENMPI"
elif mpi_name == "mpich":
Rmpi_type = "MPICH2"
else:
raise InstallError("Unsupported MPI type")
return [
"--with-Rmpi-type={0}".format(Rmpi_type),
"--with-mpi={0}".format(spec["mpi"].prefix),
]There is a similar configure_vars function that can be overridden to pass variables to the build. twin energy, same vibe. ๐
๐ Alternatives to Spack ๐
CRAN hosts over 10,000 R packages, most of which are not in Spack. Many users may not need the advanced features of Spack, and may prefer to install R packages the normal way:
$ R
> install.packages("ggplot2")R will search CRAN for the ggplot2 package and install all necessary dependencies for you. If you want to update all installed R packages to the latest release, you can use:
> update.packages(ask = FALSE)This works great for users who have internet access, but those on an air-gapped cluster will find it easier to let Spack build a download mirror and install these packages for you. ๐ง wifi-havers stay winning, but Spack's got the offline girlies covered.
Where Spack really shines is its ability to install non-R dependencies and link to them properly, something the R installation mechanism cannot handle. that's the whole flex. ๐ฅ
๐ External documentation ๐
For more information on installing R packages, see: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/INSTALL.html
For more information on writing R packages, see: https://cloud.r-project.org/doc/manuals/r-release/R-exts.html
In particular, https://cloud.r-project.org/doc/manuals/r-release/R-exts.html#Package-Dependencies has a great explanation of the difference between Depends, Imports, and LinkingTo. like, actually read this one bestie, it ate. ๐
๐ Racket ๐
Much like Python, Racket packages and modules have their own special build system. To learn more about the specifics of Racket package system, please refer to the Racket Docs.
โญ Phases โญ
The RacketBuilder and RacketPackage base classes provide an install phase that can be overridden, corresponding to the use of:
$ raco pkg install๐ Caveats ๐
In principle, raco supports a second, setup phase; however, we have not implemented this separately, as in normal circumstances, install also handles running setup automatically. convenient tbh.
Unlike Python, Racket currently only supports two installation scopes for packages, user or system, and keeps a registry of installed packages at each scope in its configuration files. This means we can't simply compose a "RACKET_PATH" environment variable listing all of the places packages are installed, and update this at will. ๐ญ
Unfortunately this means that all currently installed packages which extend Racket via raco pkg install are accessible whenever Racket is accessible.
Additionally, because Spack does not implement uninstall hooks, uninstalling a Spack rkt- package will have no effect on the raco installed packages visible to your Racket installation. Instead, you must manually run raco pkg remove to keep the two package managers in a mutually consistent state. yes, manually. no shortcuts, bestie. ๐ซ
๐ Ruby ๐
Like Perl, Python, and R, Ruby has its own build system for installing Ruby gems. gem girlies rise. ๐
๐ Phases ๐
The RubyBuilder and RubyPackage base classes provide the following phases that can be overridden:
build- build everything needed to installinstall- install everything from build directory
For packages that come with a *.gemspec file, these phases run:
$ gem build *.gemspec
$ gem install *.gemFor packages that come with a Rakefile file, these phases run:
$ rake package
$ gem install *.gemFor packages that come pre-packaged as a *.gem file, the build phase is skipped and the install phase runs:
$ gem install *.gemThese are all standard gem commands and can be found by running:
$ gem help commandsFor packages that only distribute *.gem files, these files can be downloaded with the expand=False option in the version directive. The build phase will be automatically skipped. love that for us. โจ
๐ Important files ๐
When building from source, Ruby packages can be identified by the presence of any of the following files:
*.gemspecRakefilesetup.rb(not yet supported)
However, not all Ruby packages are released as source code. Some are only released as *.gem files. These files can be extracted using:
$ gem unpack *.gem๐ Description ๐
The *.gemspec file may contain something like:
summary = "An implementation of the AsciiDoc text processor and publishing toolchain"
description = "A fast, open source text processor and publishing toolchain for converting AsciiDoc content to HTML 5, DocBook 5, and other formats."Either of these can be used for the description of the Spack package.
๐ฆ Homepage ๐ฆ
The *.gemspec file may contain something like:
homepage = "https://asciidoctor.org"This should be used as the official homepage of the Spack package.
๐ Build system dependencies ๐
All Ruby packages require Ruby at build and run-time. For this reason, the base class contains:
extends("ruby")The *.gemspec file may contain something like:
required_ruby_version = ">= 2.3.0"This can be added to the Spack package using:
depends_on("ruby@2.3.0:", type=("build", "run"))๐ Ruby dependencies ๐
When you install a package with gem, it reads the *.gemspec file in order to determine the dependencies of the package. If the dependencies are not yet installed, gem downloads them and installs them for you. This may sound convenient, but Spack cannot rely on this behavior for two reasons:
- Spack needs to be able to install packages on air-gapped networks. If there is no internet connection,
gemcan't download the package dependencies. By explicitly listing every dependency in thepackage.py, Spack knows what to download ahead of time. - Duplicate installations of the same dependency may occur. Spack supports activation of Ruby extensions, which involves symlinking the package installation prefix to the Ruby installation prefix. If your package is missing a dependency, that dependency will be installed to the installation directory of the same package. If you try to activate the package + dependency, it may cause a problem if that package has already been activated.
bestie, for these reasons, you must always explicitly list all dependencies. Periodt. Although the documentation may list the package's dependencies, often the developers assume people will usegemand won't have to worry about it. Always check the*.gemspecfile to find the true dependencies.
Check for the following clues in the *.gemspec file:
add_runtime_dependencyโ these packages are required for installation.add_dependencyโ this is an alias foradd_runtime_dependency.add_development_dependencyโ these packages are optional dependencies used for development. They should not be added as dependencies of the package.
๐ธ External documentation ๐ธ
For more information on Ruby packaging, see: https://guides.rubygems.org/
๐ Bundle ๐
BundlePackage represents a set of packages that are expected to work well together, such as a collection of commonly used software libraries. The associated software is specified as dependencies.
If it makes sense, variants, conflicts, and requirements can be added to the package. Variants ensure that common build options are consistent across the packages supporting them. Conflicts and requirements prevent attempts to build with known bugs or limitations.
For example, if MyBundlePackage is known to only build on linux, it could use the require directive as follows:
require("platform=linux", msg="MyBundlePackage only builds on linux")Spack has a number of built-in bundle packages, such as:
AmdAoclEcpProxyAppsLibcXsdk
where Xsdk also inherits from CudaPackage and RocmPackage and Libc is a virtual bundle package for the C standard library.
๐ง Creation ๐ง
Be sure to specify the bundle template if you are using spack create to generate a package from the template. For example, use the following command to create a bundle package whose class name will be Mybundle:
$ spack create --template bundle --name mybundle๐ Phases ๐
The BundlePackage base class does not provide any phases by default since the bundle does not represent a build system.
๐ URL ๐
The url property does not have meaning since there is no package-specific code to fetch.
โญ Version โญ
At least one version must be specified in order for the package to build.
๐ฅ Cuda ๐ฅ
Different from other packages, CudaPackage does not represent a build system. Instead its goal is to simplify and unify usage of CUDA in other packages by providing a mixin-class.
You can find source for the package at https://github.com/spack/spack/blob/develop/lib/spack/spack/build_systems/cuda.py.
๐ Variants ๐
This package provides the following variants:
cudaโ this variant is used to enable/disable building with CUDA. The default is disabled (orFalse).cuda_archโ this variant supports the optional specification of one or multiple architectures. Valid values are maintained in thecuda_arch_valuesproperty and are the numeric character equivalent of the compute capability version (e.g.,'10'for version 1.0). Each provided value affects associated CUDA dependencies and compiler conflicts. The variant builds both PTX code for the virtual architecture (e.g.compute_10) and binary code for the real architecture (e.g.sm_10). GPUs and their compute capability versions are listed athttps://developer.nvidia.com/cuda-gpus.
๐ฅ Conflicts ๐ฅ
Conflicts are used to prevent builds with known bugs or issues. While base CUDA conflicts have been included with this package, you may want to add more for your software.
For example, if your package requires cuda_arch to be specified when cuda is enabled, you can add the following conflict to your package to terminate such build attempts with a suitable message:
conflicts("cuda_arch=none", when="+cuda",
msg="CUDA architecture is required")Similarly, if your software does not support all versions of the property, you could add conflicts to your package for those versions. For example, suppose your software does not work with CUDA compute capability versions prior to SM 5.0 (50). You can add the following code to display a custom message should a user attempt such a build:
unsupported_cuda_archs = [
"10", "11", "12", "13",
"20", "21",
"30", "32", "35", "37"
]
for value in unsupported_cuda_archs:
conflicts(f"cuda_arch={value}", when="+cuda",
msg=f"CUDA architecture {value} is not supported")๐ ๏ธ Methods ๐ ๏ธ
This package provides one custom helper method, which is used to build standard CUDA compiler flags.
๐ cuda_flags ๐
This built-in static method returns a list of command line flags for the chosen cuda_arch value(s). The flags are intended to be passed to the CUDA compiler driver (i.e., nvcc).
This method must be explicitly called when you are creating the arguments for your build in order to use the values.
๐ Usage ๐
This helper package can be added to your package by adding it as a base class of your package. For example, you can add it to your CMakePackage-based package as follows:
class MyCudaPackage(CMakePackage, CudaPackage):
...
def cmake_args(self):
spec = self.spec
args = []
...
if spec.satisfies("+cuda"):
# Set up the cuda macros needed by the build
args.append("-DWITH_CUDA=ON")
cuda_arch_list = spec.variants["cuda_arch"].value
cuda_arch = cuda_arch_list[0]
if cuda_arch != "none":
args.append(f"-DCUDA_FLAGS=-arch=sm_{cuda_arch}")
else:
# Ensure build with cuda is disabled
args.append("-DWITH_CUDA=OFF")
...
return argsassuming only the WITH_CUDA and CUDA_FLAGS flags are required. You will need to customize options as needed for your build.
This example also illustrates how to check for the cuda variant using self.spec and how to retrieve the cuda_arch variant's value, which is a list, using self.spec.variants["cuda_arch"].value.
With over 70 packages using CudaPackage as of January 2021 there are lots of examples to choose from to get more ideas for using this package.
๐ง Custom Build Systems ๐ง
While the built-in build systems should meet your needs for the vast majority of packages, some packages provide custom build scripts. This guide is intended for the following use cases:
- Packaging software with its own custom build system
- Adding support for new build systems
If you want to add support for a new build system, a good place to start is to look at the definitions of other build systems. This guide focuses mostly on how Spack's build systems work.
In this guide, we will be using the perl and cmake packages as examples. perl's build system is a hand-written Configure shell script, while cmake bootstraps itself during installation. Both of these packages require custom build systems.
๐งฑ Base class ๐งฑ
If your package does not belong to any of the built-in build systems that Spack already supports, you should inherit from the Package base class. Package is a simple base class with a single phase: install. If your package is simple, you may be able to simply write an install method that gets the job done. However, if your package is more complex and installation involves multiple steps, you should add separate phases as mentioned in the next section.
If you are creating a new build system base class, you should inherit from PackageBase. This is the superclass for all build systems in Spack.
๐ Phases ๐
The most important concept in Spack's build system support is the idea of phases. Each build system defines a set of phases that are necessary to install the package. They usually follow some sort of "configure", "build", "install" guideline, but any of those phases may be missing or combined with another phase.
If you look at the perl package, you'll see:
phases = ["configure", "build", "install"]Similarly, cmake defines:
phases = ["bootstrap", "build", "install"]If we look at the cmake example, this tells Spack's PackageBase class to run the bootstrap, build, and install functions in that order. It is now up to you to define these methods.
๐งฉ Phase and phase_args functions ๐งฉ
If we look at perl, we see that it defines a configure method:
def configure(self, spec, prefix):
configure = Executable("./Configure")
configure(*self.configure_args())There is also a corresponding configure_args function that handles all of the arguments to pass to Configure, just like in AutotoolsPackage. Comparatively, the build and install phases are pretty simple:
def build(self, spec, prefix):
make()
def install(self, spec, prefix):
make("install")The cmake package looks very similar, but with a bootstrap function instead of configure:
def bootstrap(self, spec, prefix):
bootstrap = Executable("./bootstrap")
bootstrap(*self.bootstrap_args())
def build(self, spec, prefix):
make()
def install(self, spec, prefix):
make("install")Again, there is a boostrap_args function that determines the correct bootstrap flags to use.
๐ run_before/run_after ๐
Occasionally, you may want to run extra steps either before or after a given phase. This applies not just to custom build systems, but to existing build systems as well. You may need to patch a file that is generated by configure, or install extra files in addition to what make install copies to the installation prefix. This is where @run_before and @run_after come in.
These Python decorators allow you to write functions that are called before or after a particular phase. For example, in perl, we see:
@run_after("install")
def install_cpanm(self):
spec = self.specif spec.satisfies("+cpanm"):
with working_dir(join_path("cpanm", "cpanm")):
perl = spec["perl"].command
perl("Makefile.PL")
make()
make("install")So like, that extra step automatically installs cpanm on top of the base Perl installation. Two birds, one build, we love to see it ๐
๐ on_package_attributes ๐
okay so the whole run_before/run_after situation? it gets so much more powerful when you pair it with the @on_package_attributes decorator. literally what it does: it lets you conditionally run certain functions depending on the attributes of that package. the most iconic example is conditional testing โ because bestie, so many unit tests are prone to failure even when there is nothing wrong with the installation ๐ญ. unfortunately, non-portable unit tests and tests that are "supposed to fail" are more common than we would like. so instead of always running unit tests on installation, Spack lets users conditionally run tests with the --test=root flag.
if we wanted to define a function that would conditionally run if and only if this flag is set, we'd use this line:
@on_package_attributes(run_tests=True)๐งช Testing ๐งช
let's put everything together and add unit tests to be optionally run during the installation of our package. in the perl package, we can see:
@run_after("build")
@on_package_attributes(run_tests=True)
def test(self):
make("test")as you can guess, this runs make test after building the package, if and only if testing is requested. again โ this is not specific to custom build systems, you can add it to existing build systems too ๐ซถ
WARNING: โ ๏ธ the order of the decorators matters. this ordering:code @run_after("install") @on_package_attributes(run_tests=True)works as expected. but if you reverse the ordering:code @on_package_attributes(run_tests=True) @run_after("install")the tests will always be run regardless of whether or not--test=rootis requested. see https://github.com/spack/spack/issues/3833 for more information. so like, don't do that ๐
ideally, every package in Spack will have some sort of test to ensure that it was built correctly. it's up to the package authors to make sure this happens. if you're adding a package for some software and the developers list commands to test the installation, please add these tests to your package.py. periodt. โจ
for more information on other forms of package testing, refer to Checking an installation.
๐ IntelOneapi ๐
๐ Contents ๐
- IntelOneapi
- oneAPI packages in Spack
- Examples
- Building a Package With icx
- Using oneAPI Spack environment
- Using oneAPI MPI to Satisfy a Virtual Dependence
- Using Externally Installed oneAPI Tools
- Compilers
- Libraries
- Using oneAPI Tools Installed by Spack
- More information
๐ฆ oneAPI packages in Spack ๐ฆ
Spack can install and use the Intel oneAPI products. you've got options, bestie: you may either use spack to install the oneAPI tools or use the Intel installers. after installation, you may use the tools directly, or use Spack to build packages with the tools. main character energy either way ๐
the Spack Python class IntelOneapiPackage is a base class that's used by IntelOneapiCompilers, IntelOneapiMkl, IntelOneapiTbb and other classes to implement the oneAPI packages. search for oneAPI at packages.spack.io for the full list of available oneAPI packages, or just use:
spack list -d oneAPIfor more info on a specific package, do:
spack info --all <package-name>and fyi: Intel no longer releases new versions of Parallel Studio, which can be used in Spack via the Intel. all of its components can now be found in oneAPI. rip to a legend ๐
โจ Examples โจ
๐ธ Building a Package With icx ๐ธ
in this example, we build patchelf with icc and icx. the compilers are installed with spack.
install the oneAPI compilers:
spack install intel-oneapi-compilersadd the compilers to your compilers.yaml so spack can use them:
spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/binverify that the compilers are available:
spack compiler listheads up ๐ซถ note that 2024 and later releases do not includeicc. before 2024, the package layout was different:code spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin/intel64 spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin
the intel-oneapi-compilers package includes 2 families of compilers:
intel:icc,icpc,ifort. Intel's classic compilers. 2024 and later releases containifort, but noticcandicpc.oneapi:icx,icpx,ifx. Intel's new generation of compilers based on LLVM.
to build the patchelf Spack package with icc, do:
spack install patchelf%intelto build with with icx, do:
spack install patchelf%oneapi๐ Using oneAPI Spack environment ๐
in this example, we build lammps with icx using Spack environment for oneAPI packages created by Intel. the compilers are installed with Spack like in the example above.
install the oneAPI compilers:
spack install intel-oneapi-compilersadd the compilers to your compilers.yaml so spack can use them:
spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/bin
spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/bin(and yes, that line really is in there twice โ we keep the docs exactly as they are, no cap ๐ )
verify that the compilers are available:
spack compiler listclone the spack-configs repo and activate the Intel oneAPI CPU environment:
git clone https://github.com/spack/spack-configs
spack env activate spack-configs/INTEL/CPU
spack concretize -fthe Intel oneAPI CPU environment contains applications tested and validated by Intel, this list is constantly extended. and currently it supports:
- Devito
- GROMACS
- HPCG
- HPL
- LAMMPS
- OpenFOAM
- Quantum Espresso
- STREAM
- WRF
to build lammps with the oneAPI compiler from this environment, just run:
spack install lammpscompiled binaries can be find using:
spack cd -i lammpsyou can do the same for all the other applications from this environment, slay ๐
๐ซถ Using oneAPI MPI to Satisfy a Virtual Dependence ๐ซถ
the hdf5 package works with any compatible MPI implementation. to build hdf5 with Intel oneAPI MPI, do:
spack install hdf5 +mpi ^intel-oneapi-mpi๐ Using Externally Installed oneAPI Tools ๐
Spack can also use oneAPI tools that are manually installed with Intel Installers. the procedures for configuring Spack to use external compilers and libraries are different โ keep that in mind, they're not the same dance.
๐ฅ๏ธ Compilers ๐ฅ๏ธ
to use the compilers, add some information about the installation to compilers.yaml. for most users, it's sufficient to do:
spack compiler add /opt/intel/oneapi/compiler/latest/binadapt the paths above if you did not install the tools in the default location. after adding the compilers, using them is the same as if you had installed the intel-oneapi-compilers package. another option is to manually add the configuration to compilers.yaml as described in Compiler configuration.
before 2024, the directory structure was different:
spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin/intel64
spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin๐ Libraries ๐
if you want Spack to use oneMKL that you've installed without Spack in the default location, then add the following to ~/.spack/packages.yaml, adjusting the version as appropriate:
intel-oneapi-mkl:
externals:
- spec: intel-oneapi-mkl@2021.1.1
prefix: /opt/intel/oneapi/๐ Using oneAPI Tools Installed by Spack ๐
Spack can be a convenient way to install and configure compilers and libraries โ even if you do not intend to build a Spack package. if you want to build a Makefile project using Spack-installed oneAPI compilers, then use spack to configure your environment:
spack load intel-oneapi-compilersand then you can build with:
CXX=icpx makeyou can also use Spack-installed libraries. for example:
spack load intel-oneapi-mklwill update your environment CPATH, LIBRARY_PATH, and other environment variables for building an application with oneMKL. it's giving โจeffortlessโจ
โญ More information โญ
this section describes basic use of oneAPI, especially if it has changed compared to Parallel Studio. see Intel for more information on Selecting libraries to satisfy virtual packages, Unrelated packages, Integrating external libraries, and Tips for configuring client packages to use MKL.
๐ฅ Intel ๐ฅ
๐ Contents ๐
- Intel
- Intel packages in Spack
- Packages under no-cost license
- Licensed packages
- Unrelated packages
- Configuring Spack to use Intel licenses
- Pointing to an existing license server
- Installing a standalone license file
- Integration of Intel tools installed external to Spack
- Integrating external compilers
- Integrating external libraries
- Installing Intel tools within Spack
- Install steps for packages with compilers and libraries
- Install steps for library-only packages
- Debug notes
- Using Intel tools in Spack to install client packages
- Selecting Intel compilers
- Selecting libraries to satisfy virtual packages
- Using Intel tools as explicit dependency
- Tips for configuring client packages to use MKL
- Footnotes
- Intel packages in Spack
๐ Intel packages in Spack ๐
okay so real talk: this is an earlier version of Intel software development tools and has now been replaced by Intel oneAPI Toolkits. so like, historical icon behavior ๐ฐ๏ธ
Spack can install and use several software development products offered by Intel. some of these are available under no-cost terms, others require a paid license. ๐ธ all of them share the same basic steps for configuration, installation, and, where applicable, license management. the Spack Python class IntelPackage implements these steps.
Spack interacts with Intel tools in several routes, like it does for any other package:
- Accept system-provided tools after you declare them to Spack as external packages.
- Install the products for you as internal packages in Spack.
- Use the packages, regardless of installation route, to install what we'll call client packages for you, this being Spack's primary purpose.
an auxiliary route follows from route 2, as it would for most Spack packages, namely:
- Make Spack-installed Intel tools available outside of Spack for ad-hoc use, typically through Spack-managed modulefiles.
this document covers routes 1 through 3. ๐ซถ
๐ Packages under no-cost license ๐
Okay so this is actually the good-news part, bestie. Intel's standalone performance library products โ most famously MKL and MPI โ have been available under a simplified license since 2017 [fn1]. Slay. They're packaged in Spack as:
intel-mklโ Math Kernel Library (linear algebra and FFT) โจintel-mpiโ The Intel-MPI implementation (derived from MPICH)intel-ippโ Primitives for image-, signal-, and data-processingintel-daalโ Machine learning and data analytics
Some earlier versions of these libraries were released under a paid license though, so heads up: for those older versions, the license must be available at installation time of the products and during compilation of client packages. Like, both. No skipping.
Here's the cute part: the library packages work so well with the Intel compilers but they do not require them โ those packages can just as well be used with other compilers. The Intel compiler invocation commands offer custom options to simplify linking Intel libraries (sometimes considerably), but Spack always uses fairly explicit linkage anyway. iykyk ๐
๐ Licensed packages ๐
Now for the ones that want your credit card. Intel's core software development products that provide compilers, analyzers, and optimizers do require a paid license. In Spack, they're packaged as:
intel-parallel-studioโ the entire suite of compilers and librariesintelโ a subset containing just the compilers and the Intel-MPI runtime [fn2]
The license is needed at installation time and to compile client packages, but never to merely run any resulting binaries. (Iconic, honestly.) The license status for a given Spack package is normally specified in the package code through directives like license_required (see Licensed software). For the Intel packages, however, the class code provides these directives (in exchange of forfeiting a measure of OOP purity) and takes care of idiosyncasies like historic version dependence. Not me defending the OOP purity sacrifice ๐ญ
The libraries that are provided in the standalone packages are also included in the all-encompassing intel-parallel-studio. To complicate matters a bit, that package is sold in 3 "editions", of which only the upper-tier cluster edition supports compiling MPI applications, and hence only that edition can provide the mpi virtual package. (As mentioned [fn2], all editions provide support for running MPI applications.)
The edition forms the leading part of the version number for Spack's intel* packages discussed here. This differs from the primarily numeric version numbers seen with most other Spack packages. For example, we have:
$ spack info intel-parallel-studio
...
Preferred version:
professional.2018.3 http:...
Safe versions:
professional.2018.3 http:...
...
composer.2018.3 http:...
...
cluster.2018.3 http:...
...
...The full studio suite, capable of compiling MPI applications, currently requires about 12 GB of disk space when installed (see section Install steps for packages with compilers and libraries for detailed instructions). If you need to save disk space or installation time, you could install the intel compilers-only subset (0.6 GB) and just the library packages you need, for example intel-mpi (0.5 GB) and intel-mkl (2.5 GB). Your SSD will thank you ๐ซถ
๐ธ Unrelated packages ๐ธ
The following packages do not use the Intel installer and are not in class IntelPackage that is discussed here:
intel-gpu-toolsโ Test suite and low-level tools for the Linux Direct Rendering Managerintel-mkl-dnnโ Math Kernel Library for Deep Neural Networks (CMakePackage)intel-xedโ X86 machine instructions encoder/decoderintel-tbbโ Standalone version of Intel Threading Building Blocks. Note that a TBB runtime version is included withintel-mkl, and development versions are provided by the packagesintel-parallel-studio(all editions) and itsintelsubset.
๐ฆ Configuring Spack to use Intel licenses ๐ฆ
If you wish to integrate licensed Intel products into Spack as external packages (route 1 above) we assume that their license configuration is in place and is working [fn3]. In this case, skip to section Integration of Intel tools installed external to Spack. Bye bestie, see you down there ๐
If you plan to have Spack install licensed products for you (route 2 above), the Intel product installer that Spack will run underneath must have access to a license that is either provided by a license server or as a license file. The installer may be able to locate a license that is already configured on your system. If it cannot, you must configure Spack to provide either the server location or the license file.
For authoritative information on Intel licensing, see:
- https://software.intel.com/en-us/faq/licensing
- https://software.intel.com/en-us/articles/how-do-i-manage-my-licenses
๐ฅ Pointing to an existing license server ๐ฅ
Installing and configuring a license server is outside the scope of Spack. We assume that:
- Your system administrator has a license server running.
- The license server offers valid licenses for the Intel packages of interest.
- You can access these licenses under the user id running Spack.
Be aware of the difference between (a) installing and configuring a license server, and (b) configuring client software to use a server's so-called floating licenses. We are concerned here with (b) only. The process of obtaining a license from a server for temporary use is called "checking out a license". For that, a client application such as the Intel package installer or a compiler needs to know the host name and port number of one or more license servers that it may query [fn4].
Follow one of three methods to point client software to a floating license server. Ideally, your license administrator will already have implemented one that can be used unchanged in Spack: Look for the environment variable INTEL_LICENSE_FILE or for files /opt/intel/licenses/*.lic that contain:
SERVER hostname hostid_or_ANY portnum
USE_SERVERThe relevant tokens, among possibly others, are the USE_SERVER line, intended specifically for clients, and one or more SERVER lines above it which give the network address.
If you cannot find pre-existing /opt/intel/licenses/*.lic files and the INTEL_LICENSE_FILE environment variable is not set (even after you loaded any relevant modulefiles), ask your license administrator for the server address(es) and place them in a "global" license file within your Spack directory tree as shown below). ๐
๐ Installing a standalone license file ๐
If you purchased a user-specific license, follow Intel's instructions to "activate" it for your serial number, then download the resulting license file. If needed, request to have the file re-sent to you.
Intel's license files are text files that contain tokens in the proprietary "FLEXlm" format and whose name ends in .lic. Intel installers and compilers look for license files in several locations when they run. Place your license by one of the following means, in order of decreasing preference:
- Default directory
Install your license file in the directory /opt/intel/licenses/ if you have write permission to it. This directory is inspected by all Intel tools and is therefore preferred, as no further configuration will be needed. Create the directory if it does not yet exist. For the file name, either keep the downloaded name or use another suitably plain yet descriptive name that ends in .lic. Adjust file permissions for access by licensed users.
- Directory given in environment variable
If you cannot use the default directory, but your system already has set the environment variable INTEL_LICENSE_FILE independent from Spack [fn5], then, if you have the necessary write permissions, place your license file in one of the directories mentioned in this environment variable. Adjust file permissions to match licensed users.
TIP: If your system has not yet set and used the environment variableINTEL_LICENSE_FILE, you could start using it with thespack installstage of licensed tools and subsequent client packages. You would, however, be in a bind to always set that variable in the same manner, across updates and re-installations, and perhaps accommodate additions to it. As this may be difficult in the long run, we recommend that you do not attempt to start using the variable solely for Spack. ๐
- Spack-managed file
The first time Spack encounters an Intel package that requires a license, it will initialize a Spack-global Intel-specific license file for you, as a template with instructional comments, and bring up an editor [fn6]. Spack will do this even if you have a working license elsewhere on the system.
- To proceed with an externally configured license, leave the newly templated file as is (containing comments only) and close the editor. You do not need to touch the file again.
- To configure your own standalone license, copy the contents of your downloaded license file into the opened file, save it, and close the editor.
- To use a license server (i.e., a floating network license) that is not already configured elsewhere on the system, supply your license server address(es) in the form of
SERVERandUSE_SERVERlines at the beginning of the file [fn7], in the format shown in section Pointing to an existing license server. Save the file and close the editor.
To revisit and manually edit this file, such as prior to a subsequent installation attempt, find it at $SPACK_ROOT/etc/spack/licenses/intel/intel.lic .
Spack will place symbolic links to this file in each directory where licensed Intel binaries were installed. If you kept the template unchanged, Intel tools will simply ignore it. Lowkey elegant โจ
๐ Integration of Intel tools installed external to Spack ๐
This section discusses route 1 from the introduction.
A site that already uses Intel tools, especially licensed ones, will likely have some versions already installed on the system, especially at a time when Spack is just being introduced. It will be useful to make such previously installed tools available for use by Spack as they are. How to do this varies depending on the type of the tools:
โญ Integrating external compilers โญ
For Spack to use external Intel compilers, you must tell it both where to find them and when to use them. The present section documents the "where" aspect, involving compilers.yaml and, in most cases, long absolute paths. The "when" aspect actually relates to route 3 and requires explicitly stating the compiler as a spec component (in the form foo %intel or foo %intel@compilerversion) when installing client packages or altering Spack's compiler default in packages.yaml. See section Selecting Intel compilers for details. Two separate vibes, two separate jobs, fr fr ๐
okay so โ to pull a whole new set of already-installed-elsewhere Intel compilers into Spack, you go follow the Compiler configuration section. tl;dr: set up your shell env exactly like you would to use these compilers like a normal person (usually a module load ... or a shell source ... command), then hit it with spack compiler find so Spack knows they exist. That creates a shiny new entry in a suitably scoped (and possibly brand-new) compilers.yaml file. You could hand-write that entry yourself, but ngl it's error-prone โ it's giving indentation-sensitive, multiple-data-types chaos ๐.
The Intel compilers need and use your system's native GCC compiler (gcc on most systems, clang on macOS) for certain functionality โ notably to support C++. So if you want a different GCC compiler for the Intel tools, or more generally want to set persistent flags for every single invocation of the Intel compilers, go find the compilers.yaml entry that defines your Intel compiler and, with a text editor, change one or both of these:
- At the
modules:tag, add agccmodule to the list. - At the
flags:tag, addcflags:,cxxflags:, andfflags:key-value entries.
Consult the examples under Compiler configuration and Vendor-Specific Compiler Configuration in the Spack documentation. When you're done, validate your compiler definition by running spack compiler info intel@compilerversion (swap compilerversion for the version you actually defined). vibe check passed โ
Be aware that both the GCC integration and persistent compiler flags can also be affected by an advanced third method:
- A modulefile that provides the Intel compilers for you could, for the benefit of users outside of Spack, implicitly integrate a specific gcc version via compiler flag environment variables or (hopefully not ๐) via a sneaky extra
PATHaddition.
Next, visit section Selecting Intel Compilers to learn how to tell Spack to use the newly configured compilers.
๐ธ Integrating external libraries
Configure external library-type packages (as opposed to compilers) in the files $SPACK_ROOT/etc/spack/packages.yaml or ~/.spack/packages.yaml, following the Spack documentation under External Packages.
Similar to compilers.yaml, the packages.yaml files define a package external to Spack in terms of a Spack spec, and resolve each such spec via either the paths or modules tokens to a specific pre-installed package version on the system. Since Intel tools generally need environment variables to interoperate โ which cannot be conveyed in a mere paths specification โ the modules token will be more sensible to use. It resolves the Spack-side spec to a modulefile generated and managed outside of Spack's purview, which Spack will load internally and transiently when the corresponding spec is called upon to compile client packages. it's giving middleman energy, but make it useful โจ
Unlike for compilers, where spack find compilers [spec] generates an entry in an existing or new compilers.yaml file, Spack does not offer a command to generate an entirely new packages.yaml entry. You must create new entries yourself in a text editor, though the command spack config [--scope=...] edit packages can help with selecting the proper file. See section Configuration Scopes for an explanation about the different files and section Build customization for specifics and examples for packages.yaml files.
The following example integrates packages embodied by hypothetical external modulefiles intel-mkl/18/... into Spack as packages intel-mkl@...:
$ spack config edit packagesMake sure the file begins with:
packages:Adapt the following example. Be sure to maintain the indentation:
# other content ...
intel-mkl:
externals:
- spec: "intel-mkl@2018.2.199 arch=linux-centos6-x86_64"
modules:
- intel-mkl/18/18.0.2
- spec: "intel-mkl@2018.3.222 arch=linux-centos6-x86_64"
modules:
- intel-mkl/18/18.0.3The version numbers for the intel-mkl specs defined here correspond to file and directory names that Intel uses for its products, because they were adopted and declared as such within Spack's package repository. You can inspect the versions known to your current Spack installation by:
$ spack info intel-mklUsing the same version numbers for external packages as for packages known internally is useful for clarity, but not strictly necessary. Moreover, with a packages.yaml entry, you can go beyond internally known versions. glow-up, honestly ๐
Note that the Spack spec in the example does not contain a compiler specification. This is intentional, as the Intel library packages can be used unmodified with different compilers.
A slightly more advanced example illustrates how to provide variants and how to use the buildable: False directive to prevent Spack from installing other versions or variants of the named package through its normal internal mechanism.
packages:
intel-parallel-studio:
externals:
- spec: "intel-parallel-studio@cluster.2018.2.199 +mkl+mpi+ipp+tbb+daal arch=linux-centos6-x86_64"
modules:
- intel/18/18.0.2
- spec: "intel-parallel-studio@cluster.2018.3.222 +mkl+mpi+ipp+tbb+daal arch=linux-centos6-x86_64"
modules:
- intel/18/18.0.3
buildable: FalseOne additional example illustrates the use of prefix: instead of modules:, useful when external modulefiles are not available or not suitable:
packages:
intel-parallel-studio:
externals:
- spec: "intel-parallel-studio@cluster.2018.2.199 +mkl+mpi+ipp+tbb+daal"
prefix: /opt/intel
- spec: "intel-parallel-studio@cluster.2018.3.222 +mkl+mpi+ipp+tbb+daal"
prefix: /opt/intel
buildable: FalseNote that for the Intel packages discussed here, the directory values in the prefix: entries must be the high-level and typically version-less "installation directory" that has been used by Intel's product installer. Such a directory will typically accumulate various product versions. Amongst them, Spack will select the correct version-specific product directory based on the @version spec component that each path is being defined for.
For further background and details, see External Packages.
๐ง Installing Intel tools within Spack
This section discusses route 2 from the introduction.
When a system does not yet have Intel tools installed already, or the installed versions are undesirable, Spack can install these tools like any regular Spack package for you and, with appropriate pre- and post-install configuration, use its compilers and/or libraries to install client packages. we love a self-sufficient queen ๐
๐ Install steps for packages with compilers and libraries
The packages intel-parallel-studio and intel (which is a subset of the former) are many-in-one products that contain both compilers and a set of library packages whose scope depends on the edition. Because they are general products geared towards shell environments, it can be somewhat involved to integrate these packages at their full extent into Spack. buckle up bestie ๐ซถ
Note: To install library-only packages likeintel-mkl,intel-mpi, andintel-daalfollow the next section instead.
- Review the section Configuring spack to use intel licenses.
- To install a version of
intel-parallel-studiothat provides Intel compilers at a version that you have not yet declared in Spack, the following preparatory steps are recommended:
A. Determine the compiler spec that the new intel-parallel-studio package will provide, as follows: From the package version, combine the last two digits of the version year, a literal "0" (zero), and the version component that immediately follows the year.
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Package version โ Compiler spec provided โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโค
โ intel-parallel-studio@edition.YYyy.u โ intel@yy.0.u โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโExample: The package intel-parallel-studio@cluster.2018.3 will provide the compiler with spec intel@18.0.3. math ate, no crumbs ๐
B. Add a new compiler section with the newly anticipated version at the end of a compilers.yaml file in a suitable scope. For example, run:
$ spack config --scope=user/linux edit compilersand append a stub entry:
- compiler:
target: x86_64
operating_system: centos6
modules: []
spec: intel@18.0.3
paths:
cc: /usr/bin/true
cxx: /usr/bin/true
f77: /usr/bin/true
fc: /usr/bin/trueReplace 18.0.3 with the version that you determined in the preceding step. The exact contents under paths: do not matter yet, but the paths must exist.
This temporary stub is required such that the intel-parallel-studio package can be installed for the intel compiler (which the package itself is going to provide after the installation) rather than an arbitrary system compiler. The paths given in cc, cxx, f77, fc must exist, but will never be used to build anything during the installation of intel-parallel-studio.
The reason for this stub is that intel-parallel-studio also provides the mpi and mkl packages and when concretizing a spec, Spack ensures strong consistency of the used compiler across all dependencies: [fn8]. Installing a package foo +mkl %intel will make Spack look for a package mkl %intel, which can be provided by intel-parallel-studio+mkl %intel, but not by intel-parallel-studio+mkl %gcc.
Failure to do so may result in additional installations of mkl, intel-mpi or even intel-parallel-studio as dependencies for other packages. lowkey a whole cascade, avoid it ๐ญ
- Verify that the compiler version provided by the new studio version would be used as expected if you were to compile a client package:
$ spack spec zlib %intelIf the version does not match, explicitly state the anticipated compiler version, e.g.:
$ spack spec zlib %intel@18.0.3- Install the new studio package with Spack's regular install command. Lowkey a good idea to hand it the compiler you're anticipating (the one from up there โ๏ธ) as an explicit concretization element:
$ spack install intel-parallel-studio@cluster.2018.3 %intel@18.0.3- Do the same steps as under Integrating external compilers so Spack knows all the minutiae for actually using those compilers with client packages. If you dropped a stub entry into a
compilers.yamlfile, bestie, now is the moment โ go edit it and fill in the particulars. โจ- Under
paths:, give the full paths to the actual compiler binaries (icc,ifort, etc.) sitting inside the Spack installation tree, in all their unsightly length [fn9].
- Under
To figure out the full path to the C compiler, adapt and run:
$ find `spack location -i intel-parallel-studio@cluster.2018.3` \
-name icc -type f -lsIf you get hits for both intel64 and ia32, you almost certainly want the intel64 variant. The icpc and ifort compilers live in the same directory as icc.
- Make sure you specify
modules: ['intel-parallel-studio-cluster2018.3-intel-18.0.3-HASH'](withHASHbeing the short hash as displayed when you runspack find -l intel-parallel-studio@cluster.2018.3, and the versions adapted accordingly) so the correct and complete environment for the Intel compilers actually gets loaded when they run. With modern versions of the Intel compiler you may otherwise see issues about missing libraries. Also pls note: the module name must exactly match the name as returned bymodule avail(and shown in the example above). No typos, bestie ๐ - Use the
modules:and/orcflags:tokens to further specify a suitable accompanyinggccversion, which helps pacify picky client packages that ask for C++ standards more recent than what your system-providedgccand itslibstdc++.socan handle. - If you specified a custom variant (for example
+vtune), you may want to add it as your preferred variant in the packages configuration for theintel-parallel-studiopackage, as described in Package Preferences. Otherwise you'll have to specify the variant every single timeintel-parallel-studiois being used as anmkl,fftwormpiimplementation โ just to avoid pulling in a different variant. Exhausting, I know ๐ญ - To set the Intel compilers as the default in Spack instead of the usual
%gcc, follow the section Selecting Intel compilers.
TIP: ๐ซถ Compiler packages likeintel-parallel-studiocan easily be above 10 GB in size, which can tax the disk space available for temporary files on small, busy, or restricted systems (like virtual machines). The Intel installer will stop and report insufficient space as:==> './install.sh' '--silent' 'silent.cfg' ... Missing critical prerequisite -- Not enough disk spaceAs a first remedy, clean Spack's existing staging area:$ spack clean --stagethen retry installing the large package. Spack normally cleans staging directories, but certain failures may prevent it from doing so. If the error persists, tell Spack to use an alternative location for temporary files: 1. Rundf -hto identify an alternative location on your system. 2. Tell Spack to use that location for staging. Do one of the following: - Run Spack with the environment variableTMPDIRaltered for just a single command. For example, to use your$HOMEdirectory:$ TMPDIR="$HOME/spack-stage" spack install ....This example uses Bourne shell syntax. Adapt for other shells as needed. - Alternatively, customize Spack'sbuild_stageconfiguration setting.$ spack config edit configAppend:config: build_stage: - /home/$user/spack-stageDo not duplicate theconfig:line if it's already there. Adapt the location, which here is the same as in the preceding example. 3. Retry installing the large package.
Install steps for library-only packages ๐
To install library-only packages like intel-mkl, intel-mpi, and intel-daal, follow the steps right here. For packages that contain a compiler, follow the previous section instead, okay?
- For pre-2017 product releases, review the section Configuring Spack to use Intel licenses.
- Inspect the package spec. Specify an explicit compiler if necessary, e.g.:
$ spack spec intel-mpi@2018.3.199
$ spack spec intel-mpi@2018.3.199 %intelCheck that the package will use the compiler flavor and version you expect. Vibe check your spec, basically โจ
- Install the package normally within Spack. Use the same spec as in the previous command, i.e., as general or as specific as needed:
$ spack install intel-mpi@2018.3.199
$ spack install intel-mpi@2018.3.199 %intel@18- To prepare the new packages for use with client packages, follow Selecting libraries to satisfy virtual packages.
Debug notes ๐
- You can trigger a whole wall of additional diagnostics using Spack options, e.g.:
$ spack --debug -v install intel-mpiThe --debug option can also be useful while installing client packages (see below ๐) to confirm the integration of the Intel tools in Spack โ notably MKL and MPI.
- The
.spack/subdirectory of an installedIntelPackagewill contain, besides Spack's usual archival items, a copy of thesilent.cfgfile that was passed to the Intel installer:
$ grep COMPONENTS ...intel-mpi...<hash>/.spack/silent.cfg
COMPONENTS=ALL- If an installation error occurs, Spack will normally clean up and remove a partially installed target directory. You can direct Spack to keep it using
--keep-prefix, e.g.:
$ spack install --keep-prefix intel-mpiYou must, however, remove such partial installations prior to subsequent installation attempts. Otherwise the Intel installer will behave incorrectly. No cap, that part matters ๐
Using Intel tools in Spack to install client packages ๐
Finally, this section pertains to route 3 from the introduction.
Once Intel tools are installed within Spack as external or internal packages, they can be used as intended for installing client packages.
Selecting Intel compilers ๐
Select Intel compilers to compile client packages, like any compiler in Spack, by one of the following means:
- Request the Intel compilers explicitly in the client spec, e.g.:
$ spack install libxc@3.0.0%intel- Alternatively, request Intel compilers implicitly via package preferences. Configure the order of compilers in the appropriate
packages.yamlfile, under either anall:or client-package-specific entry, in acompiler:list. Consult the Spack documentation for Configuring Package Preferences and Package Preferences.
Example: etc/spack/packages.yaml might simply contain:
packages:
all:
compiler: [ intel, gcc, ]To be more specific, you can state partial or full compiler version numbers, for example:
packages:
all:
compiler: [ intel@18, intel@17, gcc@4.4.7, gcc@4.9.3, gcc@7.3.0, ]Selecting libraries to satisfy virtual packages ๐ฆ
Intel packages โ whether integrated into Spack as external packages or installed within Spack โ can be called upon to satisfy the requirement of a client package for a library that's available from different providers. The relevant virtual packages for Intel are blas, lapack, scalapack, and mpi.
In both integration routes, Intel packages can have optional variants which alter the list of virtual packages they can satisfy. For Spack-external packages, the active variants are a combination of the defaults declared in Spack's package repository and the spec it's declared as in packages.yaml. Needless to say, those should match the components that are actually present in the external product installation. Likewise, for Spack-internal packages, the active variants are determined, persistently at installation time, from the defaults in the repository and the spec selected to be installed.
To have Intel packages satisfy virtual package requests for all or selected client packages, edit the packages.yaml file. Customize, either in the all: or a more specific entry, a providers: dictionary whose keys are the virtual packages and whose values are the Spack specs that satisfy the virtual package, in order of decreasing preference. To learn more about the providers: settings, see the Spack tutorial for Configuring Package Preferences and the section Package Preferences.
Example: this following fairly minimal example for packages.yaml shows how to exclusively use the standalone intel-mkl package for all the linear algebra virtual packages in Spack, and intel-mpi as the preferred MPI implementation. Other providers can still be chosen on a per-package basis. Periodt. ๐
packages:
all:
providers:
mpi: [intel-mpi]
blas: [intel-mkl]
lapack: [intel-mkl]
scalapack: [intel-mkl]If you have access to the intel-parallel-studio@cluster edition, you can use instead:
all:
providers:
mpi: [intel-parallel-studio+mpi]
# Note: +mpi vs. +mkl
blas: [intel-parallel-studio+mkl]
lapack: [intel-parallel-studio+mkl]
scalapack: [intel-parallel-studio+mkl]If you installed intel-parallel-studio within Spack ("route 2"), make sure you followed the special installation step to ensure that its virtual packages match the compilers it provides. โญ
Using Intel tools as explicit dependency ๐
With the proper installation as detailed above, no special steps should be required when a client package specifically (and thus deliberately) requests an Intel package as a dependency โ this being one of the target use cases for Spack. Slay, honestly. ๐ซถ
Tips for configuring client packages to use MKL ๐
The Math Kernel Library (MKL) is provided by several Intel packages, currently intel-parallel-studio when variant +mkl is active (it is by default) and the standalone intel-mkl. Because of these different provider packages, a virtual mkl package is declared in Spack.
- To use MKL-specific APIs in a client package:
Declare a dependency on mkl, rather than a specific provider like intel-mkl. Declare the dependency either absolutely or conditionally based on variants that your package might have declared:
...and that's where mkl comes in, bestie โ here's how you actually use it ๐
# Examples for absolute and conditional dependencies:
depends_on("mkl")
depends_on("mkl", when="+mkl")
depends_on("mkl", when="fftw=mkl")The MKLROOT environment variable (part of the documented API, no cap) gets set during all stages of client package installation, and it's available to both the Spack packaging code and the client code. Both. Iconic.
- To use MKL as a provider for BLAS, LAPACK, or ScaLAPACK โ so the packages that provide
mklalso provide the narrower virtualblas,lapack, andscalapackpackages. Peep the relevant Packaging Guide section for the intro. To portably use these virtual packages, construct preprocessor and linker option strings in your package configuration code using the package functions.headersand.libs, together with utility functions from these classes:llnl.util.filesystem.FileList,llnl.util.filesystem.HeaderList,llnl.util.filesystem.LibraryList.
โ ๏ธ TIP: do not use constructs like.prefix.includeor.prefix.libwith Intel or any other implementation ofblas,lapack, andscalapack. Just don't, babe.
For example, for an AutotoolsPackage use .libs.ld_flags to transform the library file list into linker options passed to ./configure:
def configure_args(self):
args = []
...
args.append("--with-blas=%s" % self.spec["blas"].libs.ld_flags)
args.append("--with-lapack=%s" % self.spec["lapack"].libs.ld_flags)
...๐ธ TIP: even though.ld_flagsreturns a string of multiple words, do not put quotes around options like--with-blas=...โ because Spack passes them to./configurewithout invoking a shell. No shell, no quotes, no drama.
Likewise, in a MakefilePackage or a similar package that doesn't use AutoTools, you may need to hand over include and link options for use on command lines or in environment variables. For example, to generate an option string of the form -I<dir>, use:
self.spec["blas"].headers.include_flagsand to generate linker options (-L<dir> -llibname ...), use the same thing as above:
self.spec["blas"].libs.ld_flagsSee MakefilePackage and, more generally, the Packaging Guide for background and further examples. ๐
๐ Footnotes
[fn1]โ strictly speaking, versions from 2017.2 onward.[fn2]โ theintelpackage intentionally does not have a+mpivariant, since it's meant to be small. The native installer will always add MPI runtime components because it follows defaults defined in the download package โ even whenintel-parallel-studio ~mpihas been requested.
For intel-parallel-studio +mpi, the class function .IntelPackage.pset_components will include "intel-mpi intel-imb" in a list of component patterns passed to the Intel installer. The installer will extend each pattern word with an implied glob-like * to resolve it to package names that are actually present in the product BOM. As a side effect, this pattern approach accommodates occasional package name changes โ like capturing both intel-mpirt and intel-mpi-rt ๐
[fn3]โ how could the external installation have succeeded otherwise? ๐[fn4]โ according to Intel's documentation, there's supposedly a way to install a product using a network license even when a FLEXlm server isn't running: specify the license in the formport@serverhostin theINTEL_LICENSE_FILEenvironment variable. All other means of specifying a network license require that the license server be up. Them's the rules.[fn5]โ despite the name,INTEL_LICENSE_FILEcan hold several and diverse entries. They can be either directories (presumed to contain*.licfiles), file names, or network locations in the formport@host(on Linux and Mac), with all items separated by":"(on Linux and Mac).[fn6]โ should said editor turn out to bevi, you better be in a position to know how to use it ๐[fn7]โ comment lines in FLEXlm files, indicated by#as the first non-whitespace character on the line, are generally allowed anywhere in the file. There have been reports, however, that as of 2018,SERVERandUSE_SERVERlines must precede any comment lines. Gross, but noted.[fn8]โ Spack's close coupling of installed packages to compilers, which both necessitates the detour for installingintel-parallel-studioand largely limits any of its provided virtual packages to a single compiler, heavily favors recommending to install Intel Parallel Studio outside of Spack and declare it for Spack inpackages.yamlby a compiler-less spec.[fn9]โ with some effort, you can convince Spack to use shorter paths.
๐จ WARNING: altering the naming scheme means that Spack will lose track of all packages it has installed for you so far. That said, the time is right for this kind of customization when you're defining a new set of compilers.
The relevant tunables are:
- Set the
install_treelocation inconfig.yaml(see doc). - Set the hash length in
install-path-scheme, also inconfig.yaml(q.v.). - You'll want to set the same hash length for module files if you have Spack produce them for you, under
projectionsinmodules.yaml.
๐ง ROCm
The ROCmPackage is not a build system but a helper package. Like CudaPackage, it provides standard variants, dependencies, and conflicts to make building packages with GPUs easier โ except for AMD, in this case. ๐
You can find the source for this package (plus suggestions for setting up your compilers.yaml and packages.yaml files) at https://github.com/spack/spack/blob/develop/lib/spack/spack/build_systems/rocm.py.
โจ Variants
This package provides the following variants:
rocmโ this variant enables/disables building with rocm. Default is disabled (orFalse).amdgpu_targetโ this variant supports the optional specification of the AMD GPU architecture. Valid values are the names of the GPUs (e.g.,gfx701), which are maintained in theamdgpu_targetsproperty.
๐ Dependencies
This package defines basic rocm dependencies, including llvm and hip.
๐ฅ Conflicts
Conflicts are used to prevent builds with known bugs or issues. This package already requires that the amdgpu_target always be specified for rocm builds. It also defines a conflict that prevents builds with an amdgpu_target when rocm is disabled.
Refer to Conflicts for more info on package conflicts.
๐ ๏ธ Methods
This package provides one custom helper method, used to build standard AMD hip compiler flags.
hip_flagsโ this built-in static method returns the appropriately formatted--amdgpu-targetbuild option forhipcc.
This method must be explicitly called when you're creating the arguments for your build, in order to use the values.
๐ซ Usage
This helper package can be added to your package by adding it as a base class of your package. For example, you can add it to your CMakePackage-based package like this:
class MyRocmPackage(CMakePackage, ROCmPackage):
...
# Ensure +rocm and amdgpu_targets are passed to dependencies
depends_on("mydeppackage", when="+rocm")
for val in ROCmPackage.amdgpu_targets:
depends_on(f"mydeppackage amdgpu_target={val}",
when=f"amdgpu_target={val}")
...
def cmake_args(self):
spec = self.spec
args = []
...
if spec.satisfies("+rocm"):
# Set up the hip macros needed by the build
args.extend([
"-DENABLE_HIP=ON",
f"-DHIP_ROOT_DIR={spec['hip'].prefix}"])
rocm_archs = spec.variants["amdgpu_target"].value
if "none" not in rocm_archs:
args.append(f"-DHIP_HIPCC_FLAGS=--amdgpu-target={','.join(rocm_archs}")
else:
# Ensure build with hip is disabled
args.append("-DENABLE_HIP=OFF")
...
return args
......assuming only the ENABLE_HIP, HIP_ROOT_DIR, and HIP_HIPCC_FLAGS macros are required to be set, and the only dependency needing rocm options is mydeppackage. You'll need to customize the flags as needed for your build. ๐ซถ
This example also shows how to check for the rocm variant using self.spec, and how to retrieve the amdgpu_target variant's value using self.spec.variants["amdgpu_target"].value.
All five packages using ROCmPackage as of January 2021 also use the CudaPackage. So it's genuinely worth peeking at those packages to get ideas for making a package that supports both cuda and rocm. Bestie behaviour tbh.
๐ฆ Sourceforge
SourceforgePackage is a mixin-class. It automatically sets the URL based on a list of Sourceforge mirrors listed in sourceforge_mirror_path, which defaults to a half dozen known mirrors. Refer to the package source (https://github.com/spack/spack/blob/develop/lib/spack/spack/build_systems/sourceforge.py) for the current list of mirrors Spack uses.
๐ช Methods
This package provides a method for populating mirror URLs.
urlsโ this method returns a list of possible URLs for package source. It's decorated withproperty, so its results are treated as a package attribute.
Refer to https://spack.readthedocs.io/en/latest/packaging_guide.html#mirrors-of-the-main-url for info on how Spack uses the urls attribute during fetching.
๐ Usage
This helper package can be added to your package by adding it as a base class of your package, then defining the relative location of an archive file for one version of your software.
class MyPackage(AutotoolsPackage, SourceforgePackage):
...
sourceforge_mirror_path = "my-package/mypackage.1.0.0.tar.gz"
...Over 40 packages are using SourceforcePackage this mix-in as of July 2022, so there are multiple packages to choose from if you want a real example. Go shop around, it's giving reference library โจ
For reference, the Build System API docs give you a list of build systems and methods/attributes that can be overridden. Curious about the implementation of a particular build system? You can view the source code by running:
$ spack edit --build-system autotoolsThat'll open up the AutotoolsPackage definition in your favorite editor. And if you're working with a less common build system like QMake, SCons, or Waf, it may be useful to see examples from other packages. You can find examples quick-quick by running:
$ cd var/spack/repos/builtin/packages
$ grep -l QMakePackage */package.pyYou can then view these packages with spack edit โ go peek, bestie, it's all right there ๐
This guide is lowkey designed to be the sidekick to the Build System API docs โ it's got examples of how to override the methods everyone actually overrides, plus rules of thumb and lil suggestions for package devs who've never touched a particular build system before. No gatekeeping here ๐ซถ
๐ DEVELOPER GUIDE ๐
Okay so this guide is for the people who wanna work on Spack itself. If you're just here to write packages, skrrt over to the Packaging Guide instead, that's your lane ๐ธ
And it's assumed you've already read the Basic Usage and Packaging Guide sections and that you're comfy with the concepts in them. Not comfy yet? Genuinely, go read those first, it'll save you so much confusion, no cap.
โจ Overview โจ
Spack is designed around three separate roles, and honestly the way they split it up is iconic:
- Users, who need to install software without knowing all the details about how it's built.
- Packagers who know how a particular software package is built and encode that info in package files.
- Developers who work on Spack itself, add new features, and try to make the jobs of packagers and users easier.
A "user" could be a person installing software in their home directory, or an admin installing software to a shared directory on a shared machine. A "packager" could be an admin who wants to automate software builds, or an app developer who wants to make their software more accessible to users. It's giving range ๐
So as you'd expect, there are many types of users with different levels of sophistication, and Spack is built to handle both the simple and the complex use cases for packages. Someone who only knows they need a certain package should be able to type something simple like spack install <package name> and get the package they want. And if a user wants a specific version, particular compilers, or several builds with different configurations? Also possible, with a minimal amount of extra spec โจ
Which brings us to the two key concepts in Spack's software design:
- Specs: expressions for describing builds of software, and
- Packages: Python modules that build software according to a spec.
A package is a template for building particular software, and a spec is a descriptor for one or more instances of that template. Users express the config they want using a spec, and a package turns that spec into a complete build. Chef's kiss ๐ค
The obvious difficulty with this design โ and yeah, we're gonna say it โ is that users under-specify what they want. To build a software package, the package object needs a complete specification. So in Spack, if a spec describes only one instance of a package, we say it's concrete. If a spec could describe many instances (i.e. it's under-specified in one way or another), we say it's abstract.
Spack's whole job is to take an abstract spec from the user, find a concrete spec that satisfies the constraints, and hand the task of building the software off to the package object. The rest of this doc is basically the story of all the pieces that come together to make that happen ๐ฌ
๐ Directory Structure ๐
So you can get familiar with the project, let's start with a high-level view of Spack's directory structure. Behold:
spack/ <- installation root
bin/
spack <- main spack executable
etc/
spack/ <- Spack config files.
Can be overridden by files in ~/.spack.
var/
spack/ <- build & stage directories
repos/ <- contains package repositories
builtin/ <- pkg repository that comes with Spack
repo.yaml <- descriptor for the builtin repository
packages/ <- directories under here contain packages
cache/ <- saves resources downloaded during installs
opt/
spack/ <- packages are installed here
lib/
spack/
docs/ <- source for this documentation
env/ <- compiler wrappers for build environment
external/ <- external libs included in Spack distro
llnl/ <- some general-use libraries
spack/ <- spack module; contains Python code
build_systems/ <- modules for different build systems
cmd/ <- each file in here is a spack subcommand
compilers/ <- compiler description files
container/ <- module for spack containerize
hooks/ <- hook modules to run at different points
modules/ <- modules for lmod, tcl, etc.
operating_systems/ <- operating system modules
platforms/ <- different spack platforms
reporters/ <- reporters like cdash, junit
schema/ <- schemas to validate data structures
solver/ <- the spack solver
test/ <- unit test modules
util/ <- common codeSpack is designed so it could live inside a standard UNIX directory hierarchy, so lib, var, and opt all contain a spack subdirectory โ you know, in case Spack is installed alongside other software ๐ฏ Most of the interesting parts of Spack live in lib/spack.
Spack has one directory layout and there is no install process. Wild, right? Most Python programs don't look like this (they're out here using distutils, setup.py, etc.) but the goal was to make Spack very easy to use. The simple layout spares users from needing to install Spack into a Python environment. So many users don't have write access to a Python installation, and installing an entire new instance of Python just to bootstrap Spack would be so complicated ๐ญ Users should not have to install a big, complicated package to use the thing that's supposed to spare them from the details of big, complicated packages. Periodt. The end result is that Spack works out of the box: clone it, add bin to your PATH, and you're ready to go ๐
๐ง Code Structure ๐ง
This section gives an overview of the various Python modules in Spack, grouped by functionality. Organized? We love to see it ๐
๐ฆ Package-related modules ๐ฆ
spack.package_base
Contains the PackageBase class, which is the superclass for all packages in Spack. The parent of everyone, the main character fr fr ๐
spack.util.naming
Contains functions for mapping between Spack package names, Python module names, and Python class names. Functions like mod_to_class() handle mapping package module names to class names.
spack.directives
Directives are functions that can be called inside a package definition to modify the package, like depends_on() and provides(). See Dependencies and Virtual dependencies.
spack.multimethod
Implementation of the @when decorator, which allows multimethods in packages. Fancy โจ
๐ฆ Spec-related modules ๐ฆ
spack.spec
Contains Spec. Also implements most of the logic for concretization of specs.
spack.parser
Contains SpecParser and functions related to parsing specs.
spack.concretize
Contains Concretizer implementation, which allows site administrators to change Spack's Concretization Policies.
spack.version
Implements a simple Version class with simple comparison semantics. Also implements VersionRange and VersionList. All three are comparable with each other and offer union and intersection operations. Spack uses these classes to compare versions and to manage version constraints on specs. Comparison semantics are similar to the LooseVersion class in distutils and to the way RPM compares version strings. Ngl, this one's kind of a baddie ๐
spack.compilers
Submodules contains descriptors for all valid compilers in Spack. This is used by the build system to set up the build environment.
โ ๏ธ WARNING: Not yet implemented. Currently has two compiler descriptions, but compilers aren't fully integrated with the build process yet. Yeah, it's giving "work in progress" ๐ don't come for us.
๐ Build environment ๐
spack.stage
Handles creating temporary directories for builds.
spack.build_environment
This contains utility functions used by the compiler wrapper script, cc.
spack.directory_layout
Classes that control the way an installation directory is laid out. Create more implementations of this to change the hierarchy and naming scheme in $spack_prefix/opt ๐๏ธ
โก Spack Subcommands โก
spack.cmd
Each module in this package implements a Spack subcommand. See writing commands for details.
๐งช Unit tests ๐งช
spack.test
Implements Spack's test suite. Add a module and put its name in the test suite in __init__.py to add more unit tests. Test your stuff bestie, we're begging ๐
๐ธ Other Modules ๐ธ
spack.url
URL parsing, for deducing names and versions of packages from tarball URLs.
spack.error
SpackError, the base class for Spack's exception hierarchy.
llnl.util.tty
Basic output functions for all of the messages Spack writes to the terminal.
llnl.util.tty.color
Implements a color formatting syntax used by spack.tty. (Colour!! We love colour ๐)
llnl.util
In this package are a number of utility modules for the rest of Spack.
๐ Spec objects ๐
๐ Package objects ๐
Most spack commands look something like this:
- Parse an abstract spec (or specs) from the command line,
- Normalize the spec based on information in package files,
- Concretize the spec according to some customizable policies,
- Instantiate a package based on the spec, and
- Call methods (e.g.,
install()) on the package object.
The information in Package files is used at all stages in this process. Every single stage, it's that serious ๐ฅ
โ๏ธ Writing commands โ๏ธ
Adding a new command to Spack is easy. Simply add a <name>.py file to lib/spack/spack/cmd/, where <name> is the name of the subcommand. At the bare minimum, two functions are required in this file:
๐ ๏ธ setup_parser() ๐ ๏ธ
Unless your command doesn't accept any arguments, a setup_parser() function is required to define what arguments and flags your command takes. See the Argparse documentation for more details on how to add arguments.
Some commands have a set of subcommands, like spack compiler find or spack module lmod refresh. You can add subparsers to your parser to handle this. Check out spack edit --command compiler for an example of this. See, it's all in there ๐
A lot of commands take the same arguments and flags. These arguments should be defined in lib/spack/spack/cmd/common/arguments.py so that they don't need to be redefined in multiple commands. DRY, bestie, DRY ๐ง
๐ง <name>()
ok so for Spack to actually run your command, it goes looking for a function with the exact same name as your command inside <name>.py. that's your command's main character function, and it can totally call other helper methods to handle the common tasks. ๐
real talk bestie: before you add a whole new command, sit with yourself for a sec and ask if it's actually necessary. sometimes the thing you want can just be added to an existing command. also โ add unit tests for it, fr fr. if it isn't used very frequently, changes to the rest of Spack can break your command and nobody will even notice, because there weren't enough unit tests to stop it from happening. ๐ญ
and whenever you add, remove, or rename a command โ or the flags on an existing command โ make sure you update Spack's Bash tab completion script too. don't leave it hanging. ๐ซถ
๐ช Writing Hooks
a hook is a callback, and it makes it easy to design functions that run for different events. the way we do it: define hook types, then insert them at different places in the spack code base. whenever a hook type triggers via a function call, we find all the hooks of that type and run them.
Spack defines hooks by way of a module at lib/spack/spack/hooks, where we can define types of hooks in the __init__.py, and then python files in that folder can use hook functions. those files get parsed automatically, so if you write a new file for some integration โ like lib/spack/spack/hooks/myintegration.py โ you can write hook functions in that file and they'll be automatically detected and run whenever your hook gets called. iconic. this section covers the basic kind of hooks and how to write them. โจ
๐ Types of Hooks
the following hooks are currently implemented, so you โ the developer โ can drop hooks in at different stages of a spack install or similar. if there's a hook you want that's missing, you can propose adding a new one. ๐
pre_install(spec)
a pre_install hook runs within the install subprocess, directly before the install starts. it expects a single argument of a spec.
post_install(spec, explicit=None)
a post_install hook runs within the install subprocess, directly after the install finishes โ but before the build stage is removed and before the spec is registered in the database. it expects two arguments: spec, and an optional boolean indicating whether this spec is being installed explicitly.
pre_uninstall(spec) and post_uninstall(spec)
these hooks are currently used for cleaning up module files after uninstall.
โจ Adding a New Hook Type
adding a new hook type is literally so simple! in lib/spack/spack/hooks/__init__.py you just create a new HookRunner named to match your new hook. for example, let's say you want to add a new hook called post_log_write to trigger after anything is written to a logger. you'd add it like this:
# pre/post install and run by the install subprocess
pre_install = HookRunner('pre_install')
post_install = HookRunner('post_install')
# hooks related to logging
post_log_write = HookRunner('post_log_write') # <- here is my new hook!then you decide what arguments your hook would expect. since this one's about logging, let's say you want a message and a level. that means you add a python file to the lib/spack/spack/hooks folder with one or more callbacks intended to be triggered by this hook. you might use your new hook like this:
def post_log_write(message, level):
"""Do something custom with the message and level every time we write
to the log
"""
print('running post_log_write!')to use the hook, we'd call it something like this somewhere in the logic that does logging. in this example we use it outside of a logger that's already defined:
import spack.hooks
# We do something here to generate a logger and message
spack.hooks.post_log_write(message, logger.level)now, this is not me saying this would be the best way to implement an integration with the logger โ you'd probably want to write a custom logger, or you could have the hook defined within the logger โ but it serves as an example of writing a hook. ๐ฆ
๐งช Unit tests
Unit testing
๐ป Developer environment
โ ๏ธ WARNING: This is an experimental feature. It is expected to change and you should not use it in a production environment.
when installing a package, we currently have support to export environment variables that specify adding debug flags to the build. by default, a package install will build without any debug flag. but if you want to add them, you can export:
export SPACK_ADD_DEBUG_FLAGS=true
spack install zlibwant custom flags? export an additional variable too:
export SPACK_ADD_DEBUG_FLAGS=true
export SPACK_DEBUG_FLAGS="-g"
spack install zlibthese environment variables will eventually be integrated into spack so they're set from the command line. ๐
๐ Developer commands
๐ spack doc
(this one's just listed here as a developer command โ nothing extra attached to it in this section โจ)
spack style
spack style exists to help you, the developer user, check imports and style with mypy, flake8, isort, and (soon) black. to run all the style checks, simply do:
$ spack styleto run automatic fixes for isort, you can do:
$ spack style --fixyou do not need any of these Python packages installed on your system for the checks to work! Spack will bootstrap install them from packages for you. ๐
spack unit-test
see the contributor guide section on spack unit-test.
spack python
spack python is a command that lets you import and debug things as if you were in a Spack interactive shell. with no arguments, it's similar to a normal interactive Python shell โ except you can import spack and any other Spack modules:
$ spack python
Spack version 0.10.0
Python 2.7.13, Linux x86_64
>>> from spack.version import Version
>>> a = Version('1.2.3')
>>> b = Version('1_2_3')
>>> a == b
True
>>> c = Version('1.2.3b')
>>> c > a
True
>>>prefer using an IPython interpreter? given that IPython is installed, you can specify the interpreter with -i:
$ spack python -i ipython
Python 3.8.3 (default, May 19 2020, 18:47:26)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.17.0 -- An enhanced Interactive Python. Type '?' for help.
Spack version 0.16.0
Python 3.8.3, Linux x86_64
In [1]:with either interpreter you can run a single command:
$ spack python -c 'from spack.spec import Spec; Spec("python").concretized()'
...
$ spack python -i ipython -c 'from spack.spec import Spec; Spec("python").concretized()'
Out[1]: ...or a file:
$ spack python ~/test_fetching.py
$ spack python -i ipython ~/test_fetching.pyjust like you would with the normal python command. ๐๐
spack blame
spack blame is a way to quickly see contributors to packages or files in the spack repository. you should provide a target package name or file name to the command. here's an example asking to see contributions for the package python:
$ spack blame python
LAST_COMMIT LINES % AUTHOR EMAIL
2 weeks ago 3 0.3 Mickey Mouse <cheddar@gmouse.org>
a month ago 927 99.7 Minnie Mouse <swiss@mouse.org>
2 weeks ago 930 100.0by default you get a table view (shown above) sorted by date of contribution, with the most recent contribution at the top. if you want to sort by percentage of code contribution instead, add -p:
$ spack blame -p pythonand to see the git blame view, add -g instead:
$ spack blame -g pythonfinally, to get a json export of the data, add --json:
$ spack blame --json pythonspack url
a package containing a single URL can be used to download several different versions of the package. ever wondered how that works? all of the magic is in spack.url. this module contains methods for extracting the name and version of a package from its URL. the name is what spack create uses to guess the name of the package. and by determining the version from the URL, Spack can replace it with other versions to figure out where to download them from.
the regular expressions in parse_name_offset and parse_version_offset are what extract the name and version โ but they aren't perfect. in order to debug Spack's URL parsing support, the spack url command exists. ๐ต๏ธโโ๏ธ
spack url parse
if you need to debug a single URL, you can use this:
$ spack url parse http://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.0.tar.gz
==> Parsing URL: http://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.0.tar.gz
==> Matched version regex 0: r'^[a-zA-Z+._-]+[._-]v?(\\d[\\d._-]*)$'
==> Matched name regex 10: r'^([A-Za-z\\d+\\._-]+)$'
==> Detected:
http://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.0.tar.gz
---- ~~~~~
name: ruby
version: 2.2.0
==> Substituting version 9.9.9b:
http://cache.ruby-lang.org/pub/ruby/2.2/ruby-9.9.9b.tar.gz
---- ~~~~~~you'll notice the name and version of this URL are correctly detected, and you can even see which regular expressions it was matched to. however โ notice that when it substitutes the version number in, it doesn't replace the 2.2 with 9.9 where we'd expect 9.9.9b to live. this particular package may require a list_url or url_for_version function. ๐
this command also accepts a --spider flag. if provided, Spack searches for other versions of the package and prints the matching URLs.
spack url list
this command lists every URL in every package in Spack. if given the --color and --extrapolation flags, it also colors the part of the string that it detected to be the name and version. the --incorrect-name and --incorrect-version flags can be used to print URLs that were not being parsed correctly. ๐จ
spack url summary
this command attempts to parse every URL for every package in Spack and prints a summary of how many of them are being correctly parsed. it also prints a histogram showing which regular expressions are being matched and how frequently:
$ spack url summary
==> Generating a summary of URL parsing in Spack...
Total URLs found: 7898
Names correctly parsed: 6866/7898 (86.93%)
Versions correctly parsed: 6944/7898 (87.92%)
==> Statistics on name regular expressions:...and here's the next scoreboard, bestie โ this one's scoring the regexes that pull names out of URLs ๐
Index Right Wrong Total Regular Expression
0 1842 365 2207 r'github\\.com/[^/]+/([^/]+)'
1 6 0 6 r'gitlab[^/]+/api/v4/projects/[^/]+%2F([^/]+)'
2 65 27 92 r'gitlab[^/]+/(?!api/v4/projects)[^/]+/([^/]+)'
3 16 6 22 r'bitbucket\\.org/[^/]+/([^/]+)'
4 4 0 4 r'pypi\\.(?:python\\.org|io)/packages/source/[A-Za-z\\d]/([^/]+)'
6 13 1 14 r'\\?f=([A-Za-z\\d+-]+)$'
7 21 0 21 r'\\?package=([A-Za-z\\d+-]+)'
9 2 1 3 r'([^/]+)/download.php$'
10 4897 602 5499 r'^([A-Za-z\\d+\\._-]+)$'and the version regex scoreboard, which is giving main character energy โจ:
==> Statistics on version regular expressions:
Index Right Wrong Total Regular Expression
0 4956 197 5153 r'^[a-zA-Z+._-]+[._-]v?(\\d[\\d._-]*)$'
1 1425 67 1492 r'^v?(\\d[\\d._-]*)$'
2 14 38 52 r'^[a-zA-Z+]*(\\d[\\da-zA-Z]*)$'
3 10 17 27 r'^[a-zA-Z+-]*(\\d[\\da-zA-Z-]*)$'
4 7 128 135 r'^[a-zA-Z+_]*(\\d[\\da-zA-Z_]*)$'
5 61 28 89 r'^[a-zA-Z+.]*(\\d[\\da-zA-Z.]*)$'
6 277 10 287 r'^[a-zA-Z\\d+-]+-v?(\\d[\\da-zA-Z.]*)$'
7 1 0 1 r'^[a-zA-Z\\d+-]+-v?(\\d[\\da-zA-Z_]*)$'
8 34 1 35 r'^[a-zA-Z\\d+_]+_v?(\\d[\\da-zA-Z.]*)$'
9 0 2 2 r'^[a-zA-Z\\d+_]+\\.v?(\\d[\\da-zA-Z.]*)$'
10 0 1 1 r'^[a-zA-Z\\d+]+_r?(\\d[\\da-zA-Z-]*)$'
11 31 81 112 r'^(?:[a-zA-Z\\d+-]+-)?v?(\\d[\\da-zA-Z.-]*)$'
12 3 0 3 r'^[a-zA-Z+]+v?(\\d[\\da-zA-Z.-]*)$'
13 11 4 15 r'^[a-zA-Z\\d+_]+-v?(\\d[\\da-zA-Z.]*)$'
14 28 7 35 r'^[a-zA-Z\\d+.]+_v?(\\d[\\da-zA-Z.-]*)$'
15 1 0 1 r'^[a-zA-Z\\d+-]+-v?(\\d[\\da-zA-Z._]*)$'
16 3 1 4 r'^[a-zA-Z\\d+._]+-v?(\\d[\\da-zA-Z.]*)$'
17 5 1 6 r'^[a-zA-Z+-]+(\\d[\\da-zA-Z._]*)$'
18 1 2 3 r'^[a-zA-Z\\d+_-]+-v?(\\d[\\da-zA-Z.]*)$'
19 0 1 1 r'bzr(\\d[\\da-zA-Z._-]*)$'
20 10 0 10 r'[?&](?:sha|ref|version)=[a-zA-Z\\d+-]*[_-]?v?(\\d[\\da-zA-Z._-]*)$'
21 35 0 35 r'[?&](?:filename|f|get)=[a-zA-Z\\d+-]+[_-]v?(\\d[\\da-zA-Z.]*)'
22 14 1 15 r'github\\.com/[^/]+/[^/]+/releases/download/[a-zA-Z+._-]*v?(\\d[\\da-zA-Z._-]*)/'
23 17 254 271 r'(\\d[\\da-zA-Z._-]*)/[^/]+$'This command is literally essential for anyone adding or changing the regular expressions that parse names and versions. Run it before and after your change and you can see with your own eyes that your regex fixes more packages than it breaks โ no cap, that's the whole vibe check ๐
๐ฅ Profiling ๐ฅ
Spack has some limited built-in support for profiling, and can report statistics using standard Python timing tools. To use this feature, supply --profile to Spack on the command line, before any subcommands โ like, before everything, bestie, it's gotta be up front ๐ซถ
spack --profile
spack --profile output looks like this:
$ spack --profile graph hdf5 os=SUSE target=x86_64
...The bottom of the output shows the top most time consuming functions, slowest on top. The profiling support is from Python's built-in tool, cProfile. Slay, built-in tooling ๐โจ
๐ Releases ๐
This section documents Spack's release process. It is intended for project maintainers, as the tasks described here require maintainer privileges on the Spack repository. For others, we hope this section at least provides some insight into how the Spack project works. Peek behind the curtain, bestie ๐ฆ
Release branches
There are currently two types of Spack releases: major releases (0.17.0, 0.18.0, etc.) and point releases (0.17.1, 0.17.2, 0.17.3, etc.). Here is a diagram of how Spack release branches work:
o branch: develop (latest version, v0.19.0.dev0)
|
o
| o branch: releases/v0.18, tag: v0.18.1
o |
| o tag: v0.18.0
o |
| o
|/
o
|
o
| o branch: releases/v0.17, tag: v0.17.2
o |
| o tag: v0.17.1
o |
| o tag: v0.17.0
o |
| o
|/
oThe develop branch has the latest contributions, and nearly all pull requests target develop. The develop branch will report that its version is that of the next major release with a .dev0 suffix.
Each Spack release series also has a corresponding branch, e.g. releases/v0.18 has 0.18.x versions of Spack, and releases/v0.17 has 0.17.x versions. A major release is the first tagged version on a release branch. Minor releases are back-ported from develop onto release branches. This is typically done by cherry-picking bugfix commits off of develop.
To avoid version churn for users of a release series, minor releases should not make changes that would change the concretization of packages. They should generally only contain fixes to the Spack core. However, sometimes priorities are such that new functionality needs to be added to a minor release. It happens ๐
Both major and minor releases are tagged. As a convenience, we also tag the latest release as releases/latest, so that users can easily check it out to get the latest stable version. See Updating releases/latest for more details.
NOTE: Older spack releases were merged back into develop so that we could do fancy things with tags, but since tarballs and many git checkouts do not have tags, this proved overly complex and confusing. We have since converted to using PEP 440 compliant versions. See here for details. ๐ (the lore is deep)
๐๏ธ Scheduling work for releases ๐๏ธ
We schedule work for releases by creating GitHub projects. At any time, there may be several open release projects. For example, below are two releases (from some past version of the page linked above): [image]
This image shows one release in progress for 0.15.1 and another for 0.16.0. Each of these releases has a project board containing issues and pull requests. GitHub shows a status bar with completed work in green, work in progress in purple, and work not started yet in gray, so it's fairly easy to see progress. Color-coded progress, we love to see it ๐
Spack's project boards are not firm commitments, so we move work between releases frequently. If we need to make a release and some tasks are not yet done, we will simply move them to the next minor or major release, rather than delaying the release to complete them. Flexible queen behavior ๐
For more on using GitHub project boards, see GitHub's documentation.
๐ Making major releases ๐
Assuming a project board has already been created and all required work completed, the steps to make the major release are:
- Create two new project boards:
- One for the next major release
- One for the next point release
- Move any optional tasks that are not done to one of the new project boards.
In general, small bugfixes should go to the next point release. Major features, refactors, and changes that could affect concretization should go in the next major release.
- Create a branch for the release, based on
develop:
$ git checkout -b releases/v0.15 developFor a version vX.Y.Z, the branch's name should be releases/vX.Y. That is, you should create a releases/vX.Y branch if you are preparing the X.Y.0 release.
- Remove the
dev0development release segment from the version tuple inlib/spack/spack/__init__.py.
The version number itself should already be correct and should not be modified.
- Update
CHANGELOG.mdwith major highlights in bullet form.
Use proper markdown formatting, like this example from 0.15.0.
- Push the release branch to GitHub.
- Make sure CI passes on the release branch, including:
- Regular unit tests
- Build tests
- The E4S pipeline at
gitlab.spack.io
If CI is not passing, submit pull requests to develop as normal and keep rebasing the release branch on develop until CI passes.
- Make sure the entire documentation is up to date. If documentation is outdated submit pull requests to
developas normal and keep rebasing the release branch ondevelop. ๐ - Bump the major version in the
developbranch.
Create a pull request targeting the develop branch, bumping the major version in lib/spack/spack/__init__.py with a dev0 release segment. For instance when you have just released v0.15.0, set the version to (0, 16, 0, 'dev0') on develop.
- Follow the steps in Publishing a release on GitHub.
- Follow the steps in Updating releases/latest.
- Follow the steps in Announcing a release.
๐ Making point releases ๐
Assuming a project board has already been created and all required work completed, the steps to make the point release are:
- Create a new project board for the next point release.
- Move any optional tasks that are not done to the next project board.
- Check out the release branch (it should already exist).
For the X.Y.Z release, the release branch is called releases/vX.Y. For v0.15.1, you would check out releases/v0.15:
$ git checkout releases/v0.15- If a pull request to the release branch named Backports vX.Y.Z is not already in the project, create it. This pull request ought to be created as early as possible when working on a release project, so that we can build the release commits incrementally, and identify potential conflicts at an early stage. Early vibes only ๐ธ
- Cherry-pick each pull request in the Done column of the release project board onto the Backports vX.Y.Z pull request.
This is usually fairly simple since we squash the commits from the vast majority of pull requests. That means there is only one commit per pull request to cherry-pick. For example, this pull request has three commits, but they were squashed into a single commit on merge. You can see the commit that was created here: [image]
You can easily cherry pick it like this (assuming you already have the release branch checked out):
$ git cherry-pick 7e46da7For pull requests that were rebased (or not squashed), you'll need to cherry-pick each associated commit individually.
WARNING: It is important to cherry-pick commits in the order they happened, otherwise you can get conflicts while cherry-picking. When cherry-picking look at the merge date, not the number of the pull request or the date it was opened. ๐จ Order matters, bestie โ do not freestyle this one.
Sometimes you'll still get merge conflicts even after you cherry-picked all the commits in order, and honestly? that's lowkey expected ๐ญ. What it usually means is that some other pull request is sitting in between โ the one you're trying to pick depends on it. When that happens you gotta make a judgment call about those pull requests. Weigh up how many files are affected and or what the resulting differences look like.
- If the dependency changes are small, you might just cherry-pick it too. If you do that, add the task to the release board.
- If the changes are large, then you may decide this fix is not worth including in a point release โ and in that case you should remove the task from the release project.
- You can always decide to manually back-port the fix to the release branch if neither of the above options makes sense, but that can require a lot of work. It's seldom the right choice.
- When all the commits from the project board are cherry-picked into the
Backports vX.Y.Zpull request, you can push a commit to:- Bump the version in
lib/spack/spack/__init__.py. - Update
CHANGELOG.mdwith a list of the changes.
- Bump the version in
Typically this is a summary of the commits you cherry-picked onto the release branch. Go look at the changelog from 0.14.1 for the vibe.
- Merge the
Backports vX.Y.ZPR with the Rebase and merge strategy. This one matters โ it's what keeps track, in the release branch, of all the commits that were cherry-picked. ๐ซถ - Make sure CI passes on the release branch, including:
- Regular unit tests
- Build tests
- The E4S pipeline at
gitlab.spack.io
If CI does not pass, you'll need to figure out why and make changes to the release branch until it does. You can make more commits, modify or remove cherry-picked commits, or cherry-pick more from develop to make it happen. Whatever works ๐
- Follow the steps in Publishing a release on GitHub.
- Follow the steps in Updating releases/latest.
- Follow the steps in Announcing a release.
- Submit a PR to update the
CHANGELOGin thedevelopbranch with the addition of this point release.
Publishing a release on GitHub
- Create the release in GitHub.
- Go to
github.com/spack/spack/releasesand click Draft a new release. - Set Tag version to the name of the tag that will be created.
- Go to
The name should start with v and contain all three parts of the version (e.g. v0.15.0 or v0.15.1).
- Set Target to the
releases/vX.Ybranch (e.g.,releases/v0.15). - Set Release title to
vX.Y.Zto match the tag (e.g.,v0.15.1). - Paste the latest release markdown from your
CHANGELOG.mdfile as the text. - Save the draft so you can keep coming back to it while you prepare the release.
- When you're ready to finalize the release, click Publish release.
- Immediately after publishing, go back to
github.com/spack/spack/releasesand download the auto-generated.tar.gzfile for the release. It's the Source code (tar.gz) link. - Click Edit on the release you just made and attach the downloaded release tarball as a binary. This does two things:
- Makes sure the hash of our releases does not change over time.
GitHub sometimes annoyingly changes the way they generate tarballs, which can result in the hashes changing if you rely on the auto-generated tarball links. ๐
- Gets download counts on releases visible through the GitHub API.
GitHub tracks downloads of artifacts, but not the source links. See the releases page and search for download_count to see this.
- Go to
readthedocs.organd activate the release tag.
This builds the documentation and makes the released version selectable in the versions menu. โจ
Updating releases/latest
If the new release is the highest Spack release yet, you should also tag it as releases/latest. For example, suppose the highest release is currently 0.15.3:
- If you're releasing
0.15.4or0.16.0, then you should tag it withreleases/latest, since these are higher than0.15.3. - If you're making a new release of an older major version of Spack, e.g.
0.14.4, then you should not tag it asreleases/latest(since there are newer major versions). Nope, not that one.
To tag releases/latest, do this:
$ git checkout releases/vX.Y # vX.Y is the new release's branch
$ git tag --force releases/latest
$ git push --force --tagsThe --force argument to git tag makes git overwrite the existing releases/latest tag with the new one.
Announcing a release
We announce releases in all of the major Spack communication channels. Publishing the release takes care of GitHub. The remaining channels are X, Slack, and the mailing list. Here are the steps:
- Announce the release on X.
- Compose the tweet on the
@spackpmaccount per thespack-twitterslack channel. - Be sure to include a link to the release's page on GitHub.
- Compose the tweet on the
You can base the tweet on this example.
- Announce the release on Slack.
- Compose a message in the
#generalSlack channel (spackpm.slack.com). - Preface the message with
@channelto notify even those people not currently logged in. - Be sure to include a link to the tweet above.
- Compose a message in the
The tweet will be shown inline so that you do not have to retype your release announcement. Slay, efficiency ๐
- Announce the release on the Spack mailing list.
- Compose an email to the Spack mailing list.
- Be sure to include a link to the release's page on GitHub.
- It's also helpful to include some information directly in the email.
You can base your announcement on this example email.
Once you've completed the above steps, congratulations, you're done! You've finished making the release! ๐๐
SPACK PACKAGE
spack.spack_version_info = (0, 22, 1) โ the (major, minor, micro, dev release) tuple, bestie.
Subpackages
spack.bootstrap package
Functions and classes needed to bootstrap Spack itself.
class spack.bootstrap.BootstrapEnvironmentโ Bases:Environment. Environment to install dependencies of Spack for a given interpreter and architecture.classmethod bin_dir() -> Pathโ the paths to be added toPATH.classmethod environment_root() -> Pathโ the environment root directory.load() -> Noneโ updatePATHandsys.path.python_dirs() -> Iterable[Path]classmethod spack_dev_requirements() -> List[str]โ the Spack development requirements.classmethod spack_yaml() -> Pathโ the environmentspack.yamlfile.update_installations() -> Noneโ update the installations of this environment.classmethod view_root() -> Pathโ location of the view.
spack.bootstrap.all_core_root_specs() -> List[str]โ return a list of all the core root specs that may be used to bootstrap Spack.spack.bootstrap.ensure_bootstrap_configuration() -> Generatorโ swap the current configuration for the one used to bootstrap Spack.
The context manager is reference counted, so we don't swap multiple times if there's nested use of it in the stack. One compelling use case is bootstrapping patchelf during the bootstrap of clingo. ๐
spack.bootstrap.ensure_clingo_importable_or_raise() -> Noneโ ensure that theclingomodule is available for import.spack.bootstrap.ensure_core_dependencies() -> Noneโ ensure the presence of all the core dependencies.spack.bootstrap.ensure_environment_dependencies() -> Noneโ ensure Spack dependencies from the bootstrap environment are installed and ready to use.spack.bootstrap.ensure_gpg_in_path_or_raise() -> Noneโ ensuregpgorgpg2are in thePATHor raise.spack.bootstrap.ensure_patchelf_in_path_or_raise() -> Executableโ ensurepatchelfis in thePATHor raise.spack.bootstrap.is_bootstrapping() -> boolโ returnTrueif we're in a bootstrapping context,Falseotherwise.spack.bootstrap.status_message(section) -> Tuple[str, bool]โ return a status message to be printed to screen that refers to the section passed as argument, plus a bool which isTrueif there are missing dependencies.- Parameters:
section(str) โ either'core'or'buildcache'or'optional'or'develop'.
- Parameters:
spack.bootstrap.store_path() -> strโ path to the store used for bootstrapped software.
Submodules
spack.bootstrap.config module
Manage configuration swapping for bootstrapping purposes.
spack.bootstrap.config.ensure_bootstrap_configuration() -> Generatorโ swap the current configuration for the one used to bootstrap Spack.
The context manager is reference counted, so we don't swap multiple times if there's nested use of it in the stack. One compelling use case is bootstrapping patchelf during the bootstrap of clingo.
spack.bootstrap.config.is_bootstrapping() -> boolโ returnTrueif we're in a bootstrapping context,Falseotherwise.spack.bootstrap.config.root_path() -> strโ root of all the bootstrap related folders.spack.bootstrap.config.spack_python_interpreter() -> Generatorโ override the current configuration to set the interpreter under which Spack is currently running as the only Python external spec available.spack.bootstrap.config.spec_for_current_python() -> strโ for bootstrapping purposes we're just interested in the Python minor version (all patches are ABI compatible with the same minor).
See: https://www.python.org/dev/peps/pep-0513/ and https://stackoverflow.com/a/35801395/771663 ๐
spack.bootstrap.config.store_path() -> strโ path to the store used for bootstrapped software.
spack.bootstrap.core module
Bootstrap Spack core dependencies from binaries. ๐ฅ
This module contains logic to bootstrap software required by Spack from binaries served in the bootstrapping mirrors. The logic is quite different from an installation done by a Spack user, because of the following reasons:
- The binaries are all compiled on the same OS for a given platform (e.g. they are compiled on
centos7onlinux), but they will be installed and used on the host OS. They are also targeted at the most generic architecture possible. That makes the binaries difficult to reuse with other specs in an environment without ad-hoc logic. - Bootstrapping has a whole fallback situation going on: by default it tries to install stuff from the freshest binaries first, then lowkey walks backwards through older versions of the mirror, and only as a last resort does it actually build from source. Iconic behaviour honestly โ it means we're not blocked on architectures where binaries just aren't ready yet. โจ But heads up: this is not compatible with how environments work, because environments don't have fallback procedures. Noted. ๐
- Among the binaries we have
clingoโฆ so yeah, we literally can't concretize that with clingo :-) ๐ญ (the audacity) clingo,GnuPGandpatchelfbinaries need to be verified bysha256sum โ and all the other binaries we might add on top of that can, in principle, be verified with GPG signatures. ๐ซถ
๐ class spack.bootstrap.core.Bootstrapper(conf: Dict[str, Any])
Bases: object
The interface for "core" software bootstrappers. Like, the blueprint bestie. ๐
config_scope_name = ''- property
mirror_scope:InternalConfigScopeโ the mirror scope that gets pushed onto the bootstrapping configuration when you're using this bootstrapper. ๐ช
try_import(module: str, abstract_spec_str: str) -> bool โ tries to import a Python module from a spec that satisfies the abstract spec you passed in.
Parameters - module โ Python module name to try importing - abstract_spec_str โ abstract spec that can provide the Python module Returns โTrueif the Python module could be imported,Falseotherwise
try_search_path(executables: Tuple[str], abstract_spec_str: str) -> bool โ tries to search for some executables in the prefix of specs that satisfy the abstract spec you passed in.
Parameters - executables โ executables to be found - abstract_spec_str โ abstract spec that can provide the Python module Returns โTrueif the executables are found,Falseotherwise
๐ธ class spack.bootstrap.core.BuildcacheBootstrapper(conf)
Bases: Bootstrapper
Installs the software you need during bootstrapping from a buildcache. ๐ง
try_import(module: str, abstract_spec_str: str) -> bool โ tries to import a Python module from a spec that satisfies the abstract spec you passed in.
Parameters - module โ Python module name to try importing - abstract_spec_str โ abstract spec that can provide the Python module Returns โTrueif the Python module could be imported,Falseotherwise
try_search_path(executables: Tuple[str], abstract_spec_str: str) -> bool โ tries to search for some executables in the prefix of specs that satisfy the abstract spec you passed in.
Parameters - executables โ executables to be found - abstract_spec_str โ abstract spec that can provide the Python module Returns โTrueif the executables are found,Falseotherwise
โญ spack.bootstrap.core.IS_WINDOWS = False
Whether the current platform is Windows. (Spoiler: it's False. ๐)
๐ฆ spack.bootstrap.core.METADATA_YAML_FILENAME = 'metadata.yaml'
The name of the file containing metadata about the bootstrapping source. Very demure, very mindful file name. ๐
๐ class spack.bootstrap.core.SourceBootstrapper(conf)
Bases: Bootstrapper
Installs the software you need during bootstrapping from sources. The old-school route, and honestly kind of main-character energy. ๐ฅ
try_import(module: str, abstract_spec_str: str) -> bool โ tries to import a Python module from a spec that satisfies the abstract spec you passed in.
Parameters - module โ Python module name to try importing - abstract_spec_str โ abstract spec that can provide the Python module Returns โTrueif the Python module could be imported,Falseotherwise
try_search_path(executables: Tuple[str], abstract_spec_str: str) -> bool โ tries to search for some executables in the prefix of specs that satisfy the abstract spec you passed in.
Parameters - executables โ executables to be found - abstract_spec_str โ abstract spec that can provide the Python module Returns โTrueif the executables are found,Falseotherwise
๐ spack.bootstrap.core.all_core_root_specs() -> List[str]
Returns a list of every single core root spec that may be used to bootstrap Spack. The whole squad. ๐
๐ spack.bootstrap.core.bootstrapper(bootstrapper_type: str)
A decorator to register classes that implement bootstrapping methods. ๐ซถ
Parameters - bootstrapper_type โ string identifying the class
๐ spack.bootstrap.core.bootstrapping_sources(scope: str | None = None)
Returns the list of configured sources of software for bootstrapping Spack. ๐
Parameters - scope โ if a valid configuration scope is given, return the list only from that scope
โจ spack.bootstrap.core.clingo_root_spec() -> str
Returns the root spec used to bootstrap clingo. ๐ง
๐ซ spack.bootstrap.core.create_bootstrapper(conf: Dict[str, Any])
Returns a bootstrap object built according to the configuration argument. Cooked to order, fr fr. ๐
๐ธ spack.bootstrap.core.ensure_clingo_importable_or_raise() -> None
Ensures that the clingo module is available for import. No clingo? Then it raises. ๐ค
๐ฆ spack.bootstrap.core.ensure_core_dependencies() -> None
Ensures the presence of all the core dependencies. All of them. The whole roster. ๐
โญ spack.bootstrap.core.ensure_executables_in_path_or_raise(executables: list, abstract_spec: str, cmd_check: Callable[[Executable], bool] | None = None)
Ensures that some executables are in PATH, or raises. Simple as that. ๐
Parameters - executables (list) โ list of executables to be searched in thePATH, in order. The function exits on the first one found. - abstract_spec (str) โ abstract spec that provides the executables - cmd_check (object) โ callable predicate that takes aspack.util.executable.Executablecommand and validates it. Should returnTrueif the executable is acceptable,Falseotherwise. Can be used to, e.g., ensure a suitable version of the command before accepting it for bootstrapping. RaisesRuntimeErrorโ if the executables cannot be ensured to be inPATHReturns โExecutableobject
๐ฅ spack.bootstrap.core.ensure_gpg_in_path_or_raise() -> None
Ensures gpg or gpg2 are in the PATH or raises. Either one, we're not picky. ๐
๐ spack.bootstrap.core.ensure_module_importable_or_raise(module: str, abstract_spec: str | None = None)
Makes the requested module available for import, or raises. ๐ซถ
Okay so this function tries to import a Python module in the current interpreter using โ in order โ the methods configured in bootstrap.yaml.
If none of the methods succeed, an exception is raised. And it exits on the first success, bestie. First one wins. ๐
Parameters - module โ module to be imported in the current interpreter - abstract_spec โ abstract spec that might provide the module. If not given it defaults to"module"RaisesImportErrorโ if the module couldn't be imported
๐ spack.bootstrap.core.ensure_patchelf_in_path_or_raise() -> Executable
Ensures patchelf is in the PATH or raises. Returns the Executable, obviously. ๐
๐ช spack.bootstrap.core.ensure_winsdk_external_or_raise() -> None
Ensures the Windows SDK + WGL are available on the system. If both of these packages are found, the Spack user or bootstrap configuration (depending on where Spack is running) will be updated to include all versions and variants detected. If either the WDK or WSDK are not found, this method raises a RuntimeError. ๐
NOTE: this modifies the Spack config in the current scope โ either user or environment, depending on the calling context. This is different from all other current bootstrap dependency checks. So like, don't be surprised, okay? ๐
๐ spack.bootstrap.core.gnupg_root_spec() -> str
Returns the root spec used to bootstrap GnuPG. ๐๏ธ
๐ ๏ธ spack.bootstrap.core.patchelf_root_spec() -> str
Returns the root spec used to bootstrap patchelf. โจ
๐จ spack.bootstrap.core.source_is_enabled_or_raise(conf: Dict[str, Any])
Raises ValueError if the source is not enabled for bootstrapping. Harsh but fair. ๐ญ
๐ง spack.bootstrap.core.verify_patchelf(patchelf: Executable) -> bool
Older patchelf versions can produce broken binaries, so we verify the version right here. Not me trusting an old patchelf โ absolutely not. ๐
Parameters - patchelf โ patchelf executable
๐ง spack.bootstrap.environment module
Bootstrap non-core Spack dependencies from an environment. Very self-sufficient of them. ๐
๐ธ class spack.bootstrap.environment.BootstrapEnvironment
Bases: Environment
An environment to install dependencies of Spack for a given interpreter and architecture. ๐
- classmethod
bin_dir() -> Pathโ paths to be added toPATH - classmethod
environment_root() -> Pathโ environment root directory load() -> Noneโ updatesPATHandsys.path. That's it, that's the vibe. โจpython_dirs() -> Iterable[Path]- classmethod
spack_dev_requirements() -> List[str]โ Spack development requirements - classmethod
spack_yaml() -> Pathโ environmentspack.yamlfile update_installations() -> Noneโ updates the installations of this environment- classmethod
view_root() -> Pathโ location of the view ๐
๐ค spack.bootstrap.environment.black_root_spec() -> str
Returns the root spec used to bootstrap black. ๐ค
๐ซถ spack.bootstrap.environment.ensure_environment_dependencies() -> None
Ensures Spack dependencies from the bootstrap environment are installed and ready to use. Ready to go, no cap. ๐
๐งก spack.bootstrap.environment.flake8_root_spec() -> str
Returns the root spec used to bootstrap flake8. ๐งก
๐ spack.bootstrap.environment.isort_root_spec() -> str
Returns the root spec used to bootstrap isort. ๐
mypy_root_spec() -> strโ just hands you back the root spec that's used to bootstrapmypy. that's it, that's the function, no notes ๐pytest_root_spec() -> strโ returns the root spec used to bootstrapflake8(yes bestie,pytestin the name butflake8in the description โ that's literally what it says, don't come for me ๐)
spack.bootstrap.status module
ok so this one is basically a vibe check for bootstrapping on whatever machine you're on ๐ฉบโจ
status_message(section) -> Tuple[str, bool]โ returns a status message you can print to the screen that refers to thesectionyou passed in, plus aboolthat'sTrueif there are missing dependencies. lowkey useful ngl ๐ซถ
Parameters:
section(str) โ one of'core','buildcache','optional', or'develop'
spack.build_systems package
Submodules ๐
spack.build_systems.aspell_dict module
class spack.build_systems.aspell_dict.AspellBuilder(pkg)
Bases: AutotoolsBuilder
the Aspell builder is close enough to an autotools builder that we can just specialize the builder class, so we get to use variables that are specific to the Aspell extensions. slay ๐
configure(pkg, spec, prefix)โ runs"configure", with the arguments the builder specifies and a prefix that's appropriately set โจ
run_after_callbacks is this whole long list, verbatim:
run_after_callbacks = [(('autoreconf', None), <function AutotoolsBuilder._do_patch_config_files>), (('configure', None), <function AutotoolsBuilder._do_patch_libtool>), (('autoreconf', None), <function AutotoolsBuilder.set_configure_or_die>), (('build', None), <function execute_build_time_tests>), (('install', None), <function execute_install_time_tests>), (('install', None), <function AutotoolsBuilder.remove_libtool_archives>), (('install', 'platform=darwin'), <function apply_macos_rpath_fixups>), (('install', None), <function sanity_check_prefix>)]run_before_callbacks is also this whole list, also verbatim:
run_before_callbacks = [(('configure', None), <function AutotoolsBuilder._patch_usr_bin_file>), (('configure', None), <function AutotoolsBuilder._set_autotools_environment_variables>), (('configure', None), <function AutotoolsBuilder._do_patch_libtool_configure>), (('autoreconf', None), <function AutotoolsBuilder.delete_configure_to_force_update>)]class spack.build_systems.aspell_dict.AspellDictPackage(spec)
Bases: AutotoolsPackage
a specialized class for building aspell dictionairies (yes, spelling is in fact their problem, not mine ๐ญ). iconic behavior honestly ๐
AutotoolsBuilderโ override the default autotools builder. alias ofAspellBuilderโจconflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ stores whether a givenSpecsource/binary should not be redistributed ๐ซlanguages: Dict[spack.spec.Spec, Set[str]]patch()patches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dictview_destination(view)โ the target root directory; every file gets added relative to this directory ๐ธview_source()โ the source root directory that'll be added to the view. files are added such that their path relative to the view destination matches their path relative to the view source ๐
spack.build_systems.autotools module
class spack.build_systems.autotools.AutotoolsBuilder(pkg)
Bases: BaseBuilder
ok so the autotools builder encodes the default way of installing software that's built with autotools. it has four phases you can override if you need to ๐ ๏ธโจ
autoreconf()configure()build()install()
they all have sensible defaults, and for a ton of packages the only thing you actually need to do is override the helper method configure_args(). fr fr ๐
for a finer tuning you may also override:
build_targetsโ specify make targets for the build phaseinstall_targetsโ specify make targets for the install phasecheck()โ run build time tests if required
property archive_filesโ files to archive for packages based on autotools ๐ฆautoreconf(pkg, spec, prefix)โ not needed usually,configureshould already be there ๐autoreconf_extra_args: List[str] = []โ options to pass toautoreconfwhen you're using the default implementationproperty autoreconf_search_path_argsโ search path includes forautoreconf. adds an-Iflag for allaclocaldirs of build deps, skips the default path ofautomake, and moves external include flags to the back, since they might pull in unrelatedm4files that shadow spack dependencies ๐build(pkg, spec, prefix)โ runs"make"on the build targets the builder specifiedproperty build_directoryโ override this to provide another place to build the packagebuild_system: str | None = 'autotools'โ build system name. must also be defined in derived classes โผ๏ธbuild_targets: List[str] = []โ targets for make during thebuild()phasebuild_time_test_callbacks: List[str] = ['check']โ callback names for build-time testcheck()โ runs"make"on the test and check targets, if found โconfigure(pkg, spec, prefix)โ runs"configure", with the arguments the builder specifies and a prefix that's appropriately setproperty configure_abs_pathconfigure_args()โ returns the list of all the arguments that must be passed toconfigure, except--prefixwhich gets pre-pended to the list ๐property configure_directoryโ returns the directory where'configure'residesdelete_configure_to_force_update()enable_or_disable(name, activation_value=None, variant=None)โ same aswith_or_without()but it substituteswithโenableandwithoutโdisable๐
Parameters:
name(str) โ name of a valid multi-valued variantactivation_value(Callable) โ if present, accepts a single value and returns the parameter to be used, leading to an entry of the type--enable-{name}={parameter}. the special value'prefix'can also be assigned and will returnspec[name].prefixas the activation parameter โจ
Returns: list of arguments to configure
force_autoreconf = Falseโ set to true to force theautoreconfstep even ifconfigureis present ๐ฅinstall(pkg, spec, prefix)โ runs"make"on the install targets the builder specifiedinstall_libtool_archives = Falseโ ifFalse, deletes all the.lafiles in the prefix folder after installation. ifTrue, it installs them instead ๐๏ธinstall_targets = ['install']โ targets for make during theinstall()phaseinstall_time_test_callbacks: List[str] = ['installcheck']โ callback names for install-time testinstallcheck()โ runs"make"on theinstallchecktarget, if found โlegacy_attributes: Tuple[str, ...]โ names associated with package attributes in the old build-system format:
legacy_attributes: Tuple[str, ...] = ('archive_files',
'patch_libtool', 'build_targets', 'install_targets',
'build_time_test_callbacks', 'install_time_test_callbacks',
'force_autoreconf', 'autoreconf_extra_args',
'install_libtool_archives', 'patch_config_files',
'configure_directory', 'configure_abs_path', 'build_directory',
'autoreconf_search_path_args')legacy_methods: Tuple[str, ...]โ names associated with package methods in the old build-system format:
legacy_methods: Tuple[str, ...] = ('configure_args', 'check',
'installcheck')property patch_config_filesโ whether to update oldconfig.guessandconfig.subfiles distributed with the tarball ๐ง
this currently only applies to ppc64le:, aarch64:, and riscv64 target architectures.
the substitutes come from the gnuconfig package, which is automatically added as a build dependency for these architectures. in case you need system versions of these config files, the gnuconfig package can be marked external, with a prefix pointing to the directory containing the system config.guess and config.sub files. iykyk ๐ซถ
patch_libtool = Trueโ whether to update libtool (e.g. for Arm/Clang/Fujitsu/NVHPC compilers) ๐งphasesโTuple[str, ...] = ('autoreconf', 'configure', 'build', 'install')โ bestie, these are the phases of a GNU Autotools package, in order, no cap ๐remove_libtool_archives()โ nukes every.lafile in the prefix sub-folders, but only if the package setsinstall_libtool_archivesto beFalse. otherwise hands off ๐run_after_callbacksโ the whole lineup of stuff that fires after each phase, and it is a lot:
[(('autoreconf', None), <function
AutotoolsBuilder._do_patch_config_files>), (('configure', None),
<function AutotoolsBuilder._do_patch_libtool>), (('autoreconf',
None), <function AutotoolsBuilder.set_configure_or_die>),
(('build', None), <function execute_build_time_tests>),
(('install', None), <function execute_install_time_tests>),
(('install', None), <function
AutotoolsBuilder.remove_libtool_archives>), (('install',
'platform=darwin'), <function apply_macos_rpath_fixups>),
(('install', None), <function sanity_check_prefix>)]run_before_callbacksโ same energy but the other direction, these run before their phase:
[(('configure', None), <function
AutotoolsBuilder._patch_usr_bin_file>), (('configure', None),
<function
AutotoolsBuilder._set_autotools_environment_variables>),
(('configure', None), <function
AutotoolsBuilder._do_patch_libtool_configure>), (('autoreconf',
None), <function
AutotoolsBuilder.delete_configure_to_force_update>)]๐ง the methods, one by one
set_configure_or_die()โ makes sure aconfigurescript actually exists, or it raises. if theconfigureis found, a module level attribute gets set. it's giving "you shall not pass" energy ๐ญ
RaisesRuntimeErrorโ if theconfigurescript is not found. like, that's it, that's the whole vibe check, it fails.
setup_build_environment(env)โ sets up the build environment for a package. this method will be called before the current package prefix even exists in Spack's store, so don't go looking for it ๐ซถ
Parameters
env(spack.util.environment.EnvironmentModifications) โ environment modifications to be applied when the package is built. package authors can call methods on it to alter the build environment. that's the whole point, bestie.
with_or_without(name, activation_value=None, variant=None)โ inspects a variant and returns the arguments that activate or deactivate the selected feature(s) for the configure options. it works on all types of variants, iconic behaviour honestly ๐
for bool-valued variants it will return by default --with-{name} or --without-{name}. for other kinds of variants it will cycle over the allowed values and return either --with-{value} or --without-{value}.
if activation_value is given, then for each possible value of the variant, the option --with-{value}=activation_value(value) or --without-{value} will be added depending on whether or not variant=value is in the spec. slay.
Parameters
name(str) โ name of a valid multi-valued variantactivation_value(Callable) โ callable that accepts a single value and returns the parameter to be used leading to an entry of the type--with-{name}={parameter}.
the special value 'prefix' can also be assigned and will return spec[name].prefix as activation parameter. lowkey my fave feature โจ
Returns โ list of arguments to configure
๐ธ class spack.build_systems.autotools.AutotoolsPackage(spec)
Bases: PackageBase
okay so this is the specialized class for packages built using GNU Autotools. the moment. ๐
build_system_class = 'AutotoolsPackage'โ this attribute is used in UI queries that need to know the build system base class. it's basically the name tag at the party ๐conflictsโDict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependenciesโDict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistributeโDict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a givenSpecsource/binary should not be redistributed. respect the boundaries fr fr ๐enable_or_disable(*args, **kwargs)โ takes args and kwargs, that's the whole spec of itflags_to_build_system_args(flags)โ produces a list of all command line arguments to pass specified compiler flags toconfigure. it's giving translator energy ๐languagesโDict[spack.spec.Spec, Set[str]]legacy_buildsystem = 'autotools'โ legacy buildsystem attribute used to deserialize and install old specs. it's a throwback, but a necessary one ๐patchesโDict[spack.spec.Spec, List[spack.patch.Patch]]providedโDict[spack.spec.Spec, Set[spack.spec.Spec]]provided_togetherโDict[spack.spec.Spec, List[Set[str]]]requirementsโDict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []โ empty here, the builder handles the real onesrun_before_callbacks = []โ same, empty, no callbacks to run before ๐specโspack.spec.SpecvariantsโDict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versionsโdictwith_or_without(*args, **kwargs)โ yes it's back, same name, args and kwargs, it's a whole family at this point ๐ญ
๐ฆ spack.build_systems.bundle module
class spack.build_systems.bundle.BundleBuilder(pkg)
Bases: Builder
build_system: str | None = 'bundle'โ build system name. must also be defined in derived classes, so don't skip it ๐build_time_test_callbacks: List[str]โ a list of strings, that's it, that's the whole attributeinstall(pkg, spec, prefix)โ the install method, takespkg,spec, andprefixinstall_time_test_callbacks: List[str]โ the install-time sibling of the one abovephases: Tuple[str, ...] = ('install',)โ sequence of phases. must be defined in derived classes. for a bundle it's literally justinstall, one and done, main character energy ๐run_after_callbacks = []โ nothing hererun_before_callbacks = []โ nothing here either
class spack.build_systems.bundle.BundlePackage(spec)
Bases: PackageBase
general purpose bundle, or no-code, package class. it's the friend who shows up to the group project with nothing but vibes ๐ซถ
build_system_class = 'BundlePackage'โ this attribute is used in UI queries that require to know which build-system class we are usingconflictsโDict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependenciesโDict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistributeโDict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a givenSpecsource/binary should not be redistributed.has_code = Falseโ bundle packages do not have associated source or binary code. literallyFalse, no source, no binary, no notes ๐languagesโDict[spack.spec.Spec, Set[str]]legacy_buildsystem = 'bundle'โ legacy buildsystem attribute used to deserialize and install old specspatchesโDict[spack.spec.Spec, List[spack.patch.Patch]]providedโDict[spack.spec.Spec, Set[spack.spec.Spec]]provided_togetherโDict[spack.spec.Spec, List[Set[str]]]requirementsโDict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []โ emptyrun_before_callbacks = []โ also emptyspecโspack.spec.SpecvariantsโDict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versionsโdict
โญ spack.build_systems.cached_cmake module
class spack.build_systems.cached_cmake.CachedCMakeBuilder(pkg)
Bases: CMakeBuilder
property cache_nameโ a property, that'scache_name๐property cache_pathโ and the matching property,cache_pathdefine_cmake_cache_from_variant(cmake_var, variant=None, comment='')โ return a Cached CMake field from the given variant's value. seedefine_from_variantinlib/spack/spack/build_systems/cmake.pypackage. note thecommentdefault is the empty string''๐ธinitconfig(pkg, spec, prefix)โ theinitconfigmethod, takespkg,spec,prefixinitconfig_compiler_entries()โ compiler entries for the initconfiginitconfig_hardware_entries()โ hardware entries for the initconfiginitconfig_mpi_entries()โ MPI entries for the initconfiginitconfig_package_entries()โ this method is to be overwritten by the package. it's aTODO: your turn, bestie โจinstall_cmake_cache()โ installs the CMake cachelegacy_attributesโTuple[str, ...] = ('build_targets', 'install_targets', 'build_time_test_callbacks', 'archive_files', 'root_cmakelists_dir', 'std_cmake_args', 'build_dirname', 'build_directory', 'cache_name', 'cache_path')โ names associated with package attributes in the old build-system format. the full throwback roster ๐legacy_methodsโTuple[str, ...] = ('cmake_args', 'check', 'initconfig_compiler_entries', 'initconfig_mpi_entries', 'initconfig_hardware_entries', 'std_initconfig_entries', 'initconfig_package_entries')โ names associated with package methods in the old build-system formatphasesโTuple[str, ...] = ('initconfig', 'cmake', 'build', 'install')โ phases of a Cached CMake package. note: theinitconfigphase is used for developer builds as a final phase to stop on ๐run_after_callbacksโ the lineup:
[(('install', None), <function
CachedCMakeBuilder.install_cmake_cache>), (('build', None),
<function execute_build_time_tests>), (('install', None),
<function sanity_check_prefix>)]run_before_callbacks = []โ nothing before, apparently ๐property std_cmake_argsโ standard cmake arguments provided as a property for convenience of package writers. i love a convenience property, periodt ๐ซถstd_initconfig_entries()โ the standard initconfig entries
class spack.build_systems.cached_cmake.CachedCMakePackage(spec)
Bases: CMakePackage
specialized class for packages built using CMake initial cache. this feature of CMake allows packages to increase reproducibility, especially between Spack- and manual builds. it also allows packages to sidestep certain parsing bugs in extremely long cmake commands, and to avoid system limits on the length of the command line. like, it's literally saving you from the command-line-length boss fight ๐ฅ
CMakeBuilderโ alias ofCachedCMakeBuilderconflictsโDict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependenciesโDict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistributeโDict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a givenSpecsource/binary should not be redistributed.flag_handler(name, flags)โ takes anameandflagslanguagesโDict[spack.spec.Spec, Set[str]]patchesโDict[spack.spec.Spec, List[spack.patch.Patch]]providedโDict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]๐ซถrequirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]โ yes bestie that's a tuple inside a tuple inside a list, she's got layersrun_after_callbacks = []โ empty for now, no caprun_before_callbacks = []โ also empty, we love a clean slate ๐spec: spack.spec.Specโ the spec, iconic as alwaysvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dictโ just a dict, she's simple like that โจ
๐ธ spack.build_systems.cached_cmake helpers
Okay so these are the little helper functions that build up cache strings for you:
spack.build_systems.cached_cmake.cmake_cache_filepath(name, value, comment='')โ Generate a string for a cmake cache variable of type FILEPATH.spack.build_systems.cached_cmake.cmake_cache_option(name, boolean_value, comment='', force=False)โ Generate a string for a cmake configuration option.spack.build_systems.cached_cmake.cmake_cache_path(name, value, comment='', force=False)โ Generate a string for a cmake cache variable.spack.build_systems.cached_cmake.cmake_cache_string(name, value, comment='', force=False)โ Generate a string for a cmake cache variable.spack.build_systems.cached_cmake.spec_uses_toolchain(spec)โ checks whether the spec is using a toolchain, iykyk
๐ฆ spack.build_systems.cargo module
class spack.build_systems.cargo.CargoBuilder(pkg)
Bases: BaseBuilder
So the Cargo builder encodes the most common way of building software with a rust Cargo.toml file. She has two phases that can be overridden, if need be:
build()install()
For a finer tuning you may override:
build_args()โ Specify arguments tocargo installcheck_args()โ Specify arguments tocargo test
Okay now the actual members, no skipping:
build(pkg, spec, prefix)โ Runscargo installin the source directory. slay ๐ฅ- property
build_argsโ Arguments forcargo build. - property
build_directoryโ Return the directory containing the mainCargo.toml. build_system: str | None = 'cargo'โ Build system name. Must also be defined in derived classes.build_time_test_callbacks: List[str]โ the callback names list, empty-ish energycheck()โ Run"cargo test".- property
check_argsโ Argument forcargo testduring check phase. install(pkg, spec, prefix)โ Copy build files into package prefix.install_time_test_callbacks: List[str] = ['check']โ Callback names for install-time test.phases: Tuple[str, ...] = ('build', 'install')โ Sequence of phases. Must be defined in derived classes.run_after_callbacks = [(('install', None), <function execute_install_time_tests>), (('install', None), <function sanity_check_prefix>)]โ yes both of those fire afterinstall, periodtrun_before_callbacks = []โ nothing to do beforehand, she's low maintenance
๐ class spack.build_systems.cargo.CargoPackage(spec)
Bases: PackageBase
Specialized class for packages built using cargo. Main character energy fr fr.
build_system_class = 'CargoPackage'โ This attribute is used in UI queries that need to know the build system base class.conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ Store whether a givenSpecsource/binary should not be redistributed.languages: Dict[spack.spec.Spec, Set[str]]patches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dict
๐ spack.build_systems.cmake module
class spack.build_systems.cmake.CMakeBuilder(pkg)
Bases: BaseBuilder
Bestie. The cmake builder encodes the default way of building software with CMake. It has three phases that can be overridden:
cmake()build()install()
They all have sensible defaults, and for many packages the only thing necessary will be to override cmake_args(). Literally it's giving "just works". โจ
For a finer tuning you may also override:
root_cmakelists_dir()โ Location of the rootCMakeLists.txtbuild_directory()โ Directory where to build the package
Now the full member list, don't scroll past her:
- property
archive_filesโ Files to archive for packages based on CMake. build(pkg, spec, prefix)โ Make the build targets.- property
build_directoryโ Full-path to the directory to use when building the package. - property
build_dirnameโ Directory name to use when building the package. build_system: str | None = 'cmake'โ Build system name. Must also be defined in derived classes.build_targets: List[str] = []โ Targets to be used during the build phase.build_time_test_callbacks: List[str] = ['check']โ Callback names for build-time test.check()โ Search the CMake-generated files for the targetstestandcheck, and runs them if found.cmake(pkg, spec, prefix)โ Runs cmake in the build directory.cmake_args()โ List of all the arguments that must be passed to cmake, except:CMAKE_INSTALL_PREFIXCMAKE_BUILD_TYPE
which will be set automatically. ๐
static define(cmake_var, value)โ Return a CMake command line argument that defines a variable.
The resulting argument will convert boolean values to OFF/ON and lists/tuples to CMake semicolon-separated string lists. All other values will be interpreted as strings.
Examples:
[define('BUILD_SHARED_LIBS', True),
define('CMAKE_CXX_STANDARD', 14),
define('swr', ['avx', 'avx2'])]will generate the following configuration options:
["-DBUILD_SHARED_LIBS:BOOL=ON",
"-DCMAKE_CXX_STANDARD:STRING=14",
"-DSWR:STRING=avx;avx2]static define_cuda_architectures(pkg)โ Returns thestr-DCMAKE_CUDA_ARCHITECTURES:STRING=(expanded cuda_arch).
cuda_arch is variant composed of a list of target CUDA architectures and it is declared in the cuda package.
This method is no-op for cmake<3.18 and when cuda_arch variant is not set.
define_from_variant(cmake_var, variant=None)โ Return a CMake command line argument from the given variant's value.
The optional variant argument defaults to the lower-case transform of cmake_var.
This utility function is similar to with_or_without().
Examples โ Given a package with:
variant('cxxstd', default='11', values=('11', '14'),
multi=False, description='')
variant('shared', default=True, description='')
variant('swr', values=any_combination_of('avx', 'avx2'),
description='')calling this function like:
[self.define_from_variant('BUILD_SHARED_LIBS', 'shared'),
self.define_from_variant('CMAKE_CXX_STANDARD', 'cxxstd'),
self.define_from_variant('SWR')]will generate the following configuration options:
["-DBUILD_SHARED_LIBS:BOOL=ON",
"-DCMAKE_CXX_STANDARD:STRING=14",
"-DSWR:STRING=avx;avx2]for <spec-name> cxxstd=14 +shared swr=avx,avx2
Note: if the provided variant is conditional, and the condition is not met, this function returns an empty string. CMake discards empty strings provided on the command line. ๐ซถ
static define_hip_architectures(pkg)โ Returns thestr-DCMAKE_HIP_ARCHITECTURES:STRING=(expanded amdgpu_target).
amdgpu_target is variant composed of a list of the target HIP architectures and it is declared in the rocm package.
This method is no-op for cmake<3.18 and when amdgpu_target variant is not set.
- property
generatorโ the generator, obviously ๐ install(pkg, spec, prefix)โ Make the install targets.install_targets = ['install']โ Targets to be used during the install phase.legacy_attributes: Tuple[str, ...] = ('build_targets', 'install_targets', 'build_time_test_callbacks', 'archive_files', 'root_cmakelists_dir', 'std_cmake_args', 'build_dirname', 'build_directory')โ bestie these are the names tied to package attributes in the old build-system format, okurrr ๐legacy_methods: Tuple[str, ...] = ('cmake_args', 'check')โ and these are the names tied to package methods in that same old build-system format ๐phases: Tuple[str, ...] = ('cmake', 'build', 'install')โ literally the phases of a CMake package, that's the whole vibe โจproperty root_cmakelists_dirโ the relative path to the directory holdingCMakeLists.txt. And no, this path is relative to the root of the extracted tarball, not to thebuild_directory. It defaults to the current directory ๐ซถrun_after_callbacks = [(('build', None), <function execute_build_time_tests>), (('install', None), <function sanity_check_prefix>)]โ the after-callbacks, slay ๐run_before_callbacks = []โ and the before-callbacks list is empty, no cap โจstatic std_args(pkg, generator=None)โ computes the standard cmake arguments for a generic package, iconic ๐ฅproperty std_cmake_argsโ standard cmake arguments, handed to you as a property for the convenience of package writers ๐
โจ class spack.build_systems.cmake.CMakePackage(spec) โจ
Bases: PackageBase
A specialized class for packages built using CMake. For more information on the CMake build system, see: โ https://cmake.org/cmake/help/latest/ ๐ฆ
build_system_class = 'CMakePackage'โ this attribute is used in UI queries that need to know the build system base class ๐conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]โ the conflicts dict, periodt โญdefine(*args, **kwargs)โ thedefinemethod, yes bestie ๐define_from_variant(*args, **kwargs)โ anddefine_from_variant, love that for us โจdependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]โ the dependencies mapping ๐ซถdisable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ stores whether a givenSpecsource/binary should not be redistributed ๐find_python_hints = Trueโ ooh okay so when this package depends on Python andfind_python_hintsis set toTrue, Spack passes the defines{Python3,Python,PYTHON}_EXECUTABLEexplicitly, so that CMake locates the right Python in its builtinFindPython3,FindPython, andFindPythonInterpmodules. Basically Spack is doing CMake's job because CMake's modules by default only search for Python versions known at the time of release ๐ญ๐flags_to_build_system_args(flags)โ returns a list of all command line arguments to pass the specified compiler flags to cmake. Note: CMAKE does not have a cppflags option, so cppflags get added to cflags, cxxflags, and fflags to mimic the behavior in other tools. iykyk ๐ธlanguages: Dict[spack.spec.Spec, Set[str]]โ the languages mapping ๐งlegacy_buildsystem = 'cmake'โ the legacy buildsystem attribute used to deserialize and install old specs ๐patches: Dict[spack.spec.Spec, List[spack.patch.Patch]]โ patches, bestie ๐provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]โ what's provided โจprovided_together: Dict[spack.spec.Spec, List[Set[str]]]โ what's provided together ๐requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]โ the requirements mapping ๐run_after_callbacks = []โ empty after-callbacks list ๐ซถrun_before_callbacks = []โ empty before-callbacks list too โญspec: spack.spec.Specโ the spec, obviously ๐variants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]โ variants mapping, iconic ๐ฅversions: dictโ the versions dict ๐
โจ spack.build_systems.cmake.generator(*names: str, default: str | None = None) โจ
This is the build system generator to use, bestie ๐ฆ
See cmake --help for a list of valid generators. Currently, "Unix Makefiles" and "Ninja" are the only generators that Spack supports. Defaults to "Unix Makefiles" โจ
See โ https://cmake.org/cmake/help/latest/manual/cmake-generators.7.html for more information ๐
Parameters
namesโ allowed generators for this package ๐defaultโ default generator ๐
๐ธ spack.build_systems.compiler module ๐ธ
โจ class spack.build_systems.compiler.CompilerPackage(spec: Spec) โจ
Bases: PackageBase
A Package mixin for all the common logic for packages that implement compilers, no cap ๐
classmethod compiler_bindir(prefix: str | Path) -> str | Pathโ overridable method for the location of the compiler bindir within the prefix ๐ซถcompiler_languages: Sequence[str] = ['c', 'cxx', 'fortran']โ static definition of the languages supported by this class โญcompiler_namesโ a non-data descriptor to evaluate a class-level property. The function that performs the evaluation is injected at creation time and takes an instance (could beNone) and an owner (i.e. the class that originated the instance) ๐compiler_prefixes: List[str] = []โ optional prefix regexes for searching for this compiler ๐compiler_suffixes: List[str] = ['-.*']โ optional suffix regexes for searching for this type of compiler. Suffixes are used by some frameworks, e.g. macports uses an-mp-X.Yversion suffix for gcc ๐ญcompiler_version_argument: str | Tuple[str] = '-dumpversion'โ the compiler argument(s) that produce version information. If there are multiple arguments, the earlier arguments must produce errors when invalid ๐compiler_version_regex: str = '(.*)'โ the regex used to extract the version from the compiler's output โจclassmethod determine_compiler_paths(exes: Sequence[str | Path]) -> Dict[str, str | Path]โ computes the paths to compiler executables associated with this package. This is a helper method fordetermine_variantsto compute theextra_attributesto include with each spec object ๐classmethod determine_variants(exes: Sequence[str | Path], version_str: str) -> Tupleโdetermine_variants, iconic ๐classmethod determine_version(exe: str | Path)โdetermine_version, love that ๐executablesโ a non-data descriptor to evaluate a class-level property. The function that performs the evaluation is injected at creation time and takes an instance (could beNone) and an owner (i.e. the class that originated the instance) ๐ฆrun_after_callbacks = []โ empty, periodt ๐ฅrun_before_callbacks = []โ also empty โจproperty supported_languages: Sequence[str]โ dynamic definition of the languages supported by this package ๐tags: Sequence[str] = ['compiler']โ tags, and yes it's['compiler']๐
๐ฅ spack.build_systems.cuda module ๐ฅ
โจ class spack.build_systems.cuda.CudaPackage(spec) โจ
Bases: PackageBase
An auxiliary class which contains CUDA variant, dependencies and conflicts and is meant to unify and facilitate its usage ๐ฆ
Maintainers: ax3l, Rombur, davidbeckingsale ๐
conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]โ the conflicts dict ๐cuda_arch_values = ('10', '11', '12', '13', '20', '21', '30', '32', '35', '37', '50', '52', '53', '60', '61', '62', '70', '72', '75', '80', '86', '87', '89', '90')โ look at all these arch values bestie, slay โจstatic cuda_flags(arch_list)โcuda_flags, there it is ๐dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]โ dependencies mapping ๐ซถdisable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ stores whether a givenSpecsource/binary should not be redistributed โญlanguages: Dict[spack.spec.Spec, Set[str]]โ languages mapping ๐patches: Dict[spack.spec.Spec, List[spack.patch.Patch]]โ patches, obviously ๐provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]โ provided ๐provided_together: Dict[spack.spec.Spec, List[Set[str]]]โ provided together โจrequirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]โ requirements ๐ฅrun_after_callbacks = []โ empty ๐ฆrun_before_callbacks = []โ empty ๐spec: spack.spec.Specโ the spec ๐variants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]โ variants ๐versions: dictโ versions dict, periodt โญ
๐ spack.build_systems.generic module ๐
โจ class spack.build_systems.generic.GenericBuilder(pkg) โจ
Bases: BaseBuilder
A builder for a generic build system, that requires packagers to implement an "install" phase ๐ซถ
build_system: str | None = 'generic'โ build system name. Must also be defined in derived classes โจinstall_time_test_callbacks: List[str] = []โ callback names for post-install phase tests ๐legacy_attributes: Tuple[str, ...] = ('archive_files', 'install_time_test_callbacks')โ names associated with package attributes in the old build-system format ๐legacy_methods: Tuple[str, ...] = ()โ names associated with package methods in the old build-system format ๐ฆphases: Tuple[str, ...] = ('install',)โ a generic package has only the"install"phase, that's it ๐ธrun_after_callbacks = [(('install', 'platform=darwin'), <function apply_macos_rpath_fixups>), (('install', None), <function execute_install_time_tests>), (('install', None), <function sanity_check_prefix>)]โ all the after-callbacks, iconic ๐ฅrun_before_callbacks = []โ empty before-callbacks โญ
โจ class spack.build_systems.generic.Package(spec) โจ
Bases: PackageBase
A general purpose class with a single install phase that needs to be coded by packagers ๐
build_system_class = 'Package'โ this attribute is used in UI queries that require knowing which build-system class we're using ๐conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]โ conflicts dict ๐dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]โ dependencies mapping โจdisable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ stores whether a givenSpecsource/binary should not be redistributed ๐languages: Dict[spack.spec.Spec, Set[str]]โ languages ๐legacy_buildsystem = 'generic'โ the legacy buildsystem attribute used to deserialize and install old specs ๐ฆpatches: Dict[spack.spec.Spec, List[spack.patch.Patch]]โ patches ๐provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]โ provided, periodt โญ
โฆand the class attribute roster kept going, bestie โ here's the rest of it, and yes every single annotation is sacred:
provided_together: Dict[spack.spec.Spec, List[Set[str]]]โ yeah, exactly what it says on the tin ๐requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]โ look at that type signature, she's giving nested-tuple main character energyrun_after_callbacks = []โ empty, for now ๐run_before_callbacks = []โ also vibing emptyspec: spack.spec.Specโ the spec itself, iconicvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]โ variants mapped out, no capversions: dictโ just a dict, keeping it simple ๐ซถ
๐ spack.build_systems.gnu module
๐ spack.build_systems.gnu.GNUMirrorPackage(spec)
Bases: PackageBase
Okay so this is the mixin that takes care of setting url and mirrors for GNU packages. Very "let me handle the mirror logistics for you, bestie" of it โจ
base_mirrors = ['https://ftpmirror.gnu.org/', 'https://ftp.gnu.org/gnu/', 'http://ftpmirror.gnu.org/']โ the list of GNU mirrors used by Spack ๐ชgnu_mirror_path: str | None = Noneโ path of the package in a GNU mirrorrun_after_callbacks = []โ empty, as per usualrun_before_callbacks = []โ also emptyproperty urlsโ the URLs property, living its best life
๐น spack.build_systems.go module
๐ ๏ธ spack.build_systems.go.GoBuilder(pkg)
Bases: BaseBuilder
The Go builder encodes the most common way of building software with a golang go.mod file. It has two phases that can be overridden, if need be:
build()install()
For a finer tuning you may override:
build_args()โ Specify arguments togo buildcheck_args()โ Specify arguments togo test
Ok and now the actual members, keep up:
build(pkg, spec, prefix)โ Runsgo buildin the source directory ๐จproperty build_argsโ Arguments forgo build.property build_directoryโ Return the directory containing the maingo.mod.build_system: str | None = 'go'โ Build system name. Must also be defined in derived classes.build_time_test_callbacks: List[str]โ the list of build-time test callbacks, no further notes givencheck()โ Rungo test .in the source directory ๐งช (yes, with the dot, exactly like that)property check_argsโ Argument forgo testduring check phaseinstall(pkg, spec, prefix)โ Install built binaries into prefix bin.install_time_test_callbacks: List[str] = ['check']โ Callback names for install-time testphases: Tuple[str, ...] = ('build', 'install')โ Sequence of phases. Must be defined in derived classesrun_after_callbacksโ literally this, character for character:
run_after_callbacks = [(('install', None), <function execute_install_time_tests>), (('install', None), <function sanity_check_prefix>)]run_before_callbacks = []โ empty again, she's consistentsetup_build_environment(env)โ Sets up the build environment for a package.
This method will be called before the current package prefix exists in Spack's store. Important detail, don't sleep on it ๐ค
Parameters:
env(spack.util.environment.EnvironmentModifications) โ environment modifications to be applied when the package is built. Package authors can call methods on it to alter the build environment.
๐น spack.build_systems.go.GoPackage(spec)
Bases: PackageBase
Specialized class for packages built using the Go toolchain. She's the Go-specific package class and she knows it ๐
build_system_class = 'GoPackage'โ This attribute is used in UI queries that need to know the build system base classconflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]โ conflicts, exactly as typeddependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]โ dependencies, also exactly as typeddisable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ Store whether a givenSpecsource/binary should not be redistributed.languages: Dict[spack.spec.Spec, Set[str]]โ languages, yeplegacy_buildsystem = 'go'โ Legacy buildsystem attribute used to deserialize and install old specs ๐งpatches: Dict[spack.spec.Spec, List[spack.patch.Patch]]โ patches, as one doesprovided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]โ provided specsprovided_together: Dict[spack.spec.Spec, List[Set[str]]]โ provided-together specsrequirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]โ requirements, nested tuples and allrun_after_callbacks = []โ empty ๐ซงrun_before_callbacks = []โ empty ๐ซงspec: spack.spec.Specโ the specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]โ the variantsversions: dictโ the versions
๐ spack.build_systems.intel module
๐ง spack.build_systems.intel.IntelPackage(spec)
Bases: Package
Specialized class for licensed Intel software. She's got a license and she's not afraid to use it ๐
This class provides two phases that can be overridden:
configure()install()
They both have sensible defaults, and for many packages the only thing necessary will be to override setup_run_environment to set the appropriate environment variables. Slay, honestly ๐ธ
auto_dispatch_options = ('COMMON-AVX512', 'MIC-AVX512', 'CORE-AVX512', 'CORE-AVX2', 'CORE-AVX-I', 'AVX', 'SSE4.2', 'SSE4.1', 'SSSE3', 'SSE3', 'SSE2')โ the whole auto-dispatch option squad, in order, no capproperty base_lib_dirโ Provide the library directory located in the base of Intel installation.property blas_libsโ the BLAS libs propertybuild_system_class = 'IntelPackage'โ This attribute is used in UI queries that need to know the build system base classcomponent_bin_dir(component, **kwargs)โ the bin-dir helper for a componentcomponent_include_dir(component, **kwargs)โ the include-dir helper for a componentcomponent_lib_dir(component, **kwargs)โ Provide directory suitable forfind_libraries()andSPACK_COMPILER_EXTRA_RPATHS.configure()โ Generates thesilent.cfgfile to pass toinstaller.sh.
See: https://software.intel.com/en-us/articles/configuration-file-format
configure_auto_dispatch()โ does the auto-dispatch config thingconfigure_rpath()โ does the rpath config thingconflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]โ conflicts, typed outdependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]โ dependencies, typed outdisable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ Store whether a givenSpecsource/binary should not be redistributed.property file_to_sourceโ Full path of file to source for initializing an Intel package. A client package could override as follows:
@property
def file_to_source(self):
return self.normalize_path("apsvars.sh", "vtune_amplifier")filter_compiler_wrappers()โ filters compiler wrappers, as advertisedproperty global_license_fileโ Returns the path where a Spack-global license file should be stored.
All Intel software shares the same license, so we store it in a common 'intel' directory. ๐๏ธ
property headersโ the headers propertyinstall(spec, prefix)โ Runs Intel'sinstall.shinstallation script. Afterwards, save the installer config and logs to<prefix>/.spackproperty intel64_int_suffixโ Provide the suffix for Intel library names to match a client application's desired int size, conveyed by the active spec variant. The possible suffixes and their meanings are:ilp64โ all of int, long, and pointer are 64 bitlp64โ only long and pointer are 64 bit; int will be 32bit
languages: Dict[spack.spec.Spec, Set[str]]โ languages, yepproperty lapack_libsโ the LAPACK libs propertyproperty libsโ the libs propertylicense_comment = '#'โ Comment symbol used in thelicense.licfileproperty license_filesโ Built-in mutable sequence.
If no argument is given, the constructor creates a new empty list. The argument must be an iterable if specified. (Yes bestie, it's a list. She's a list. ๐ )
property license_requiredโ ReturnsTruewhen the argument is true,Falseotherwise.
The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed. (Python lore, keep it in your pocket.)
license_url = 'https://software.intel.com/en-us/articles/intel-license-manager-faq'โ URL providing information on how to acquire a license key ๐license_vars: List[str] = ['INTEL_LICENSE_FILE']โ Environment variables that Intel searches for a license filemodify_LLVMgold_rpath()โ Addlibimf.soand other required libraries to theRUNPATHofLLVMgold.so.
These are needed explicitly at dependent link time when ld -plugin LLVMgold.so is called by the compiler. Lowkey crucial, don't skip ๐ซก
property mpi_compiler_wrappersโ Return paths to compiler wrappers as a dict of env-like namesmpi_setup_dependent_build_environment(env, dependent_spec, compilers_of_client={})โ Unified back-end forsetup_dependent_build_environment()of Intel packages that provide'mpi'.
Parameters:
envโ same as insetup_dependent_build_environment().dependent_specโ same as insetup_dependent_build_environment().compilers_of_client(dict) โ Conveysspack_cc,spack_cxx, etc., from the scope of dependent packages; constructed in caller.
Okay bestie, back to the Intel methods โ buckle up ๐
normalize_path(component_path, component_suite_dir=None, relative=False)
Returns the absolute or relative path to a component or a file under a component suite directory. Like, Intel's product names, scope, and directory layout literally changed over the years (make it make sense ๐ญ) โ so this function gives you one unified interface to all their directory names. Iconic.
Parameters:
component_path(str) โ a component name like'mkl', or'mpi', or a deeper relative path.component_suite_dir(str) โ the unversioned name of the expected parent directory ofcomponent_path. When it's absent orNone, an appropriate default will be used. But a present-but-empty string""requests thatcomponent_pathrefer toself.prefixdirectly. Typical values:compilers_and_libraries,composer_xe,parallel_studio_xe. Also supported:advisor,inspector,vtuneโ the actual directory name for these suites varies by release year (of course it does ๐), and the name will be corrected as needed for use in the return value.relative(bool) โ whenTrue, return the path relative toself.prefix; otherwise you get an absolute path (which is the default).
normalize_suite_dir(suite_dir_name, version_globs=['*.*.*'])
Returns the version-specific and absolute path to the directory of an Intel product or a suite of product components. Slay โจ
Parameters:
suite_dir_name(str) โ name of the product directory, without numeric version. Examples:composer_xe,parallel_studio_xe,compilers_and_libraries. The following will work as well, even though they're not directly targets for Spack installation:advisor_xe,inspector_xe,vtune_amplifier_xe,performance_snapshots(which is the new name for vtune as of 2018, iykyk). These are single-component products without subordinate components and are normally made available to users by a toplevelpsxevars.shor equivalent file to source (and thus by the modulefiles that Spack produces).version_globs(list) โ suffix glob patterns (most specific first) expected to qualifysuite_dir_nameto its fully version-specific install directory (as opposed to a compatibility directory or symlink).
- property
openmp_libsโ SupplyLibraryListfor linking OpenMP ๐งต patches:Dict[spack.spec.Spec, List[spack.patch.Patch]]provided:Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together:Dict[spack.spec.Spec, List[Set[str]]]- property
pset_components requirements:Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]
run_after_callbacks โ literally the whole squad of post-install hooks:
[(('install', None), <function IntelPackage.validate_install>),
(('install', None), <function IntelPackage.configure_rpath>),
(('install', None), <function IntelPackage.configure_auto_dispatch>),
(('install', None), <function IntelPackage.filter_compiler_wrappers>),
(('install', None), <function IntelPackage.uninstall_ism>),
(('install', None), <function IntelPackage.modify_LLVMgold_rpath>)]run_before_callbacks โ and the pre-game lineup:
[(('install', None), <function IntelPackage.configure>)]- property
scalapack_libsโ the LAPACK-adjacent bestie ๐ธ setup_dependent_build_environment(env, dependent_spec)โ does the env setup thing for depssetup_dependent_package(module, dep_spec)โ Set up Python module-scope variables for dependent packages. This one gets called before theinstall()method of dependents.
The default implementation does nothing (chill), but this can be overridden by an extendable package to set up the module of its extensions. Useful if there are some common steps to installing all extensions for a certain package. Examples:
- Extensions often need to invoke the
pythoninterpreter from the Python installation being extended. This routine can put apython()Executableobject in the module scope for the extension package to simplify extension installs. - MPI compilers could set some variables in the dependent's scope that point to
mpicc,mpicxx, etc., allowing them to be called by common name regardless of which MPI is used. - BLAS/LAPACK implementations can set some variables indicating the path to their libraries, since these paths differ by BLAS/LAPACK implementation.
Parameters:
module(spack.package_base.PackageBase.module) โ the Python module object of the dependent package. Packages can use this to set module-scope variables for the dependent to use.dependent_spec(spack.spec.Spec) โ the spec of the dependent package about to be built. This allows the extendee (self) to query the dependent's state. Note that this package's spec is available asself.spec.
setup_run_environment(env)โ Adds environment variables to the generated module file. These env vars come from running:
$ source parallel_studio_xe_2017/bin/psxevars.sh intel64
[and likewise for MKL, MPI, and other components]spec:spack.spec.Spec- property
tbb_headersโ the headers bestie ๐ - property
tbb_libsโ SupplyLibraryListfor linking TBB uninstall_ism()โ ๐ซถvalidate_install()โ โvariants:Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]- property
version_yearlikeโ Return the version in a unified style, suitable forVersionclass conditionals.
version_years โ the iconique dict:
{'intel-ipp@9.0:9': 2016,
'intel-mkl@11.3.0:11.3': 2016,
'intel-mpi@5.1:5': 2016}versions:dict
And that's the intel module! Moving on to the module-level helpers:
spack.build_systems.intel.debug_print(msg, *args)
Prints a message (usu. a variable) and the callers' names for a couple of stack frames. Like a little debug receipt ๐งพ
spack.build_systems.intel.raise_lib_error(*args)
Bails out with an error message. Shows args after the first as one per line, tab-indented โ super useful for long paths to line up and stand out. Aesthetic error messages, we love to see it ๐
๐ฆ spack.build_systems.lua module
class spack.build_systems.lua.LuaBuilder(pkg)
Bases: Builder
build_system: str | None = 'lua'โ Build system name. Must also be defined in derived classes.build_time_test_callbacks: List[str]generate_luarocks_config(pkg, spec, prefix)install(pkg, spec, prefix)install_time_test_callbacks: List[str]legacy_attributes: Tuple[str, ...] = ()โ Names associated with package attributes in the old build-system format.legacy_methods: Tuple[str, ...] = ('luarocks_args',)โ Names associated with package methods in the old build-system format.luarocks_args()โ the args handler, luarooocks style ๐ชจphases: Tuple[str, ...] = ('unpack', 'generate_luarocks_config', 'preprocess', 'install')โ Sequence of phases. Must be defined in derived classes.preprocess(pkg, spec, prefix)โ Override this to preprocess source before building with luarocks. Do it, bestie โจrun_after_callbacks = []run_before_callbacks = []setup_build_environment(env)โ Sets up the build environment for a package. This method will be called before the current package prefix exists in Spack's store.
Parameters:
env(spack.util.environment.EnvironmentModifications) โ environment modifications to be applied when the package is built. Package authors can call methods on it to alter the build environment.
unpack(pkg, spec, prefix)
class spack.build_systems.lua.LuaPackage(spec)
Bases: PackageBase
Specialized class for lua packages ๐ (it's giving niche but make it fashion)
build_system_class = 'LuaPackage'โ This attribute is used in UI queries that need to know the build system base class.conflicts:Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies:Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute:Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ Store whether a given Spec source/binary should not be redistributed. Respect the boundaries, periodt ๐languages:Dict[spack.spec.Spec, Set[str]]legacy_buildsystem = 'lua'โ Legacy buildsystem attribute used to deserialize and install old specs.list_depth = 1โ Link depth to whichlist_urlshould be searched for new versions.- property
luaโ the lua interpreter bestie ๐ - property
luarocksโ the rocks themselves ๐ชจ patches:Dict[spack.spec.Spec, List[spack.patch.Patch]]provided:Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together:Dict[spack.spec.Spec, List[Set[str]]]requirements:Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []โ empty, calm, quiet ๐ฟrun_before_callbacks = []โ also empty, also calmspec:spack.spec.Specvariants:Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions:dict
๐ง spack.build_systems.makefile module
class spack.build_systems.makefile.MakefileBuilder(pkg)
Bases: BaseBuilder โ lowkey the most straightforward build system bestie and we respect her for it ๐
okay so the Makefile builder is literally the blueprint for the most common way software gets built with Makefiles, no cap. It's got three phases you can override if you really need to:
edit()build()install()
Most of the time you're gonna override the edit() phase (which, fun fact, is a no-op by default), and the other two come with sensible defaults already. Slay.
For finer tuning you can also override:
build_targetsโ specify make targets for the build phaseinstall_targetsโ specify make targets for the install phasebuild_directory()โ directory where the Makefile is locatedbuild(pkg, spec, prefix)โ run"make"on the build targets specified by the builder.build_directory(property) โ return the directory containing the main Makefile.build_system: str | None = 'makefile'โ build system name. Must also be defined in derived classes.build_targets: List[str] = []โ targets for make during thebuild()phase.build_time_test_callbacks: List[str] = ['check']โ callback names for build-time test.check()โ run"make"on the test and check targets, if found.edit(pkg, spec, prefix)โ edit the Makefile before calling make. The default is a no-op.install(pkg, spec, prefix)โ run"make"on the install targets specified by the builder.install_targets = ['install']โ targets for make during theinstall()phase.install_time_test_callbacks: List[str] = ['installcheck']โ callback names for install-time test.installcheck()โ searches the Makefile for aninstallchecktarget and runs it if found.legacy_attributes: Tuple[str, ...] = ('build_targets', 'install_targets', 'build_time_test_callbacks', 'install_time_test_callbacks', 'build_directory')โ names associated with package attributes in the old build-system format.legacy_methods: Tuple[str, ...] = ('check', 'installcheck')โ names associated with package methods in the old build-system format.phases: Tuple[str, ...] = ('edit', 'build', 'install')โ sequence of phases. Must be defined in derived classes.run_after_callbacksโ the whole roster, verbatim:
[(('build', None), <function execute_build_time_tests>), (('install', None), <function execute_install_time_tests>), (('install', 'platform=darwin'), <function apply_macos_rpath_fixups>), (('install', None), <function sanity_check_prefix>)]run_before_callbacks = []โ nothing here, bestie. Empty list, iconic.
๐ธ class spack.build_systems.makefile.MakefilePackage(spec)
Bases: PackageBase
A specialized class for packages built using Makefiles, periodt.
build_system_class = 'MakefilePackage'โ this attribute is used in UI queries that need to know the build system base class.conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a given Spec source/binary should not be redistributed.languages: Dict[spack.spec.Spec, Set[str]]legacy_buildsystem = 'makefile'โ legacy buildsystem attribute used to deserialize and install old specs.patches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[โspack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dict
โจ spack.build_systems.maven module โจ
๐ฆ class spack.build_systems.maven.MavenBuilder(pkg)
Bases: BaseBuilder
The Maven builder encodes the default way to build software with Maven, and honestly it's giving efficiency. It has two phases that can be overridden, if need be:
build()install()
build(pkg, spec, prefix)โ compile code and package into a JAR file. cute and functional.build_args()โ list of args to pass to the build phase.build_directory(property) โ the directory containing thepom.xmlfile.build_system: str | None = 'maven'โ build system name. Must also be defined in derived classes.build_time_test_callbacks: List[str]install(pkg, spec, prefix)โ copy to installation prefix.install_time_test_callbacks: List[str]legacy_attributes: Tuple[str, ...] = ('build_directory',)โ names associated with package attributes in the old build-system format.legacy_methods: Tuple[str, ...] = ('build_args',)โ names associated with package methods in the old build-system format.phases: Tuple[str, ...] = ('build', 'install')โ sequence of phases. Must be defined in derived classes.run_after_callbacks = [(('install', None), <function sanity_check_prefix>)]run_before_callbacks = []โ empty, as per usual.
๐ class spack.build_systems.maven.MavenPackage(spec)
Bases: PackageBase
A specialized class for packages that are built using the Maven build system. See https://maven.apache.org/index.html for more information โ go read it, bestie, it's lowkey helpful.
build_system_class = 'MavenPackage'conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a given Spec source/binary should not be redistributed.languages: Dict[spack.spec.Spec, Set[str]]legacy_buildsystem = 'maven'โ legacy buildsystem attribute used to deserialize and install old specs.patches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[โspack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dict
๐ง spack.build_systems.meson module ๐ง
๐ class spack.build_systems.meson.MesonBuilder(pkg)
Bases: BaseBuilder
The Meson builder encodes the default way to build software with Meson. The builder has three phases that can be overridden, if need be:
meson()build()install()
They all have sensible defaults, and for many packages the only thing necessary will be to override meson_args(). Lazy-girl build config, we love to see it.
For a finer tuning you may also override:
root_mesonlists_dir()โ location of the rootMesonLists.txtbuild_directory()โ directory where to build the packagearchive_files(property) โ files to archive for packages based on Meson.build(pkg, spec, prefix)โ make the build targets.build_directory(property) โ directory to use when building the package.build_dirname(property) โ returns the directory name to use when building the package.build_system: str | None = 'meson'โ build system name. Must also be defined in derived classes.build_targets: List[str] = []build_time_test_callbacks: List[str] = ['check']check()โ search Meson-generated files for the target test and run it if found.install(pkg, spec, prefix)โ make the install targets.install_targets = ['install']legacy_attributes: Tuple[str, ...] = ('build_targets', 'install_targets', 'build_time_test_callbacks', 'root_mesonlists_dir', 'std_meson_args', 'build_directory')โ these are the names that were package attributes back in the old build-system format, iykyk ๐legacy_methods: Tuple[str, ...] = ('meson_args', 'check')โ and these are the names that were package methods in the old build-system format โจmeson(pkg, spec, prefix)โ runsmesoninside the build directory ๐ ๏ธmeson_args()โ the list of arguments that must be passed tomesonโฆ except for these icons right here, because spack handles them for you:--prefix--libdir--buildtype--strip--default_library
those five get set automatically, no cap ๐
phases: Tuple[str, ...] = ('meson', 'build', 'install')โ the sequence of phases. Must be defined in derived classes, fr fr ๐ธ- property
root_mesonlists_dirโ the relative path to the directory that containsmeson.build๐ฆ This path is relative to the root of the extracted tarball, not to thebuild_directory. Defaults to the current directory. run_after_callbacks = [(('build', None), <function execute_build_time_tests>), (('install', None), <function sanity_check_prefix>)]โ literally just vibes and callbacks โจrun_before_callbacks = []โ empty, nothing to see here ๐- static
std_args(pkg)โ the standardmesonarguments for a generic package ๐ง - property
std_meson_argsโ the standardmesonarguments, handed to you as a property purely for the convenience of package writers ๐ซถ
๐ class spack.build_systems.meson.MesonPackage(spec)
Bases: PackageBase
The specialized class for packages built with Meson ๐ If you want more info on the Meson build system, the docs live at https://mesonbuild.com/ โ go treat yourself ๐
build_system_class = 'MesonPackage'โ this attribute is used in UI queries that need to know the build system base class โญconflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ stores whether a givenSpecsource/binary should not be redistributed ๐ซflags_to_build_system_args(flags)โ produces a list of all the command line arguments needed to pass the specified compiler flags tomeson๐ฅlanguages: Dict[spack.spec.Spec, Set[str]]legacy_buildsystem = 'meson'โ the legacy buildsystem attribute, used to deserialize and install old specs ๐ฐ๏ธpatches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dict
๐ช spack.build_systems.msbuild module
๐ class spack.build_systems.msbuild.MSBuildBuilder(pkg)
Bases: BaseBuilder
Okay so the MSBuild builder encodes the most common way of building software with Microsoft's MSBuild tool ๐ ๏ธ It has two phases, and you can override both of them if you need to:
build()install()
Highkey it's usually necessary to override the install() phase, because so many packages with MSBuild systems neglect to provide an install target ๐ญ The default install phase will attempt to invoke an install target from MSBuild โ and if none exists, that results in a build failure ๐
For finer tuning you may override:
build_targetsโ specify msbuild targets for the build phase ๐ฏinstall_targetsโ specify msbuild targets for the install phase ๐ฏbuild_directory()โ the directory where the projectsln/vcxprojis located ๐
Onwards, bestie:
build(pkg, spec, prefix)โ runs"msbuild"on the build targets specified by the builder ๐จ- property
build_directoryโ returns the directory that contains the MSBuild solution orvcxproj๐ build_system: str | None = 'msbuild'โ build system name. Must also be defined in derived classes โจbuild_targets: List[str] = []โ targets formakeduring thebuild()phase ๐define(msbuild_arg, value)โ yes queen, define it ๐define_targets(*targets)โ targets, plural, as a treat ๐install(pkg, spec, prefix)โ runs"msbuild"on the install targets specified by the builder. This isINSTALLby default ๐ซถinstall_targets: List[str] = ['INSTALL']โ targets formsbuildduring theinstall()phase โญmsbuild_args()โ define build arguments for MSBuild. This is an empty list by default ๐ Individual packages should override it to specify MSBuild args for the command line.PlatformToolsetis already defined and can be controlled via thetoolchain_versionproperty ๐งmsbuild_install_args()โ define install arguments for MSBuild outside of theINSTALLtarget. This is the same asmsbuild_argsby default ๐ฏphases: Tuple[str, ...] = ('build', 'install')โ the sequence of phases. Must be defined in derived classes ๐ธrun_after_callbacks = [(('install', None), <function sanity_check_prefix>)]โ the install-phase vibe check, literally โจrun_before_callbacks = []โ empty ๐- property
std_msbuild_argsโ returns the commonmsbuildcl arguments โ for now it's just the toolchain ๐งฐ - property
toolchain_versionโ returns the currently targeted version of the MSVC toolchain ๐ Override this method to select a specific version of the toolchain, or to change the selection heuristics. The default is whatever version ofmsvchas been selected by concretization ๐
๐ class spack.build_systems.msbuild.MSBuildPackage(spec)
Bases: PackageBase
The specialized class for packages built with Visual Studio project files or solutions ๐ชโจ
build_system_class = 'MSBuildPackage'โ this attribute is used in UI queries that need to know the build system base class โญconflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ stores whether a givenSpecsource/binary should not be redistributed ๐ซlanguages: Dict[spack.spec.Spec, Set[str]]patches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dict
๐ธ spack.build_systems.nmake module
๐ง class spack.build_systems.nmake.NMakeBuilder(pkg)
Bases: BaseBuilder
So the NMake builder encodes the most common way of building software with Microsoft's NMake tool ๐ It has two phases that can be overridden, if need be:
build()install()
It is usually necessary to override the install() phase, because many packages with NMake systems neglect to provide an install target ๐ญ The default install phase will attempt to invoke an install target from NMake. If none exists, this will result in a build failure ๐
For a finer tuning you may override:
build_targetsโ specify nmake targets for the build phase ๐ฏinstall_targetsโ specify nmake targets for the install phase ๐ฏbuild_directory()โ the directory where the project makefile is located ๐build(pkg, spec, prefix)โ runs"nmake"on whatever build targets the builder told it about. that's it, that's the job ๐property build_directoryโ hands you back the directory that's holding the makefile. lowkey the most useful property here fr.build_system: str | None = 'nmake'โ the build system's name. heads up bestie: this must also be defined in derived classes, no skipping โจbuild_targets: List[str] = []โ the targets for make during thebuild()phase. empty list by default, your move ๐define(nmake_arg, value)โ lil helper that formats arguments for the nmake command line. it's giving "let me handle the syntax for you" ๐ซถproperty ignore_quotesโ controls whether Spack warns you about quoted arguments passed to build utilities. if this isTrue, spack will not warn about quotes. super useful when there's a space in the path, or when build scripts require quoted arguments. genuinely a lifesaver in those cases ๐ญinstall(pkg, spec, prefix)โ runs"nmake"on the install targets the builder specified. this isINSTALLby default, iconic behaviour ๐install_targets: List[str] = ['INSTALL']โ the targets for make during theinstall()phase. yes, the default is literallyINSTALL, periodt.property makefile_nameโ name of the current makefile. rn this is an empty value. but if a project defines it, it gets used with the/fargument so nmake gets an explicit makefile. this is useful when there are multiple nmake files chilling in the same directory ๐๏ธproperty makefile_rootโ the relative path to the directory containing the nmake makefile. important detail: this path is relative to the root of the extracted tarball, not to thebuild_directory. defaults to the current directory. don't mix that up bestie ๐nmake_args()โ define build arguments to NMake. empty list by default. individual packages should override this to specify NMake args to the command line ๐ธnmake_install_args()โ define arguments that are appropriate only for the install phase to NMake. empty list by default. individual packages should override to specify NMake args to the command line โจoverride_env(var_name, new_value)โ helper method that formats arguments for overriding env variables on the nmake command line. returns a properly formatted argument ๐งphases: Tuple[str, ...] = ('build', 'install')โ the sequence of phases. must be defined in derived classes, no cap.run_after_callbacks = [(('install', None), <function sanity_check_prefix>)]โ yep, that's the actual value.sanity_check_prefixruns afterinstall, as one does.run_before_callbacks = []โ nothing here, it's empty, moving on ๐ฆproperty std_nmake_argsโ returns the list of standard arguments handed to NMake. currently that's only/NOLOGO. short and sweet.
class spack.build_systems.nmake.NMakePackage(spec)
Bases: PackageBase
specialized class for packages built using a Makefiles. (yes, that's what it says ๐)
build_system_class = 'NMakePackage'โ this attribute is used in UI queries that need to know the build system base class. it's giving main character metadata ๐conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ stores whether a given Spec source/binary should not be redistributed. respect the boundaries ๐ซlanguages: Dict[spack.spec.Spec, Set[str]]patches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dict
๐ spack.build_systems.octave module
class spack.build_systems.octave.OctaveBuilder(pkg)
Bases: BaseBuilder
the octave builder provides the following phases that can be overridden:
install()
okay so one phase, but she's doing her best ๐
build_system: str | None = 'octave'โ build system name. must also be defined in derived classes, same rule as always โจbuild_time_test_callbacks: List[str]install(pkg, spec, prefix)โ install the package from the archive file. that's the whole phase, slay ๐ฆinstall_time_test_callbacks: List[str]legacy_attributes: Tuple[str, ...] = ()โ names associated with package attributes in the old build-system format. legacy stuff, we keep it around ๐ซถlegacy_methods: Tuple[str, ...] = ()โ names associated with package methods in the old build-system format. same vibe.phases: Tuple[str, ...] = ('install',)โ sequence of phases. must be defined in derived classes. just the one, iconique.run_after_callbacks = [(('install', None), <function sanity_check_prefix>)]run_before_callbacks = []setup_build_environment(env)โ sets up the build environment for a package. this method gets called before the current package prefix exists in Spack's store. Parameters:env(spack.util.environment.EnvironmentModifications) โ the environment modifications to be applied when the package is built. package authors can call methods on it to alter the build environment. so much power in your hands ngl ๐
class spack.build_systems.octave.OctavePackage(spec)
Bases: PackageBase
specialized class for Octave packages. see https://www.gnu.org/software/octave/doc/v4.2.0/Installing-and-Removing-Packages.html for more info, iykyk ๐
build_system_class = 'OctavePackage'conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a given Spec source/binary should not be redistributed ๐ซlanguages: Dict[spack.spec.Spec, Set[str]]legacy_buildsystem = 'octave'โ legacy buildsystem attribute used to deserialize and install old specs. yes, "legacy", we're honouring our elders ๐patches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dict
๐ง spack.build_systems.oneapi module
common utilities for managing intel oneapi packages. very "we have a whole module for this" of them ๐
spack.build_systems.oneapi.INTEL_MATH_LIBRARIES = ('intel-mkl', 'intel-oneapi-mkl', 'intel-parallel-studio') โ a tuple of Intel math libraries, exported to packages. three of them, named exactly like that, no cap โจ
class spack.build_systems.oneapi.IntelOneApiLibraryPackage(spec)
Bases: IntelOneApiPackage
base class for Intel oneAPI library packages. contains some convenient default implementations for libraries. if something different is needed, implement the method directly in the package. your package, your rules ๐ซถ
conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a given Spec source/binary should not be redistributed.header_directories(dirs)property headerslanguages: Dict[spack.spec.Spec, Set[str]]property libsopenmp_libs()โ supply aLibraryListfor linking OpenMP. that's the whole vibe ๐ฅpatches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dict
class spack.build_systems.oneapi.IntelOneApiLibraryPackageWithSdk(spec)
Bases: IntelOneApiPackage
base class for Intel oneAPI library packages with SDK components. contains some convenient default implementations for libraries that expose functionality in sdk subdirectories. if something different is needed, implement the method directly in the package. same energy, extra sdk ๐
conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a given Spec source/binary should not be redistributed.property headerslanguages: Dict[spack.spec.Spec, Set[str]]property libspatches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dict
notice howheader_directories(dirs)andopenmp_libs()show up in the plain library class but not in theWithSdkone? yeah, that's just how it is, don't come for me, i'm only the messenger ๐
class spack.build_systems.oneapi.IntelOneApiPackage(spec)
Bases: Package
base class for Intel oneAPI packages. the mothership ๐
c = 'platform=windows:'โ yep. literally that. that's a real line and i will not be elaborating ๐property component_dirโ the subdirectory for this component in the install prefix. know your place, component ๐๏ธproperty component_prefixโ path to component<prefix>/<component>/<version>. there it is, the whole shape of it, precious ๐conflictsโDict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]๐dependenciesโDict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistributeโDict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ literally stores whether a givenSpecsource/binary should not be redistributed. no cap, it remembers that for you ๐ซถenv_script_args(property) โ extra arguments to pass to thevars.shscript โจhomepageโstr | None = 'https://software.intel.com/oneapi'โ the package homepage where users can find more info about the package ๐ฆinstall(spec, prefix)โ the install moment, iconic as ever ๐install_component(installer_path)โ shared install method for all oneapi packages. like, every single one, same method, periodt ๐languagesโDict[spack.spec.Spec, Set[str]]patchesโDict[spack.spec.Spec, List[spack.patch.Patch]]providedโDict[spack.spec.Spec, Set[spack.spec.Spec]]provided_togetherโDict[spack.spec.Spec, List[Set[str]]]requirementsโDict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []โ empty, bestie. nothing queued up ๐run_before_callbacks = []โ also empty ๐setup_run_environment(env)โ adds environment variables to the generated module file ๐ธ
these env vars come from running:
$ source {prefix}/{component}/{version}/env/vars.shiykyk, it's sourcing the vars.sh script, that's the whole vibe โญ
spec: spack.spec.Specโ the spec, obviously ๐symlink_dir(src, dest)โ symlinks a dir fromsrctodest๐งstatic update_description(cls)โ updates oneapi package descriptions with common text โจv2_layout(property) โ returns true if this version implements the v2 directory layout ๐ฅv2_layout_versions(property) โ the versions that implement the v2 directory layout ๐variantsโDict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versionsโdict
๐ class spack.build_systems.oneapi.IntelOneApiStaticLibraryList(static_libs, dynamic_libs)
Bases: object
provides ld_flags when static linking is needed ๐ซถ
okay so here's the situation: oneapi puts static and dynamic libraries in the same directory, so -l will default to finding the dynamic library. the fix? use absolute paths, as recommended by oneapi documentation. ngl that's just how it is ๐ญ
this lets both static and dynamic libraries be supplied by the package. iconic flexibility โจ
directories(property) โ the directories, as promised ๐ld_flags(property) โ the ld flags ๐link_flags(property) โ the link flags ๐search_flags(property) โ the search flags ๐
๐ช spack.build_systems.perl module
class spack.build_systems.perl.PerlBuilder(pkg)
Bases: BaseBuilder
the perl builder provides four phases that can be overridden, if required:
configure()build()check()install()
the default methods use, in order of preference:
Makefile.PL,Build.PL.
some packages may need to override configure_args(), which produces a list of arguments for configure(). ๐ฆ
heads up bestie: arguments should not include the installation base directory. fr fr, don't do it ๐
build(pkg, spec, prefix)โ builds a Perl package ๐ธbuild_executable(property) โ returns the executable method to build the perl package โจbuild_method(property) โ searches the package for either aMakefile.PLorBuild.PL๐
note: raisesRuntimeErrorโ if neitherMakefile.PLnorBuild.PLexist ๐ญ like, there's literally nothing to build from
build_system: str | None = 'perl'โ build system name. must also be defined in derived classes ๐build_time_test_callbacks: List[str] = ['check']โ callback names for build-time test ๐check()โ runs built-in tests of a Perl package ๐งconfigure(pkg, spec, prefix)โ runMakefile.PLorBuild.PLwith arguments consisting of an appropriate installation base directory followed by the list returned byconfigure_args(). lowkey elegant โญconfigure_args()โ list of arguments passed toconfigure(). ๐ซถ
arguments should not include the installation base directory, which is prepended automatically. (yes, automatically, so don't bother ๐)
fix_shebang()โ fixes up the shebang line, bestie ๐install(pkg, spec, prefix)โ installs a Perl package ๐install_time_test_callbacks: List[str]โ the install-time test callback names ๐legacy_attributes: Tuple[str, ...] = ()โ names associated with package attributes in the old build-system format ๐legacy_methods: Tuple[str, ...] = ('configure_args', 'check', 'test_use')โ names associated with package methods in the old build-system format ๐phases: Tuple[str, ...] = ('configure', 'build', 'install')โ phases of a Perl package ๐run_after_callbacksโ literally:
[(('configure', None), <function PerlBuilder.fix_shebang>), (('build', None), <function execute_build_time_tests>), (('install', None), <function sanity_check_prefix>)]run_before_callbacks = []โ nothing before, everything after, that's the vibe โจ
class spack.build_systems.perl.PerlPackage(spec)
Bases: PackageBase
specialized class for packages that are built using Perl ๐ชโจ
build_system_class = 'PerlPackage'โ this attribute is used in UI queries that need to know the build system base class ๐conflictsโDict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependenciesโDict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistributeโDict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a givenSpecsource/binary should not be redistributed ๐ซถlanguagesโDict[spack.spec.Spec, Set[str]]legacy_buildsystem = 'perl'โ legacy buildsystem attribute used to deserialize and install old specs ๐patchesโDict[spack.spec.Spec, List[spack.patch.Patch]]providedโDict[spack.spec.Spec, Set[spack.spec.Spec]]provided_togetherโDict[spack.spec.Spec, List[Set[str]]]requirementsโDict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []โ empty ๐run_before_callbacks = []โ also empty ๐ธskip_modules: Iterable[str](property) โ names of modules that should be skipped when running tests ๐ง
these are a subset of use_modules.
Returns: list of strings of module names. โจ
spec: spack.spec.Specโ the spec ๐test_use()โ tests'use module'๐use_modules: Iterable[str](property) โ names of the package's perl modules ๐variantsโDict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versionsโdict
๐ spack.build_systems.python module
class spack.build_systems.python.PythonExtension(spec)
Bases: PackageBase
add_files_to_view(view, merge_map, skip_if_exists=True)โ given a map of package files to destination paths in the view, add the files to the view. ๐
by default this adds all files. alternative implementations may skip some files, for example if other packages linked into the view already include the file. iykyk ๐ฆ
Parameters
view(spack.filesystem_view.FilesystemView) โ the view that's updatedmerge_map(dict) โ maps absolute source paths to absolute dest paths for all files in from this package.skip_if_exists(bool) โ whenTrue, don't link files in view when they already exist. whenFalse, always link files, without checking if they already exist. periodt โจ
conflictsโDict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependenciesโDict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistributeโDict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a givenSpecsource/binary should not be redistributed ๐ซถget_external_python_for_prefix()โ for an external package that extends python, find the most likely spec for the python it depends on ๐
first search: an "installed" external that shares a prefix with this package second search: a configured external that shares a prefix with this package third search: search this prefix for a python package
Returns: the external Spec for python most likely to be compatible with self.spec
Return type: spack.spec.Spec ๐
import_modules: Iterable[str](property) โ names of modules that the Python package provides ๐ฆ
these are used to test whether or not the installation succeeded. these names generally come from running:
>> import setuptools
>> setuptools.find_packages()in the source tarball directory. if the module names are incorrectly detected, this property can be overridden by the package. ngl that's a handy escape hatch ๐ญ
Returns: list of strings of module names. โญ
languagesโDict[spack.spec.Spec, Set[str]]patchesโDict[spack.spec.Spec, List[spack.patch.Patch]]providedโDict[spack.spec.Spec, Set[spack.spec.Spec]]provided_togetherโDict[spack.spec.Spec, List[Set[str]]]remove_files_from_view(view, merge_map)โ given a map of package files to files currently linked in the view, remove the files from the view. ๐งน
the default implementation removes all files. alternative implementations may not remove all files. for example if two packages include the same file, it should only be removed when both packages are removed. like, shared custody of a file, cute ๐
requirementsโDict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []โ empty, bestie ๐run_before_callbacks = []โ empty too ๐ธskip_modules: Iterable[str](property) โ names of modules that should be skipped when running tests.- these are a subset of
import_modules, bestie. And if a module has submodules, those get skipped too โ soa.bis skipped whenais already in there. iykyk ๐
Returns โ list of strings of module names.
spec: spack.spec.Specโ the spec, obviously ๐ซถtest_imports() -> Noneโ attempts to import the modules of the installed package. fingers crossed fr fr ๐update_external_dependencies(extendee_spec=None)โ makes sure every external python package has a python dependency. like, if another package in the DAG depends on python, we use that python for the dependency of the external. If there isn't one, we just assume the externalPythonPackageis installed into the same directory as the python it depends on. lowkey a vibe โจvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]โ yes queen, dict of variants ๐versions: dictโ it's a dict, that's the whole post ๐view_file_conflicts(view, merge_map)โ reports all the file conflicts, excepting special cases for python. specifically (and this is the important bit, don't @ me), it does not report errors for duplicate__init__.pyfiles for packages in the same namespace. iconic behavior honestly ๐ธ
๐ฆ class spack.build_systems.python.PythonPackage(spec)
Bases: PythonExtension
This is the specialized class for packages that are built using pip. she's the main character of the python build systems, periodt ๐
build_system_class = 'PythonPackage'โ literally the name of the build system class โจ- property
headers: HeaderListโ discovers header files inplatlib. go find them bestie ๐ homepage: str | None = Noneโ package homepage where users can find more information about the package. like a little linktree for your package ๐install_time_test_callbacks = ['test']โ callback names for install-time test ๐งชlegacy_buildsystem = 'python_pip'โ the legacy buildsystem attribute used to deserialize and install old specs. she's vintage, she's archival ๐- property
libs: LibraryListโ discovers libraries inplatlib. libraries found, slay ๐ list_url: str | None = Noneโ default list URL, aka the place to find available versions ๐py_namespace: str | None = Noneโ the python namespace, defaults to nothing ๐ซถpypi: str | None = Noneโ package name, version, and extension on PyPI ๐- property
python_specโ getspython-venvif it exists, orpythonotherwise. whichever one's available, bestie ๐ run_after_callbacks = []โ empty list, no callbacks after ๐run_before_callbacks = []โ empty list, no callbacks before ๐url = Noneโ no url set here, nope โญ
๐ฅ class spack.build_systems.python.PythonPipBuilder(pkg)
Bases: BaseBuilder
- property
build_directory: strโ the root directory of the Python package. this is usually the directory containing one of the following files:pyproject.tomlsetup.cfgsetup.py
find the file, find the directory, that's the vibe check ๐โจ
build_system: str | None = 'python_pip'โ build system name. it must also be defined in derived classes, no skipping ๐build_time_test_callbacks: List[str]โ a list of strings, callback names, you know the drill ๐งconfig_settings(spec: Spec, prefix: Prefix) -> Mapping[str, object]โ configuration settings to be passed to the PEP 517 build backend. heads up though: it requirespip22.1 or newer for keys that appear only a single time, orpip23.1 or newer if the same key appears multiple times. so check your pip version, bestie, seriously ๐ญ
Parameters:
specโ build spec.prefixโ installation prefix.
Returns โ possibly nested dictionary of KEY, VALUE settings. nested dicts, we love a structure ๐ซถ
global_options(spec: Spec, prefix: Prefix) -> Iterable[str]โ extra global options to be supplied to thesetup.pycall before theinstallorbdist_wheelcommand. but ngl, deprecated in pip 23.1 โ she's aging out, pour one out ๐
Parameters:
specโ build spec.prefixโ installation prefix.
Returns โ list of options.
install(pkg: PythonPackage, spec: Spec, prefix: Prefix) -> Noneโ installs everything from the build directory. that's the whole job, and she ate ๐ฅinstall_options(spec: Spec, prefix: Prefix) -> Iterable[str]โ extra arguments to be supplied to thesetup.py installcommand. โ ๏ธ requirespip23.0 or older, so don't come at her with a fresh pip ๐ธ
Parameters:
specโ build spec.prefixโ installation prefix.
Returns โ list of options.
install_time_test_callbacks: List[str] = ['test']โ callback names for install-time test, again ๐งชlegacy_attributes: Tuple[str, ...] = ('archive_files', 'build_directory', 'install_time_test_callbacks')โ names associated with package attributes in the old build-system format. the legacy trio, we respect them ๐legacy_long_methods = ('install_options', 'global_options', 'config_settings')โ same aslegacy_methods, but the signature is different. plot twist ๐ณlegacy_methods: Tuple[str, ...] = ('test',)โ names associated with package methods in the old build-system format. it's justtest, and honestly? same ๐phases: Tuple[str, ...] = ('install',)โ sequence of phases. it must be defined in derived classes, no like actually ๐ซถrun_after_callbacks = [(('install', None), <function execute_install_time_tests>), (('install', None), <function sanity_check_prefix>)]โ afterinstallruns the install-time tests, then the sanity check on the prefix. two for one, we love efficiency โจrun_before_callbacks = []โ nothing before, empty list ๐static std_args(cls) -> List[str]โ the standard args, as a list of strings ๐
๐ง spack.build_systems.qmake module
๐ class spack.build_systems.qmake.QMakeBuilder(pkg)
Bases: BaseBuilder
The qmake builder provides three phases that can be overridden, bestie, and here they are in order:
qmake()build()install()
They all have sensible defaults, and for many packages the only thing necessary will be to override qmake_args(). literally just one override and you're done, she's low-maintenance like that ๐
build(pkg, spec, prefix)โ makes the build targets. go build, go ๐โโ๏ธ- property
build_directoryโ the directory containing the*.profile. cute little pro file ๐ธ build_system: str | None = 'qmake'โ build system name. must also be defined in derived classes ๐build_time_test_callbacks: List[str] = ['check']โ callback names for build-time test ๐งชcheck()โ searches theMakefilefor achecktarget and runs it if found. if there's a check target, she checks. simple as ๐ซถinstall(pkg, spec, prefix)โ makes the install targets ๐ฆinstall_time_test_callbacks: List[str]โ list of strings for the install-time test callbacks ๐legacy_attributes: Tuple[str, ...] = ('build_directory', 'build_time_test_callbacks')โ names associated with package attributes in the old build-system format. legacy duo, still iconic ๐legacy_methods: Tuple[str, ...] = ('qmake_args', 'check')โ names associated with package methods in the old build-system format ๐phases: Tuple[str, ...] = ('qmake', 'build', 'install')โ sequence of phases. must be defined in derived classes, not optional bestie ๐ฆqmake(pkg, spec, prefix)โ runs qmake to configure the project and generate aMakefile. configure + generate, the whole pregame โจqmake_args()โ list of arguments passed to qmake. all your args in one place ๐ซถrun_after_callbacks = [(('build', None), <function execute_build_time_tests>), (('install', None), <function sanity_check_prefix>)]โ build-time tests afterbuild, sanity check afterinstall. she's thorough ๐run_before_callbacks = []โ empty, nothing runs before ๐
๐ class spack.build_systems.qmake.QMakePackage(spec)
Bases: PackageBase
Specialized class for packages that are built using qmake. for more info on the qmake build system, go look at http://doc.qt.io/qt-5/qmake-manual.html โ the docs are right there bestie ๐
build_system_class = 'QMakePackage'โ this attribute is used in UI queries that need to know the build system base class. it's giving helpful metadata ๐conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]โ conflicts, all mapped out ๐dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]โ the dependencies dict ๐ซถdisable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ stores whether a givenSpecsource/binary should not be redistributed. important! respect the flag ๐ฅlanguages: Dict[spack.spec.Spec, Set[str]]โ languages, as a set of strings ๐ฃ๏ธlegacy_buildsystem = 'qmake'โ legacy buildsystem attribute used to deserialize and install old specs. throwback ๐patches: Dict[spack.spec.Spec, List[spack.patch.Patch]]โ the patches ๐ฉนprovided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]โ what's provided ๐provided_together: Dict[spack.spec.Spec, List[Set[str]]]โ provided together โ different thing, don't mix them up bestie ๐ซถrequirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]โ requirements, nested tuples and all ๐ญrun_after_callbacks = []โ empty ๐ธrun_before_callbacks = []โ empty ๐spec: spack.spec.Specโ the spec ๐variants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]โ variants โจversions: dictโ versions, it's a dict, we've been over this ๐
๐ spack.build_systems.r module
๐ฆ class spack.build_systems.r.RBuilder(pkg)
Bases: GenericBuilder
The R builder provides a single phase that can be overridden, and it's literally just:
install()
It has sensible defaults, and for many packages the only thing necessary will be to add dependencies. that's it, that's the whole build system, she's chill like that ๐
configure_args()โ arguments to pass to install via--configure-args๐ซถconfigure_vars()โ arguments to pass to install via--configure-varsโจinstall(pkg, spec, prefix)โ installs an R package. do it, queen ๐ฆlegacy_methods: Tuple[str, ...] = ('configure_args', 'configure_vars')โ names associated with package methods in the old build-system format ๐run_after_callbacks = [(('install', 'platform=darwin'), <function apply_macos_rpath_fixups>), (('install', None), <function execute_install_time_tests>), (('install', None), <function sanity_check_prefix>)]โ okay so afterinstallonplatform=darwinwe apply the macOS rpath fixups, then the install-time tests, then the sanity check on the prefix. three callbacks, one phase, she's booked and busy ๐ฅrun_before_callbacks = []โ empty, nothing before ๐
๐ class spack.build_systems.r.RPackage(spec)
Bases: Package
Specialized class for packages that are built using R. for more info on the R build system, check https://stat.ethz.ch/R-manual/R-devel/library/utils/html/INSTALL.html โ yes the URL is long, yes it's real ๐ธ
GenericBuilderโ alias ofRBuilder. same girl, different name ๐bioc: str | None = Noneโ the Bioconductor thing, defaults to nothing ๐งฌbuild_system_class = 'RPackage'โ this attribute is used in UI queries that need to know the build system base class. iconic metadata move ๐cran: str | None = Noneโ the CRAN thing, also defaults to nothing ๐ซถ- property
git homepage: str | None = Noneโ the package's homepage, aka the cute lil website where curious besties go to learn more about it ๐list_url: str | None = Noneโ the default list URL, aka the spot where you go hunting for which versions even exist ๐run_after_callbacks = []โ empty list, nothing hooked up here yet, it's giving "clean slate" โจrun_before_callbacks = []โ also empty, matching energy ๐ฆurl = Noneโ no url, no problem, very TBD of it ๐
๐ง spack.build_systems.racket module
๐ธ spack.build_systems.racket.RacketBuilder(pkg)
Bases: Builder
Okay so the Racket builder is serving exactly one phase, and bestie you can override it: install. That's the whole menu, and honestly? Iconic minimalism ๐
build_directory(property) โ the build directory, as a property, slaybuild_system: str | None = 'racket'โ build system name. Heads up: it must also be defined in derived classes, no skipping ๐build_time_test_callbacks: List[str] = ['check']โ callback names for build-time testinstall(pkg, spec, prefix)โ install everything from the build directory. That's it, that's the job ๐legacy_attributes: Tuple[str, ...] = ('build_directory', 'build_time_test_callbacks', 'subdirectory')โ names associated with package attributes in the old build-system formatlegacy_methods: Tuple[str, ...] = ()โ names associated with package methods in the old build-system format (empty, so, nothing)phases: Tuple[str, ...] = ('install',)โ sequence of phases. Must be defined in derived classes, fr frracket_name: str | None = Noneโ the racket name, currently vibing atNonerun_after_callbacks = []run_before_callbacks = []subdirectory(property) โ the subdirectory, as a property
๐ธ spack.build_systems.racket.RacketPackage(spec)
Bases: PackageBase
A specialized class for packages built with Racket's raco pkg install and raco setup commands. It's giving "I know exactly what I'm doing" ๐ซถ
build_system_class = 'RacketPackage'โ the build system class namehomepage: str | None = Noneโ package homepage where users can find more information about the packagelegacy_buildsystem = 'racket'โ legacy buildsystem attribute used to deserialize and install old specsparallel = Trueโ by default we build in parallel. Subclasses can override this ๐racket_name: str | None = Nonerun_after_callbacks = []run_before_callbacks = []
๐ spack.build_systems.rocm module
๐ spack.build_systems.rocm.ROCmPackage(spec)
Bases: PackageBase
The auxiliary class that holds the ROCm variant, dependencies and conflicts, and exists to unify and facilitate its usage. It closely mimics CudaPackage, bestie โ same DNA, different GPU ๐งฌ
Maintainers: dtaller
amdgpu_targetsโ the full roster of GPU targets, no cap:
('gfx701', 'gfx801', 'gfx802', 'gfx803', 'gfx900', 'gfx900:xnack-', 'gfx902', 'gfx904', 'gfx906', 'gfx906:xnack-', 'gfx908', 'gfx908:xnack-', 'gfx909', 'gfx90a', 'gfx90a:xnack-', 'gfx90a:xnack+', 'gfx90c', 'gfx940', 'gfx941', 'gfx942', 'gfx1010', 'gfx1011', 'gfx1012', 'gfx1013', 'gfx1030', 'gfx1031', 'gfx1032', 'gfx1033', 'gfx1034', 'gfx1035', 'gfx1036', 'gfx1100', 'gfx1101', 'gfx1102', 'gfx1103')asan_on(env: EnvironmentModifications)โ the asan hook, takingenv: EnvironmentModificationsconflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]โ the conflicts dictdependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]โ the dependencies dictdisable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a given Spec source/binary should not be redistributed ๐ฅstatic hip_flags(amdgpu_target)โ static method, takesamdgpu_targetlanguages: Dict[spack.spec.Spec, Set[str]]โ the languages dictpatches: Dict[spack.spec.Spec, List[spack.patch.Patch]]โ the patches dictprovided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]โ the provided dictprovided_together: Dict[spack.spec.Spec, List[Set[str]]]โ the provided-together dictrequirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]โ the requirements dictrun_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specโ the spec, obviouslyvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]โ the variants dictversions: dictโ the versions, as a plain olddict
๐ฆ spack.build_systems.ruby module
๐ธ spack.build_systems.ruby.RubyBuilder(pkg)
Bases: BaseBuilder
The Ruby builder is serving two phases, and you can override either of them if you need to:
build()install()
build(pkg, spec, prefix)โ build a Ruby gem ๐build_system: str | None = 'ruby'โ build system name. Must also be defined in derived classes, lowkey importantbuild_time_test_callbacks: List[str]โ callback names for build-time testinstall(pkg, spec, prefix)โ install a Ruby gem. The ruby package setsGEM_HOMEto tell gem where to install to ๐install_time_test_callbacks: List[str]โ callback names for install-time testlegacy_attributes: Tuple[str, ...] = ()โ names associated with package attributes in the old build-system formatlegacy_methods: Tuple[str, ...] = ()โ names associated with package methods in the old build-system formatphases: Tuple[str, ...] = ('build', 'install')โ sequence of phases. Must be defined in derived classesrun_after_callbacks = [(('install', None), <function sanity_check_prefix>)]โ yes, thesanity_check_prefixfunction runs afterinstall, keeping us honest โจrun_before_callbacks = []
๐ธ spack.build_systems.ruby.RubyPackage(spec)
Bases: PackageBase
A specialized class for building Ruby gems. Very demure, very gem ๐
build_system_class = 'RubyPackage'โ this attribute is used in UI queries that need to know the build system base classconflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]โ the conflicts dictdependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]โ the dependencies dictdisable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a given Spec source/binary should not be redistributedlanguages: Dict[spack.spec.Spec, Set[str]]โ the languages dictlegacy_buildsystem = 'ruby'โ legacy buildsystem attribute used to deserialize and install old specspatches: Dict[spack.spec.Spec, List[spack.patch.Patch]]โ the patches dictprovided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]โ the provided dictprovided_together: Dict[spack.spec.Spec, List[Set[str]]]โ the provided-together dictrequirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]โ the requirements dictrun_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specโ the specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]โ the variants dictversions: dictโ the versions
๐ฅ spack.build_systems.scons module
๐ spack.build_systems.scons.SConsBuilder(pkg)
Bases: BaseBuilder
The SCons builder provides these phases, and yes you can override them:
build()install()
Real talk though: packages that use SCons as a build system are less uniform than packages using other build systems ๐ญ Developers can add custom subcommands or variables that control the build, so you will likely need to override build_args() to pass the appropriate variables. That's the whole tea โ
build(pkg, spec, prefix)โ build the packagebuild_args(spec, prefix)โ arguments to pass to buildbuild_system: str | None = 'scons'โ build system name. Must also be defined in derived classesbuild_test()โ run unit tests after build. By default, does nothing. Override this if you want to add package-specific tests, bestie ๐ซถbuild_time_test_callbacks: List[str] = ['build_test']โ callback names for build-time testinstall(pkg, spec, prefix)โ install the packageinstall_args(spec, prefix)โ arguments to pass to installinstall_time_test_callbacks: List[str]โ callback names for install-time testlegacy_attributes: Tuple[str, ...] = ('build_time_test_callbacks',)โ names associated with package attributes in the old build-system formatlegacy_long_methods = ('build_args', 'install_args')โ same aslegacy_methods, but the signature is different ๐legacy_methods: Tuple[str, ...] = ('build_test',)โ names associated with package methods in the old build-system formatphases: Tuple[str, ...] = ('build', 'install')โ phases of a SCons packagerun_after_callbacks = [(('build', None), <function execute_build_time_tests>), (('install', None), <function sanity_check_prefix>)]โ build-time tests afterbuild, prefix sanity check afterinstall, we love a routine โจrun_before_callbacks = []
๐ spack.build_systems.scons.SConsPackage(spec)
Bases: PackageBase
A specialized class for packages built using SCons. See http://scons.org/documentation.html for more information ๐
build_system_class = 'SConsPackage'โ to be used in UI queries that require to know which build-system class we are usingconflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]โ the conflicts dictdependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]โ the dependencies dictdisable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ store whether a given Spec source/binary should not be redistributedlanguages: Dict[spack.spec.Spec, Set[str]]โ the languages dictlegacy_buildsystem = 'scons'โ legacy buildsystem attribute used to deserialize and install old specspatches: Dict[spack.spec.Spec, List[spack.patch.Patch]]โ the patches dictprovided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]โ the provided dictprovided_together: Dict[spack.spec.Spec, List[Set[str]]]โ the provided-together dictrequirements: Dict[spack.spec.Spec, List[Tuple[Tuple[spack.spec.Spec, ...], str, str | None]]]โ the requirements dictrun_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specโ the specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]โ the variants dictversions: dictโ the versions
๐
spack.build_systems.sip module
๐ง spack.build_systems.sip.SIPBuilder(pkg)
Bases: BaseBuilder
The SIP builder provides the following phases that can be overridden:
configurebuildinstall
The configure phase already adds a whole set of default flags for you, no cap. Want to see more options? Just run sip-build --help ๐
build(pkg, spec, prefix)โ builds the package. that's it, that's the job โจbuild_args()โ the arguments you pass to build ๐ซถbuild_directory = 'build'โ literally the build directory, it's'build'build_system: str | None = 'sip'โ the build system name ๐ and yes it must also be defined in derived classes, don't skip thatbuild_time_test_callbacks: List[str]โ the callback list, v importantconfigure(pkg, spec, prefix)โ configures the packageconfigure_args()โ arguments to pass to configureinstall(pkg, spec, prefix)โ installs the package, slayinstall_args()โ arguments to pass to installinstall_time_test_callbacks: List[str]โ more callbacks, love that for uslegacy_attributes: Tuple[str, ...] = ('build_targets', 'install_targets', 'build_time_test_callbacks', 'install_time_test_callbacks', 'build_directory')โ the names associated with package attributes in the old build-system format ๐legacy_methods: Tuple[str, ...] = ('configure_args', 'build_args', 'install_args')โ names associated with package methods in the old build-system formatphases: Tuple[str, ...] = ('configure', 'build', 'install')โ the sequence of phases, and yeah it must be defined in derived classesrun_after_callbacks = [(('install', None), <function execute_install_time_tests>), (('install', None), <function sanity_check_prefix>)]run_before_callbacks = []
โจ class spack.build_systems.sip.SIPPackage(spec) โจ
Bases: PackageBase
Okay so this is the specialized class for packages that get built with the SIP build system ๐ See https://www.riverbankcomputing.com/software/sip/intro for the whole rundown.
build_system_class = 'SIPPackage'conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ stores whether a given Spec source/binary should not be redistributed ๐property import_modulesโ the names of modules that the Python package provides ๐ These are used to test whether or not the installation succeeded. These names generally come from running:
>> import setuptools
>> setuptools.find_packages()in the source tarball directory. If the module names are incorrectly detected, this property can be overridden by the package.
Returns: list of strings of module names Return type: list
install_time_test_callbacks = ['test_imports']โ callback names for install-time testing โญlanguages: Dict[spack.spec.Spec, Set[str]]legacy_buildsystem = 'sip'โ legacy buildsystem attribute used to deserialize and install old specspatches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]python(*args, **kwargs)โ thepythonExecutable ๐requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[โspack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []sip_module = 'sip'โ name of the private sip module to install alongside the packagespec: spack.spec.Spectest_imports()โ attempts to import modules of the installed package ๐งชvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dict
๐ spack.build_systems.sourceforge module ๐
class spack.build_systems.sourceforge.SourceforgePackage(spec)
Bases: PackageBase
This mixin handles setting url and mirrors for Sourceforge packages, bestie ๐ฆ
base_mirrors = ['https://prdownloads.sourceforge.net/', 'https://freefr.dl.sourceforge.net/', 'https://netcologne.dl.sourceforge.net/', 'https://pilotfiber.dl.sourceforge.net/', 'https://downloads.sourceforge.net/', 'http://kent.dl.sourceforge.net/sourceforge/']โ the list of Sourceforge mirrors Spack uses ๐ซrun_after_callbacks = []run_before_callbacks = []sourceforge_mirror_path: str | None = Noneโ the path of the package in a Sourceforge mirrorproperty urls
๐ธ spack.build_systems.sourceware module ๐ธ
class spack.build_systems.sourceware.SourcewarePackage(spec)
Bases: PackageBase
Mixin that takes care of setting url and mirrors for Sourceware.org packages ๐
base_mirrors = ['https://sourceware.org/pub/', 'https://mirrors.kernel.org/sourceware/', 'https://ftp.gwdg.de/pub/linux/sources.redhat.com/']โ the list of Sourceware mirrors Spack usesrun_after_callbacks = []run_before_callbacks = []sourceware_mirror_path: str | None = Noneโ the path of the package in a Sourceware mirrorproperty urls
๐ฅ spack.build_systems.waf module ๐ฅ
class spack.build_systems.waf.WafBuilder(pkg)
Bases: BaseBuilder
The WAF builder gives you the following phases, and you can totally override them ๐
- configure
- build
- install
These are all standard Waf commands and you can find them by running:
$ python waf --helpEach phase provides a function <phase> that runs:
$ python waf -j<jobs> <phase>where <jobs> is the number of parallel jobs to build with โจ Each phase also has a <phase_args> function that can pass arguments to this call. All of these functions are empty except for the configure_args function โ that one passes --prefix=/path/to/installation/prefix ๐
build(pkg, spec, prefix)โ executes the buildbuild_args()โ arguments to pass to buildproperty build_directoryโ the directory containing the waf filebuild_system: str | None = 'waf'โ build system name, and it must also be defined in derived classes (iykyk)build_test()โ run unit tests after build ๐งช By default it does nothing. Override this if you want to add package-specific tests.build_time_test_callbacks: List[str] = ['build_test']configure(pkg, spec, prefix)โ configures the projectconfigure_args()โ arguments to pass to configureinstall(pkg, spec, prefix)โ installs the targets on the systeminstall_args()โ arguments to pass to installinstall_test()โ run unit tests after install ๐ By default it does nothing. Override this if you want to add package-specific tests.install_time_test_callbacks: List[str] = ['install_test']legacy_attributes: Tuple[str, ...] = ('build_time_test_callbacks', 'build_directory', 'install_time_test_callbacks')โ names associated with package attributes in the old build-system formatlegacy_methods: Tuple[str, ...] = ('build_test', 'install_test', 'configure_args', 'build_args', 'install_args', 'build_test', 'install_test')โ names associated with package methods in the old build-system format (yes,build_testandinstall_testshow up twice in there, that's the original, don't @ me ๐)phases: Tuple[str, ...] = ('configure', 'build', 'install')โ sequence of phases, must be defined in derived classespython(*args, **kwargs)โ thepythonExecutablerun_after_callbacks = [(('build', None), <function execute_build_time_tests>), (('install', None), <function execute_install_time_tests>), (('install', None), <function sanity_check_prefix>)]run_before_callbacks = []waf(*args, **kwargs)โ runs thewafExecutable ๐ ๏ธ
class spack.build_systems.waf.WafPackage(spec)
Bases: PackageBase
Specialized class for packages built using the Waf build system. See https://waf.io/book/ for the whole story ๐
build_system_class = 'WafPackage'conflicts: Dict[spack.spec.Spec, List[Tuple[spack.spec.Spec, str | None]]]dependencies: Dict[spack.spec.Spec, Dict[str, spack.dependency.Dependency]]disable_redistribute: Dict['spack.spec.Spec', 'spack.directives.DisableRedistribute']โ stores whether a given Spec source/binary should not be redistributed ๐languages: Dict[spack.spec.Spec, Set[str]]legacy_buildsystem = 'waf'โ legacy buildsystem attribute used to deserialize and install old specspatches: Dict[spack.spec.Spec, List[spack.patch.Patch]]provided: Dict[spack.spec.Spec, Set[spack.spec.Spec]]provided_together: Dict[spack.spec.Spec, List[Set[str]]]requirements: Dict[spack.spec.Spec, List[Tuple[Tuple[โspack.spec.Spec, ...], str, str | None]]]run_after_callbacks = []run_before_callbacks = []spec: spack.spec.Specvariants: Dict[str, Tuple[spack.variant.Variant, spack.spec.Spec]]versions: dict
๐ฆ spack.build_systems.xorg module ๐ฆ
class spack.build_systems.xorg.XorgPackage(spec)
Bases: PackageBase
Mixin that takes care of setting url and mirrors for x.org packages ๐
base_mirrors = ['https://www.x.org/archive/individual/', 'https://mirrors.ircam.fr/pub/x.org/individual/', 'https://mirror.transip.net/xorg/individual/', 'ftp://ftp.freedesktop.org/pub/xorg/individual/', 'http://xorg.mirrors.pair.com/individual/']โ the mirror list, iconicrun_after_callbacks = []run_before_callbacks = []property urlsxorg_mirror_path: str | None = Noneโ the path of the package in an x.org mirror
๐ spack.cmd package ๐
exception spack.cmd.CommandNameError(name)
Bases: SpackError
An exception class that gets thrown for impermissible command names ๐ญ like if the name just isn't allowed, this is what you get.
spack.cmd.PythonNameError(name)
Bases: SpackError
The exception that gets yeeted whenever somebody hands over a python name that is, uh, not it โ impermissible python names, straight up.
spack.cmd.all_commands()
Gives you a sorted list of literally every spack command. ๐
How does she do it? She goes and lists the lib/spack/spack/cmd directory and finds the commands chillin' in there to build the list. And the iconic part: she does not actually import those python files โ she just takes the names. Efficiency queen. โจ
spack.cmd.cmd_name(python_name)
Swaps the module name (the one with _) into a command name (the one with -). That's the whole gig.
spack.cmd.disambiguate_spec(spec, env, local=False, installed=True, first=False)
You hand her a spec, she figures out which installed package it's actually referring to. Detective mode. ๐
Parameters
spec(spack.spec.Spec) โ a spec to disambiguateenv(spack.environment.Environment) โ a spack environment, if one is active, orNoneif no environment is activelocal(bool) โ do not search chained spack instancesinstalled(boolorspack.database.InstallStatusorIterable) โ install status argument passed to database query. Seespack.database.Database._queryfor details.
spack.cmd.disambiguate_spec_from_hashes(spec, hashes, local=False, installed=True, first=False)
Same energy as above, except you also hand her a list of hashes โ she gives you back the concrete spec that your spec is pointing at. ๐ฏ
Parameters
spec(spack.spec.Spec) โ a spec to disambiguatehashes(Iterable) โ a set of hashes of specs among which to disambiguatelocal(bool) โ do not search chained spack instancesinstalled(boolorspack.database.InstallStatusorIterable) โ install status argument passed to database query. Seespack.database.Database._queryfor details.
spack.cmd.display_specs(specs, args=None, **kwargs)
Displays human-readable specs with formatting you can customize. It's giving aesthetic.
Prints the supplied specs to your screen, formatted according to the arguments you gave her.
Specs get grouped by architecture and compiler, and columnized if that's even possible.
Options can add more info to the default display. You can pass options either as keyword arguments or as an argparse namespace โ and ngl, keyword arguments take precedence over whatever's sitting in the argparse namespace. ๐
Parameters
specs(list) โ the specs to displayargs(argparse.NamespaceorNone) โ namespace containing formatting arguments
Keyword Arguments
paths(bool) โ Show paths with each displayed specdeps(bool) โ Display dependencies with specslong(bool) โ Display short hashes with specsvery_long(bool) โ Display full hashes with specs (supersedeslong)namespaces(bool) โ Print namespaces along with namesshow_flags(bool) โ Show compiler flags with specsvariants(bool) โ Show variants with specsindent(int) โ indent each line this muchgroups(bool) โ display specs grouped by arch/compiler (defaultTrue)decorator(Callable) โ function to call to decorate specsall_headers(bool) โ show headers even when arch/compiler aren't definedoutput(IO) โ A file object to write to. Default issys.stdout
spack.cmd.display_specs_as_json(specs, deps=False)
Converts specs into a list of json records. Machine-readable slay. ๐
spack.cmd.ensure_single_spec_or_die(spec, matching_specs)
No description in the source, bestie โ she's just there, doing her thing, existing. ๐ซถ
spack.cmd.extant_file(f)
An argparse type for files that actually exist. Like, exist exist. ๐
spack.cmd.filter_loaded_specs(specs)
Filters a list of specs and returns only the ones that are currently loaded. If you're not loaded, you're not invited. ๐
spack.cmd.find_environment(args)
Finds the active environment from your args or from an environment variable.
She checks for an environment in this exact order:
- via
spack -e ENVorspack -D DIR(arguments) - via a path in the
spack.environment.spack_env_varenvironment variable.
If an environment is found, she reads it in. If not, you get None. No environment, no problem (kinda). โจ
Parameters args (argparse.Namespace) โ argparse namespace with command arguments
Returns a found environment, or None
Return type (spack.environment.Environment)
spack.cmd.first_line(docstring)
Returns the first line of the docstring. Just the first one. Don't push it. ๐
spack.cmd.get_command(cmd_name)
Imports the command function associated with cmd_name.
The function's name is derived from cmd_name using python_name().
Parameters cmd_name (str) โ name of the command (contains -, not _).
spack.cmd.get_module(cmd_name)
Imports the module for a particular command name and hands it back to you.
Parameters cmd_name (str) โ name of the command for which to get a module (contains -, not _).
spack.cmd.gray_hash(spec, length)
No description here either โ she's mysterious like that. ๐ฆ
spack.cmd.is_git_repo(path)
Also no description. Iconic, unbothered, no notes. ๐
spack.cmd.iter_groups(specs, indent, all_headers)
Breaks a list of specs into groups indexed by arch/compiler. Very organize, much tidy. ๐ง
spack.cmd.matching_spec_from_env(spec)
Returns a concrete spec matching whatever's available in the environment. If nothing matching shows up in the environment โ or if there's literally no active environment at all โ she returns the spec you gave her, but concretized. Make it make sense? She does. ๐ฅ
spack.cmd.parse_specs(args: str | List[str], concretize: bool = False, tests: bool = False) -> List[Spec]
A convenience function for parsing arguments from specs. She handles the common exceptions and dies if there are errors. Not me โ she's dramatic but correct. ๐
spack.cmd.print_how_many_pkgs(specs, pkg_type='', suffix='')
Give her a list of specs and she'll print a message about how many specs are in that list.
Parameters
specs(list) โ depending on how many items are in this list, choose the plural or singular form of the word "package"pkg_type(str) โ the output string will mention this provided category, e.g. ifpkg_typeis"installed"then the message would be"3 installed packages"
spack.cmd.python_name(cmd_name)
Converts - into _ in the command name, so you get a valid identifier. ๐ช
spack.cmd.quote_kvp(string: str) -> str
For strings like name=value or name==value, quotes and escapes the value if needed.
Fr fr, this one's a compromise to respect quoting of key-value pairs on the CLI. The shell strips quotes from quoted arguments, so there is genuinely no way to know exactly how CLI arguments were quoted. To compensate, she re-adds quotes around anything starting with name= or name==, and assumes the rest of the argument is the value. This covers the common cases of passing flags, e.g. cflags="-O2 -g" on the command line. โจ
spack.cmd.remove_options(parser, *options)
Removes some options from a parser. Breakups happen. ๐
spack.cmd.require_active_env(cmd_name)
Used by commands to get the active environment.
If no environment is found, she prints an error message saying the calling command needs an active environment. (Read: you must have an environment. Non-negotiable, bestie.)
Parameters cmd_name (str) โ name of calling command
Returns the active environment
Return type (spack.environment.Environment)
spack.cmd.require_cmd_name(cname)
Requires that the name you handed over is a valid command name (per cmd_name()). Useful for checking parameters as function prerequisites. Gatekeeping, but make it helpful. ๐
spack.cmd.require_python_name(pname)
Requires that the provided name is a valid python name (per python_name()). Also useful for checking parameters as function prerequisites. ๐
spack.cmd.spack_is_git_repo()
Ensures that this instance of Spack is a git clone. That's it, that's the vibe check. ๐ธ
๐ฆ Subpackages
spack.cmd.common package
spack.cmd.common.shell_init_instructions(cmd, equivalent)
Prints out instructions for users to initialize shell support. Because sometimes you gotta set the vibe up before the vibe can happen. ๐
Parameters
cmd(str) โ the command the user tried to run that requires shell support in order to workequivalent(str) โ a command they can run instead, without enabling shell support
๐งฉ Submodules
spack.cmd.common.arguments module
spack.cmd.common.arguments.add_common_arguments(parser, list_of_arguments)
Extends a parser with extra arguments. The more the merrier, honestly.
Parameters
parserโ parser to be extendedlist_of_argumentsโ arguments to be added to the parser
spack.cmd.common.confirmation module
spack.cmd.common.confirmation.confirm_action(specs: List[Spec], participle: str, noun: str)
Displays the list of specs about to be acted on and asks for confirmation first. Consent matters! ๐
Parameters
specsโ specs to be removedparticipleโ action expressed as a participle, e.g."uninstalled"nounโ action expressed as a noun, e.g."uninstallation"
spack.cmd.common.env_utility module
class spack.cmd.common.env_utility.AreDepsInstalledVisitor(context: Context = Context.BUILD)
Bases: object
accept(item)
Goes here, does the accepting thing. ๐ซก
neighbors(item)
Goes here, does the neighbors thing. Gotta know the whole friend group. ๐
spack.cmd.common.env_utility.emulate_env_utility(cmd_name, context: Context, args)
No description in the source โ she's lowkey keeping to herself on this one. ๐ฆ
spack.cmd.common.env_utility.setup_parser(subparser)
Also no description. Mysterious main character energy, honestly. ๐
spack.cmd.modules package
Implementation details of the spack module command. (Yes, details โ this is the backstage, not the show. ๐ญ)
spack.cmd.modules.MultipleSpecsMatch
Bases: Exception
Raised when multiple specs match a constraint, in a context where that is not allowed. Too many matches, and nobody asked for that. ๐ญ
spack.cmd.modules.NoSpecMatches
Bases: Exception
Raised when no spec matches a constraint, in a context where this is not allowed. Empty-handed and it's not cute. ๐
spack.cmd.modules.add_loads_arguments(subparser)
No description provided โ she just adds those loads arguments and minds her business. โจ
spack.cmd.modules.callbacks = {'find': <function find>, 'loads': <function loads>, 'refresh': <function refresh>, 'rm': <function rm>}
A dictionary populated with the list of sub-commands. Each sub-command must be callable and accept 3 arguments:
module_type: the type of module it refers tospecs: the list of specs to be processed
argsโ namespace containing the parsed command line arguments ๐ซถ
okay so the spack.cmd.modules functions keep rolling like this:
spack.cmd.modules.check_module_set_name(name)โ no extra docstring here, just the signature doing its thing ๐spack.cmd.modules.find(module_type, specs, args)โ Retrieve paths or use names of module files โจspack.cmd.modules.loads(module_type, specs, args, out=None)โ Prompt the list of modules associated with a list of specs ๐
spack.cmd.modules.modules_cmd(parser, args, module_type,
callbacks={'find': <function find>, 'loads': <function loads>,
'refresh': <function refresh>, 'rm': <function rm>})โ this is the callback-dispatcher of the group chat, and yes the dict is part of the whole vibe ๐ฅ
spack.cmd.modules.one_spec_or_raise(specs)โ Ensures exactly one spec has been selected, or raises the appropriate exception. so real talk: one spec, or catch these exceptions ๐spack.cmd.modules.refresh(module_type, specs, args)โ Regenerates the module files for every spec in specs and every module type in module types. glow-up for the module files โจspack.cmd.modules.rm(module_type, specs, args)โ Deletes the module files associated with every spec in specs, for every module type in module types. bye bestie ๐ซถspack.cmd.modules.setup_parser(subparser)โ parser setup, no extra commentary in the source ๐
๐ Submodules ๐
๐ธ spack.cmd.modules.lmod module ๐ธ
spack.cmd.modules.lmod.add_command(parser, command_dict)โ adds the command, no extra docstring ๐spack.cmd.modules.lmod.setdefault(module_type, specs, args)โ set the default module file, when multiple are present ๐
๐ฆ spack.cmd.modules.tcl module ๐ฆ
spack.cmd.modules.tcl.add_command(parser, command_dict)โ adds the command, no extra docstring ๐spack.cmd.modules.tcl.setdefault(module_type, specs, args)โ set the default module file, when multiple are present ๐
๐ฆ Submodules ๐ฆ
๐ spack.cmd.add module ๐
spack.cmd.add.add(parser, args)โ the mainaddentry point, no extra docstring โจspack.cmd.add.setup_parser(subparser)โ parser setup, no extra docstring ๐
๐ง spack.cmd.arch module ๐ง
spack.cmd.arch.arch(parser, args)โ the mainarchentry point, no extra docstring โจspack.cmd.arch.display_targets(targets)โ Prints a human readable list of the targets passed as argument. literally a little target parade ๐spack.cmd.arch.setup_parser(subparser)โ parser setup, no extra docstring ๐
๐ฅ spack.cmd.audit module ๐ฅ
spack.cmd.audit.audit(parser, args)โ no extra docstring ๐spack.cmd.audit.configs(parser, args)โ no extra docstring ๐spack.cmd.audit.externals(parser, args)โ no extra docstring ๐spack.cmd.audit.list(parser, args)โ no extra docstring ๐spack.cmd.audit.packages(parser, args)โ no extra docstring ๐spack.cmd.audit.packages_https(parser, args)โ no extra docstring ๐spack.cmd.audit.setup_parser(subparser)โ parser setup, no extra docstring ๐
๐ spack.cmd.blame module ๐
spack.cmd.blame.blame(parser, args)โ no extra docstring ๐spack.cmd.blame.dump_json(rows, last_mod, total_lines, emails)โ Dump the blame as a json object to the terminal. json, lowercase, we respect it ๐ซถspack.cmd.blame.print_table(rows, last_mod, total_lines, emails)โ Given a set of rows with authors and lines, print a table. table time, bestie โจspack.cmd.blame.setup_parser(subparser)โ parser setup, no extra docstring ๐
๐ spack.cmd.bootstrap module ๐
spack.cmd.bootstrap.LOCAL_MIRROR_DIR = 'bootstrap_cache'โ Subdirectory where to create the mirror ๐spack.cmd.bootstrap.bootstrap(parser, args)โ no extra docstring ๐spack.cmd.bootstrap.setup_parser(subparser)โ parser setup, no extra docstring ๐
๐ spack.cmd.build_env module ๐
spack.cmd.build_env.build_env(parser, args)โ no extra docstring ๐
๐ spack.cmd.buildcache module ๐
class spack.cmd.buildcache.NoPool โ Bases: object ๐
map(func, args)โ no extra docstring โจstarmap(func, args)โ no extra docstring โจspack.cmd.buildcache.buildcache(parser, args)โ no extra docstring ๐spack.cmd.buildcache.check_fn(args: Namespace)โ check specs against remote binary mirror(s) to see if any need to be rebuilt.
real talk: this command uses the process exit code to indicate its result, specifically, if the exit code is non-zero, then at least one of the indicated specs needs to be rebuilt. the exit code is the whole message ๐
spack.cmd.buildcache.copy_buildcache_file(src_url, dest_url, local_path=None)โ Copy from source url to destination url ๐ซถspack.cmd.buildcache.download_fn(args)โ download buildcache entry from a remote mirror to local folder.
this command uses the process exit code to indicate its result, specifically, a non-zero exit code indicates that the command failed to download at least one of the required buildcache components. no cap, exit code says it all ๐ญ
spack.cmd.buildcache.get_buildcache_name_fn(args)โ get name (prefix) of buildcache entries for this spec โจspack.cmd.buildcache.install_fn(args)โ install from a binary package ๐spack.cmd.buildcache.keys_fn(args)โ get public keys available on mirrors ๐spack.cmd.buildcache.list_fn(args)โ list binary packages available from mirrors ๐spack.cmd.buildcache.manifest_copy(manifest_file_list, dest_mirror=None)โ Read manifest files containing information about specific specs to copy from source to destination, remove duplicates since any binary packge for a given hash should be the same as any other, and copy all files specified in the manifest files. yes,packgeโ we keep the lore ๐spack.cmd.buildcache.preview_fn(args)โ analyze an installed spec and reports whether executables and libraries are relocatable ๐งspack.cmd.buildcache.push_fn(args)โ create a binary package and push it to a mirror ๐ฅspack.cmd.buildcache.save_specfile_fn(args)โ get full spec for dependencies and write them to files in the specified output directory.
uses exit code to signal success or failure. an exit code of zero means the command was likely successful. if any errors or exceptions are encountered, or if expected command-line arguments are not provided, then the exit code will be non-zero. zero = good vibes, non-zero = bestie we have a situation ๐
spack.cmd.buildcache.setup_parser(subparser: ArgumentParser)โ parser setup, no extra docstring ๐spack.cmd.buildcache.sync_fn(args)โ sync binaries (and associated metadata) from one mirror to another.
requires an active environment in order to know which specs to sync. active environment required, bestie ๐ธ
spack.cmd.buildcache.update_index(mirror: Mirror, update_keys=False)โ no extra docstring ๐spack.cmd.buildcache.update_index_fn(args)โ update a buildcache index โจ
๐ฆ spack.cmd.cd module ๐ฆ
spack.cmd.cd.cd(parser, args)โ no extra docstring ๐spack.cmd.cd.setup_parser(subparser)โ This is for decoration --spack cdis used through spack's shell support. This allowsspack cdto print a descriptive help message when called with-h. decoration with a purpose, honestly ๐
๐ spack.cmd.change module ๐
spack.cmd.change.change(parser, args)โ no extra docstring ๐spack.cmd.change.setup_parser(subparser)โ parser setup, no extra docstring ๐
๐ spack.cmd.checksum module ๐
spack.cmd.checksum.add_versions_to_package(pkg: PackageBase, version_lines: str, is_batch: bool)โ Add checksumed versions to a package's instructions and open a user's editor so they may double check the work of the function.
Parameters
pkg(spack.package_base.PackageBase) โ A package class for a given package in Spack.version_lines(str) โ A string of rendered version lines.
spack.cmd.checksum.checksum(parser, args)โ no extra docstring ๐spack.cmd.checksum.print_checksum_status(pkg: PackageBase, version_hashes: dict)โ Verify checksums present inversion_hashesagainst those present in the package's instructions.
Parameters
pkg(spack.package_base.PackageBase) โ A package class for a given package in Spack.version_hashes(dict) โ A dictionary of the form:version -> checksum.
spack.cmd.checksum.setup_parser(subparser)โ parser setup, no extra docstring ๐
๐ spack.cmd.ci module ๐
spack.cmd.ci.ci(parser, args)โ no extra docstring ๐spack.cmd.ci.ci_generate(args)โ generate jobs file from a CI-aware spack file.
if you want to report the results on CDash, you will need to set the SPACK_CDASH_AUTH_TOKEN before invoking this command. the value must be the CDash authorization token needed to create a build group and register all generated jobs under it. token first, then slay ๐
spack.cmd.ci.ci_rebuild(args)โ rebuild a spec if it is not on the remote mirror.
check a single spec against the remote mirror, and rebuild it from source if the mirror does not contain the hash. mirror says no? source says yes ๐ฅ
spack.cmd.ci.ci_reindex(args)โ rebuild the buildcache index for the remote mirror.
use the active, gitlab-enabled environment to rebuild the buildcache index for the associated mirror. active gitlab-enabled environment required โจ
spack.cmd.ci.ci_reproduce(args)โ generate instructions for reproducing the spec rebuild job.
artifacts of the provided gitlab pipeline rebuild job's URL will be used to derive instructions for reproducing the build locally. local reproduction instructions, coming right up ๐ซถ
spack.cmd.ci.deindent(desc)โ no extra docstring ๐spack.cmd.ci.setup_parser(subparser)โ parser setup, no extra docstring ๐spack.cmd.ci.unicode_escape(path: str) -> strโ Returns transformed path with any unicode characters replaced with their corresponding escapes ๐ง
โญ spack.cmd.clean module โญ
class spack.cmd.clean.AllClean
class spack.cmd.clean.AllClean(option_strings, dest, nargs=None,
const=None, default=None, type=None, choices=None, required=False,
help=None, metavar=None, deprecated=False)Bases: Action ๐
Activates flags -s -d -f -m and -p simultaneously. all the flags at once, iconic ๐
spack.cmd.clean.clean(parser, args)โ no extra docstring ๐spack.cmd.clean.remove_python_cache()โ no extra docstring โจspack.cmd.clean.setup_parser(subparser)โ parser setup, no extra docstring ๐
๐ธ spack.cmd.clone module ๐ธ
spack.cmd.clone.clone(parser, args)โ no extra docstring ๐spack.cmd.clone.get_origin_info(remote)โ no extra docstring โจspack.cmd.clone.setup_parser(subparser)โ parser setup, no extra docstring ๐
๐ spack.cmd.commands module ๐
class spack.cmd.commands.BashCompletionWriter
class spack.cmd.commands.BashCompletionWriter(prog: str, out: ~typing.IO
= <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>,
aliases: bool = False)Bases: ArgparseWriter ๐
Write argparse output as bash programmable tab completion. yes bestie, tab completion is a love language โจ
body(positionals: Sequence[str], optionals: Sequence[str], subcommands: Sequence[str]) -> strโ Return the body of the function.
Parameters
positionalsโ List of positional arguments.optionalsโ List of optional arguments.subcommandsโ List of subcommand parsers.
Returns: Function body.
end_function(prog: str) -> strโ Return the syntax needed to end a function definition.
Parameters
progโ Program name
Returns: Function definition ending.
format(cmd: Command) -> strโ Return the string representation of a single node in the parser tree.
Parameters
cmdโ Parsed information about a command or subcommand.
Returns: String representation of this subcommand.
optionals(optionals: Sequence[str]) -> strโ Return the syntax for reporting optional flags.
Parameters
optionalsโ List of optional arguments.
Returns: Syntax for optional flags.
positionals(positionals: Sequence[str]) -> strโ Return the syntax for reporting positional arguments.
Parameters
positionalsโ List of positional arguments.
Returns โ syntax for positional arguments. ๐ซ
โจ start_function
start_function(prog: str) -> strReturns the syntax you need to begin a function definition โ like the opening line, the vibe-setter, the "once upon a time" of a function. ๐ซถ
progโ program name.- Returns โ the function definition beginning.
๐ subcommands
subcommands(subcommands: Sequence[str]) -> strSpits out the syntax for reporting subcommands. Iconic little dispatcher energy. ๐
subcommandsโ list of subcommand parsers.- Returns โ syntax for subcommand parsers.
๐
class spack.cmd.commands.FishCompletionWriter(...)
class spack.cmd.commands.FishCompletionWriter(prog: str, out: ~typing.IO = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, aliases: bool = False)Bases: ArgparseWriter
Writes argparse output as bash programmable tab completion. Yes bestie, fish writer making bash completions โ it's giving crossover episode. ๐โจ
๐ complete
complete(prog: str, positionals: List[Tuple[str, Iterable[Any] | None, int | str | None, str]], optionals: List[Tuple[Sequence[str], List[str], str, int | str | None, str]], subcommands: List[Tuple[ArgumentParser, str, str]]) -> strReturns all the completion commands. The whole squad. ๐
progโ program name.positionalsโ list of positional arguments.optionalsโ list of optional arguments.subcommandsโ list of subcommand parsers.- Returns โ completion command.
๐ static complete_head
static complete_head(prog: str, index: int | None = None, nargs: int | str | None = None) -> strReturns the head of the completion command โ the opening act, the main character's entrance. ๐ฌ
progโ program name.indexโ index of positional argument.nargsโ number of arguments.- Returns โ head of the completion command.
๐ฆ format
format(cmd: Command) -> strReturns the string representation of a single node in the parser tree. One node, one moment, one slay. ๐ธ
cmdโ parsed information about a command or subcommand.- Returns โ string representation of a node.
๐ optionals
optionals(prog: str, optionals: List[Tuple[Sequence[str], List[str], str, int | str | None, str]]) -> strReturns the completion for optional arguments โ you know, the flags with the little dashes. ๐
progโ program name.optionalsโ list of optional arguments.- Returns โ completion command.
๐ optspecs
optspecs(prog: str, optionals: List[Tuple[Sequence[str], List[str], str, int | str | None, str]]) -> strReads the optionals and returns the command to set optspec. Setting the variable, setting the tone. ๐
progโ program name.optionalsโ list of optional arguments.- Returns โ command to set
optspecvariable.
๐ positionals
positionals(prog: str, positionals: List[Tuple[str, Iterable[Any] | None, int | str | None, str]]) -> strReturns the completion for positional arguments. No dashes, all vibes. โจ
progโ program name.positionalsโ list of positional arguments.- Returns โ completion command.
๐ prog_comment
prog_comment(prog: str) -> strReturns a comment line for the command. A little annotative garnish. ๐ท
progโ program name.- Returns โ comment line.
๐ง subcommands
subcommands(prog: str, subcommands: List[Tuple[ArgumentParser, str, str]]) -> strReturns the completion for subcommands โ the whole extended family tree. ๐ณ
progโ program name.subcommandsโ list of subcommand parsers.- Returns โ completion command.
๐ฅ class spack.cmd.commands.SpackArgparseRstWriter(...)
class spack.cmd.commands.SpackArgparseRstWriter(prog: str, out: ~typing.IO = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, aliases: bool = False, documented_commands: ~typing.Set[str] = {}, rst_levels: ~typing.Sequence[str] = ['-', '-', '^', '~', ':', '`'])Bases: ArgparseRstWriter
An RST writer tailored for spack documentation โ bespoke, custom-fit, no off-the-rack docs here bestie. ๐๐
โญ usage
usage(usage: str) -> strExample usage of a command. Show, don't tell. ๐
usageโ command usage.- Returns โ usage of a command.
๐ class spack.cmd.commands.SubcommandWriter(...)
class spack.cmd.commands.SubcommandWriter(prog: str, out: ~typing.IO = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, aliases: bool = False)Bases: ArgparseWriter
Writes argparse output as a list of subcommands โ a cute little roster. ๐โจ
๐ธ format
format(cmd: Command) -> strReturns the string representation of a single node in the parser tree. ๐ฑ
cmdโ parsed information about a command or subcommand.- Returns โ string representation of this subcommand.
๐ซ spack.cmd.commands.bash
spack.cmd.commands.bash(args: Namespace, out: IO) -> NoneBash tab-completion script. Press Tab, receive blessings. ๐
argsโ command-line arguments.outโ file object to write to.
๐ฆ spack.cmd.commands.commands
spack.cmd.commands.commands(parser: ArgumentParser, args: Namespace) -> NoneThe main function that calls the formatter functions โ the group-chat admin summoning everyone. ๐ฃ
parserโ argument parser.argsโ command-line arguments.
๐ spack.cmd.commands.fish
spack.cmd.commands.fish(args, out)(No extra notes attached to this one โ it just exists and serves fish-completion realness. ๐ซง)
๐
spack.cmd.commands.formatter
spack.cmd.commands.formatter(func: Callable[[Namespace, IO], None]) -> Callable[[Namespace, IO], None]Decorator used to register formatters. Wrap it up, put a bow on it, add it to the registry. ๐
funcโ formatting function.- Returns โ the same function.
๐ spack.cmd.commands.formatters
spack.cmd.commands.formatters: Dict[str, Callable[[Namespace, IO], None]] = {'bash': <function bash>, 'fish': <function fish>, 'names': <function names>, 'rst': <function rst>, 'subcommands': <function subcommands>}The list of command formatters โ literally the whole lineup. ๐
๐ spack.cmd.commands.names
spack.cmd.commands.names(args: Namespace, out: IO) -> NoneSimple list of top-level commands. Just the names, no drama. โจ
argsโ command-line arguments.outโ file object to write to.
๐ท spack.cmd.commands.prepend_header
spack.cmd.commands.prepend_header(args: Namespace, out: IO) -> NonePrepends header text at the beginning of a file โ the opening credits. ๐ฌ
argsโ command-line arguments.outโ file object to write to.
๐ง spack.cmd.commands.rst
spack.cmd.commands.rst(args: Namespace, out: IO) -> NoneReStructuredText documentation of subcommands. Docs, but make it structured. ๐
argsโ command-line arguments.outโ file object to write to.
๐ spack.cmd.commands.rst_index
spack.cmd.commands.rst_index(out: IO) -> NoneGenerates an index of all commands. The table of contents is so back. ๐
outโ file object to write to.
๐ spack.cmd.commands.setup_parser
spack.cmd.commands.setup_parser(subparser: ArgumentParser) -> NoneSets up the argument parser. Prep work, but make it cute. ๐ ๏ธ
subparserโ preliminary argument parser.
โญ spack.cmd.commands.subcommands
spack.cmd.commands.subcommands(args: Namespace, out: IO) -> NoneHierarchical tree of subcommands โ the family tree, fully rendered. ๐ณ
argsโ command-line arguments.outโ file object to write to.
๐ฅ spack.cmd.commands.update_completion
spack.cmd.commands.update_completion(parser: ArgumentParser, args: Namespace) -> NoneIterates through the shells and updates the standard completion files.
It's a convenience method โ so you don't have to call this command a million times, and so developers can update completion without suffering. You're welcome. ๐
parserโ argument parser.argsโ command-line arguments.
๐ spack.cmd.commands.update_completion_args
spack.cmd.commands.update_completion_args: Dict[str, Dict[str, Any]] = {'bash': {'aliases': True, 'format': 'bash', 'header': '/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/share/spack/bash/spack-completion.bash', 'update': '/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/share/spack/spack-completion.bash'}, 'fish': {'aliases': True, 'format': 'fish', 'header': '/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/share/spack/fish/spack-completion.fish', 'update': '/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/share/spack/spack-completion.fish'}}The standard arguments for updating completion scripts โ we iterate through these when called with --update-completion. ๐ซถ
๐ป spack.cmd.compiler module
spack.cmd.compiler.compiler
spack.cmd.compiler.compiler(parser, args)(The plain entry point โ parser and args, no extra commentary attached. ๐คทโ๏ธ)
๐ spack.cmd.compiler.compiler_find
spack.cmd.compiler.compiler_find(args)Searches either $PATH or a list of paths OR MODULES for compilers, and adds them to Spack's configuration. Scavenger hunt but for compilers. ๐ต๏ธโ๏ธ
๐ฌ spack.cmd.compiler.compiler_info
spack.cmd.compiler.compiler_info(args)Prints info about all compilers matching a spec. The compiler's whole lore, dumped. ๐
๐ spack.cmd.compiler.compiler_list
spack.cmd.compiler.compiler_list(args)(Just the function โ no description in the source, so no invented tea from me. โ)
๐๏ธ spack.cmd.compiler.compiler_remove
spack.cmd.compiler.compiler_remove(args)(No description attached in the original either โ it does what the name says, but I'm not adding claims. ๐)
๐ ๏ธ spack.cmd.compiler.setup_parser
spack.cmd.compiler.setup_parser(subparser)(Sets up the parser โ that's the whole note. ๐)
๐งฉ spack.cmd.compilers module
spack.cmd.compilers.compilers
spack.cmd.compilers.compilers(parser, args)(Existence confirmed; no extra blurb given. โจ)
spack.cmd.compilers.setup_parser
spack.cmd.compilers.setup_parser(subparser)(Same deal โ parser setup, no commentary. ๐ซ)
๐ฎ spack.cmd.concretize module
spack.cmd.concretize.concretize
spack.cmd.concretize.concretize(parser, args)(The concretizer's command entry โ nothing extra attached. ๐ซง)
spack.cmd.concretize.setup_parser
spack.cmd.concretize.setup_parser(subparser)(Parser setup, au naturel. ๐ธ)
โ๏ธ spack.cmd.config module
spack.cmd.config.config
spack.cmd.config.config(parser, args)(The config command's front door โ no extra notes. ๐ช)
โ spack.cmd.config.config_add
spack.cmd.config.config_add(args)Adds the given configuration to the specified config scope.
Heads up: this is a stateful operation that edits the config files. Real changes, real files โ not just vibes. โ ๏ธ
๐ต๏ธโ๏ธ spack.cmd.config.config_blame
spack.cmd.config.config_blame(args)Prints out line-by-line blame of merged YAML. Who wrote this line? Found out. ๐
๐ spack.cmd.config.config_change
spack.cmd.config.config_change(args)(No description in the source โ just the signature. ๐โ๏ธ)
โ๏ธ spack.cmd.config.config_edit
spack.cmd.config.config_edit(args)Edits the configuration file for a specific scope and config section.
With no arguments and an active environment, it edits the spack.yaml for the active environment. Your environment, your edit. ๐
๐ค spack.cmd.config.config_get
spack.cmd.config.config_get(args)Dumps the merged YAML configuration for a specific section.
With no arguments and an active environment, it prints the contents of the environment's manifest file (spack.yaml). Show me everything. ๐
๐ spack.cmd.config.config_list
spack.cmd.config.config_list(args)Lists the possible configuration sections.
Used primarily for shell tab completion scripts. Humble, but essential. ๐
๐๏ธ spack.cmd.config.config_prefer_upstream
spack.cmd.config.config_prefer_upstream(args)Generates a packages config based on the configuration of all upstream installs. Upstream, but make it config. โจ
๐ฆ spack.cmd.config.config_remove(args)
okay so this one removes whatever configuration you point it at, from the config scope you specified ๐๏ธโจ. and bestie, listen โ it's a stateful operation: it works by literally editing the config files. like, on disk. no undo, no take-backsies ๐ so maybe peek at your files first, yeah?
๐ซถ spack.cmd.config.config_revert(args)
ngl the manpage gives this one zero description โ just the signature and nothing else ๐ so I'm not gonna sit here and invent behaviour for it. facts only, periodt.
๐ซถ spack.cmd.config.config_update(args)
same vibe as above โ no description on this one either. it exists, it's listed, that's literally all we get ๐
โญ spack.cmd.config.print_configuration(args, *, blame: bool) -> None
again: no description in the source. the signature's the whole story here, iykyk ๐ธ
๐ spack.cmd.config.print_flattened_configuration(*, blame: bool) -> None
this one's iconic โ it prints a flattened version of the configuration to stdout ๐คโจ flat, like, unfolded, all the layers smoothed out.
Parameters:
blameโ ifTrue, it shows file provenance for each entry in the configuration ๐ so you can see exactly which file each bit came from. detective mode, fr fr ๐
๐ง spack.cmd.config.setup_parser(subparser)
no description given. that's it, that's the entry ๐
๐ spack.cmd.containerize module
๐ spack.cmd.containerize.containerize(parser, args)
no description in the manpage for this one โ just the signature, sitting there being mysterious ๐
๐ spack.cmd.containerize.setup_parser(subparser)
also no description. we love a lowkey module ๐
๐ spack.cmd.create module
buckle up bestie, this module is stacked โ it's the whole template zoo for spack create ๐ฆ every package type gets its own little template class with default values baked in. let's go, one by one ๐ธ
๐ class spack.cmd.create.AutoreconfPackageTemplate(name, url, versions)
Bases: PackageTemplate
this gives you the appropriate overrides for Autotools-based packages that do not come with a configure script โ so you're bootstrapping the whole autotools situation yourself ๐ค
base_class_name = 'AutotoolsPackage'
body_def:
def autoreconf(self, spec, prefix):
# FIXME: Modify the autoreconf method as necessary
autoreconf("--install", "--verbose", "--force")
def configure_args(self):
# FIXME: Add arguments other than --prefix
# FIXME: If not needed delete this function
args = []
return argsโ ๏ธ heads up โ in the original source this is one single string literal with literal \n escapes in it. the layout above is that string, unescaped and readable, so don't go copy-pasting the quotes wrong ๐
dependencies:
depends_on("autoconf", type="build")
depends_on("automake", type="build")
depends_on("libtool", type="build")
depends_on("m4", type="build")
# FIXME: Add additional dependencies if required.
# depends_on("foo")๐ class spack.cmd.create.AutotoolsPackageTemplate(name, url, versions)
Bases: PackageTemplate
and this is the sibling ๐ซถ โ appropriate overrides for Autotools-based packages that do come with a configure script. fancy โจ
base_class_name = 'AutotoolsPackage'
body_def:
def configure_args(self):
# FIXME: Add arguments other than --prefix
# FIXME: If not needed delete this function
args = []
return args๐ class spack.cmd.create.BazelPackageTemplate(name, url, versions)
Bases: PackageTemplate
appropriate overrides for Bazel-based packages ๐งฑ
body_def:
def install(self, spec, prefix):
# FIXME: Add logic to build and install here.
bazel()dependencies:
# FIXME: Add additional dependencies if required.
depends_on("bazel", type="build")๐ class spack.cmd.create.BuildSystemGuesser
Bases: object
okay this one's lowkey the main character ๐ต๏ธโโ๏ธ an instance of BuildSystemGuesser gives you a callable object to use during spack create. by passing this object to spack checksum, we can take a peek at the fetched tarball and discern the build system it uses ๐ iconic behaviour, honestly.
๐ class spack.cmd.create.BundlePackageTemplate(name, versions)
Bases: object
gives you the default values for a bundle package file template โ note the args, there's no url here, because there's nothing to download ๐
base_class_name = 'BundlePackage'
body_def:
# There is no need for install() since there is no code.dependencies:
# FIXME: Add dependencies if required.
# depends_on("foo")url_def:
# There is no URL since there is no code to download.write(pkg_path) โ writes the new package file. ๐
๐ class spack.cmd.create.CMakePackageTemplate(name, url, versions)
Bases: PackageTemplate
appropriate overrides for CMake-based packages ๐
base_class_name = 'CMakePackage'
body_def:
def cmake_args(self):
# FIXME: Add arguments other
# FIXME: than CMAKE_INSTALL_PREFIX and CMAKE_BUILD_TYPE
# FIXME: If not needed delete this function
args = []
return args๐ class spack.cmd.create.CargoPackageTemplate(name, url, versions)
Bases: PackageTemplate
appropriate overrides for cargo-based packages ๐ฆ (yes, that kind of cargo bestie)
base_class_name = 'CargoPackage'body_def = ''โ literally empty. nothing to override. immaculate minimalism โจ
๐ class spack.cmd.create.GoPackageTemplate(name, url, versions)
Bases: PackageTemplate
appropriate overrides for Go-module-based packages ๐น
base_class_name = 'GoPackage'body_def = ''โ again, empty. clean. no notes ๐
๐ class spack.cmd.create.IntelPackageTemplate(name, url, versions)
Bases: PackageTemplate
appropriate overrides for licensed Intel software ๐
base_class_name = 'IntelPackage'
body_def:
# FIXME: Override `setup_environment` if necessary.๐ class spack.cmd.create.LuaPackageTemplate(name, url, *args, **kwargs)
Bases: PackageTemplate
appropriate overrides for LuaRocks-based packages ๐
base_class_name = 'LuaPackage'
body_def:
def luarocks_args(self):
# FIXME: Add arguments to `luarocks make` other than rockspec path
# FIXME: If not needed delete this function
args = []
return args๐ class spack.cmd.create.MakefilePackageTemplate(name, url, versions)
Bases: PackageTemplate
appropriate overrides for Makefile packages ๐งต
base_class_name = 'MakefilePackage'
body_def:
def edit(self, spec, prefix):
# FIXME: Edit the Makefile if necessary
# FIXME: If not needed delete this function
# makefile = FileFilter("Makefile")
# makefile.filter("CC = .*", "CC = cc")
pass๐ class spack.cmd.create.MavenPackageTemplate(name, url, versions)
Bases: PackageTemplate
appropriate overrides for Maven-based packages โ
base_class_name = 'MavenPackage'
body_def:
def build(self, spec, prefix):
# FIXME: If not needed delete this function
pass๐ class spack.cmd.create.MesonPackageTemplate(name, url, versions)
Bases: PackageTemplate
appropriate overrides for meson-based packages ๐ง
base_class_name = 'MesonPackage'
body_def:
def meson_args(self):
# FIXME: If not needed delete this function
args = []
return args๐ class spack.cmd.create.OctavePackageTemplate(name, *args, **kwargs)
Bases: PackageTemplate
appropriate overrides for octave packages ๐
base_class_name = 'OctavePackage'
dependencies:
extends("octave")
# FIXME: Add additional dependencies if required.
# depends_on("octave-foo", type=("build", "run"))๐ class spack.cmd.create.PackageTemplate(name, url, versions)
Bases: BundlePackageTemplate
the generic one, the blueprint, the mother of templates ๐ธ gives you the default values used for the package file template.
base_class_name = 'Package'
body_def:
def install(self, spec, prefix):
# FIXME: Unknown build system
make()
make("install")url_line:
url = "{url}"๐ class spack.cmd.create.PerlbuildPackageTemplate(name, *args, **kwargs)
Bases: PerlmakePackageTemplate
appropriate overrides for Perl extensions that come with a Build.PL instead of a Makefile.PL ๐ช
dependencies:
depends_on("perl-module-build", type="build")
# FIXME: Add additional dependencies if required:
# depends_on("perl-foo", type=("build", "run"))๐ class spack.cmd.create.PerlmakePackageTemplate(name, *args, **kwargs)
Bases: PackageTemplate
appropriate overrides for Perl extensions that come with a Makefile.PL ๐ช
base_class_name = 'PerlPackage'
body_def:
def configure_args(self):
# FIXME: Add non-standard arguments
# FIXME: If not needed delete this function
args = []
return argsdependencies:
# FIXME: Add dependencies if required:
# depends_on("perl-foo", type=("build", "run"))๐ class spack.cmd.create.PythonPackageTemplate(name, url, *args, **kwargs)
Bases: PackageTemplate
appropriate overrides for python extensions ๐๐
base_class_name = 'PythonPackage'
body_def:
def config_settings(self, spec, prefix):
# FIXME: Add configuration settings to be passed to the build backend
# FIXME: If not needed, delete this function
settings = {}
return settingsdependencies:
# FIXME: Only add the python/pip/wheel dependencies if you need specific versions
# or need to change the dependency type. Generic python/pip/wheel dependencies are
# added implicity by the PythonPackage base class.
# depends_on("python@2.X:2.Y,3.Z:", type=("build", "run"))
# depends_on("py-pip@X.Y:", type="build")
# depends_on("py-wheel@X.Y:", type="build")
# FIXME: Add a build backend, usually defined in pyproject.toml. If no such file
# exists, use setuptools.
# depends_on("py-setuptools", type="build")
# depends_on("py-hatchling", type="build")
# depends_on("py-flit-core", type="build")
# depends_on("py-poetry-core", type="build")
# FIXME: Add additional dependencies if required.
# depends_on("py-foo", type=("build", "run"))๐ซถ just so we're clear โ that implicity typo is straight from the original, bestie. I'm not fixing it. preservation era ๐
๐ class spack.cmd.create.QMakePackageTemplate(name, url, versions)
Bases: PackageTemplate
appropriate overrides for QMake-based packages ๐ฏ
base_class_name = 'QMakePackage'
body_def:
def qmake_args(self):
# FIXME: If not needed delete this function
args = []
return args๐ class spack.cmd.create.RPackageTemplate(name, url, *args, **kwargs)
Bases: PackageTemplate
appropriate overrides for R extensions ๐
base_class_name = 'RPackage'
body_def:
def configure_args(self):
# FIXME: Add arguments to pass to install via --configure-args
# FIXME: If not needed delete this function
args = []
return argsdependencies:
# FIXME: Add dependencies if required.
# depends_on("r-foo", type=("build", "run"))๐ class spack.cmd.create.RacketPackageTemplate(name, url, *args, **kwargs)
Bases: PackageTemplate
and last up in this batch โ appropriate overrides for Racket extensions ๐พ (yes, it says "approriate" in the original, and yes, I'm keeping that too โจ)
base_class_name = 'RacketPackage'
body_def:
# FIXME: specify the name of the package,
# as it should appear to ``raco pkg install``
name = "{0}"
# FIXME: set to true if published on pkgs.racket-lang.org
# pkgs = False
# FIXME: specify path to the root directory of the
# package, if not the base directory
# subdirectory = Nonesee that {0}? that's a real format placeholder, so keep it exactly as-is when you're poking at the template ๐ซถ
โฆand RacketPackageTemplate wraps up its whole vibe with these two class attributes:
dependencies = ' # FIXME: Add dependencies if required. Only add
the racket dependency\n # if you need specific versions. A
generic racket dependency is\n # added implicity by the
RacketPackage class.\n # depends_on("racket@8.3:", type=("build",
"run"))'
url_line = ' # FIXME: set the proper location from which to fetch
your package\n git = "git@github.com:example/example.git"\n '๐งถ The spack.cmd.create template squad (continued)
spack.cmd.create.RubyPackageTemplate(name, *args, **kwargs)โ Bases:PackageTemplate. Provides appropriate overrides for Ruby packages ๐
base_class_name = 'RubyPackage'body_def = ' def build(self, spec, prefix):\n # FIXME: If not
needed delete this function\n pass'dependencies = ' # FIXME: Add dependencies if required. Only add
the ruby dependency\n # if you need specific versions. A generic
ruby dependency is\n # added implicity by the RubyPackage
class.\n # depends_on("ruby@X.Y.Z:", type=("build", "run"))\n #
depends_on("ruby-foo", type=("build", "run"))'spack.cmd.create.SIPPackageTemplate(name, *args, **kwargs)โ Bases:PackageTemplate. Provides appropriate overrides for SIP packages. ๐
base_class_name = 'SIPPackage'body_def = ' def configure_args(self, spec, prefix):\n # FIXME:
Add arguments other than --bindir and --destdir\n # FIXME: If not
needed delete this function\n args = []\n return args'spack.cmd.create.SconsPackageTemplate(name, url, versions)โ Bases:PackageTemplate. Provides appropriate overrides for SCons-based packages ๐ ๏ธ
base_class_name = 'SConsPackage'body_def = ' def build_args(self, spec, prefix):\n # FIXME: Add
arguments to pass to build.\n # FIXME: If not needed delete this
function\n args = []\n return args'spack.cmd.create.WafPackageTemplate(name, url, versions)โ Bases:PackageTemplate. Provides appropriate override for Waf-based packages ๐พ
base_class_name = 'WafPackage'body_def = ' # FIXME: Override configure_args(), build_args(),\n
# or install_args() if necessary.'โจ spack.cmd.create helper functions
spack.cmd.create.create(parser, args)โ the big boss function of the module, no cap ๐spack.cmd.create.get_build_system(template, url, guesser)โ Determine the build system template. If a template is specified, always use that. Otherwise, if a URL is provided, download the tarball and peek inside to guess what build system it uses. Otherwise, use a generic template by default. Literally a vibe check on the tarball ๐
Parameters
template(str) โ the--templateargument given tospack createurl(str) โ theurlargument given tospack createargs(argparse.Namespace) โ the arguments given tospack createguesser(BuildSystemGuesser) โ thefirst_stage_functiongiven tospack checksumwhich records the build system it detects
Returns โ the name of the build system template to use
Return type โ str
spack.cmd.create.get_name(name, url)โ Get the name of the package based on the supplied arguments. If a name was provided, always use that. Otherwise, if a URL was provided, extract the name from that. Otherwise, use a default. She's resourceful like that ๐ซถ
Parameters
name(str) โ the explicit--nameargument given tospack createurl(str) โ theurlargument given tospack create
Returns โ the name of the package
Return type โ str
spack.cmd.create.get_repository(args, name)โ Returns aRepoobject that will allow us to determine the path where the new package file should be created. Finding its little home on disk ๐
Parameters
args(argparse.Namespace) โ the arguments given tospack createname(str) โ the name of the package to create
Returns โ a Repo object capable of determining the path to the package file
Return type โ spack.repo.Repo
spack.cmd.create.get_url(url)โ Get the URL to use. Uses a default URL if none is provided. No URL? No problem ๐โโ๏ธ
Parameters
url(str) โ theurlargument tospack create
Returns โ the URL of the package
Return type โ str
spack.cmd.create.get_versions(args, name)โ Returns a list of versions and hashes for a package. Also returns aBuildSystemGuesserobject. Returns default values if no URL is provided. Double trouble, double return ๐
Parameters
args(argparse.Namespace) โ the arguments given tospack createname(str) โ the name of the package
Returns โ versions and hashes, and a BuildSystemGuesser object
Return type โ tuple
spack.cmd.create.setup_parser(subparser)โ wires up the CLI, as always ๐ง
๐ spack.cmd.debug module
spack.cmd.debug.create_db_tarball(args)spack.cmd.debug.debug(parser, args)spack.cmd.debug.report(args)spack.cmd.debug.setup_parser(subparser)
๐ซฅ spack.cmd.deconcretize module
spack.cmd.deconcretize.deconcretize(parser, args)spack.cmd.deconcretize.deconcretize_specs(args, specs)spack.cmd.deconcretize.get_deconcretize_list(args: Namespace, specs: List[Spec], env: Environment) -> List[Spec]โ Get list of environment roots to deconcretize (yes that's the whole description, she's concise ๐ )spack.cmd.deconcretize.setup_parser(subparser)
๐งฉ spack.cmd.dependencies module
spack.cmd.dependencies.dependencies(parser, args)spack.cmd.dependencies.setup_parser(subparser)
๐ฏ spack.cmd.dependents module
spack.cmd.dependents.dependents(parser, args)spack.cmd.dependents.get_dependents(pkg_name, ideps, transitive=False, dependents=None)โ Get all dependents for a package. Who's relying on who, the group chat of packages fr fr ๐ฑ
Parameters
pkg_name(str) โ name of the package whose dependents should be returnedideps(dict) โ dictionary of dependents, frominverted_dependencies()transitive(bool or None) โ return transitive dependents whenTrue
spack.cmd.dependents.inverted_dependencies()โ Iterate through all packages and return a dictionary mapping package names to possible dependencies. Virtual packages are included as sources, so that you can query dependents of, e.g.,mpi, but virtuals are not included as actual dependents. Subtle but iconic ๐ฅspack.cmd.dependents.setup_parser(subparser)
๐๏ธ spack.cmd.deprecate module
Deprecate one Spack install in favor of another.
Spack packages of different configurations cannot be installed to the same location. However, in some circumstances (e.g. security patches) old installations should never be used again. In these cases, we will mark the old installation as deprecated, remove it, and link another installation into its place. ๐ซก
It is up to the user to ensure binary compatibility between the deprecated installation and its deprecator. Like, genuinely โ that part is on you, bestie. ๐ญ
spack.cmd.deprecate.deprecate(parser, args)โ Deprecate one spec in favor of anotherspack.cmd.deprecate.setup_parser(sp)
๐ฃ spack.cmd.dev_build module
spack.cmd.dev_build.dev_build(self, args)spack.cmd.dev_build.setup_parser(subparser)
๐ฑ spack.cmd.develop module
spack.cmd.develop.develop(parser, args)spack.cmd.develop.setup_parser(subparser)
๐ spack.cmd.diff module
spack.cmd.diff.compare_specs(a, b, to_string=False, color=None, ignore_packages=None)โ Generate a comparison, including diffs (for each side) and an intersection. We can either print the result to the console, or parse into a json object for the user to save. We return an object that shows the differences, intersection, and names for a pair of specsaandb. Spot the difference, spec edition โจ
Parameters
a(spack.spec.Spec) โ the first spec to compareb(spack.spec.Spec) โ the second spec to comparea_name(str) โ the name of specab_name(str) โ the name of specbto_string(bool) โ return an object that can be json dumpedcolor(bool) โ whether to format the names for the console
spack.cmd.diff.diff(parser, args)spack.cmd.diff.flatten(functions)โ Given a list of ASP functions, convert into a list ofkey: valuetuples. We are squashing whatever is after the first index into one string for easier parsing in the interface. Compression era ๐๏ธspack.cmd.diff.print_difference(c, attributes='all', out=None)โ Print the difference. Given a diffset for A and a diffset for B, print red/green diffs to show the differences. green for slay, red for... not slay ๐spack.cmd.diff.setup_parser(subparser)spack.cmd.diff.shift(asp_function)โ Transformsattr("foo", "bar")intofoo("bar"). A lil glow-up for your ASP function ๐
๐ spack.cmd.docs module
spack.cmd.docs.docs(parser, args)
โ๏ธ spack.cmd.edit module
spack.cmd.edit.edit(parser, args)spack.cmd.edit.edit_package(name, repo_path, namespace)โ Opens the requested package file in your favorite$EDITOR. Time to get hands-on ๐ฉโ๐ป
Parameters
name(str) โ the name of the packagerepo_path(str) โ the path to the repository containing this packagenamespace(str) โ a valid namespace registered with Spack
spack.cmd.edit.setup_parser(subparser)
๐ spack.cmd.env module
class spack.cmd.env.ViewActionโ Bases:object๐static actions()disable = 'disable'enable = 'enable'regenerate = 'regenerate'
spack.cmd.env.create_temp_env_directory()โ Returns the path of a temporary directory in which to create an environment. A little pop-up home for your env ๐๏ธspack.cmd.env.env(parser, args)โ Look for a function calledenvironment_<name>and call it. Name-based dispatch, she's clever like that ๐งspack.cmd.env.env_activate(args)spack.cmd.env.env_activate_setup_parser(subparser)โ set the current environmentspack.cmd.env.env_create(args)spack.cmd.env.env_create_setup_parser(subparser)โ create a new environmentspack.cmd.env.env_deactivate(args)spack.cmd.env.env_deactivate_setup_parser(subparser)โ deactivate any active environment in the shellspack.cmd.env.env_depfile(args)spack.cmd.env.env_depfile_setup_parser(subparser)โ generate a depfile from the concrete environment specsspack.cmd.env.env_list(args)spack.cmd.env.env_list_setup_parser(subparser)โ lists the managed environments for you, bestie โจspack.cmd.env.env_loads(args)โ the actualloadsworkhorse, no setup needed ๐spack.cmd.env.env_loads_setup_parser(subparser)โ lists modules for an installed environment '(see spack module loads)' ๐spack.cmd.env.env_remove(args)โ Remove a named environment.
This removes an environment that's actually managed by Spack. Important vibe check: directory environments and manifests embedded in repositories should be removed manually, okay? Don't come for them with this ๐
spack.cmd.env.env_remove_setup_parser(subparser)โ remove an existing environment ๐ซถspack.cmd.env.env_rename(args)โ Rename an environment.
This renames a managed environment, or moves an anonymous one. iconic little glow-up โจ
spack.cmd.env.env_rename_setup_parser(subparser)โ rename an existing environment ๐ธspack.cmd.env.env_revert(args)โ the revert worker himself, lowkey carrying the whole rollback ๐ญspack.cmd.env.env_revert_setup_parser(subparser)โ restore environments to their state before update โ basically the undo button, periodt ๐spack.cmd.env.env_status(args)โ the status checker doing its thing ๐spack.cmd.env.env_status_setup_parser(subparser)โ print whether there is an active environment โจspack.cmd.env.env_update(args)โ the update worker, slay ๐ฅspack.cmd.env.env_update_setup_parser(subparser)โ update environments to the latest format ๐ฆspack.cmd.env.env_view(args)โ the view handler, doing the most for you ๐spack.cmd.env.env_view_setup_parser(subparser)โ manage a view associated with the environment ๐spack.cmd.env.setup_parser(subparser)โ sets up the parser for the wholeenvsquad ๐ซถspack.cmd.env.subcommand_functions = {}โ Dictionary mapping subcommand names and aliases to functions. It's giving lookup table realness ๐spack.cmd.env.subcommands = ['activate', 'deactivate', 'create', ['remove', 'rm'], ['rename', 'mv'], ['list', 'ls'], ['status', 'st'], 'loads', 'view', 'update', 'revert', 'depfile']โ List of subcommands ofspack env, and yes the aliases are right there in their little nested lists, we love a two-for-one ๐
๐งฉ spack.cmd.extensions module
spack.cmd.extensions.extensions(parser, args)โ the main extensions function, doing its thing โจspack.cmd.extensions.setup_parser(subparser)โ parser setup for extensions, no cap ๐
๐ spack.cmd.external module
exception spack.cmd.external.NoManifestFileError(message, long_message=None)โ Bases:SpackError. The moment when the manifest file is justโฆ not there ๐spack.cmd.external.external(parser, args)โ the external command entry point, periodt ๐spack.cmd.external.external_find(args)โ finds externals, iykyk ๐spack.cmd.external.external_list(args)โ lists externals for you, so helpful โจspack.cmd.external.external_read_cray_manifest(args)โ reads the Cray manifest, very Cray of it ๐spack.cmd.external.packages_to_search_for(*, names: List[str] | None, tags: List[str], exclude: List[str] | None)โ decides which packages to search for, withnames,tags, andexclude๐ซถspack.cmd.external.setup_parser(subparser)โ parser setup for external, slay ๐ฆ
๐ฅ spack.cmd.fetch module
spack.cmd.fetch.fetch(parser, args)โ the fetch function, getting your stuff ๐ฅspack.cmd.fetch.setup_parser(subparser)โ parser setup for fetch, iconic โจ
๐ spack.cmd.find module
spack.cmd.find.display_env(env, args, decorator, results)โ Display extra find output when running in an environment.
Find in an environment outputs 2 or 3 sections:
- Root specs
- Concretized roots (if asked for with
-c) - Installed specs
It's giving tiered content, bestie ๐
spack.cmd.find.find(parser, args)โ the find command's main character energy ๐spack.cmd.find.make_env_decorator(env)โ Create a function for decorating specs when in an environment. Little decorator factory, we stan ๐spack.cmd.find.query_arguments(args)โ pulls out the query arguments ๐ธspack.cmd.find.setup_parser(subparser)โ parser setup for find, no cap ๐
๐๏ธ spack.cmd.gc module
spack.cmd.gc.gc(parser, args)โ the garbage collector command, here to clean up your mess ๐ญspack.cmd.gc.roots_from_environments(args, active_env)โ grabs roots from environments, takingactive_envinto account ๐spack.cmd.gc.setup_parser(subparser)โ parser setup for gc, periodt ๐
๐ spack.cmd.gpg module
spack.cmd.gpg.gpg(parser, args)โ the gpg command dispatcher, doing the crypto things ๐ฅspack.cmd.gpg.gpg_create(args)โ create a new key ๐spack.cmd.gpg.gpg_export(args)โ export a gpg key, optionally including secret key. Ooh secret key, very sneaky ๐spack.cmd.gpg.gpg_init(args)โ add the default keys to the keyring โจspack.cmd.gpg.gpg_list(args)โ list keys available in the keyring ๐spack.cmd.gpg.gpg_publish(args)โ publish public keys to a build cache ๐ขspack.cmd.gpg.gpg_sign(args)โ sign a package, put your name on it ๐spack.cmd.gpg.gpg_trust(args)โ add a key to the keyring ๐ซถspack.cmd.gpg.gpg_untrust(args)โ remove a key from the keyring, bye bestie ๐spack.cmd.gpg.gpg_verify(args)โ verify a signed package โspack.cmd.gpg.setup_parser(subparser)โ parser setup for gpg, iconic ๐
๐ spack.cmd.graph module
spack.cmd.graph.graph(parser, args)โ the graph command, making your dependency art ๐ธspack.cmd.graph.setup_parser(subparser)โ parser setup for graph, slay โจ
โ spack.cmd.help module
spack.cmd.help.help(parser, args)โ the help command, here to answer your questions ๐ซถspack.cmd.help.setup_parser(subparser)โ parser setup for help, iconic ๐
โน๏ธ spack.cmd.info module
class spack.cmd.info.VariantFormatter(pkg)โ Bases:object.default(v)โ the default formatter move ๐property linesโ a property calledlinesโจ
spack.cmd.info.info(parser, args)โ the info command entry point, main character ๐spack.cmd.info.license(s)โ the license helper, making it pretty ๐spack.cmd.info.padder(str_list, extra=0)โ Return a function to pad elements of a list.extra=0by default, we love a sensible default ๐ซถspack.cmd.info.print_dependencies(pkg, args)โ output build, link, and run package dependencies ๐spack.cmd.info.print_detectable(pkg, args)โ output information on external detection ๐spack.cmd.info.print_licenses(pkg, args)โ Output the licenses of the project. ๐spack.cmd.info.print_maintainers(pkg, args)โ output package maintainers ๐spack.cmd.info.print_phases(pkg, args)โ output installation phases ๐ชspack.cmd.info.print_tags(pkg, args)โ output package tags ๐ท๏ธspack.cmd.info.print_tests(pkg, args)โ output relevant build-time and stand-alone tests ๐งชspack.cmd.info.print_variants(pkg, args)โ output variants ๐๏ธspack.cmd.info.print_variants_by_name(pkg)โ sorts those variants by name, so tidy โจspack.cmd.info.print_variants_grouped_by_when(pkg)โ groups variants by theirwhen, love a lil organization ๐ธspack.cmd.info.print_versions(pkg, args)โ output versions ๐ฆspack.cmd.info.print_virtuals(pkg, args)โ output virtual packages ๐spack.cmd.info.section_title(s)โ makes a section title, cute ๐spack.cmd.info.setup_parser(subparser)โ parser setup for info, slay ๐spack.cmd.info.variant(s)โ the variant helper, periodt ๐ฆspack.cmd.info.version(s)โ the version helper, iconic โญ
๐ ๏ธ spack.cmd.install module
spack.cmd.install.cache_opt(default_opt, use_buildcache)โ the cache option helper, doing buildcache math ๐ฅspack.cmd.install.compute_tests_install_kwargs(specs, cli_test_arg)โ Translate the test cli argument into the proper install argument. It's giving translator ๐spack.cmd.install.concrete_specs_from_cli(args, install_kwargs)โ Return abstract and concrete spec parsed from the command line. Both of them, we contain multitudes ๐ซถspack.cmd.install.concrete_specs_from_file(args)โ Return the list of concrete specs read from files. ๐spack.cmd.install.default_log_file(spec)โ Computes the default filename for the log file and creates the corresponding directory if not present. So proactive, love that ๐spack.cmd.install.install(parser, args)โ the install command itself, the moment ๐spack.cmd.install.install_kwargs_from_args(args)โ Translate command line arguments into a dictionary that will be passed to the package installer. Dict-making realness ๐spack.cmd.install.install_with_active_env(env: Environment, args, install_kwargs, reporter_factory)โ the path when you have an active env ๐spack.cmd.install.install_without_active_env(args, install_kwargs, reporter_factory)โ the path when you don't, also valid ๐spack.cmd.install.report_filename(args: Namespace, specs: List[Spec]) -> strโ Return the filename to be used for reporting to JUnit or CDash format. Two formats, fancy ๐spack.cmd.install.require_user_confirmation_for_overwrite(concrete_specs, args)โ makes sure you actually meant to overwrite, protective queen ๐ก๏ธspack.cmd.install.setup_parser(subparser)โ parser setup for install, slay โจ
๐ spack.cmd.license module
class spack.cmd.license.LicenseErrorโ Bases:object.add_error(error)โ adds an error to the pile ๐error_messages()โ returns the error messages ๐ฌhas_errors()โ tells you whether there are errors, yes/no energy โ
spack.cmd.license.apache2_mit_spdx = '(Apache-2.0 OR MIT)'โ Spack's license identifier. That's the whole license vibe ๐spack.cmd.license.latest_year = 2025โ Latest year that copyright applies. UPDATE THIS when bumping copyright โ literally a note to future you, don't ignore it bestie ๐ญspack.cmd.license.lgpl_exceptions = ['lib/spack/spack/cmd/license.py', 'lib/spack/spack/test/cmd/license.py']โ licensed files that can have LGPL language in them so far, just this command -- so it can find LGPL things elsewhere ๐ซถspack.cmd.license.license(parser, args)โ the license command entry point, main character energy ๐spack.cmd.license.license_line_regexes = ['Copyright 2013-(2024|2025) Lawrence Livermore National Security, LLC and other', '(Spack|sbang) [Pp]roject [Dd]evelopers\\. See the top-level COPYRIGHT file for details.', 'SPDX-License-Identifier: \\(Apache-2\\.0 OR MIT\\)']โ regexes for valid license lines at tops of files ๐spack.cmd.license.license_lines = 7โ SPDX license id must appear in the first<license_lines>lines of a file. Seven lines, count 'em โจspack.cmd.license.licensed_files = ['^bin/spack$', '^bin/spack\\.bat$', '^bin/spack\\.ps1$', '^bin/spack_pwsh\\.ps1$', '^bin/sbang$', '^bin/spack-python$', '^bin/haspywin\\.py$', '^lib/spack/spack_installable/main\\.py$', '^lib/spack/spack/(?!(test/)?util/unparse).*\\.py$', '^lib/spack/spack/.*\\.sh$', '^lib/spack/spack/.*\\.lp$', '^lib/spack/llnl/.*\\.py$', '^lib/spack/env/cc$', '^lib/spack/spack/test/data/style/broken.dummy', '^lib/spack/spack/test/data/unparse/.*\\.txt', '^lib/spack/docs/(?!command_index|spack|llnl).*\\.rst$', '^lib/spack/docs/.*\\.py$', '^lib/spack/docs/spack.yaml$', '^lib/spack/external/__init__.py$', '^share/spack/.*\\.sh$', '^share/spack/.*\\.bash$', '^share/spack/.*\\.csh$', '^share/spack/.*\\.fish$', 'share/spack/setup-env\\.ps1$', '^share/spack/qa/run-[^/]*$', '^share/spack/bash/spack-completion.in$', '^.github/actions/.*\\.py$', '^var/spack/repos/.*/package.py$']โ regular expressions for licensed files. That's the whole list, no skips, we keep every single one ๐ซถspack.cmd.license.list_files(args)โ list files in spack that should have license headers ๐spack.cmd.license.setup_parser(subparser)โ parser setup for license, iconic ๐spack.cmd.license.update_copyright_year(args)โ update copyright for the current year in all licensed files ๐๏ธspack.cmd.license.verify(args)โ verify that files in spack have the right license header โ
๐ spack.cmd.list module
spack.cmd.list.filter_by_name(pkgs, args)โ Filters the sequence of packages according to user prescriptions.
Parameters:
pkgs-- sequence of packagesargs-- parsed command line arguments
Returns: filtered and sorted list of packages. Tidy and sorted, we love ๐ธ
spack.cmd.list.formatter(func)โ Decorator used to register formatters. Registration realness ๐spack.cmd.list.get_dependencies(pkg)โ grabs a package's dependencies ๐spack.cmd.list.github_url(pkg)โ Link to a package file on github. ๐spack.cmd.list.html(pkg_names, out)โ Print out information on all packages in Sphinx HTML.
This is intended to be inlined directly into Sphinx documentation. We write HTML instead of RST for speed; generating RST from all packages causes the Sphinx build to take forever. Including this as raw HTML is much faster. Speed queen behavior โก
spack.cmd.list.list(parser, args)โ the list command itself, the moment ๐spack.cmd.list.name_only(pkgs, out)โ just the names, no extra, minimal queen โจspack.cmd.list.rows_for_ncols(elts, ncols)โ Print out rows in a table withncolsofeltslaid out vertically. Table math, we love to see it ๐spack.cmd.list.setup_parser(subparser)โ parser setup for list, slay ๐ฆspack.cmd.list.version_json(pkg_names, out)โ Print all packages with their latest versions. JSON realness ๐ฆ
๐ฆ spack.cmd.load module
spack.cmd.load.load(parser, args)โ the load command, doing the module thing ๐ฅspack.cmd.load.setup_parser(subparser)โ Parser is only constructed so that this prints a nice help message with-h. That's the whole reason, and honestly? Valid ๐
๐งญ spack.cmd.location module
spack.cmd.location.location(parser, args)spack.cmd.location.setup_parser(subparser)
Bestie, that's the whole vibe here โ two lil' functions carrying the spack location command. ๐
๐ spack.cmd.log_parse module
spack.cmd.log_parse.log_parse(parser, args)spack.cmd.log_parse.setup_parser(subparser)
๐ spack.cmd.logs module
spack.cmd.logs.logs(parser, args)spack.cmd.logs.setup_parser(subparser)
๐ซถ spack.cmd.maintainers module
spack.cmd.maintainers.maintained_packages()spack.cmd.maintainers.maintainers(parser, args)spack.cmd.maintainers.maintainers_to_packages(users=None)spack.cmd.maintainers.packages_to_maintainers(package_names=None)spack.cmd.maintainers.setup_parser(subparser)spack.cmd.maintainers.union_values(dictionary)โ given a dictionary whose values are Collections, return their union. Lowkey the most useful three lines of set math ever. ๐งฎ- Parameters:
dictionary(dict) โ dictionary whose values are all collections. - Returns: the union of all collections in the dictionary's values.
- Return type:
(set)
- Parameters:
๐งฐ spack.cmd.make_installer module
spack.cmd.make_installer.make_installer(parser, args)โ use CMake to generate a WIX installer in the newly created build directory. Windows girlies, this one's for y'all. ๐ชspack.cmd.make_installer.setup_parser(subparser)spack.cmd.make_installer.txt_to_rtf(file_path)
๐ท๏ธ spack.cmd.mark module
spack.cmd.mark.do_mark(specs, explicit)โ marks all the specs in a list. Periodt, they're marked now.- Parameters:
specs(list) โ list of specs to be markedexplicit(bool) โ whether to mark specs as explicitly installed
- Parameters:
spack.cmd.mark.find_matching_specs(specs, allow_multiple_matches=False)โ returns a list of specs matching the not-necessarily-concretized specs given from the CLI. It's giving "match me maybe." ๐- Parameters:
specs(list) โ list of specs to be matched against installed packagesallow_multiple_matches(bool) โ ifTrue, multiple matches are admitted
- Returns: list of specs
- Parameters:
spack.cmd.mark.mark(parser, args)spack.cmd.mark.mark_specs(args, specs)spack.cmd.mark.setup_parser(subparser)
๐ช spack.cmd.mirror module
class spack.cmd.mirror.IncludeFilter(args)โ Bases:objectspack.cmd.mirror.all_specs_with_all_versions()spack.cmd.mirror.concrete_specs_from_cli_or_file(args)spack.cmd.mirror.concrete_specs_from_environment()spack.cmd.mirror.concrete_specs_from_user(args)โ return the list of concrete specs that the user wants to mirror. The list is passed either from the command line or from a text file. Either way, we gotchu. โจspack.cmd.mirror.create_mirror_for_all_specs(mirror_specs, path, skip_unstable_versions)spack.cmd.mirror.create_mirror_for_individual_specs(mirror_specs, path, skip_unstable_versions)spack.cmd.mirror.extend_with_additional_versions(specs, num_versions)spack.cmd.mirror.extend_with_dependencies(specs)โ extend the input list by adding all the dependencies explicitly. Gotta bring the whole friend group. ๐ฏspack.cmd.mirror.filter_externals(specs)spack.cmd.mirror.mirror(parser, args)spack.cmd.mirror.mirror_add(args)โ add a mirror to Spackspack.cmd.mirror.mirror_create(args)โ create a directory to be used as a spack mirror, and fill it with package archivesspack.cmd.mirror.mirror_destroy(args)โ given a URL, recursively delete everything under it. โ ๏ธ Recursively. Everything. Under it. Be so serious right now before you run this.spack.cmd.mirror.mirror_list(args)โ print out available mirrors to the consolespack.cmd.mirror.mirror_remove(args)โ remove a mirror by namespack.cmd.mirror.mirror_set(args)โ configure the connection details of a mirrorspack.cmd.mirror.mirror_set_url(args)โ change the URL of a mirrorspack.cmd.mirror.process_mirror_stats(present, mirrored, error)spack.cmd.mirror.setup_parser(subparser)spack.cmd.mirror.specs_from_text_file(filename, concretize=False)โ return a list of specs read from a text file. The file should contain one spec per line. ๐- Parameters:
filename(str) โ name of the file containing the abstract specs.concretize(bool) โ ifTrue, concretize the specs before returning the list.
- Parameters:
spack.cmd.mirror.versions_per_spec(args)โ return how many versions should be mirrored per spec.
๐งฉ spack.cmd.module module
spack.cmd.module.module(parser, args)spack.cmd.module.setup_parser(subparser)
๐ฉน spack.cmd.patch module
spack.cmd.patch.patch(parser, args)spack.cmd.patch.setup_parser(subparser)
๐ฆ spack.cmd.pkg module
spack.cmd.pkg.get_grep(required=False)โ get a grep command to use withspack pkg grep.spack.cmd.pkg.pkg(parser, args, unknown_args)spack.cmd.pkg.pkg_add(args)โ add a package to the git stage withgit addspack.cmd.pkg.pkg_added(args)โ show packages added since a commitspack.cmd.pkg.pkg_changed(args)โ show packages changed since a commitspack.cmd.pkg.pkg_diff(args)โ compare packages available in two different git revisionsspack.cmd.pkg.pkg_grep(args, unknown_args)โ grep for strings inpackage.pyfiles from all repositoriesspack.cmd.pkg.pkg_hash(args)โ dump canonical source code hash for a package specspack.cmd.pkg.pkg_list(args)โ list packages associated with a particular spack git revisionspack.cmd.pkg.pkg_removed(args)โ show packages removed since a commitspack.cmd.pkg.pkg_source(args)โ dump source code for a packagespack.cmd.pkg.setup_parser(subparser)
๐ spack.cmd.providers module
spack.cmd.providers.providers(parser, args)spack.cmd.providers.setup_parser(subparser)
๐ spack.cmd.pydoc module
spack.cmd.pydoc.pydoc(parser, args)spack.cmd.pydoc.setup_parser(subparser)
๐ spack.cmd.python module
spack.cmd.python.ipython_interpreter(args)โ an ipython interpreter is intended to be interactive, so it doesn't support running a script or arguments. It's giving "just vibe with me, don't hand me homework." ๐ซงspack.cmd.python.propagate_exceptions_from(console)โ setsys.excepthookto let uncaught exceptions return1to the shell.- Parameters:
console(code.InteractiveConsole) โ the console that needs a change insys.excepthook
- Parameters:
spack.cmd.python.python(parser, args, unknown_args)spack.cmd.python.python_interpreter(args)โ a python interpreter is the default interpreterspack.cmd.python.setup_parser(subparser)
๐ spack.cmd.reindex module
spack.cmd.reindex.reindex(parser, args)
๐๏ธ spack.cmd.remove module
spack.cmd.remove.remove(parser, args)spack.cmd.remove.setup_parser(subparser)
๐๏ธ spack.cmd.repo module
spack.cmd.repo.repo(parser, args)spack.cmd.repo.repo_add(args)โ add a package source to Spack's configurationspack.cmd.repo.repo_create(args)โ create a new package repositoryspack.cmd.repo.repo_list(args)โ show registered repositories and their namespacesspack.cmd.repo.repo_remove(args)โ remove a repository from Spack's configurationspack.cmd.repo.setup_parser(subparser)
๐ spack.cmd.resource module
spack.cmd.resource.resource(parser, args)spack.cmd.resource.resource_list(args)โ list all resources known to spack (currently just patches)spack.cmd.resource.resource_show(args)โ show a resource, identified by its checksumspack.cmd.resource.setup_parser(subparser)
โป๏ธ spack.cmd.restage module
spack.cmd.restage.restage(parser, args)spack.cmd.restage.setup_parser(subparser)
๐ง spack.cmd.solve module
spack.cmd.solve.setup_parser(subparser)spack.cmd.solve.show_options=('asp', 'opt', 'output', 'solutions')โ output optionsspack.cmd.solve.solve(parser, args)
๐ฌ spack.cmd.spec module
spack.cmd.spec.setup_parser(subparser)spack.cmd.spec.spec(parser, args)
๐ฌ spack.cmd.stage module
spack.cmd.stage.setup_parser(subparser)spack.cmd.stage.stage(parser, args)
๐
spack.cmd.style module
Okay this one is the whole glam squad, buckle up. ๐
spack.cmd.style.changed_files(base='develop', untracked=True, all_files=False, root=None)โ get the list of changed files in the Spack repository.- Parameters:
base(str) โ name of base branch to evaluate differences with.untracked(bool) โ include untracked files in the list.all_files(bool) โ list all files in the repository.root(str) โ use this directory instead of the Spack prefix.
- Parameters:
spack.cmd.style.cwd_relative(path, args)โ translate a prefix-relative path to a current-working-directory-relative one.spack.cmd.style.exclude_directories=['lib/spack/external']โ list of directories to exclude from checks โ relative to the spack rootspack.cmd.style.grouper(iterable, n, fillvalue=None)โ collect data into fixed-length chunks or blocksspack.cmd.style.is_package(f)โ whether flake8 should consider a file as a core file or a package.
We run flake8 with different exceptions for the core and for packages, since we allow from spack import * and poking globals into packages.
spack.cmd.style.missing_tools(tools_to_run)spack.cmd.style.mypy_ignores=['[annotation-unchecked]']โ warnings to ignore in mypyspack.cmd.style.print_style_header(file_list, args, tools_to_run)spack.cmd.style.print_tool_header(tool)spack.cmd.style.print_tool_result(tool, returncode)spack.cmd.style.rewrite_and_print_output(output, args, re_obj=re.compile('^(.+):([0-9]+):'), replacement='{0}:{1}:')โ rewrite output with the<file>:<line>:format to respect path args (yes, "ouput" โ we keep the typo energy, it's iconic ๐)spack.cmd.style.run_black(black_cmd, file_list, args)spack.cmd.style.run_flake8(flake8_cmd, file_list, args)spack.cmd.style.run_isort(isort_cmd, file_list, args)spack.cmd.style.run_mypy(mypy_cmd, file_list, args)spack.cmd.style.setup_parser(subparser)spack.cmd.style.style(parser, args)class spack.cmd.style.tool(name, required=False)โ Bases:objectspack.cmd.style.tool_names=['isort', 'black', 'flake8', 'mypy']โ order in which tools should be run. flake8 is last so that it can double-check the results of other tools (if, e.g.,--fixwas provided). The list maps an executable name to a method to ensure the tool is bootstrapped or present in the environment.spack.cmd.style.tools={'black': (<function run_black>, False), 'flake8': (<function run_flake8>, True), 'isort': (<function run_isort>, False), 'mypy': (<function run_mypy>, False)}โ tools we run in spack style
The order is not random, bestie โ flake8 goes last on purpose so it can double-check everything the other tools just did. That's called teamwork. ๐ซถ
spack.cmd.style.validate_toolset(arg_value)โ validates the--tooland--skipargs for us (these are the sets of tools, optionally comma-separated, bestie ๐ ).
๐ท๏ธ spack.cmd.tags module
spack.cmd.tags.report_tags(category, tags)โ reports the tags, no cap.spack.cmd.tags.setup_parser(subparser)โ sets up the parser, obviously โจ.spack.cmd.tags.tags(parser, args)โ the actualtagscommand runner.
๐งช spack.cmd.test module
spack.cmd.test.create_reporter(args, specs_to_test, test_suite)โ builds a reporter for ya.spack.cmd.test.report_filename(args, test_suite)โ figures out where the report file goes.spack.cmd.test.setup_parser(subparser)โ parser setup, iconic.spack.cmd.test.test(parser, args)โ the maintestentry point.spack.cmd.test.test_find(args)โ find tests that are running or have available results ๐.- displays aliases for tests that have them, otherwise test suite content hashes (iykyk).
spack.cmd.test.test_list(args)โ list installed packages that have available tests ๐.spack.cmd.test.test_remove(args)โ remove results from Spack test suite(s) (default all).- if no test suite is listed, remove results for all suites ๐.
- removed tests can no longer be accessed for results or status, and will not appear in
spack test listresults.
spack.cmd.test.test_results(args)โ get the results from Spack test suite(s) (default all) ๐.spack.cmd.test.test_run(args)โ run tests for the specified installed packages ๐ฅ.- if no specs are listed, run tests for all packages in the current environment or all installed packages if there is no active environment.
spack.cmd.test.test_status(args)โ get the current status for the specified Spack test suite(s) โญ.
๐ธ spack.cmd.test_env module
spack.cmd.test_env.test_env(parser, args)โ runs thetest-envcommand, slay.
๐ spack.cmd.tutorial module
spack.cmd.tutorial.setup_parser(subparser)โ parser setup, periodt.spack.cmd.tutorial.tutorial(parser, args)โ runs thetutorialcommand, bestie ๐ซถ.
๐ง spack.cmd.undevelop module
spack.cmd.undevelop.setup_parser(subparser)โ parser setup, as ever โจ.spack.cmd.undevelop.undevelop(parser, args)โ theundevelopcommand runner.
๐๏ธ spack.cmd.uninstall module
spack.cmd.uninstall.all_specs_in_env(env: Environment, specs: List[Spec]) -> List[Spec]โ given a list of specs, return those that are in the env ๐.spack.cmd.uninstall.dependent_environments(specs: List[Spec], current_env: Environment | None = None) -> Dict[Environment, List[Spec]]โ works out which environments depend on these specs.spack.cmd.uninstall.do_uninstall(specs: List[Spec], force: bool = False)โ does the actual uninstalling, no dramas.spack.cmd.uninstall.find_matching_specs(env: Environment | None, specs: List[Spec], allow_multiple_matches: bool = False, origin=None) -> List[Spec]โ returns a list of specs matching the not necessarily concretized specs given from cli โจ.
Parameters
envโ optional active environmentspecsโ list of specs to be matched against installed packagesallow_multiple_matchesโ ifTruemultiple matches are admitted
Returns list of specs
Return type list
spack.cmd.uninstall.get_uninstall_list(args, specs: List[Spec], env: Environment | None)โ returns unordereduninstall_listandremove_list: these may overlap (some things may be both uninstalled and removed from the current environment) ๐.- it is assumed we are in an environment if
--removeis specified (this method raises an exception otherwise) โ so like, don't do that bestie ๐.
- it is assumed we are in an environment if
spack.cmd.uninstall.installed_dependents(specs: List[Spec]) -> List[Spec]โ finds the installed dependents of these specs.spack.cmd.uninstall.setup_parser(subparser)โ parser setup, the usual ๐.spack.cmd.uninstall.uninstall(parser, args)โ theuninstallcommand runner.spack.cmd.uninstall.uninstall_specs(args, specs)โ uninstall the given specs.
๐งช spack.cmd.unit_test module
spack.cmd.unit_test.add_back_pytest_args(args, unknown_args)โ add parsed pytest args, unknown args, and remainder together ๐ซถ.- we add some basic pytest arguments to the Spack parser to ensure that they show up in the short help, so we have to reassemble things here.
spack.cmd.unit_test.do_list(args, extra_args)โ print a list of tests than what pytest offers ๐.spack.cmd.unit_test.setup_parser(subparser)โ parser setup, iconic.spack.cmd.unit_test.unit_test(parser, args, unknown_args)โ runs theunit-testcommand.
๐ spack.cmd.unload module
spack.cmd.unload.setup_parser(subparser)โ parser is only constructed so that this prints a nice help message with-hโจ.spack.cmd.unload.unload(parser, args)โ unload spack packages from the user environment ๐.
๐ spack.cmd.url module
spack.cmd.url.name_parsed_correctly(pkg, name)โ determine if the name of a package was correctly parsed ๐ค.
Parameters
pkg(spack.package_base.PackageBase) โ the Spack packagename(str) โ the name that was extracted from the URL
Returns True if the name was correctly parsed, else False
Return type bool
spack.cmd.url.print_name_and_version(url)โ prints a URL. Underlines the detected name with dashes and the detected version with tildes ๐ .
Parameters
url(str) โ the url to parse
spack.cmd.url.remove_prefix(pkg_name)โ remove build system prefix ('py-','perl-', etc.) from a package name ๐.- after determining a name,
spack createdetermines a build system. Some build systems prepend a special string to the front of the name. Since this can't be guessed from the URL, it would be unfair to say that these names are incorrectly parsed, so we remove them.
- after determining a name,
Parameters
pkg_name(str) โ the name of the package
Returns the name of the package with any build system prefix removed
Return type str
spack.cmd.url.remove_separators(version)โ remove separator characters ('.','_', and'-') from a version โจ.- a version like
1.2.3may be displayed as1_2_3in the URL. Make sure1.2.3,1-2-3,1_2_3, and123are considered equal. Unfortunately, this also means that1.23and12.3are equal ๐ญ.
- a version like
Parameters
version(strorspack.version.Version) โ a version
Returns the version with all separator characters removed
Return type str
spack.cmd.url.setup_parser(subparser)โ parser setup, always ๐ซถ.spack.cmd.url.url(parser, args)โ theurlcommand runner.spack.cmd.url.url_list(args)โ lists URLs.spack.cmd.url.url_list_parsing(args, urls, url, pkg)โ helper function forurl_list()๐.
Parameters
args(argparse.Namespace) โ the arguments given tospack url listurls(set) โ list of URLs that have already been addedurl(strorNone) โ a URL to potentially add tourlsdepending onargspkg(spack.package_base.PackageBase) โ the Spack package
Returns the updated set of urls
Return type set
spack.cmd.url.url_parse(args)โ parses URLs.spack.cmd.url.url_stats(args)โ gets URL stats, no cap.spack.cmd.url.url_summary(args)โ summarises URLs ๐ธ.spack.cmd.url.version_parsed_correctly(pkg, version)โ determine if the version of a package was correctly parsed ๐.
Parameters
pkg(spack.package_base.PackageBase) โ the Spack packageversion(str) โ the version that was extracted from the URL
Returns True if the name was correctly parsed, else False
Return type bool
โ
spack.cmd.verify module
spack.cmd.verify.setup_parser(subparser)โ parser setup, periodt.spack.cmd.verify.verify(parser, args)โ theverifycommand runner, iconic.
๐ข spack.cmd.versions module
spack.cmd.versions.setup_parser(subparser)โ parser setup โจ.spack.cmd.versions.versions(parser, args)โ theversionscommand runner.
๐ช spack.cmd.view module
Produces a "view" of a Spack DAG ๐.
A "view" is file hierarchy representing the union of a number of Spack-installed package file hierarchies. The union is formed from:
- specs resolved from the package names given by the user (the seeds) ๐ฑ
- all dependencies of the seeds unless user specifies
--no-dependencies - less any specs with names matching the regular expressions given by
--exclude
The view can be built and tore down via a number of methods (the "actions"):
symlink:: a file system view which is a directory hierarchy that is the union of the hierarchies of the installed packages in the DAG where installed files are referenced via symlinks ๐.hardlink:: like the symlink view but hardlinks are used.statlink:: a view producing a status report of a symlink or hardlink view.
The file system view concept is imspired by Nix, implemented by โ brett.viren@gmail.com ca 2016 ๐.
All operations on views are performed via proxy objects such as YamlFilesystemView.
spack.cmd.view.disambiguate_in_view(specs, view)โ when dealing with querying actions (remove/status) we only need to disambiguate among specs in the view ๐ค.spack.cmd.view.setup_parser(sp)โ parser setup, as ever ๐.spack.cmd.view.view(parser, args)โ produce a view of a set of packages ๐ฆ.
๐ ๏ธ spack.compilers package
This module contains functions related to finding compilers on the system and configuring Spack to use multiple compilers ๐ฅ.
class spack.compilers.CacheReference(val)โ bases:object.- this acts as a hashable reference to any object (regardless of whether the object itself is hashable) and also prevents the object from being garbage-collected (so if two
CacheReferenceobjects are equal, they will refer to the same object, since it will not have been gc'ed since the creation of the firstCacheReference) ๐ .
- this acts as a hashable reference to any object (regardless of whether the object itself is hashable) and also prevents the object from being garbage-collected (so if two
exception spack.compilers.CompilerDuplicateError(compiler_spec, arch_spec)โ bases:SpackError.class spack.compilers.CompilerID(os, compiler_name, version)โ bases:tuple.- gathers the attribute values by which a detected compiler is considered unique in Spack โจ.
os: the operating systemcompiler_name: the name of the compiler (e.g.'gcc','clang', etc.)version: the version of the compiler
compiler_nameโ alias for field number 1osโ alias for field number 0versionโ alias for field number 2
- gathers the attribute values by which a detected compiler is considered unique in Spack โจ.
exception spack.compilers.CompilerSpecInsufficientlySpecificError(compiler_spec)โ bases:SpackError.class spack.compilers.DetectVersionArgs(id, variation, language, path)โ bases:tuple.- groups together the arguments needed by
detect_version. The four entries in the tuple are: id: an instance of theCompilerIDnamed tuple (version can be set toNoneas it will be detected later) ๐ธ
- groups together the arguments needed by
variationโ aNameVariationfor the file that's being tested ๐languageโ the compiler language we're testing, and it's one of'cc','cxx','fc','f77'(yes, exactly those four, no freestyling)pathโ the full path to the executable being tested ๐
Then the little aliases, which are literally just shortcuts into the tuple positions:
idโ alias for field number 0languageโ alias for field number 2pathโ alias for field number 3variationโ alias for field number 1
exception spack.compilers.InvalidCompilerConfigurationError(compiler_spec)
Bases: SpackError
It's giving "your compiler config is not it" energy ๐ญ โ Spack raises this when the compiler configuration just doesn't add up.
exception spack.compilers.NameVariation(prefix, suffix)
Bases: tuple
Variations on a matched compiler name. Cute little tuple of two things:
prefixโ alias for field number 0suffixโ alias for field number 1
exception spack.compilers.NoCompilerForSpecError(compiler_spec, target)
Bases: SpackError
No compiler matched the spec you asked for. Tragic, but at least it tells you. ๐
exception spack.compilers.NoCompilersError
Bases: SpackError
Literally zero compilers. Like an empty group chat. ๐ซ
exception spack.compilers.UnknownCompilerError(compiler_name)
Bases: SpackError
You dropped a compiler name Spack doesn't recognize. Not iconic of you, ngl.
spack.compilers.add_compilers_to_config(compilers, scope=None)โ Add compilers to the config for the specified architecture. โจ- Parameters
compilersโ a list ofCompilerobjects.scopeโ configuration scope to modify.
- Parameters
spack.compilers.all_compiler_module_names() -> List[str]โ returns the module names, that's it, that's the tweet.spack.compilers.all_compiler_names() -> List[str]โ same vibe, but names.spack.compilers.all_compiler_specs(scope=None, init_config=True)โ all the compiler specs. Defaults are right there in the signature, don't @ me.spack.compilers.all_compiler_types()โ all the compiler types. ๐งspack.compilers.all_compilers(scope=None, init_config=True)โ every compiler, scope and init config optional.spack.compilers.all_compilers_config(configuration: Configuration, *, scope: str | None = None, init_config: bool = True) -> List[Compiler]โ Return a set of specs for all the compiler versions currently available to build with. These are instances ofCompilerSpec. ๐spack.compilers.all_compilers_from(configuration, scope=None, init_config=True)โ pulls the compilers out of a config, with the usualscopeandinit_configknobs.spack.compilers.all_os_classes()โ Return the list of classes for all operating systems available on this platform. ๐ฅ๏ธspack.compilers.arguments_to_detect_version_fn(operating_system: OperatingSystem, paths: List[str]) -> List[DetectVersionArgs]โ Returns a list ofDetectVersionArgstuples to be used in a corresponding function to detect compiler versions.
The operating_system instance can customize the behavior of this function by providing a method called with the same name. iykyk ๐
- Parameters
operating_systemโ the operating system on which we are looking for compilerspathsโ paths to search for compilers
- Returns
- List of
DetectVersionArgstuples. Each item in the list will be later mapped to the corresponding function call to detect the version of the compilers in this OS.
- List of
spack.compilers.class_for_compiler_name(compiler_name)โ Given a compiler module name, get the correspondingCompilerclass. โจspack.compilers.compiler_config_files()โ the compiler config files. Short and sweet.spack.compilers.compiler_for_spec(cspec_like, *args, **kwargs)โ takes a spec-ish thing plus*argsand**kwargs, returns the compiler.spack.compilers.compiler_from_dict(items)โ builds a compiler out of a dict. ๐spack.compilers.compiler_specs_for_arch(arch_spec, scope=None)โ compiler specs for an arch, optional scope.spack.compilers.compilers_for_arch(arch_spec, scope=None)โ the compilers themselves for an arch, optional scope.spack.compilers.compilers_for_spec(cspec_like, *args, **kwargs)โ compilers matching a spec-ish argument.spack.compilers.detect_version(detect_version_args: DetectVersionArgs) -> Tuple[DetectVersionArgs | None, str | None]โ Computes the version of a compiler and adds it to the information passed as input.
As this function is meant to be executed by worker processes it won't raise any exception but instead will return a (value, error) tuple that needs to be checked by the code dispatching the calls. โ ๏ธ So bestie, please actually check that tuple, don't just vibe past it.
- Parameters
detect_version_argsโ information on the compiler for which we should detect the version.
- Returns
- A
(DetectVersionArgs, error)tuple. IferrorisNonethe version of the compiler was computed correctly and the first argument of the tuple will contain it. Otherwiseerroris a string containing an explanation on why the version couldn't be computed.
- A
spack.compilers.find(cspec_like, *args, **kwargs)โ find, with*argsand**kwargs. Go off.spack.compilers.find_compilers(path_hints: List[str] | None = None, *, mixed_toolchain=False) -> List[Compiler]โ Return the list of compilers found in the paths given as arguments.- Parameters
path_hintsโ list of path hints where to look for. A sensible default based on thePATHenvironment variable will be used if the value isNonemixed_toolchainโ allow mixing compilers from different toolchains if otherwise missing for a certain language
- Parameters
spack.compilers.find_new_compilers(path_hints: List[str] | None = None, scope: str | None = None, *, mixed_toolchain: bool = False)โ Same asfind_compilersbut return only the compilers that are not already incompilers.yaml. Fresh finds only, no repeats. ๐- Parameters
path_hintsโ list of path hints where to look for. A sensible default based on thePATHenvironment variable will be used if the value isNonescopeโ scope to look for a compiler. IfNoneconsider the merged configuration.mixed_toolchainโ allow mixing compilers from different toolchains if otherwise missing for a certain language
- Parameters
spack.compilers.find_specs_by_arch(cspec_like, *args, **kwargs)โ find specs by arch, plus the usual pass-through arguments.spack.compilers.get_compiler_config(configuration: Configuration, *, scope: str | None = None, init_config: bool = False) -> List[Dict]โ Return the compiler configuration for the specified architecture. Note theinit_configdefault isFalsehere, lowkey important ๐spack.compilers.get_compiler_config_from_packages(configuration: Configuration, *, scope: str | None = None) -> List[Dict]โ Return the compiler configuration frompackages.yaml. ๐ฆspack.compilers.get_compiler_duplicates(cspec_like, *args, **kwargs)โ the duplicate compilers situation. Yes it happens.spack.compilers.get_compilers(config, cspec=None, arch_spec=None)โ grab compilers out of a config, filtered by an optionalcspecand/orarch_spec.spack.compilers.is_mixed_toolchain(compiler)โ ReturnsTrueif the current compiler is a mixed toolchain,Falseotherwise. ๐- Parameters
compiler(spack.compiler.Compiler) โ a valid compiler object
- Parameters
spack.compilers.make_compiler_list(detected_versions: List[DetectVersionArgs], mixed_toolchain: bool = False) -> List[Compiler]โ Process a list of detected versions and turn them into a list of compiler specs.- Parameters
detected_versionsโ list ofDetectVersionArgscontaining a valid versionmixed_toolchainโ allow mixing compilers from different toolchains if langauge is missing (and yes, that typo is verbatim, we don't fix, we preserve ๐ซถ)
- Returns
- list of
Compilerobjects
- list of
- Return type
list
- Parameters
spack.compilers.make_mixed_toolchain(compilers: List[Tuple[CompilerID, NameVariation, dict]]) -> Noneโ Add missing compilers across toolchains when they are missing for a particular language. This currently only adds the most sensiblegfortranto(apple)-clangif it doesn't have a fortran compiler (noflang). ๐spack.compilers.pkg_spec_for_compiler(cspec)โ Return the spec of the package that provides the compiler. ๐spack.compilers.remove_compiler_from_config(cspec_like, *args, **kwargs)โ yeets a compiler out of the config.spack.compilers.select_new_compilers(compilers, scope=None)โ Given a list of compilers, remove those that are already defined in the configuration.spack.compilers.supported(cspec_like, *args, **kwargs)โ is this compiler spec supported, with*argsand**kwargspassed along.spack.compilers.supported_compilers() -> List[str]โ Return a set of names of compilers supported by Spack.
See available_compilers() to get a list of all the available versions of supported compilers. ๐
spack.compilers.supported_compilers_for_host_platform() -> List[str]โ Return a set of compiler class objects supported by Spack that are also supported by the current host platform. ๐ปspack.compilers.supported_compilers_for_platform(platform: Platform) -> List[str]โ Return a set of compiler class objects supported by Spack that are also supported by the provided platform.- Parameters
platform(str) โ string representation of platform for which compiler compatability should be determined (typo preserved, as always ๐ )
- Parameters
๐ธ SUBMODULES ๐ธ
๐ผ spack.compilers.aocc module ๐ผ
class spack.compilers.aocc.Aocc(cspec, operating_system, target, paths, modules=None, alias=None, environment=None, extra_rpaths=None, enable_implicit_rpaths=None, **kwargs)
Bases: Compiler
The AMD-flavored one. Let's get into the member era โจ
PrgEnv: str | None = 'PrgEnv-aocc'PrgEnv_compiler: str | None = 'aocc'- property
c11_flagโ the C11 flag vibe. - property
c99_flagโ the C99 flag vibe. cc_names: List[str] = ['clang']- property
cc_pic_flagโ Returns the flag used by the C compiler to produce Position Independent Code (PIC). ๐ - property
cflagsโ the C flags. - property
cxx11_flagโ C++11 flag. - property
cxx14_flagโ C++14 flag. - property
cxx17_flagโ C++17 flag. cxx_names: List[str] = ['clang++']- property
cxx_pic_flagโ Returns the flag used by the C++ compiler to produce Position Independent Code (PIC). ๐ - property
cxxflagsโ the C++ flags. - property
debug_flagsโ the debug flags. - classmethod
extract_version_from_output(output)โ Extracts the version from compiler's output. ๐ f77_names: List[str] = ['flang']- property
f77_pic_flagโ Returns the flag used by the F77 compiler to produce Position Independent Code (PIC). ๐ fc_names: List[str] = ['flang']- property
fc_pic_flagโ Returns the flag used by the FC compiler to produce Position Independent Code (PIC). ๐ - property
fflagsโ the Fortran flags. - property
link_pathsโ the link paths. - property
openmp_flagโ the OpenMP flag. - property
opt_flagsโ the optimization flags. required_libs = ['libclang']- property
stdcxx_libsโ the standard C++ libs. - property
verbose_flagโ This property should be overridden in the compiler subclass if a verbose flag is available.
If it is not overridden, it is assumed to not be supported. ๐คทโโ๏ธ
version_argument = '--version'โ Compiler argument that produces version information. ๐ท๏ธ
spack.compilers.apple_clang module ๐
Okay so Apple's Clang gets its own lil' class and honestly? It's basically regular Clang wearing an Apple-branded hoodie.
class spack.compilers.apple_clang.AppleClang(cspec, operating_system, target, paths, modules=None, alias=None, environment=None, extra_rpaths=None, enable_implicit_rpaths=None, **kwargs)Bases: Clang
It's got all the standard-flag properties you'd expect, each one a property:
c11_flagโ the C11 flag, as a property โจc17_flagโ C17 flag, propertyc23_flagโ C23 flag, propertyc99_flagโ C99 flag, propertycxx11_flagโ C++11 flag, propertycxx14_flagโ C++14 flag, propertycxx17_flagโ C++17 flag, propertycxx20_flagโ C++20 flag, propertycxx23_flagโ C++23 flag, property
Then there's:
classmethod extract_version_from_output(output)โ pulls the version straight out of the compiler's output, cute and to the point ๐
And the one plain attribute:
openmp_flag = '-Xpreprocessor -fopenmp'โ no cap, that's literally the whole value, two flags in a trench coat ๐งฅ
spack.compilers.arm module ๐ช
class spack.compilers.arm.Arm(cspec, operating_system, target, paths, modules=None, alias=None, environment=None, extra_rpaths=None, enable_implicit_rpaths=None, **kwargs)Bases: Compiler
Language-standard properties first:
c11_flagโ propertyc99_flagโ propertycxx11_flagโ propertycxx14_flagโ propertycxx17_flagโ property
Compiler name lists (these are List[str], bestie):
cc_names: List[str] = ['armclang']cxx_names: List[str] = ['armclang++']f77_names: List[str] = ['armflang']fc_names: List[str] = ['armflang']
PIC flags, one per language, all properties:
cc_pic_flagโ returns the flag the C compiler uses to make Position Independent Code (PIC).cxx_pic_flagโ same deal but for the C++ compiler and PIC.f77_pic_flagโ the F77 compiler's PIC flag, yep.fc_pic_flagโ and the FC compiler's PIC flag ๐
Version helpers as classmethods:
classmethod f77_version(f77)classmethod fc_version(fc)
Link paths, verbatim:
link_paths = {'cc': 'arm/armclang', 'cxx': 'arm/armclang++',
'f77': 'arm/armflang', 'fc': 'arm/armflang'}More properties:
openmp_flagโ propertyopt_flagsโ propertyverbose_flagโ this one should be overridden in the compiler subclass if a verbose flag exists. If it's not overridden, the assumption is that it's simply not supported. So don't come for it ๐
And the stats:
required_libs = ['libclang', 'libflang']version_argument = '--version'โ the compiler argument that spits out version info.version_regex = 'Arm C\\/C\\+\\+\\/Fortran Compiler version ([\\d\\.]+) 'โ the regex that scoops the version out of the output.
spack.compilers.cce module ๐ฅ
class spack.compilers.cce.Cce(*args, **kwargs)Bases: Compiler
Cray compiler environment compiler. Say that three times fast, I dare you.
Programmable-environment bits:
PrgEnv: str | None = 'PrgEnv-cray'PrgEnv_compiler: str | None = 'cce'
Standard-flag properties:
c11_flagโ propertyc99_flagโ propertycxx11_flagโ propertycxx14_flagโ propertycxx17_flagโ property
Compiler names:
cc_names: List[str] = ['craycc']cxx_names: List[str] = ['crayCC']f77_names: List[str] = ['crayftn']fc_names: List[str] = ['crayftn']
PIC flags, all properties:
cc_pic_flagโ the flag the C compiler uses to emit Position Independent Code (PIC).cxx_pic_flagโ the C++ compiler's PIC flag.f77_pic_flagโ the F77 compiler's PIC flag.fc_pic_flagโ the FC compiler's PIC flag, periodt ๐
Extra properties:
debug_flagsโ propertyis_clang_basedโ propertylink_pathsโ property (yes, a property here, not a plain dict)openmp_flagโ propertystdcxx_libsโ propertyverbose_flagโ should be overridden in the compiler subclass if a verbose flag is available; if you leave it alone, it's assumed unsupported. Respect the default ๐ซถ
Patterns and versions:
suffixes = ['-mp-\\d\\.\\d']version_argument = '--version'โ the argument that produces version info.version_regex = '[Cc]ray (?:clang|C :|C\\+\\+ :|Fortran :) [Vv]ersion.*?(\\d+(\\.\\d+)+)'โ the regex that digs the version out of the output.
spack.compilers.clang module ๐ฆ
class spack.compilers.clang.Clang(cspec, operating_system, target, paths, modules=None, alias=None, environment=None, extra_rpaths=None, enable_implicit_rpaths=None, **kwargs)Bases: Compiler
Standard-flag properties, all of 'em:
c11_flagโ propertyc17_flagโ propertyc23_flagโ propertyc99_flagโ propertycxx11_flagโ propertycxx14_flagโ propertycxx17_flagโ property
Names:
cc_names: List[str] = ['clang']cxx_names: List[str] = ['clang++']f77_names: List[str] = ['flang-new', 'flang']โ two of them, newest first, slay ๐fc_names: List[str] = ['flang-new', 'flang']โ same duo here.
PIC flags:
cc_pic_flagโ the C compiler's flag for Position Independent Code (PIC).cxx_pic_flagโ the C++ compiler's PIC flag.f77_pic_flagโ the F77 compiler's PIC flag.fc_pic_flagโ the FC compiler's PIC flag.
Everything else:
debug_flagsโ propertyclassmethod extract_version_from_output(output)โ extracts the version from the compiler's output, literally its whole job.link_pathsโ propertyopenmp_flag = '-fopenmp'opt_flagsโ propertyrequired_libs = ['libclang']verbose_flagโ override this in a compiler subclass if a verbose flag exists; if not overridden, it's assumed unsupported. No flag, no problem ๐version_argument = '--version'โ the argument that makes the compiler confess its version.
Then there are two module-level lists for mixed-Fortran setups, and they are such a mood:
spack.compilers.clang.f77_mapping = [('gfortran', 'clang/gfortran'), ('xlf_r', 'xl_r/xlf_r'), ('xlf', 'xl/xlf'), ('pgfortran', 'pgi/pgfortran'), ('ifort', 'intel/ifort')]compiler symlink mappings for mixed f77 compilers โ that's what it is, no notes.
spack.compilers.clang.fc_mapping = [('gfortran', 'clang/gfortran'), ('xlf90_r', 'xl_r/xlf90_r'), ('xlf90', 'xl/xlf90'), ('pgfortran', 'pgi/pgfortran'), ('ifort', 'intel/ifort')]compiler symlink mappings for mixed f90/fc compilers โ the fc-flavoured twin ๐
spack.compilers.fj module ๐ธ
class spack.compilers.fj.Fj(cspec, operating_system, target, paths, modules=None, alias=None, environment=None, extra_rpaths=None, enable_implicit_rpaths=None, **kwargs)Bases: Compiler
Fujitsu, bestie. Let's go.
c11_flagโ propertyc99_flagโ propertycxx11_flagโ propertycxx14_flagโ propertycxx17_flagโ propertycxx98_flagโ property (yes, 98 gets a mention too ๐)
Names:
cc_names: List[str] = ['fcc']cxx_names: List[str] = ['FCC']โ capital letters, case matters, iykykf77_names: List[str] = ['frt']fc_names: List[str] = ['frt']
PIC flags, as properties:
cc_pic_flagโ C compiler's Position Independent Code (PIC) flag.cxx_pic_flagโ C++ compiler's PIC flag.f77_pic_flagโ F77 compiler's PIC flag.fc_pic_flagโ FC compiler's PIC flag.
The rest:
debug_flagsโ propertyopenmp_flagโ propertyopt_flagsโ propertyverbose_flagโ should be overridden in the compiler subclass if a verbose flag is available; otherwise it's assumed not supported, and that's the vibe.link_paths = {'cc': 'fj/fcc', 'cxx': 'fj/case-insensitive/FCC', 'f77': 'fj/frt', 'fc': 'fj/frt'}โ note thecase-insensitivein there, iconic.required_libs = ['libfj90i', 'libfj90f', 'libfjsrcinfo']โ all three, don't @ me.version_argument = '--version'โ version info argument.version_regex = '\\((?:FCC|FRT)\\) ([a-z\\d.]+)'โ the regex that extracts the version from the output.
spack.compilers.gcc module ๐ง
class spack.compilers.gcc.Gcc(cspec, operating_system, target, paths, modules=None, alias=None, environment=None, extra_rpaths=None, enable_implicit_rpaths=None, **kwargs)Bases: Compiler
The people's compiler, fr fr.
Programmable-environment fields:
PrgEnv: str | None = 'PrgEnv-gnu'PrgEnv_compiler: str | None = 'gcc'
Standard-flag properties โ the whole collection:
c11_flagโ propertyc99_flagโ propertycxx11_flagโ propertycxx14_flagโ propertycxx17_flagโ propertycxx20_flagโ propertycxx23_flagโ propertycxx98_flagโ property
Names:
cc_names: List[str] = ['gcc']cxx_names: List[str] = ['g++']f77_names: List[str] = ['gfortran']fc_names: List[str] = ['gfortran']
PIC flags:
cc_pic_flagโ the C compiler's Position Independent Code (PIC) flag.cxx_pic_flagโ the C++ compiler's PIC flag.f77_pic_flagโ the F77 compiler's PIC flag.fc_pic_flagโ the FC compiler's PIC flag.
Also:
debug_flagsโ propertyclassmethod default_version(cc)โ okay this one has lore. Older versions of gcc use the-dumpversionoption, and the output looks like this:
4.4.7In GCC 7, that option was changed so it only returns the major version of the compiler:
7So a new -dumpfullversion option was added that gives us what we actually want:
7.2.0classmethod f77_version(f77)โ classmethod for the f77 version.classmethod fc_version(fc)โ okay so this is how we sniff out the gfortran version, and bestie, it is a journey. Older gfortrans use the-dumpversionoption and the output is giving classic compiler-core:
GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)
Copyright (C) 2010 Free Software Foundation, Inc.or, if your compiler is feeling minimalist:
4.8.5then GCC 7 showed up and said "actually no" and changed the option to only return the major version:
7so they had to add a whole new -dumpfullversion option to get us the thing we actually want:
7.2.0iconic behaviour, truly. ๐
link_paths = {'cc': 'gcc/gcc', 'cxx': 'gcc/g++', 'f77': 'gcc/gfortran', 'fc': 'gcc/gfortran'}โ the map of which compilers live where, all nice and literal.property openmp_flagโ the flag for OpenMP, as the name so rudely gives away.property opt_flagsโ optimization flags, come through. ๐ฅproperty prefixโ queries the compiler for its install prefix. This is the install path as reported by the compiler itself. And before you come for me: the paths forcc,cxx, etc. are not enough to find the install prefix, because those can be symlinks, wrappers, or just filenames instead of absolute paths. ๐required_libs = ['libgcc', 'libgfortran']โ the libs we need, no cap.property stdcxx_libsโ the C++ standard library bits.suffixes = ['-mp-\\d+(?:\\.\\d+)?', '-\\d+(?:\\.\\d+)?', '\\d\\d']โ the suffixes, exactly as they are, don't touch them.property verbose_flagโ so this one should be overridden in the compiler subclass if a verbose flag is actually available. If it's not overridden? We just assume it's not supported. Simple as that. ๐ซถ
๐ธ spack.compilers.intel module ๐ธ
๐ class spack.compilers.intel.Intel(cspec, operating_system, target, paths, modules=None, alias=None, environment=None, extra_rpaths=None, enable_implicit_rpaths=None, **kwargs)
Bases: Compiler
PrgEnv: str | None = 'PrgEnv-intel'โ the Cray programming environment name.PrgEnv_compiler: str | None = 'intel'โ what the PrgEnv calls the compiler.property c11_flagโ the C11 flag. Yes bestie, we have standards.property c99_flagโ C99, the classic.cc_names: List[str] = ['icc']โ the C compiler's name, and it'sicc.property cc_pic_flagโ returns the flag used by the C compiler to produce Position Independent Code (PIC). ๐property cxx11_flagโ C++11 flag.property cxx14_flagโ C++14 flag.property cxx17_flagโ C++17 flag.cxx_names: List[str] = ['icpc']โ C++ compiler name,icpc.property cxx_pic_flagโ returns the flag used by the C++ compiler to produce Position Independent Code (PIC). โจproperty debug_flagsโ the debug flags, obviously.f77_names: List[str] = ['ifort']โ Fortran 77 compiler name,ifort.property f77_pic_flagโ returns the flag used by the F77 compiler to produce Position Independent Code (PIC). ๐ธfc_names: List[str] = ['ifort']โ Fortran compiler name, alsoifort.property fc_pic_flagโ returns the flag used by the FC compiler to produce Position Independent Code (PIC). ๐ฆlink_paths = {'cc': 'intel/icc', 'cxx': 'intel/icpc', 'f77': 'intel/ifort', 'fc': 'intel/ifort'}โ the full path map, lowkey satisfying.property openmp_flagโ OpenMP flag.property opt_flagsโ optimization flags.required_libs = ['libirc', 'libifcore', 'libifcoremt', 'libirng']โ the required libs, all four of them, don't lose any.property stdcxx_libsโ the C++ standard library flags.property verbose_flagโ should be overridden in the compiler subclass if a verbose flag is available. If it's not overridden, it's assumed not supported.version_argument = '--version'โ Compiler argument that produces version information. ๐งversion_regex = '\\((?:IFORT|ICC)\\) ([^ ]+)'โ Regex used to extract version from compiler's output.
๐ฅ spack.compilers.msvc module ๐ฅ
โญ class spack.compilers.msvc.CmdCall(*cmds)
Bases: object
Compose a call to cmd for an ordered series of cmd commands/scripts. Very "assemble the squad" energy.
property cmd_lineโ the assembled command line itself.
๐ class spack.compilers.msvc.Msvc(*args, **kwargs)
Bases: Compiler
cc_names: List[str] = ['cl']โ C compiler:cl.property cl_versionโ Cl toolset version. ๐ฉตcxx_names: List[str] = ['cl']โ C++ compiler, alsocl.f77_names: List[str] = ['ifx']โ Fortran 77 compiler:ifx.classmethod f77_version(f77)โ classmethod for the F77 version.fc_names: List[str] = ['ifx']โ Fortran compiler:ifx.classmethod fc_version(fc)โ classmethod for the FC version.property ifx_versionโ Ifx compiler version associated with this version of MSVC.ignore_version_errors: Sequence[int] = [1]โ Return values to ignore when invoking the compiler to get its version. Bestie, sometimes1just means "we got the info, chill". ๐ญis_supported_on_platform()โ whether this whole thing is even supported here.link_paths = {'cc': '', 'cxx': '', 'f77': '', 'fc': ''}โ all empty strings, no paths, that's just how MSVC rolls.property msvc_versionโ This is the VCToolset version, NOT the actual version of theclcompiler. For the CL version, queryMsvc.cl_version. Do not mix them up, I'm begging. ๐property platform_toolset_verโ This is the platform toolset version of the current MSVC compiler, i.e.142. This is different from the VC toolset version as established byshort_msvc_version.setup_custom_environment(pkg, env)โ Set environment variables for MSVC using the Microsoft-provided script. Not me trusting a vendor script, but okay. ๐property short_msvc_versionโ This is the shorthand VCToolset version of formMSVC<short-ver>, NOT the full version. For that, seeMsvc.msvc_versionorMSVC.platform_toolset_verfor the raw platform toolset version.suffixes = []โ empty list, nothing to add.version_argument = ''โ Compiler argument that produces version information. (It's empty. That's the whole vibe.)version_regex = '([1-9][0-9]*\\.[0-9]*\\.[0-9]*)'โ Regex used to extract version from compiler's output.property vs_rootโ where Visual Studio lives.
๐ class spack.compilers.msvc.VCVarsInvocation(script, arch, msvc_version)
Bases: VarsInvocation
property archโ the target architecture.command_str()โ spits out the command as a string.property sdk_verโ Accessor for the Windows SDK version property.
Note: This property may not be set by the calling context, and as such this property will return an empty string. This property will ONLY be set if the SDK package is a dependency somewhere in the Spack DAG of the package for which we are constructing an MSVC compiler env. Otherwise this property should be unset to allow the VCVARS script to use its internal heuristics to determine the appropriate SDK version. ๐
property vcvars_verโ the vcvars version.
๐ class spack.compilers.msvc.VarsInvocation(script)
Bases: object
command_str()โ the command string.property scriptโ the script.
๐ spack.compilers.msvc.get_valid_fortran_pth()
Assign maximum available fortran compiler version. (Yes, the name really is pth. We don't ask questions.) ๐
๐ฆ spack.compilers.nag module ๐ฆ
๐ class spack.compilers.nag.Nag(cspec, operating_system, target, paths, modules=None, alias=None, environment=None, extra_rpaths=None, enable_implicit_rpaths=None, **kwargs)
Bases: Compiler
cc_names: List[str] = []โ empty. No C compiler names here, periodt.property cxx11_flagโ the C++11 flag.cxx_names: List[str] = []โ also empty.property debug_flagsโ debug flags.property disable_new_dtagsโ disables new dtags.property enable_new_dtagsโ enables new dtags. (They come as a matched set, cute.)classmethod extract_version_from_output(output)โ Extracts the version from compiler's output.f77_names: List[str] = ['nagfor']โ Fortran 77 compiler:nagfor.property f77_pic_flagโ Returns the flag used by the F77 compiler to produce Position Independent Code (PIC). ๐ธproperty f77_rpath_argโ the rpath argument for F77.fc_names: List[str] = ['nagfor']โ Fortran compiler:nagfor.property fc_pic_flagโ Returns the flag used by the FC compiler to produce Position Independent Code (PIC). โจproperty fc_rpath_argโ the rpath argument for FC.link_paths = {'cc': 'cc', 'cxx': 'c++', 'f77': 'nag/nagfor', 'fc': 'nag/nagfor'}โ look at those plain little paths, love that for us.property linker_argโ Flag that need to be used to pass an argument to the linker. (Yes it says "need", it's fine.)property openmp_flagโ OpenMP flag.property opt_flagsโ optimization flags.property verbose_flagโ should be overridden in the compiler subclass if a verbose flag is available. If it's not overridden, it's assumed to not be supported. ๐ซถversion_argument = '-V'โ Compiler argument that produces version information.
โญ spack.compilers.nvhpc module โญ
๐ class spack.compilers.nvhpc.Nvhpc(cspec, operating_system, target, paths, modules=None, alias=None, environment=None, extra_rpaths=None, enable_implicit_rpaths=None, **kwargs)
Bases: Compiler
PrgEnv: str | None = 'PrgEnv-nvhpc'โ Cray environment name.PrgEnv_compiler: str | None = 'nvhpc'โ and the compiler name it uses.property c11_flagโ the C11 flag.property c99_flagโ the C99 flag.cc_names: List[str] = ['nvc']โ C compiler:nvc.property cc_pic_flagโ Returns the flag used by the C compiler to produce Position Independent Code (PIC). ๐property cxx11_flagโ C++11 flag.property cxx14_flagโ C++14 flag.property cxx17_flagโ C++17 flag.cxx_names: List[str] = ['nvc++']โ C++ compiler:nvc++.property cxx_pic_flagโ Returns the flag used by the C++ compiler to produce Position Independent Code (PIC). ๐property debug_flagsโ debug flags.f77_names: List[str] = ['nvfortran']โ Fortran 77 compiler:nvfortran.property f77_pic_flagโ Returns the flag used by the F77 compiler to produce Position Independent Code (PIC). ๐ฆfc_names: List[str] = ['nvfortran']โ Fortran compiler:nvfortran.property fc_pic_flagโ Returns the flag used by the FC compiler to produce Position Independent Code (PIC). ๐งlink_paths = {'cc': 'nvhpc/nvc', 'cxx': 'nvhpc/nvc++', 'f77': 'nvhpc/nvfortran', 'fc': 'nvhpc/nvfortran'}โ the full path map, all nested undernvhpc/.property openmp_flagโ OpenMP flag.property opt_flagsโ optimization flags.required_libs = ['libnvc', 'libnvf']โ required libs, both of them.property stdcxx_libsโ C++ standard library flags.property verbose_flagโ should be overridden in the compiler subclass if a verbose flag is available. If it isn't, it's assumed not supported.version_argument = '--version'โ Compiler argument that produces version information.version_regex = 'nv[^ ]* (?:[^ ]+ Dev-r)?([0-9.]+)(?:-[0-9]+)?'โ Regex used to extract version from compiler's output.
๐ spack.compilers.oneapi module ๐
๐ class spack.compilers.oneapi.Oneapi(cspec, operating_system, target, paths, modules=None, alias=None, environment=None, extra_rpaths=None, enable_implicit_rpaths=None, **kwargs)
Bases: Compiler
PrgEnv: str | None = 'PrgEnv-oneapi'โ Cray environment name.PrgEnv_compiler: str | None = 'oneapi'โ compiler name for the PrgEnv.property c11_flagโ the C11 flag.property c99_flagโ the C99 flag.cc_names: List[str] = ['icx']โ C compiler:icx.
โฆand that's the whole vibe of this compiler class. The rest of its props are basically โจ compiler flags on demand โจ, so let's go through them one by one ๐ซถ
property cc_pic_flagโ returns the flag the C compiler uses to produce Position Independent Code (PIC).property cxx11_flagโ listed with no description attached, it's just vibing there ๐property cxx14_flagโ same situation, no description in the source.property cxx17_flagโ also undocumented, but very much present.property cxx20_flagโ and this one too, no description, just existing.cxx_names: List[str] = ['icpx']โ the C++ compiler names, literally that list.property cxx_pic_flagโ returns the flag the C++ compiler uses to produce Position Independent Code (PIC).property debug_flagsโ no description given, it's there though ๐f77_names: List[str] = ['ifx']โ F77 compiler names.property f77_pic_flagโ returns the flag the F77 compiler uses to produce Position Independent Code (PIC).fc_names: List[str] = ['ifx']โ FC compiler names.property fc_pic_flagโ returns the flag the FC compiler uses to produce Position Independent Code (PIC).
The link paths, bestie โ she's a whole dict, look:
link_paths = {'cc': 'oneapi/icx', 'cxx': 'oneapi/icpx', 'f77':
'oneapi/ifx', 'fc': 'oneapi/ifx'}property openmp_flagโ no description, just a property doing its thing.property opt_flagsโ again, no description in the source.required_libs = ['libirc', 'libifcore', 'libifcoremt', 'libirng', 'libsvml', 'libintlc', 'libimf', 'libsycl', 'libOpenCL']โ the required libs, full list, no cuts โจsetup_custom_environment(pkg, env)โ set any environment variables necessary to use the compiler. Iconic little helper, honestly.property stdcxx_libsโ undocumented, but she's in the list.property verbose_flagโ this property should be overridden in the compiler subclass if a verbose flag is available. If it is not overridden, it is assumed to not be supported. So like, override it or get nothing ๐ซversion_argument = '--version'โ the compiler argument that produces version information.version_regexโ the regex used to extract the version from the compiler's output. Copying it exactly, no cap:
version_regex = '(?:(?:oneAPI DPC\\+\\+(?:\\/C\\+\\+)?
Compiler)|(?:\\(IFORT\\))|(?:\\(IFX\\))) (\\S+)'๐งก spack.compilers.pgi module
class spack.compilers.pgi.Pgi
class spack.compilers.pgi.Pgi(cspec, operating_system, target, paths,
modules=None, alias=None, environment=None, extra_rpaths=None,
enable_implicit_rpaths=None, **kwargs)Bases: Compiler
PrgEnv: str | None = 'PrgEnv-pgi'โ the PrgEnv value.PrgEnv_compiler: str | None = 'pgi'โ the PrgEnv compiler value.property c11_flagโ no description given.property c99_flagโ no description given either.cc_names: List[str] = ['pgcc']โ C compiler names.property cc_pic_flagโ returns the flag the C compiler uses to produce Position Independent Code (PIC).property cxx11_flagโ undocumented, present anyway.cxx_names: List[str] = ['pgc++', 'pgCC']โ C++ compiler names, both of them ๐property cxx_pic_flagโ returns the flag the C++ compiler uses to produce Position Independent Code (PIC).property debug_flagsโ no description in the source.f77_names: List[str] = ['pgfortran', 'pgf77']โ F77 compiler names.property f77_pic_flagโ returns the flag the F77 compiler uses to produce Position Independent Code (PIC).fc_names: List[str] = ['pgfortran', 'pgf95', 'pgf90']โ FC compiler names, three of 'em.property fc_pic_flagโ returns the flag the FC compiler uses to produce Position Independent Code (PIC).ignore_version_errors: Sequence[int] = [2]โ return values to ignore when invoking the compiler to get its version. Lowkey useful ๐ซถ
And the link paths:
link_paths = {'cc': 'pgi/pgcc', 'cxx': 'pgi/pgc++', 'f77':
'pgi/pgfortran', 'fc': 'pgi/pgfortran'}property openmp_flagโ no description, still there.property opt_flagsโ same, no description.required_libs = ['libpgc', 'libpgf90']โ required libs, both listed.property stdcxx_libsโ undocumented.property verbose_flagโ this property should be overridden in the compiler subclass if a verbose flag is available. If it is not overridden, it is assumed to not be supported. (Same rule as always, bestie.)version_argument = '-V'โ the compiler argument that produces version information.version_regexโ the regex used to extract the version from the compiler's output. Note the trailing space, do not touch it ๐ญ
version_regex = 'pg[^ ]* ([0-9.]+)-[0-9]+ (LLVM )?[^ ]+ target on '๐ฅ spack.compilers.rocmcc module
class spack.compilers.rocmcc.Rocmcc
class spack.compilers.rocmcc.Rocmcc(cspec, operating_system, target,
paths, modules=None, alias=None, environment=None, extra_rpaths=None,
enable_implicit_rpaths=None, **kwargs)Bases: Clang โ yes, she inherits from Clang ๐
PrgEnv: str | None = 'PrgEnv-amd'โ the PrgEnv value.PrgEnv_compiler: str | None = 'amd'โ the PrgEnv compiler value.property c11_flagโ no description.property c99_flagโ no description.cc_names: List[str] = ['amdclang']โ C compiler names.property cxx11_flagโ no description.property cxx14_flagโ no description.property cxx17_flagโ no description.cxx_names: List[str] = ['amdclang++']โ C++ compiler names.classmethod extract_version_from_output(output)โ extracts the version from the compiler's output. That's literally the whole job ๐ฏf77_names: List[str] = ['amdflang']โ F77 compiler names.classmethod f77_version(f77)โ no description given, just the signature.fc_names: List[str] = ['amdflang']โ FC compiler names.classmethod fc_version(fortran_compiler)โ no description given.property link_pathsโ undocumented property, she exists.property stdcxx_libsโ undocumented too.
๐ฆ spack.compilers.xl module
class spack.compilers.xl.Xl
class spack.compilers.xl.Xl(cspec, operating_system, target, paths,
modules=None, alias=None, environment=None, extra_rpaths=None,
enable_implicit_rpaths=None, **kwargs)Bases: Compiler
property c11_flagโ no description.property c99_flagโ no description.cc_names: List[str] = ['xlc']โ C compiler names.property cc_pic_flagโ returns the flag the C compiler uses to produce Position Independent Code (PIC).property cxx11_flagโ no description.property cxx14_flagโ no description.cxx_names: List[str] = ['xlC', 'xlc++']โ C++ compiler names.property cxx_pic_flagโ returns the flag the C++ compiler uses to produce Position Independent Code (PIC).property debug_flagsโ no description.f77_names: List[str] = ['xlf']โ F77 compiler names.property f77_pic_flagโ returns the flag the F77 compiler uses to produce Position Independent Code (PIC).classmethod f77_version(f77)โ no description given.fc_names: List[str] = ['xlf90', 'xlf95', 'xlf2003', 'xlf2008']โ FC compiler names, the whole squad ๐property fc_pic_flagโ returns the flag the FC compiler uses to produce Position Independent Code (PIC).classmethod fc_version(fc)โ no description given.property fflagsโ no description given.
Link paths, verbatim:
link_paths = {'cc': 'xl/xlc', 'cxx': 'xl/xlc++', 'f77': 'xl/xlf', 'fc':
'xl/xlf90'}property openmp_flagโ no description.property opt_flagsโ no description.property verbose_flagโ this property should be overridden in the compiler subclass if a verbose flag is available. If it is not overridden, it is assumed to not be supported.version_argument = '-qversion'โ the compiler argument that produces version information.version_regexโ the regex used to extract the version from the compiler's output:
version_regex = '([0-9]?[0-9]\\.[0-9])'๐ spack.compilers.xl_r module
class spack.compilers.xl_r.XlR
class spack.compilers.xl_r.XlR(cspec, operating_system, target, paths,
modules=None, alias=None, environment=None, extra_rpaths=None,
enable_implicit_rpaths=None, **kwargs)Bases: Xl โ she's the threaded remix of the class above, we stan ๐
cc_names: List[str] = ['xlc_r']โ C compiler names.cxx_names: List[str] = ['xlC_r', 'xlc++_r']โ C++ compiler names.f77_names: List[str] = ['xlf_r']โ F77 compiler names.fc_names: List[str] = ['xlf90_r', 'xlf95_r', 'xlf2003_r', 'xlf2008_r']โ FC compiler names, all four,_rflavoured.
link_paths = {'cc': 'xl_r/xlc_r', 'cxx': 'xl_r/xlc++_r', 'f77':
'xl_r/xlf_r', 'fc': 'xl_r/xlf90_r'}๐ณ spack.container package
A package that provides functions and classes to generate container recipes from a Spack environment. Container girl summer, fr fr ๐
spack.container.recipe(configuration, last_phase=None)
Returns a recipe that conforms to the configuration passed as input.
Parameters
configuration(dict) โ how to generate the current recipelast_phase(str) โ last phase to be printed, orNoneto print them all
spack.container.validate(configuration_file)
Validate a Spack environment YAML file that's being used to generate a recipe for a container.
Since a few attributes of the configuration must have specific values for the container recipe, this function returns a sanitized copy of the configuration in the input file. If any modification is needed, a warning will be issued. So like, bestie, read your warnings ๐
Parameters
configuration_file(str) โ path to the Spack environment YAML file
Returns
- A sanitized copy of the configuration stored in the input file
๐ Subpackages
spack.container.writers package
Writers for different kind of recipes and related convenience functions. Very "I make the templates" of them โจ
class spack.container.writers.PathContext(config, last_phase)
class spack.container.writers.PathContext(config, last_phase)Bases: Context
A generic context used to instantiate templates of recipes that install software in a common location and make it available directly via PATH.
property bootstrapโ information related to the build image.property buildโ information related to the build image.context_properties = ['depfile', 'run', 'build', 'strip', 'paths', 'manifest', 'os_packages_final', 'os_packages_build', 'os_package_update', 'labels', 'bootstrap', 'render_phase']โ the whole context property roster, all twelve of them, no skips ๐ซถproperty depfileโ no description given.property labelsโ no description given.property manifestโ thespack.yamlfile that should be used in the image.property os_package_updateโ whether or not to update the OS package manager cache.property os_packages_buildโ additional system packages that are needed at build-time.property os_packages_finalโ additional system packages that are needed at run-time.property pathsโ important paths in the image.property render_phaseโ no description given.property runโ information related to the run image.property stripโ whether or not to strip binaries in the image.template_name: str | None = Noneโ no description given, it's justNoneuntil someone sets it.
spack.container.writers.create(configuration, last_phase=None)
Returns a writer that conforms to the configuration passed as input.
Parameters
configuration(dict) โ how to generate the current recipelast_phase(str) โ last phase to be printed, orNoneto print them all
spack.container.writers.recipe(configuration, last_phase=None)
Returns a recipe that conforms to the configuration passed as input.
Parameters
configuration(dict) โ how to generate the current recipe ๐ ๏ธlast_phase(str) โ the last phase to be printedโฆ orNoneif you want all of them, no skips ๐
spack.container.writers.writer(name)
Okay so this is a decorator that registers a factory for a recipe writer โจ
The way it works: every factory takes a configuration dictionary and hands you back a properly configured writer that, when you call it, prints the corresponding recipe. That's it, that's the whole gig ๐
๐ซง Submodules
spack.container.writers.docker module
class spack.container.writers.docker.DockerContext(config, last_phase)
Bases: PathContext
This is the context used to instantiate a Dockerfile ๐ณ
context_properties = ['manifest', 'depfile', 'run', 'build', 'strip', 'paths', 'os_packages_final', 'os_packages_build', 'os_package_update', 'labels', 'bootstrap', 'render_phase']
property manifest โ the spack.yaml file that should be used in the image ๐
template_name: str | None = 'container/Dockerfile' โ name of the template used for Dockerfiles, periodt ๐
spack.container.writers.singularity module
class spack.container.writers.singularity.SingularityContext(config, last_phase)
Bases: PathContext
And this one is the context used to instantiate a Singularity definition file ๐
context_properties = ['runscript', 'startscript', 'test', 'help', 'depfile', 'run', 'build', 'strip', 'paths', 'manifest', 'os_packages_final', 'os_packages_build', 'os_package_update', 'labels', 'bootstrap', 'render_phase']
property help โ yeah, the help bit ๐ซถ
property runscript โ the runscript, bestie ๐โโ๏ธ
property singularity_config โ the Singularity config ๐ง
property startscript โ the startscript ๐ธ
template_name: str | None = 'container/singularity.def' โ name of the template used for Singularity definition files โจ
property test โ the test bit ๐งช
๐ซง Submodules
spack.container.images module
Manages the details on the images used in the various stages, lowkey the backstage crew of your container build ๐ฌ
spack.container.images.all_bootstrap_os() โ returns a list of all the OS that can be used to bootstrap Spack ๐
spack.container.images.bootstrap_template_for(image) โ takes an image, gives you the bootstrap template for it ๐งฉ
spack.container.images.build_info(image, spack_version) โ returns the name of the build image and its tag ๐ท๏ธ
Parameters:
image(str) โ image to be used at run-time. Should be of the form<image_name>:<image_tag>e.g."ubuntu:18.04"spack_version(str) โ version of Spack that we want to use to build
Returns: a tuple with (image_name, image_tag) for the build image ๐ซ
spack.container.images.checkout_command(url, ref, enforce_sha, verify) โ returns the checkout command to be used in the bootstrap phase ๐งพ
Parameters:
url(str) โ url of the Spack repositoryref(str) โ either a branch name, a tag or a commit shaenforce_sha(bool) โ if true turns everyverify(bool)
ngl bestie, the manpage literally stops at "if true turns every" for enforce_sha ๐ญ that's not me being shady, that's just what's in the source. Not me inventing the rest ๐
spack.container.images.commands_for(package_manager) โ returns the commands used to update system repositories, install system packages and clean afterwards ๐งฝ
Parameters:
package_manager(str) โ package manager to be used
Returns: a tuple of (update, install, clean) commands ๐
spack.container.images.data() โ returns a dictionary with the static data on the images ๐
The dictionary is read from a JSON file lazily, the first time this function is called. Iconic efficiency behaviour honestly โญ
spack.container.images.os_package_manager_for(image) โ returns the name of the OS package manager for the image passed as argument ๐งด
Parameters:
image(str) โ image to be used at run-time. Should be of the form<image_name>:<image_tag>e.g."ubuntu:18.04"
Returns: name of the package manager, e.g. "apt" or "yum" ๐
๐ spack.detection package
class spack.detection.DetectedPackage(spec: Spec, prefix: str)
Bases: NamedTuple
Info on a package that has been detected ๐
prefix: str โ prefix of the spec ๐
static restore(spec_str: str, prefix: str, extra_attributes: Dict[str, str] | None) -> DetectedPackage โ rebuilds one of these bad boys from its pieces ๐งต
spec: Spec โ spec that was detected ๐ฏ
spack.detection.by_path(packages_to_search: List[str], *, path_hints: List[str] | None = None, max_workers: int | None = None) -> Dict[str, List[DetectedPackage]]
Returns the list of packages that have been detected on the system, keyed by unqualified package name ๐
Parameters:
packages_to_searchโ list of packages to be detected. Each package can be either unqualified of fully qualifiedpath_hintsโ initial list of paths to be searchedmax_workersโ maximum number of workers to search for packages in parallel ๐โโ๏ธ๐จ
spack.detection.detection_tests(pkg_name: str, repository: RepoPath) -> List[Runner]
Returns a list of test runners for a given package ๐งช
Currently, detection tests are specified in a YAML file, called detection_test.yaml, alongside the package.py file.
This function reads that file to create a bunch of Runner objects. Cute, right? ๐
Parameters:
pkg_nameโ name of the package to testrepositoryโ repository where the package lives ๐
spack.detection.executable_prefix(executable_dir: str) -> str
Given a directory where an executable is found, guess the prefix (i.e. the "root" directory of that installation) and return it ๐
Parameters:
executable_dirโ directory where an executable is found
spack.detection.executables_in_path(path_hints: List[str]) -> Dict[str, str]
Get the paths of all executables available from the current PATH ๐
For convenience, this is constructed as a dictionary where the keys are the executable paths and the values are the names of the executables (i.e. the basename of the executable path).
There may be multiple paths with the same basename. In this case it is assumed there are two different instances of the executable ๐ฏโโ๏ธ
Parameters:
path_hintsโ list of paths to be searched. IfNonethe list will be constructed based on thePATHenvironment variable
spack.detection.update_configuration(detected_packages: Dict[str, List[DetectedPackage]], scope: str | None = None, buildable: bool = True) -> List[Spec]
Add the packages passed as arguments to packages.yaml ๐
Parameters:
detected_packagesโ list ofDetectedPackageobjects to be addedscopeโ configuration scope where to add the detected packagesbuildableโ whether the detected packages are buildable or not
๐ซง Submodules
spack.detection.common module
Define a common data structure to represent external packages, plus a function to update packages.yaml given a list of detected packages ๐๏ธ
Ideally, each detection method should be placed in a specific subpackage and implement at least a function that returns a list of DetectedPackage objects. The update in packages.yaml can then be done using the function provided here.
The module also contains other functions that might be useful across different detection mechanisms ๐งฐ
class spack.detection.common.DetectedPackage(spec: Spec, prefix: str)
Bases: NamedTuple
Information on a package that has been detected ๐
prefix: str โ prefix of the spec ๐
static restore(spec_str: str, prefix: str, extra_attributes: Dict[str, str] | None) -> DetectedPackage โ restores a detected package from a string, a prefix and some extra attributes ๐งต
spec: Spec โ spec that was detected ๐ฏ
class spack.detection.common.WindowsCompilerExternalPaths
Bases: object
All the Windows-compiler-flavoured path discovery, gathered in one iconic little class ๐ช
static find_windows_compiler_bundled_packages() -> List[str] โ return all MSVC compiler bundled packages ๐ฆ
static find_windows_compiler_cmake_paths() -> List[str] โ semi hard-coded search path for cmake bundled with MSVC ๐งญ
static find_windows_compiler_ninja_paths() -> List[str] โ semi hard-coded search heuristic for locating ninja bundled with MSVC ๐ฅท
static find_windows_compiler_root_paths() -> List[str] โ helper for Windows compiler installation root discovery
At the moment simply returns location of VS install paths from VSWhere. But should be extended to include more information as relevant ๐ง
class spack.detection.common.WindowsKitExternalPaths
Bases: object
Same energy, but for the Windows Kits ๐งฑ
static find_windows_driver_development_kit_paths() -> List[str] โ provides a list of all installation paths for the WDK by version and architecture ๐
static find_windows_kit_bin_paths(kit_base: str | None | list = None) -> List[str] โ returns Windows kit bin directory per version ๐๏ธ
static find_windows_kit_lib_paths(kit_base: str | None | list = None) -> List[str] โ returns Windows kit lib directory per version ๐
static find_windows_kit_reg_installed_roots_paths() -> List[str] โ the registry-installed roots, bestie ๐๏ธ
static find_windows_kit_reg_sdk_paths() -> List[str] โ and the registry SDK paths ๐งพ
static find_windows_kit_roots() -> List[str] โ return Windows kit root, typically %programfiles%Windows Kits10|11 ๐ช
spack.detection.common.compute_windows_program_path_for_package(pkg: PackageBase) -> List[str]
Given a package, attempts to compute its Windows program files location, and returns the list of best guesses ๐
Parameters:
pkgโ package for which Program Files location is to be computed
spack.detection.common.compute_windows_user_path_for_package(pkg: PackageBase) -> List[str]
Given a package attempt to compute its user scoped install location, return list of potential locations based on common heuristics ๐ธ
For more info on Windows user specific installs see: https://learn.microsoft.com/en-us/dotnet/api/system.environment.specialfolder?view=netframework-4.8 ๐ซถ
spack.detection.common.executable_prefix(executable_dir: str) -> str
Given a directory where an executable is found, guess the prefix (i.e. the "root" directory of that installation) and return it ๐
Parameters:
executable_dirโ directory where an executable is foundspack.detection.common.find_win32_additional_install_paths() -> List[str]โ okay so not every program on Windows lives on thePATH, and this lil helper returns a list of the other potential install locations. Very "there's more where that came from" energy ๐spack.detection.common.is_executable(file_path: str) -> boolโ returnsTrueif the path you handed it is the path of an executable. That's it, that's the whole vibe check โจspack.detection.common.library_prefix(library_dir: str) -> strโ give it a directory where a library was found, and it guesses the prefix (i.e. the "root" directory of that installation) and returns it. Iconic detective work fr fr ๐
Parameters:
- library_dir โ directory where a library is found
spack.detection.common.path_to_dict(search_paths: List[str])โ returns adictionary[fullpath]: basenamebuilt from a list of paths. Simple, clean, slay ๐spack.detection.common.update_configuration(detected_packages: Dict[str, List[DetectedPackage]], scope: str | None = None, buildable: bool = True) -> List[Spec]โ adds the packages you pass as arguments topackages.yaml. Bestie it's literally writing your config for you ๐ซถ
Parameters:
- detected_packages โ list of
DetectedPackageobjects to be added - scope โ configuration scope where to add the detected packages
- buildable โ whether the detected packages are buildable or not
๐ฆ spack.detection.path module ๐ฆ
Detection of software installed in the system, based on paths inspections and running executables. Very much "we looked around and found stuff" energy ๐
spack.detection.path.DETECTION_TIMEOUT = 60โ timeout used for package detection (seconds). Sixty seconds, no cap, that's the limit โณ
๐ธ class spack.detection.path.ExecutablesFinder ๐ธ
Bases: Finder
candidate_files(*, patterns: List[str], paths: List[str]) -> List[str]โ returns a list of candidate files found on the system.
Parameters:
- patterns โ search patterns to be used for matching files
- paths โ paths where to search for files
default_path_hints() -> List[str]โ no extra description on this one, it's just giving method signature and nothing else ๐prefix_from_path(*, path: str) -> strโ given a path where a file was found, returns the corresponding prefix.
Parameters:
- path โ path of a detected file
search_patterns(*, pkg: PackageBase) -> List[str]โ returns the list of patterns used to match candidate files.
Parameters:
- pkg โ package being detected
๐ class spack.detection.path.Finder ๐
Bases: object
Inspects the file-system looking for packages. Guesses places where to look using PATH. It's giving nosy bestie who finds everything ๐
candidate_files(*, patterns: List[str], paths: List[str]) -> List[str]โ returns a list of candidate files found on the system.
Parameters:
- patterns โ search patterns to be used for matching files
- paths โ paths where to search for files
default_path_hints() -> List[str]โ again, no description given, just vibes โจdetect_specs(*, pkg: PackageBase, paths: List[str]) -> List[DetectedPackage]โ given a list of files matching the search patterns, returns a list of detected specs.
Parameters:
- pkg โ package being detected
- paths โ files matching the package search patterns
find(*, pkg_name: str, initial_guess: List[str] | None = None) -> List[DetectedPackage]โ for a given package, returns a list of detected specs.
Parameters:
- pkg_name โ package being detected
- initial_guess โ initial list of paths to search from the caller if
None, default paths are searched. If this is an empty list, nothing will be searched. (Not a single thing. Zero. Nada ๐)
prefix_from_path(*, path: str) -> strโ given a path where a file was found, returns the corresponding prefix.
Parameters:
- path โ path of a detected file
search_patterns(*, pkg: PackageBase) -> List[str]โ returns the list of patterns used to match candidate files.
Parameters:
- pkg โ package being detected
๐ class spack.detection.path.LibrariesFinder ๐
Bases: Finder
Finds libraries on the system, searching by LD_LIBRARY_PATH, LIBRARY_PATH, DYLD_LIBRARY_PATH, DYLD_FALLBACK_LIBRARY_PATH, and standard system library paths. Library hunt, but make it fashion ๐
candidate_files(*, patterns: List[str], paths: List[str]) -> List[str]โ returns a list of candidate files found on the system.
Parameters:
- patterns โ search patterns to be used for matching files
- paths โ paths where to search for files
prefix_from_path(*, path: str) -> strโ given a path where a file was found, returns the corresponding prefix.
Parameters:
- path โ path of a detected file
search_patterns(*, pkg: PackageBase) -> List[str]โ returns the list of patterns used to match candidate files.
Parameters:
- pkg โ package being detected
spack.detection.path.accept_elf(path, host_compat)โ accept an ELF file if the header matches the given compat triplet. In case it's not an ELF (e.g. static library, or some arbitrary file) it falls back tois_readable_file. Plan B is always there bestie ๐spack.detection.path.by_path(packages_to_search: List[str], *, path_hints: List[str] | None = None, max_workers: int | None = None) -> Dict[str, List[DetectedPackage]]โ returns the list of packages that have been detected on the system, keyed by unqualified package name.
Parameters:
- packages_to_search โ list of packages to be detected. Each package can be either unqualified of fully qualified
- path_hints โ initial list of paths to be searched
- max_workers โ maximum number of workers to search for packages in parallel (multithreaded detection, we love to see it ๐ฅ)
spack.detection.path.common_windows_package_paths(pkg_cls=None) -> List[str]โ get the paths for common package installation location on Windows that are outside thePATH. Returns[]on unix. Windows girlies get the list, unix girlies get an empty list, that's the deal ๐ฆspack.detection.path.executables_in_path(path_hints: List[str]) -> Dict[str, str]โ get the paths of all executables available from the currentPATH.
For convenience, this is constructed as a dictionary where the keys are the executable paths and the values are the names of the executables (i.e. the basename of the executable path).
There may be multiple paths with the same basename. In this case it is assumed there are two different instances of the executable.
Parameters:
- path_hints โ list of paths to be searched. If
Nonethe list will be constructed based on thePATHenvironment variable.
spack.detection.path.file_identifier(path)โ no description in the docs for this one, just the signature, iykyk ๐spack.detection.path.libraries_in_ld_and_system_library_path(path_hints: List[str] | None = None) -> Dict[str, str]โ get the paths of all libraries available from path_hints or the following defaults:- Environment variables (Linux:
LD_LIBRARY_PATH, Darwin:DYLD_LIBRARY_PATH, andDYLD_FALLBACK_LIBRARY_PATH) - Dynamic linker default paths (glibc:
ld.so.conf, musl:ld-musl-<arch>.path) - Default system library paths.
- Environment variables (Linux:
For convenience, this is constructed as a dictionary where the keys are the library paths and the values are the names of the libraries (i.e. the basename of the library path).
There may be multiple paths with the same basename. In this case it is assumed there are two different instances of the library.
Parameters:
- path_hints (list) โ list of paths to be searched. If
Nonethe list will be constructed based on the set ofLD_LIBRARY_PATH,LIBRARY_PATH,DYLD_LIBRARY_PATH, andDYLD_FALLBACK_LIBRARY_PATHenvironment variables as well as the standard system library paths. - path_hints โ list of paths to be searched. If
None, the default system paths are used.
Yes bestie, path_hints is listed twice in the original docs. I'm not gonna pretend it isn't. Both bullets stay, periodt ๐
spack.detection.path.libraries_in_windows_paths(path_hints: List[str] | None = None) -> Dict[str, str]โ get the paths of all libraries available from the systemPATHpaths.
For more details, see libraries_in_ld_and_system_library_path regarding return type and contents.
Parameters:
- path_hints โ list of paths to be searched. If
Nonethe list will be constructed based on the set ofPATHenvironment variables as well as the standard system library paths.
๐ง spack.detection.test module ๐ง
Create and run mock e2e tests for package detection. Fake it till you make it, but like, rigorously ๐ธ
โญ class spack.detection.test.DetectionTest(pkg_name: str, layout: List[MockExecutables], results: List[ExpectedTestResult]) โญ
Bases: NamedTuple
Data structure to construct detection tests by PATH inspection.
Packages may have a YAML file containing the description of one or more detection tests to be performed. Each test creates a few mock executable scripts in a temporary folder, and checks that detection by PATH gives the expected results. Adorable and functional, the dream combo ๐
layout: List[MockExecutables]โ alias for field number 1pkg_name: strโ alias for field number 0results: List[ExpectedTestResult]โ alias for field number 2
๐ class spack.detection.test.ExpectedTestResult(spec: str, extra_attributes: Dict[str, str]) ๐
Bases: NamedTuple
Data structure to model assertions on detection tests. Receipts for what we expect to find ๐
extra_attributes: Dict[str, str]โ attributes expected in the external specspec: strโ spec to be detected
๐ class spack.detection.test.MockExecutables(executables: List[str], script: str) ๐
Bases: NamedTuple
Mock executables to be used in detection tests.
executables: List[str]โ relative paths for mock executables to be createdscript: strโ shell script for the mock executable
๐ง class spack.detection.test.Runner(*, test: DetectionTest, repository: RepoPath)
- Bases:
object - What it is: runs an external detection test. Literally its whole job. ๐
execute() -> List[Spec] โ executes a test and returns the specs that have been detected.
So like, this function sets up a test in a temporary directory, following the prescriptions in the test layout, then performs a detection by executables and hands you back the specs it detected. It's giving "set the stage, run the scene, report back" energy ๐ฌ
property expected_specs: List[Spec] โ the specs this test expects to find. ๐
๐ธ spack.detection.test.detection_tests(pkg_name: str, repository: RepoPath) -> List[Runner]
Returns a list of test runners for a given package.
So rn, detection tests live in a YAML file called detection_test.yaml, sitting right alongside the package.py file โ besties, they're neighbors ๐ซถ This function reads that file and creates a whole bunch of Runner objects out of it.
Parameters:
pkg_nameโ name of the package to testrepositoryโ repository where the package lives
๐ฆ spack.detection.test.read_detection_tests(pkg_name: str, repository: RepoPath) -> Dict[str, Any]
Returns the normalized content of the detection_tests.yaml associated with the package passed in input. โจ
And the content gets merged with the content of any package that's transitively included using the "includes" attribute. So it's not just this file, it's the whole friend group of files linked through includes. iykyk.
Parameters:
pkg_nameโ name of the package to testrepositoryโ repository in which to search for packages
๐ฟ spack.environment package
This package implements Spack environments. That's it, that's the whole vibe check. ๐
๐ spack.lock format
Okay so Spack environments have existed since Spack v0.12.0, and there have been 4 different spack.lock formats since then. All four are documented right here, no gatekeeping. ๐
The high-level format of a Spack lockfile hasn't changed much between versions, but the contents have. Lockfiles are JSON-formatted, and their top-level sections are:
_meta(object): this contains details about the file format, including:file-type: always"spack-lockfile"lockfile-version: an integer representing the lockfile format versionspecfile-version: an integer representing the spec format version (sincev0.17)
spack(object): optional, this identifies information about Spack used to concretize the environment:type: required, identifies form Spack version took (e.g.,git,release)commit: the commit if the version is from gitversion: the Spack version
roots(list): an ordered list of records representing the roots of the Spack environment. Each has two fields:hash: a Spack spec hash uniquely identifying the concrete root specspec: a string representation of the abstract spec that was concretized
concrete_specs: a dictionary containing the specs in the environment.include_concrete(dictionary): an optional dictionary that includes the roots and concrete specs from the included environments, keyed by the path to that environment
๐ Compatibility
New versions of Spack can (so far) read all old lockfile formats โ they are backward-compatible. Old versions cannot read new lockfile formats, and you'll need to upgrade Spack to use them. ๐ญ
Fr fr, if you're on an ancient Spack and you try to read a shiny new lockfile, it's not gonna go well. Upgrade first, bestie.
๐ Lockfile version compatibility across Spack versions
The original puts this in a whole grid, so here it is as a glow-up list โ same facts, cuter delivery:
v0.12:0.14โ v1 โv0.15:0.16โ v1 โ , v2 โv0.17โ v1 โ , v2 โ , v3 โv0.18:โ v1 โ , v2 โ , v3 โ , v4 โv0.22:โ v1 โ , v2 โ , v3 โ , v4 โ , v5 โ
(Blank cells in the original = no support. The โ s are the supported ones, obviously. ๐ )
๐ Version 1
When lockfiles were first created, there was only one hash in Spack: the DAG hash. This DAG hash (we'll call it the old DAG hash) did not include build dependencies โ it only included transitive link and run dependencies. Noted. ๐
The spec format at this time was keyed by name. Each spec started with a key for its name, whose value was a dictionary of other spec attributes. The lockfile put these name-keyed specs into dictionaries keyed by their DAG hash, and the spec records did not actually have a "hash" field in the lockfile โ you have to associate the hash from the key with the spec record after the fact. Sneaky, but okay. ๐
Dependencies in original lockfiles were keyed by "hash", i.e. the old DAG hash.
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 1
},
"roots": [
{
"hash": "<old_dag_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<old_dag_hash 2>",
"spec": "<abstract spec 2>"
}
],
"concrete_specs": {
"<old_dag_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": {
"depname_1": {
"hash": "<old_dag_hash for depname_1>",
"type": ["build", "link"]
},
"depname_2": {
"hash": "<old_dag_hash for depname_3>",
"type": ["build", "link"]
}
},
"hash": "<old_dag_hash 1>"
},
"<old_dag_hash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": {
"depname_3": {
"hash": "<old_dag_hash for depname_3>",
"type": ["build", "link"]
},
"depname_4": {
"hash": "<old_dag_hash for depname_4>",
"type": ["build", "link"]
},
},
"hash": "<old_dag_hash 2>"
},
}
}โญ Version 2
Version 2 changes one thing: specs in the lockfile are now keyed by build_hash instead of the old dag_hash. Specs have a hash attribute with their real DAG hash, so you can't go by the dictionary key anymore to identify a spec โ you have to read it in and look at "hash". Read the room, not the label. ๐ Dependencies are still keyed by old DAG hash.
Even though we key lockfiles by build_hash, specs in Spack were still deployed with the old, coarser DAG hash. This means that in v2 and v3 lockfiles (which are keyed by build hash), there may be multiple versions of the same spec with different build dependencies, which means they will have different build hashes but the same DAG hash. Spack would only have been able to actually install one of these. ๐
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 2
},
"roots": [
{
"hash": "<build_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<build_hash 2>",
"spec": "<abstract spec 2>"
}
],
"concrete_specs": {
"<build_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": {
"depname_1": {
"hash": "<old_dag_hash for depname_1>",
"type": ["build", "link"]
},
"depname_2": {
"hash": "<old_dag_hash for depname_3>",
"type": ["build", "link"]
}
},
"hash": "<old_dag_hash 1>",
},
"<build_hash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": {
"depname_3": {
"hash": "<old_dag_hash for depname_3>",
"type": ["build", "link"]
},
"depname_4": {
"hash": "<old_dag_hash for depname_4>",
"type": ["build", "link"]
}
},
"hash": "<old_dag_hash 2>"
}
}
}๐ Version 3
Version 3 doesn't change the top-level lockfile format, but this was when we changed the specfile format. Specs in concrete_specs are now keyed by the build hash, with no inner dictionary keyed by their package name. The package name is in a name field inside each spec dictionary. ๐ฅ
The dependencies field in the specs is a list instead of a dictionary, and each element of the list is a record with the name, dependency types, and hash of the dependency. Instead of a key called hash, dependencies are keyed by build_hash. Each spec still has a hash attribute.
Version 3 adds the specfile_version field to _meta and uses the new JSON spec format. Glow-up complete. โจ
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 3,
"specfile-version": 2
},
"roots": [
{
"hash": "<build_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<build_hash 2>",
"spec": "<abstract spec 2>"
},
],
"concrete_specs": {
"<build_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_1",
"build_hash": "<build_hash for depname_1>",
"type": ["build", "link"]
},
{
"name": "depname_2",
"build_hash": "<build_hash for depname_2>",
"type": ["build", "link"]
},
],
"hash": "<old_dag_hash 1>",
},
"<build_hash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_3",
"build_hash": "<build_hash for depname_3>",
"type": ["build", "link"]
},
{
"name": "depname_4",
"build_hash": "<build_hash for depname_4>",
"type": ["build", "link"]
},
],
"hash": "<old_dag_hash 2>"
}
}
}๐ Version 4
Okay bestie, glow-up time ๐
Version 4 removes build hashes and is keyed by the new DAG hash (hash). The hash now includes build dependencies and a canonical hash of the package.py file. Dependencies are keyed by hash (DAG hash) as well. So there are no more build_hash fields in the specs, and โ huge โ there are no more issues with lockfiles being able to store multiple specs with the same DAG hash (because the DAG hash is now finer-grained, slay). An optional spack property may be included to track version information, such as the commit or version.
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 4,
"specfile-version": 3
},
"roots": [
{
"hash": "<dag_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<dag_hash 2>",
"spec": "<abstract spec 2>"
}
],
"concrete_specs": {
"<dag_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_1",
"hash": "<dag_hash for depname_1>",
"type": ["build", "link"]
},
{
"name": "depname_2",
"hash": "<dag_hash for depname_2>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 1>",
},
"<daghash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_3",
"hash": "<dag_hash for depname_3>",
"type": ["build", "link"]
},
{
"name": "depname_4",
"hash": "<dag_hash for depname_4>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 2>"
}
}
}๐ Version 5
Version 5 doesn't change the top-level lockfile format (iconic, minimal effort), but an optional dictionary is added โจ The dictionary has the root and concrete_specs of the included environments, which are keyed by the path to that environment. And since this is optional โ if the environment does not have any included environments โ include_concrete will not be a part of the lockfile. iykyk ๐ซถ
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 5,
"specfile-version": 3
},
"roots": [
{
"hash": "<dag_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<dag_hash 2>",
"spec": "<abstract spec 2>"
}
],
"concrete_specs": {
"<dag_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_1",
"hash": "<dag_hash for depname_1>",
"type": ["build", "link"]
},
{
"name": "depname_2",
"hash": "<dag_hash for depname_2>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 1>",
},
"<daghash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_3",
"hash": "<dag_hash for depname_3>",
"type": ["build", "link"]
},
{
"name": "depname_4",
"hash": "<dag_hash for depname_4>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 2>"
}
}
"include_concrete": {
"<path to environment>": {
"roots": [
{
"hash": "<dag_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<dag_hash 2>",
"spec": "<abstract spec 2>"
}
],
"concrete_specs": {
"<dag_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_1",
"hash": "<dag_hash for depname_1>",
"type": ["build", "link"]
},
{
"name": "depname_2",
"hash": "<dag_hash for depname_2>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 1>",
},
"<daghash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_3",
"hash": "<dag_hash for depname_3>",
"type": ["build", "link"]
},
{
"name": "depname_4",
"hash": "<dag_hash for depname_4>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 2>"
}
}
}
}
}๐ฆ class spack.environment.Environment(manifest_dir: str | Path)
Bases: object
Okay so this is the Environment class โ a Spack environment, which bundles together configuration and a list of specs. It's the main character of this whole API, fr fr ๐
๐ property active
True if this environment is currently active.
๐ add(user_spec, list_name='specs')
Add a single user_spec (non-concretized) to the Environment.
Returns
Trueif the spec was added,Falseif it was already present and did not need to be added
Return type: (bool)
๐ add_view_to_env(env_mod: EnvironmentModifications, view: str) -> EnvironmentModifications
Collect the environment modifications to activate an environment using the provided view. Removes duplicate paths โจ
Parameters
env_modโ the environment modifications object that is modified.viewโ the name of the view to activate.
๐ added_specs()
Specs that are not yet installed ๐
Yields the user spec for non-concretized specs, and the concrete spec for already concretized but not yet installed specs.
๐ all_concretized_orders() -> List[str]
Returns all of the concretized order of the environment and its included environment(s).
๐ all_concretized_user_specs() -> List[Spec]
Returns all of the concretized user specs of the environment and its included environment(s).
๐ all_hashes()
Return hashes of all specs.
๐ all_matching_specs(*specs: Spec) -> List[Spec]
Returns all concretized specs in the environment satisfying any of the input specs.
๐ all_specs() -> List[Spec]
Returns a list of all concrete specs.
๐ all_specs_generator() -> Iterable[Spec]
Returns a generator for all concrete specs.
๐ change_existing_spec(change_spec: Spec, list_name: str = 'specs', match_spec: Spec | None = None, allow_changing_multiple_specs=False)
Find the spec identified by match_spec and change it to change_spec.
Parameters
change_specโ the spec properties that need to be changed. Heads up though: this will not change attributes of the matched spec unless they conflict withchange_spec. ๐list_nameโ tells you which spec list inside the environment is the one getting modified.match_specโ if you set it, it identifies the spec that should be changed. If you don't set it, it just assumes we're looking for a spec with the same name aschange_spec.
๐ธ check_views()
Checks if the environment's default view can be activated. That's the whole vibe check, bestie.
clear(re_read=False)
Clears the contents of the environment.
Parameters:
re_readโ ifTrue, do not clearnew_specs. This value can't be read from yaml, and it needs to be maintained when you're re-reading an existing environment.
concrete_roots()
Same as concretized_specs, except it returns the list of concrete roots without the associated user spec. Less baggage, same energy. โจ
concretize(force=False, tests=False)
Concretizes the user_specs in this environment.
By default it only concretizes specs that haven't been concretized yet โ unless force is True.
Lowkey important: this only modifies the environment in memory. write() is the thing that writes out a lockfile containing the concretized specs. Don't forget the write ๐ญ
Parameters:
force(bool) โ re-concretize ALL the specs, even the ones that were already concretized.tests(boolorlistorset) โFalseto run no tests,Trueto test all packages, or a list of package names to run tests for just some of them.
Returns: the list of specs that have been concretized. Each entry is a tuple of the user spec and the corresponding concretized spec.
concretized_order: List[str]
Roots associated with the last concretization, in order. A little receipt of what went down. ๐งพ
concretized_specs()
Tuples of (user spec, concrete spec) for all concrete specs. All of them, no crumbs left.
concretized_user_specs: List[Spec]
User specs from the last concretization.
deconcretize(spec: Spec, concrete: bool = True)
Removes the specified spec from the environment's concretization.
Parameters:
specโ the spec to deconcretize. It must be a root of the environment.concreteโ ifTrue, find all the instances ofspecas concrete in the environment. IfFalse, find a single instance of the abstract spec as a root of the environment.
default_view (property)
Just a property, bestie โ the source gives us no extra lore for this one.
delete_default_view() -> None
Deletes the default view associated with this environment. Bye bye ๐๐ซถ
destroy()
Removes this environment from Spack entirely. Like, gone-gone. ๐
dev_specs (property)
Another property โ dev specs, that's the whole note here.
ensure_env_directory_exists(dot_env: bool = False) -> None
Makes sure the root directory of the environment actually exists.
Parameters:
dot_envโ ifTrue, it also makes sure the<root>/.envdirectory exists.
filter_specs(reader, json_specs_by_hash, order_concretized)
Signature only on this one โ the docs don't hand us a description. ๐
get_by_hash(dag_hash: str) -> List[Spec]
Signature only again, no description in the source.
get_one_by_hash(dag_hash)
Returns the single spec from the environment that matches the hash you passed in. Raises an AssertionError if no specs match โ or if more than one matches. So bestie, make sure it's unique before you go off. ๐ฅ
has_view(view_name: str) -> bool
Signature only, no description given.
include_concrete_envs()
Copy and save the included envs' specs internally. Snapshot vibes. ๐ธ
included_concrete_envs: List[str]
Environment paths for concrete (lockfile) included environments.
included_concrete_spec_data: Dict[str, Dict[str, List[str]]]
First-level included concretized spec data from/to the lockfile.
included_concretized_order: Dict[str, List[str]]
Roots from included environments with the last concretization, in order.
included_concretized_user_specs: Dict[str, List[Spec]]
User specs from included environments from the last concretization.
included_specs_by_hash: Dict[str, Dict[str, Spec]]
Concretized specs by hash from the included environments.
included_user_specs (property) : SpecList
Included concrete user (or root) specs from the last concretization.
install_all(**install_args)
Installs all the concretized specs in an environment.
Note: this does not regenerate the views for the environment โ that's a whole separate thing you do with a call to write(). Don't skip it, fr fr.
Parameters:
install_args(dict) โ keyword install arguments.
install_specs(specs: List[Spec] | None = None, **install_args)
Signature only, bestie โ the docs give us nothing else here. ๐
invalidate_repository_cache()
Just the signature, no description in the source.
is_develop(spec)
Returns true when the spec is built from local sources.
lock_path (property)
Path to the spack.lock file in this environment.
manifest_path (property)
Path to the spack.yaml file in this environment.
manifest_uptodate_or_warn()
Emits a warning if the manifest file is not up-to-date. A gentle little nudge, not a whole crash. โญ
matching_spec(spec)
Given a spec (likely not concretized), finds a matching concretized spec in the environment.
The matching spec doesn't have to be installed in the environment, but it must be concrete. Specs added with spack add without an intervening spack concretize will not be matched โ periodt.
- If there's a single root spec that matches the provided spec, or a single dependency spec that matches the provided spec, then the concretized instance of that spec gets returned.
- If multiple root specs match the provided spec, or no root specs match and multiple dependency specs match, then this raises an error and reports all the matching specs. It will absolutely call you out. ๐ญ
regenerate_views()
Signature only, no description in the docs.
remove(query_spec, list_name='specs', force=False)
Removes specs from an environment that match a query_spec.
removed_specs()
Tuples of (user spec, concrete spec) for all the specs that will be removed on the next concretize.
repo (property)
Property with no description attached in the source.
repos_path (property)
Also a property, also no extra details given.
rm_view_from_env(env_mod: EnvironmentModifications, view: str) -> EnvironmentModifications
Collects the environment modifications needed to deactivate an environment using the provided view. It reverses the action of add_view_to_env. Rewind energy. ๐
Parameters:
env_modโ the environment modifications object that gets modified.viewโ the name of the view to deactivate.
roots()
Specs explicitly requested by the user in this environment.
Yields both added and installed specs that have user specs in spack.yaml.
scope_name (property)
Name of the config scope of this environment's manifest file.
set_included_concretized_user_specs(env_name: str, env_info: Dict[str, Dict[str, Any]], included_json_specs_by_hash: Dict[str, Dict[str, Any]]) -> Dict[str, Dict[str, Any]]
Sets all of the concretized user specs from included environments to include those from nested included environments.
Parameters:
env_nameโ the name (technically the path) of the included environment.env_infoโ included concrete environment data.included_json_specs_by_hashโ concrete spec data keyed by hash.
Returns: updated specs_by_hash. ๐ฆ
spec_lists: Dict[str, SpecList]
Specs from "spack.yaml".
specs_by_hash: Dict[str, Spec]
Concretized specs by hash.
unify (property)
Property, no description in the source. That's it, that's the tea.
uninstalled_specs()
Returns root specs that are not installed โ or that are installed but are development specs themselves, or have those among their dependencies. So it's catching strays (deps) too. ๐
update_default_view(path_or_bool: str | bool) -> None
Updates the path of the default view.
- If the argument you pass is
False, the default view is deleted, if present. The manifest will have an entry"view: false". - If the argument is
True, a default view is created, if not already present. The manifest will have an entry"view: true". If a default view is already declared, it's left untouched. - If the argument is a path, a default view pointing to that path is created, if not present already. If a default view is already declared, only its
"root"will be changed.
Parameters:
path_or_boolโ eitherTrue, orFalse, or a path.
update_environment_repository() -> None
Updates the repository associated with the environment.
update_lockfile() -> None
Signature only โ no description in the docs. ๐
update_stale_references(from_list=None)
Iterates over spec lists, updating references.
user_specs (property)
Property, no description given in the source.
view_path_default (property)
Also a property, also no extra description.
write(regenerate: bool = True) -> None
Writes an in-memory environment to its location on disk.
Writes out package files for each newly concretized spec. Also regenerates any views associated with the environment and runs post-write hooks, if regenerate is True.
Parameters:
regenerateโ regenerate views and run post-write hooks as well as writing, ifTrue. โจwrite_transaction()โ grab yourself a write lock context manager so you can use it in awithblock ๐ซถexception spack.environment.SpackEnvironmentConfigError(message, long_message=None)โ Bases:SpackEnvironmentError. The error class for things going wrong specifically with Spack environment configuration ๐exception spack.environment.SpackEnvironmentError(message, long_message=None)โ Bases:SpackError. The superclass of every error that has anything to do with Spack environments, aka the mother of all env problems ๐ญexception spack.environment.SpackEnvironmentViewError(message, long_message=None)โ Bases:SpackEnvironmentError. Errors about view generation, when your view is giving absolutely nothing ๐ธspack.environment.activate(env, use_env_repo=False)โ activate an environment โจ
To activate an environment, we add its manifest's configuration scope to the existing Spack configuration, and we set active to the current environment.
Parameters:
env(Environment) โ the environment to activateuse_env_repo(bool) โ use the packages exactly as they appear in the environment's repository
spack.environment.active(name)โTrueif the named environment is active ๐spack.environment.active_environment() -> Environment | Noneโ returns the active environment when there is anyspack.environment.all_environment_names()โ list the names of environments that currently exist ๐spack.environment.all_environments()โ generator for all managed Environmentsspack.environment.create(name: str, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) -> Environmentโ create a managed environment in Spack and return it ๐
A managed environment is created in a root directory managed by this Spack instance, so that Spack can keep track of them.
Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files.
Parameters:
nameโ name of the managed environmentinit_fileโ either a lockfile, a manifest file, orNonewith_viewโ whether a view should be maintained for the environment. If the value is a string, it specifies the path to the viewkeep_relativeโ ifTrue, develop paths are copied verbatim into the new environment file, otherwise they are made absoluteinclude_concreteโ list of concrete environment names/paths to be included
spack.environment.create_in_dir(root: str | Path, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) -> Environmentโ create an environment in the directory you pass in and return it ๐ฆ
Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files.
Parameters:
rootโ directory where to create the environment.init_fileโ either a lockfile, a manifest file, orNonewith_viewโ whether a view should be maintained for the environment. If the value is a string, it specifies the path to the viewkeep_relativeโ ifTrue, develop paths are copied verbatim into the new environment file, otherwise they are made absoluteinclude_concreteโ concrete environment names/paths to be included
spack.environment.deactivate()โ undo any configuration or repo settings modified byactivate()๐ฅspack.environment.default_manifest_yaml()โ the defaultspack.yamlfile to put in new environmentsspack.environment.display_specs(concretized_specs)โ displays the list of specs returned byEnvironment.concretize()โจ
Parameters:
concretized_specs(list) โ list of specs returned byEnvironment.concretize()
spack.environment.environment_dir_from_name(name: str, exists_ok: bool = True) -> strโ returns the directory associated with a named environment ๐
Parameters:
nameโ name of the environmentexists_okโ ifFalse, raise an error if the environment exists already
Raises SpackEnvironmentError โ if exists_ok is False and the environment exists already
spack.environment.exists(name)โ whether an environment with this name exists or not ๐spack.environment.initialize_environment_dir(environment_dir: str | Path, envfile: str | Path | None) -> Noneโ initialize an environment directory starting from an envfile ๐ง
Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files.
Parameters:
environment_dirโ directory where the environment should be placedenvfileโ manifest file or lockfile used to initialize the environment
Raises SpackEnvironmentError โ if the directory can't be initialized
spack.environment.installed_specs()โ returns the specs of packages installed in the active environment, orNoneif no packages are installed ๐spack.environment.is_env_dir(path)โ whether a directory contains a spack environmentspack.environment.is_latest_format(manifest)โ returnFalseif the manifest file exists and is not in the latest schema format ๐
Parameters:
manifest(str) โ manifest file to be analyzed
spack.environment.manifest_file(env_name_or_dir)โ return the absolute path to a manifest file given the environment name or directory ๐ซ
Parameters:
env_name_or_dir(str) โ either the name of a valid environment or a directory where a manifest file resides
Raises AssertionError โ if the environment is not found
spack.environment.no_active_environment()โ deactivate the active environment for the duration of the context. Has no effect when there is no active environment ๐spack.environment.read(name)โ get an environment with the supplied namespack.environment.root(name)โ get the root directory for an environment by name ๐spack.environment.update_yaml(manifest, backup_file)โ update a manifest file from an old format to the current one ๐
Parameters:
manifest(str) โ path to a manifest filebackup_file(str) โ file where to copy the original manifest
Returns True if the manifest was updated, False otherwise.
Raises AssertionError โ in case anything goes wrong during the update
๐ซ Submodules ๐ซ
๐ธ spack.environment.depfile module ๐ธ
This module contains the traversal logic and models that can be used to generate depfiles from an environment ๐
class spack.environment.depfile.DepfileNode(target: Spec, prereqs: List[Spec], buildcache: UseBuildCache)โ Bases:object. Contains a spec, a subset of its dependencies, and a flag whether it should be buildcache only/never/auto ๐ซถclass spack.environment.depfile.DepfileSpecVisitor(pkg_buildcache: UseBuildCache, deps_buildcache: UseBuildCache)โ Bases:object. This visitor produces an adjacency list of a (reduced) DAG, which is used to generate depfile targets with their prerequisites. Currently it only drops build deps when using buildcache only mode.
Note that the DAG could be reduced even more by dropping build edges of specs installed at the moment the depfile is generated, but that would produce stateful depfiles that would not fail when the database is wiped later ๐ฅ
accept(node)โ it's listed right there, no extra notes in the docs, that's the whole entry ๐neighbors(node)โ produce a list of spec to follow from node
class spack.environment.depfile.MakefileModel(env: Environment, roots: List[Spec], adjacency_list: List[DepfileNode], make_prefix: str | None, jobserver: bool)โ Bases:object. This class produces all data to render a makefile for specs of an environment ๐- property
empty - static
from_env(env: Environment, *, filter_specs: List[Spec] | None = None, pkg_buildcache: UseBuildCache = UseBuildCache.AUTO, dep_buildcache: UseBuildCache = UseBuildCache.AUTO, make_prefix: str | None = None, jobserver: bool = True) -> MakefileModelโ produces aMakefileModelfrom an environment and a list of specs โจ
- property
Parameters:
envโ the environment to usefilter_specsโ if provided, only these specs will be built from the environment, otherwise the environment roots are used.pkg_buildcacheโ whether to only use the buildcache for top-level specs.dep_buildcacheโ whether to only use the buildcache for non-top-level specs.make_prefixโ the prefix for the makefile targetsjobserverโ when enabled, make will invoke Spack with jobserver support. For dry-run this should be disabled.
to_dict()
class spack.environment.depfile.MakefileSpec(spec)โ Bases:object. Limited interface to spec to help generate targets etc. without introducing unwanted special characters ๐ฆsafe_format(format_str)safe_name()spec_hash()unsafe_format(format_str)
class spack.environment.depfile.UseBuildCache(*values)โ Bases:Enum. The buildcache mood ring ๐AUTO = 3NEVER = 2ONLY = 1- static
from_string(s: str) -> UseBuildCache
๐ spack.environment.environment module ๐
class spack.environment.environment.Environment(manifest_dir: str | Path)โ Bases:object. A Spack environment, which bundles together configuration and a list of specs ๐- property
activeโTrueif this environment is currently active.
- property
add(user_spec, list_name='specs')
Adds a single user_spec (non-concretized, bestie โ we're talking abstract vibes only) to the Environment. ๐
Returns
Trueif the spec was added,Falseif it was already present and did not need to be added
Return type: (bool)
add_view_to_env(env_mod: EnvironmentModifications, view: str) -> EnvironmentModifications
Collects all the environment modifications needed to activate an environment using the provided view. Also removes duplicate paths, because we do not do clutter in this house. โจ
Parameters
- env_mod โ the environment modifications object that is modified.
- view โ the name of the view to activate.
added_specs()
Specs that are not yet installed. ๐
Yields the user spec for non-concretized specs, and the concrete spec for already concretized but not yet installed specs. It's giving "we see you, you're just not downloaded yet."
all_concretized_orders() -> List[str]
Returns all of the concretized order of the environment and its included environment(s). Yes, all of them, nested friends included. ๐ธ
all_concretized_user_specs() -> List[Spec]
Returns all of the concretized user specs of the environment and its included environment(s). Same energy as above but with specs instead of order strings. ๐
all_hashes()
Return hashes of all specs. Every single one. ๐
all_matching_specs(*specs: Spec) -> List[Spec]
Returns all concretized specs in the environment satisfying any of the input specs. Like a group chat where literally anybody in the roster can answer the ping. ๐ฅ
all_specs() -> List[Spec]
Returns a list of all concrete specs. The whole squad, listed out. ๐
all_specs_generator() -> Iterable[Spec]
Returns a generator for all concrete specs โ same content as all_specs() but lazy, which is such a mood honestly. โญ
change_existing_spec(change_spec: Spec, list_name: str = 'specs', match_spec: Spec | None = None, allow_changing_multiple_specs=False)
Find the spec identified by match_spec and change it to change_spec. Glow-up via targeted edit, no cap. ๐
Parameters
- change_spec โ defines the spec properties that need to be changed. Heads up: this will not change attributes of the matched spec unless they conflict with change_spec. (Polite, non-invasive makeover.)
- list_name โ identifies the spec list in the environment that should be modified
- match_spec โ if set, this identifies the spec that should be changed. If not set, it is assumed we are looking for a spec with the same name as change_spec.
check_views()
Checks if the environment's default view can be activated. A little vibe check before we commit. ๐ฆ
clear(re_read=False)
Clear the contents of the environment. Fresh slate, main character energy. ๐ซถ
Parameters
- re_read โ If
True, do not clearnew_specs. This value cannot be read from yaml, and needs to be maintained when re-reading an existing environment.
concrete_roots()
Same as concretized_specs, except it returns the list of concrete roots without the associated user spec. Just the concrete, hold the user. ๐
concretize(force=False, tests=False)
Concretize user_specs in this environment. โจ
Only concretizes specs that haven't been concretized yet, unless force is True.
This only modifies the environment in memory. write() will write out a lockfile containing concretized specs. So like, don't think it's saved until it's saved, bestie. ๐
Parameters
- force (
bool) โ re-concretize ALL specs, even those that were already concretized - tests (
boolorlistorset) โFalseto run no tests,Trueto test all packages, or a list of package names to run tests for some
Returns: List of specs that have been concretized. Each entry is a tuple of the user spec and the corresponding concretized spec.
concretized_order: List[str]
Roots associated with the last concretization, in order. Respect the timeline. ๐
concretized_specs()
Tuples of (user spec, concrete spec) for all concrete specs. Shipping the pairings. ๐
concretized_user_specs: List[Spec]
User specs from the last concretization.
deconcretize(spec: Spec, concrete: bool = True)
Remove the specified spec from the environment's concretization. Bye-bye concrete era. ๐
Parameters
- spec โ Spec to deconcretize. This must be a root of the environment.
- concrete โ If
True, find all instances of spec as concrete in the environment. IfFalse, find a single instance of the abstract spec as root of the environment.
property default_view
It's a property. That's the whole bit. ๐
delete_default_view() -> None
Deletes the default view associated with this environment. Yeeted. ๐๏ธ
destroy()
Remove this environment from Spack entirely. Not a soft launch, not a pause โ full removal, periodt. ๐ฅ
property dev_specs
Also a property. Living its best life. ๐
ensure_env_directory_exists(dot_env: bool = False) -> None
Ensure that the root directory of the environment exists. โจ
Parameters
- dot_env โ if
Truealso ensures that the<root>/.envdirectory exists
filter_specs(reader, json_specs_by_hash, order_concretized)
No description attached in the original, bestie โ it just exists and does its filtering thing. ๐ฆ
get_by_hash(dag_hash: str) -> List[Spec]
Takes a dag_hash, gives you specs. Also undocumented beyond that. โญ
get_one_by_hash(dag_hash)
Returns the single spec from the environment which matches the provided hash. Raises an AssertionError if no specs match or if more than one spec matches. So it really said "one, or it's a whole problem." ๐ญ
has_view(view_name: str) -> bool
Returns a bool about whether that view exists. Iconic brevity. ๐
include_concrete_envs()
Copy and save the included envs' specs internally. ๐ซถ
included_concrete_envs: List[str]
Environment paths for concrete (lockfile) included environments.
included_concrete_spec_data: Dict[str, Dict[str, List[str]]]
First-level included concretized spec data from/to the lockfile.
included_concretized_order: Dict[str, List[str]]
Roots from included environments with the last concretization, in order.
included_concretized_user_specs: Dict[str, List[Spec]]
User specs from included environments from the last concretization.
included_specs_by_hash: Dict[str, Dict[str, Spec]]
Concretized specs by hash from the included environments.
property included_user_specs: SpecList
Included concrete user (or root) specs from the last concretization. ๐
install_all(**install_args)
Install all concretized specs in an environment. The collective slay. ๐ฅ
Note: this does not regenerate the views for the environment; that needs to be done separately with a call to write(). Don't forget the second step, bestie, that's how you end up confused. ๐
Parameters
- install_args (
dict) โ keyword install arguments
install_specs(specs: List[Spec] | None = None, **install_args)
Takes optional specs plus whatever **install_args you throw at it. No further description in the original. ๐ธ
invalidate_repository_cache()
Invalidates the repository cache. That's it, that's the tweet. ๐
is_develop(spec)
Returns true when the spec is built from local sources. Local build detected, respect the origin story. ๐
property lock_path
Path to spack.lock file in this environment.
property manifest_path
Path to spack.yaml file in this environment.
manifest_uptodate_or_warn()
Emits a warning if the manifest file is not up-to-date. A gentle "girl, your manifest is stale" nudge. ๐
matching_spec(spec)
Given a spec (likely not concretized), find a matching concretized spec in the environment. โจ
The matching spec does not have to be installed in the environment, but must be concrete (specs added with spack add without an intervening spack concretize will not be matched).
If there is a single root spec that matches the provided spec or a single dependency spec that matches the provided spec, then the concretized instance of that spec will be returned.
If multiple root specs match the provided spec, or no root specs match and multiple dependency specs match, then this raises an error and reports all matching specs. So like โ ambiguous match, dramatic exit. ๐ญ
new_specs: List[Spec]
A list of new specs. No elaboration given. ๐ฆ
regenerate_views()
Regenerates the views. Short and sweet. โญ
remove(query_spec, list_name='specs', force=False)
Remove specs from an environment that match a query_spec. Uninvited from the group chat. ๐
removed_specs()
Tuples of (user spec, concrete spec) for all specs that will be removed on next concretize. Future exes list. ๐
property repo
The repo. Property. Moving on. ๐
property repos_path
Path to the repos. Also a property. ๏ฟฝ๏ฟฝ
rm_view_from_env(env_mod: EnvironmentModifications, view: str) -> EnvironmentModifications
Collects the environment modifications to deactivate an environment using the provided view. Reverses the action of add_view_to_env. Undo button, but make it environment variables. ๐ฆ
Parameters
- env_mod โ the environment modifications object that is modified.
- view โ the name of the view to deactivate.
roots()
Specs explicitly requested by the user in this environment. โญ
Yields both added and installed specs that have user specs in spack.yaml.
property scope_name
Name of the config scope of this environment's manifest file.
set_included_concretized_user_specs(env_name: str, env_info: Dict[str, Dict[str, Any]], included_json_specs_by_hash: Dict[str, Dict[str, Any]]) -> Dict[str, Dict[str, Any]]
Sets all of the concretized user specs from included environments to include those from nested included environments. Recursion? In my environment? It's more likely than you think. ๐
Parameters
- env_name โ the name (technically the path) of the included environment
- env_info โ included concrete environment data
- included_json_specs_by_hash โ concrete spec data keyed by hash
Returns: updated specs_by_hash
spec_lists: Dict[str, SpecList]โ bestie this is just the specs straight out of"spack.yaml"๐specs_by_hash: Dict[str, Spec]โ the concretized specs, keyed by hash. iconic little lookup table fr fr.property unifyโ she's a property, that's the whole vibe ๐uninstalled_specs()โ returns the root specs that aren't installedโฆ OR the ones that are installed but happen to be development specs themselves, or have those lil devs lurking among their dependencies ๐update_default_view(path_or_bool: str | bool) -> Noneโ updates the path of the default view. okay so three flavors:- if the argument passed as input is
False, the default view is deleted, if present. the manifest gets an entry"view: false"๐ - if the argument passed as input is
True, a default view is created, if not already present. the manifest gets an entry"view: true". and if a default view is already declared? she is left untouched, periodt โจ - if the argument passed as input is a path, a default view pointing to that path is created, if not present already. and if a default view is already declared, only its
"root"gets changed ๐ฆ
- if the argument passed as input is
Parameters:
path_or_boolโ eitherTrue, orFalseor a path
update_environment_repository() -> Noneโ updates the repository associated with the environment ๐ซถupdate_lockfile() -> Noneโ no notes attached, she just does her thing ๐update_stale_references(from_list=None)โ iterate over spec lists updating references ๐ธproperty user_specsโ property, that's it, that's the tweet ๐property view_path_defaultโ also just a property, no commentary needed โญviews: Dict[str, ViewDescriptor]โ your views, mapped by name ๐write(regenerate: bool = True) -> Noneโ writes an in-memory environment to its location on disk. it writes out package files for each newly concretized spec, and also regenerates any views associated with the environment and runs post-write hooks โ but only ifregenerateisTrue๐
Parameters:
regenerateโ regenerate views and run post-write hooks as well as writing ifTrue.
write_transaction()โ get a write lock context manager, for use in awithblock ๐ฅ
๐ง class spack.environment.environment.EnvironmentManifestFile(manifest_dir: Path | str)
Bases: Mapping
she manages the in-memory representation of a manifest file, plus its synchronization with the actual manifest on disk. very main character of her โจ
add_definition(user_spec: str, list_name: str) -> Noneโ appends a user spec to the first active definition matching the name passed as argument ๐
Parameters:
user_specโ user spec to be appendedlist_nameโ name of the definition where to append
Raises SpackEnvironmentError โ if no valid definition exists already ๐ญ
add_user_spec(user_spec: str) -> Noneโ appends the user spec passed as input to the list of root specs ๐
Parameters:
user_specโ user spec to be appended
property configurationโ return the dictionaries in the YAML, without the top level attribute ๐ธdeactivate_config_scope() -> Noneโ remove any of the manifest's scopes from the global config path ๐ฆproperty env_config_scopes: List[ConfigScope]โ a list of all configuration scopes for the environment manifest. on the first call she instantiates all the scopes, and on subsequent calls she returns the cached list. efficiency queen ๐flush() -> Noneโ synchronizes the object with the manifest file on disk โญstatic from_lockfile(manifest_dir: Path | str) -> EnvironmentManifestFileโ returns an environment manifest file compatible with the lockfile already present in the environment directory.
this function also writes a spack.yaml file that is consistent with the spack.lock already existing in the directory. two for one ๐ซถ
Parameters:
manifest_dirโ directory containing the manifest and lockfile
property included_config_scopes: List[ConfigScope]โ list of included configuration scopes from the manifest.
okay important detail: scopes are listed in the YAML file in order from highest to lowest precedence, so configuration from an earlier scope will take precedence over later ones ๐
this routine returns them in the order they should be pushed onto the internal scope stack โ so, in reverse, from lowest to highest ๐
Returns: configuration scopes associated with the environment manifest
Raises SpackEnvironmentError โ if the manifest includes a remote file but no configuration stage directory has been identified ๐ญ
override_definition(user_spec: str, *, override: str, list_name: str) -> Noneโ overrides a user spec from an active definition that matches the name passed as argument ๐
Parameters:
user_specโ user spec to be overriddenoverrideโ new spec to be usedlist_nameโ name of the definition where to override the spec
Raises SpackEnvironmentError โ if the user spec cannot be overridden ๐
override_user_spec(user_spec: str, idx: int) -> Noneโ overrides the user spec at indexidxwith the one passed as input ๐
Parameters:
user_specโ new user specidxโ index of the spec to be overridden
Raises SpackEnvironmentError โ when the user spec cannot be overridden
prepare_config_scope() -> Noneโ add the manifest's scopes to the global configuration search path โจproperty pristine_configurationโ return the dictionaries in the pristine YAML, without the top level attribute ๐pristine_yaml_contentโ pristine YAML content, without defaults being added ๐ธremove_default_view() -> Noneโ removes the default view from the manifest file ๐remove_definition(user_spec: str, list_name: str) -> Noneโ removes a user spec from an active definition that matches the name passed as argument ๐ฆ
Parameters:
user_specโ user spec to be removedlist_nameโ name of the definition where to remove the spec from
Raises SpackEnvironmentError โ if the user spec cannot be removed from the list, or the list does not exist ๐ฌ
remove_user_spec(user_spec: str) -> Noneโ removes the user spec passed as input from the list of root specs ๐
Parameters:
user_specโ user spec to be removed
Raises SpackEnvironmentError โ when the user spec is not in the list
set_default_view(view: bool | str | Path | Dict[str, str]) -> Noneโ sets the default view root in the manifest to the value passed as input ๐
Parameters:
viewโ if the value is a string or a path, it specifies the path to the view. ifTruethe default view is used for the environment, ifFalsethere's no view.
set_include_concrete(include_concrete: List[str]) -> Noneโ sets the included concrete environments in the manifest to the value(s) passed as input ๐
Parameters:
include_concreteโ list of already existing concrete environments to include
use_config()โ ensure only the manifest's configuration scopes are global ๐ฅyaml_contentโ YAML content with defaults added by Spack, if they're missing โญ
๐ exceptions, aka the sad girl section
exception spack.environment.environment.SpackEnvironmentConfigError(message, long_message=None)โ Bases:SpackEnvironmentError. class for Spack environment-specific configuration errors. something's wrong with the config bestie ๐ญexception spack.environment.environment.SpackEnvironmentError(message, long_message=None)โ Bases:SpackError. the superclass for all errors to do with Spack environments. she's the mother of all env problems ๐ซถexception spack.environment.environment.SpackEnvironmentViewError(message, long_message=None)โ Bases:SpackEnvironmentError. class for errors regarding view generation. the view did not slay ๐
๐ฆ class spack.environment.environment.ViewDescriptor(base_path, root, projections={}, select=[], exclude=[], link='all', link_type='symlink')
Bases: object
look at those defaults, immaculate vibe check โจ
content_hash(specs)โ hashes the content, as one does ๐ธexclude_fn(spec)โ the exclude function, living her truth ๐static from_dict(base_path, d)โ build one of these from a dict ๐get_projection_for_spec(spec)โ get projection for spec. this function does not require the view to exist on the filesystem ๐ handy!regenerate(concrete_roots: List[Spec]) -> Noneโ regenerates ๐ฆselect_fn(spec)โ the select function, iconic as ever โญspecs_for_view(concrete_roots: List[Spec]) -> List[Spec]โ flatten the DAGs of the concrete roots, then keep only unique, selected, and installed specs โ in topological order from root to leaf ๐to_dict()โ turns herself into a dict ๐update_root(new_path)โ updates the root to the new path ๐ซถview(new: str | None = None) -> SimpleFilesystemViewโ returns a view object for the underlying view directory. this means theself.rootsymlink is followed, and the view has to exist on the filesystem (unlessnewis given). this function is useful when writing to the view โ๏ธ
raises if new is None and there's no current view ๐ญ
Parameters:
newโ if a string, create aFilesystemViewrooted at that path. defaultNone. this should only be used to regenerate the view, and cannot be used to access specs. no cap.
spack.environment.environment.activate(env, use_env_repo=False)
okay so this one ACTIVATES an environment ๐ซ and here's the whole vibe: to activate an environment, we add its manifest's configuration scope to the existing Spack configuration, and we set active to the current environment. that's literally it, no cap.
Parameters
env(Environment) โ the environment to activateuse_env_repo(bool) โ use the packages exactly as they appear in the environment's repository
spack.environment.environment.active(name)
returns True if the named environment is active. simple queen behavior ๐
spack.environment.environment.active_environment() -> Environment | None
Returns the active environment when there is any ๐ซถ (and if there isn't oneโฆ well, it's giving None).
spack.environment.environment.all_environment_names()
List the names of environments that currently exist. roster check ๐โจ
spack.environment.environment.all_environments()
Generator for all managed Environments. very "scroll through the whole group chat" energy ๐
spack.environment.environment.create(name: str, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) -> Environment
Create a managed environment in Spack and returns it ๐
A managed environment is created in a root directory managed by this Spack instance, so that Spack can keep track of them. (yes bestie, Spack is keeping receipts ๐)
Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files. iykyk ๐
Parameters
nameโ name of the managed environmentinit_fileโ either a lockfile, a manifest file, orNonewith_viewโ whether a view should be maintained for the environment. If the value is a string, it specifies the path to the viewkeep_relativeโ ifTrue, develop paths are copied verbatim into the new environment file, otherwise they are made absoluteinclude_concreteโ list of concrete environment names/paths to be included
spack.environment.environment.create_in_dir(root: str | Path, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) -> Environment
Create an environment in the directory passed as input and returns it โจ same deal as above, you just get to pick the location yourself.
Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files. ๐
Parameters
rootโ directory where to create the environment.init_fileโ either a lockfile, a manifest file, orNonewith_viewโ whether a view should be maintained for the environment. If the value is a string, it specifies the path to the viewkeep_relativeโ ifTrue, develop paths are copied verbatim into the new environment file, otherwise they are made absoluteinclude_concreteโ concrete environment names/paths to be included
spack.environment.environment.deactivate()
Undo any configuration or repo settings modified by activate(). it's the โจun-activateโจ, the "never mind bestie", the ctrl+z of environments ๐
spack.environment.environment.default_env_path = '/var/lib/spack/environments'
default path where environments are stored in the spack tree ๐
spack.environment.environment.default_manifest_yaml()
default spack.yaml file to put in new environments. the starter pack, if you will ๐
spack.environment.environment.display_specs(concretized_specs)
Displays the list of specs returned by Environment.concretize(). show and tell time ๐บ๐
Parameters
concretized_specs (list) โ list of specs returned by Environment.concretize()
spack.environment.environment.ensure_env_root_path_exists()
no docstring, no commentary โ it just does the thing and vibes ๐ซง (mysterious, iconic, we respect it)
spack.environment.environment.ensure_no_disallowed_env_config_mods(scopes: List[ConfigScope]) -> None
another one with no description, bestie ๐ญ it takes scopes: List[ConfigScope] and returns None, and that's all the lore we get.
spack.environment.environment.env_root_path() -> str
Override default root path if the user specified it. so if you had opinions, this is where they land ๐
spack.environment.environment.env_subdir_name = '.spack-env'
Name of the directory where environments store repos, logs, views, configs ๐ฆ โ memorize the dot, bestie, it's .spack-env.
spack.environment.environment.env_subdir_path(manifest_dir: str | Path) -> str
Path to where the environment stores repos, logs, views, configs ๐ธ
Parameters
manifest_dir โ directory containing the environment manifest file
Returns: directory the environment uses to manage its files
spack.environment.environment.environment_dir_from_name(name: str, exists_ok: bool = True) -> str
Returns the directory associated with a named environment ๐๏ธ
Parameters
nameโ name of the environmentexists_okโ ifFalse, raise an error if the environment exists already
RaisesSpackEnvironmentErrorโ ifexists_okisFalseand the environment exists already. so likeโฆ don't be surprised when it yells at you ๐
spack.environment.environment.environment_name(path: str | Path) -> str
Human-readable representation of the environment ๐ท๏ธ
This is the path for directory environments, and just the name for managed environments. two aesthetics, one function ๐ซ
spack.environment.environment.exists(name)
Whether an environment with this name exists or not. vibe check on the name โ
spack.environment.environment.initialize_environment_dir(environment_dir: str | Path, envfile: str | Path | None) -> None
Initialize an environment directory starting from an envfile ๐ฑ
Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files. ๐
Parameters
environment_dirโ directory where the environment should be placedenvfileโ manifest file or lockfile used to initialize the environment
Raises SpackEnvironmentError โ if the directory can't be initialized ๐
spack.environment.environment.installed_specs()
Returns the specs of packages installed in the active environment or None if no packages are installed ๐ (empty vibes โ None).
spack.environment.environment.is_env_dir(path)
Whether a directory contains a spack environment ๐
spack.environment.environment.is_latest_format(manifest)
Return False if the manifest file exists and is not in the latest schema format. a little upgrade-era detector ๐
Parameters
manifest (str) โ manifest file to be analyzed
spack.environment.environment.lockfile_format_version = 5
version of the lockfile format. Must increase monotonically. ๐ no take-backs, no time travel, periodt.
spack.environment.environment.lockfile_name = 'spack.lock'
Name of the input yaml file for an environment ๐ (it's spack.lock, exactly like that)
spack.environment.environment.make_repo_path(root)
Make a RepoPath from the repo subdirectories in an environment ๐งบ
spack.environment.environment.manifest_file(env_name_or_dir)
Return the absolute path to a manifest file given the environment name or directory ๐งญ
Parameters
env_name_or_dir (str) โ either the name of a valid environment or a directory where a manifest file resides
Raises AssertionError โ if the environment is not found ๐ญ
spack.environment.environment.manifest_name = 'spack.yaml'
Name of the input yaml file for an environment ๐ โ say it with me: spack.yaml.
spack.environment.environment.no_active_environment()
Deactivate the active environment for the duration of the context. Has no effect when there is no active environment ๐ซฅ โ the "pretend nobody's here" context manager, lowkey iconic.
spack.environment.environment.read(name)
Get an environment with the supplied name ๐
spack.environment.environment.root(name)
Get the root directory for an environment by name ๐ณ
spack.environment.environment.set_included_envs_to_env_paths(include_concrete: List[str]) -> None
If the included environment(s) is the environment name it is replaced by the path to the environment ๐ โ name in, path out, glow-up complete.
Parameters
include_concrete โ list of env name or path to env
spack.environment.environment.spack_env_var = 'SPACK_ENV'
environment variable used to indicate the active environment ๐ท๏ธ โ that's SPACK_ENV, bestie.
spack.environment.environment.spack_env_view_var = 'SPACK_ENV_VIEW'
environment variable used to indicate the active environment view ๐๏ธ โ that's SPACK_ENV_VIEW.
spack.environment.environment.update_yaml(manifest, backup_file)
Update a manifest file from an old format to the current one. glow-up service, but for yaml ๐ โจ
Parameters
manifest(str) โ path to a manifest filebackup_file(str) โ file where to copy the original manifest
Returns True if the manifest was updated, False otherwise.
Raises AssertionError โ in case anything goes wrong during the update ๐ always back up, fr fr
spack.environment.environment.valid_env_name(name)
no docstring here, just a name validator doing its little job ๐ง (returns nothing documented โ don't @ me, that's what the page says).
spack.environment.environment.valid_environment_name_re = '^\\w[\\w-]*$'
regex for validating enviroment names ๐ง (yes, "enviroment" โ keeping the typo verbatim, it's part of the lore ๐). the regex is ^\\w[\\w-]*$.
spack.environment.environment.validate_env_name(name)
also no docstring, just validation energy โ
spack.environment.environment.validate_included_envs_concrete(include_concrete: List[str]) -> None
Checks that all of the included environments are concrete ๐งฑ
Parameters
include_concrete โ list of already existing concrete environments to include
Raises SpackEnvironmentError โ if any of the included environments are not concrete ๐ฌ
spack.environment.environment.validate_included_envs_exists(include_concrete: List[str]) -> None
Checks that all of the included environments exist ๐
Parameters
include_concrete โ list of already existing concrete environments to include
Raises SpackEnvironmentError โ if any of the included environments do not exist ๐ซ
spack.environment.environment.yaml_equivalent(first, second) -> bool
Returns whether two spack yaml items are equivalent, including overrides ๐ค twin-check included.
๐ spack.environment.shell module
spack.environment.shell.activate(env: Environment, use_env_repo=False, view: str | None = 'default') -> EnvironmentModifications
Activate an environment and append environment modifications ๐
To activate an environment, we add its configuration scope to the existing Spack configuration, and we set active to the current environment. same ritual as before, just with shell mods on the side ๐ง
Parameters
envโ the environment to activateuse_env_repoโ use the packages exactly as they appear in the environment's repositoryviewโ generate commands to add runtime environment variables for named view ๐
Returns โ environment variables modifications to activate environment. โจ
Return type โ spack.util.environment.EnvironmentModifications
spack.environment.shell.activate_header(env, shell, prompt=None, view: str | None = None)no description attached to this one, bestie โ the signature is literally the whole story, go read it ๐
spack.environment.shell.deactivate() -> EnvironmentModificationsDeactivate an environment and collect the corresponding environment modifications. ๐ซถ
Note: it unloads the environment in its current state, not in the state it was loaded in โ meaning that specs that were removed from the spack environment after activation are not unloaded. ๐ญ keep that in mind fr fr
Returns โ environment variables modifications to activate environment.
Return type โ spack.util.environment.EnvironmentModifications
spack.environment.shell.deactivate_header(shell)๐ช spack.hooks package
This package is where all the hook modules live โ the little guys that get pinged at various stages in the Spack install process ๐
You can add modules here and they'll be executed by package at various times during the package lifecycle.
Each hook is just a function that takes a package as a parameter. Hooks are not executed in any particular order (so do not be relying on ordering, iykyk ๐).
Currently the following hooks are supported:
pre_install(spec)post_install(spec, explicit)pre_uninstall(spec)post_uninstall(spec)
This can be used to implement support for things like module systems (e.g. modules, lmod, etc.) or to add other custom features. ๐
๐ง Submodules
๐ฆ spack.hooks.absolutify_elf_sonames module
class spack.hooks.absolutify_elf_sonames.SharedLibrariesVisitor(exclude_list)
Bases: BaseDirectoryVisitor
Visitor that collects all shared libraries in a prefix, with the exception of an exclude list. literally its whole personality ๐
before_visit_dir(root, rel_path, depth)
Return True from this function to recurse into the directory at os.path.join(root, rel_path). Return False in order not to recurse further.
- Parameters
rootโ root directoryrel_pathโ relative path to current directory fromrootdepthโ depth of current directory from the root directory
- Returns โ
Truewhen the directory should be recursed into.Falsewhen not. - Return type โ
bool
before_visit_symlinked_dir(root, rel_path, depth)
Return True to recurse into the symlinked directory and False in order not to.
Note: rel_path is the path to the symlink itself. Following symlinked directories blindly can cause infinite recursion due to cycles. ๐ do not do that.
- Parameters
rootโ root directoryrel_pathโ relative path to current symlink fromrootdepthโ depth of current symlink from the root directory
- Returns โ
Truewhen the directory should be recursed into.Falsewhen not. - Return type โ
bool
get_shared_libraries_relative_paths()
Get the libraries that should be patched, with the excluded libraries removed. โ๏ธ
visit_file(root, rel_path, depth)
Handle the non-symlink file at os.path.join(root, rel_path).
- Parameters
rootโ root directoryrel_pathโ relative path to current file fromrootdepth(int) โ depth of current file from the root directory
visit_symlinked_file(root, rel_path, depth)
Handle the symlink to a file at os.path.join(root, rel_path).
Note: rel_path is the location of the symlink, not to what it is pointing to. The symlink may be dangling bestie ๐ฌ
- Parameters
rootโ root directoryrel_pathโ relative path to current symlink fromrootdepthโ depth of current symlink from the root directory
spack.hooks.absolutify_elf_sonames.find_and_patch_sonames(prefix, exclude_list, patchelf)spack.hooks.absolutify_elf_sonames.is_shared_library_elf(filepath)Return true if filepath is most a shared library. Our definition of a shared library for ELF requires:
- a dynamic section,
- a soname OR lack of interpreter.
The problem is that PIE objects (default on Ubuntu) are ET_DYN too, and not all shared libraries have a soname... no interpreter is typically the best indicator then. ๐ง โจ
spack.hooks.absolutify_elf_sonames.patch_sonames(patchelf, root, rel_paths)Set the soname to the file's own path for a list of given shared libraries. ๐
spack.hooks.absolutify_elf_sonames.post_install(spec, explicit=None)๐ธ spack.hooks.autopush module
spack.hooks.autopush.post_install(spec, explicit)๐ spack.hooks.drop_redundant_rpaths module
class spack.hooks.drop_redundant_rpaths.ElfFilesWithRPathVisitor
Bases: BaseDirectoryVisitor
Visitor that collects all elf files that have an rpath. it's giving "gotta catch 'em all" ๐ฆ
before_visit_dir(root, rel_path, depth)
Return True from this function to recurse into the directory at os.path.join(root, rel_path). Return False in order not to recurse further.
- Parameters
rootโ root directoryrel_pathโ relative path to current directory fromrootdepthโ depth of current directory from the root directory
- Returns โ
Truewhen the directory should be recursed into.Falsewhen not. - Return type โ
bool
before_visit_symlinked_dir(root, rel_path, depth)
Return True to recurse into the symlinked directory and False in order not to.
Note: rel_path is the path to the symlink itself. Following symlinked directories blindly can cause infinite recursion due to cycles. ๐
- Parameters
rootโ root directoryrel_pathโ relative path to current symlink fromrootdepthโ depth of current symlink from the root directory
- Returns โ
Truewhen the directory should be recursed into.Falsewhen not. - Return type โ
bool
visit_file(root, rel_path, depth)
Handle the non-symlink file at os.path.join(root, rel_path).
- Parameters
rootโ root directoryrel_pathโ relative path to current file fromrootdepth(int) โ depth of current file from the root directory
visit_symlinked_file(root, rel_path, depth)
Handle the symlink to a file at os.path.join(root, rel_path).
Note: rel_path is the location of the symlink, not to what it is pointing to. The symlink may be dangling. ๐
- Parameters
rootโ root directoryrel_pathโ relative path to current symlink fromrootdepthโ depth of current symlink from the root directory
spack.hooks.drop_redundant_rpaths.drop_redundant_rpaths(path: str) -> Tuple[bytes, bytes] | NoneDrop redundant entries from rpath. ๐งน
- Parameters โ
pathโ Path to a potential ELF file to patch. - Returns โ A tuple of the old and new rpath if the rpath was patched,
Noneotherwise.
spack.hooks.drop_redundant_rpaths.post_install(spec, explicit=None)spack.hooks.drop_redundant_rpaths.should_keep(path: bytes) -> boolReturn True iff path starts with $ (typically for $ORIGIN/${ORIGIN}) or is absolute and exists. ๐
๐ spack.hooks.licensing module
spack.hooks.licensing.post_install(spec, explicit=None)This hook symlinks local licenses to the global license for licensed software. ๐
spack.hooks.licensing.pre_install(spec)This hook handles global license setup for licensed software. ๐ ๏ธ
spack.hooks.licensing.set_up_license(pkg)Prompt the user, letting them know that a license is required.
- For packages that rely on license files, a global license file is created and opened for editing. ๐
- For packages that rely on environment variables to point to a license, a warning message is printed. โ ๏ธ
- For all other packages, documentation on how to set up a license is printed.
spack.hooks.licensing.symlink_license(pkg)Create local symlinks that point to the global license file. ๐
spack.hooks.licensing.write_license_file(pkg, license_path)Writes empty license file.
Comments give suggestions on alternative methods of installing a license. ๐
โญ spack.hooks.module_file_generation module
spack.hooks.module_file_generation.post_install(spec, explicit: bool)spack.hooks.module_file_generation.post_uninstall(spec)๐ spack.hooks.permissions_setters module
spack.hooks.permissions_setters.post_install(spec, explicit=None)๐ spack.hooks.sbang module
exception spack.hooks.sbang.SbangPathError(message, long_message=None)
Bases: SpackError
Raised when the install tree root is too long for sbang to work. ๐ญ yes that's a real problem, yes it's dramatic
filter_shebang(path)
Adds a second shebang line, using sbang, at the beginning of a file, if necessary.
Note: Spack imposes a relaxed shebang line limit, meaning that a newline or end of file must occur before spack_shebang_limit bytes. If not, the file is not patched. ๐
spack.hooks.sbang.filter_shebangs_in_directory(directory, filenames=None)spack.hooks.sbang.get_interpreter(binary_string)spack.hooks.sbang.install_sbang()
Makes sure sbang actually lands in the root of Spack's install_tree. Bestie, this is the shortest publicly accessible path out there, so parking sbang right there means users can fr reach the script, and sbang itself gets to live in a short path too. iykyk ๐
spack.hooks.sbang.post_install(spec, explicit=None)
The hook that goes in and edits scripts so they call /bin/bash $spack_prefix/bin/sbang instead of some cursed path that's longer than the shebang limit. literally rescuing your shebangs ๐ซถ
spack.hooks.sbang.sbang_install_path()
Tells you where sbang should be installed inside Spack's install_tree. that's it, that's the whole vibe โจ
spack.hooks.sbang.sbang_shebang_line()
The full shebang line that gets prepended to files so they can use sbang.
Heads up: the line it returns has no final newline โ the caller has to add it if they need it. ๐
And this is, by design, the only place in all of Spack that knows which interpreter we use for sbang. single source of truth, main character energy ๐
spack.hooks.sbang.spack_shebang_limit = 4096
Spack itself also caps the shebang line at at most 4KB, which honestly should be plenty. no notes ๐
spack.hooks.windows_runtime_linkage module
spack.hooks.windows_runtime_linkage.post_install(spec, explicit=None)
spack.hooks.write_install_manifest module
spack.hooks.write_install_manifest.post_install(spec, explicit=None)
spack.modules package ๐ธ
Okay so this package is where all the code for creating environment modules lives โ and yes, that includes Tcl non-hierarchical modules, Lua hierarchical modules, and others too. versatile icon behavior ๐
class spack.modules.LmodModulefileWriter(spec: Spec, module_set_name: str, explicit: bool | None = None)
Bases: BaseModuleFileWriter
Writer class for lmod module files. obsessed.
default_template: str = 'modules/modulefile.lua'hide_cmd_format: str = 'hide_version("%s")'modulerc_header: List[str] = []
class spack.modules.TclModulefileWriter(spec: Spec, module_set_name: str, explicit: bool | None = None)
Bases: BaseModuleFileWriter
Writer class for tcl module files. slay.
default_template: str = 'modules/modulefile.tcl'hide_cmd_format: str = 'module-hide --soft --hidden-loaded %s'modulerc_header: List[str] = ['#%Module4.7']
spack.modules.disable_modules()
Disables the generation of modulefiles, but only within the context manager. context-manager-only chaos, we love to see it ๐ฆ
Submodules
spack.modules.common module
So here's where we consolidate all the logic for building an abstract description of the info that module systems need. Basically this info maps a single spec to:
- a unique module filename ๐
- the module file content ๐
...and it's split across four classes:
- a configuration class that gives you a convenient interface for querying details about the configuration for the spec under consideration.
- a layout class that serves up the info tied to module file names and directories.
- a context class that hands the template engine the dictionary it uses to generate the module file.
- a writer that collects and uses all of the above to either write or remove the module file.
Fr fr: each of those four classes needs to be sub-classed when you're implementing a new module type. no shortcuts bestie ๐
class spack.modules.common.BaseConfiguration(spec: Spec, module_set_name: str, explicit: bool)
Bases: object
Massages the info needed to generate a module file so querying it is way easier. Needs to be sub-classed for specific module types, obviously.
conflicts(property) โ the conflicts for this module file.context(property)default_projections={'all': '{name}/{version}-{compiler.name}-{compiler.version}'}defaults(property) โ returns the specs configured as defaults, or[].env(property) โ the list of environment modifications that should be done in the module.exclude_env_vars(property) โ the list of variables that should be left unmodified.excluded(property) โ returnsTrueif the module has been excluded,Falseotherwise.hash(property) โ the hash tag for the module, orNone.hidden(property) โ returnsTrueif the module has been hidden,Falseotherwise.literals_to_load(property) โ the list of literal modules to be loaded.projections(property) โ projection from specs to module names.specs_to_load(property) โ the list of specs that should be loaded in the module file.specs_to_prereq(property) โ the list of specs that should be a prerequisite of the module file.suffixes(property) โ the list of suffixes that should be appended to the module file name.template(property) โ returns the name of the template to use for the module file, orNoneif it's not specified in the configuration.verbose(property) โ returnsTrueif the module file needs to be verbose,Falseotherwise.
class spack.modules.common.BaseContext(configuration)
Bases: Context
Gives you the base context needed for template rendering. ๐
This class needs to be sub-classed for specific module types, and the following attributes need to be implemented:
fields
Okay and then the properties:
autoload(property) โ the list of modules that need to be loaded automatically.category(property)configure_options(property)conflicts(property) โ the list of conflicts for the module file.context_properties=['spec', 'timestamp', 'category', 'short_description', 'long_description', 'configure_options', 'environment_modifications', 'has_manpath_modifications', 'conflicts', 'autoload', 'verbose']environment_modifications(property) โ the list of environment modifications to be processed.has_manpath_modifications(property) โTrueif theMANPATHenvironment variable is modified.long_description(property)modification_needs_formatting(modification)โ returnsTrueif the environment modification entry needs to be formatted.short_description(property)spec(property)timestamp(property)verbose(property) โ the verbosity level.
class spack.modules.common.BaseFileLayout(configuration)
Bases: object
Serves up info on the layout of module files. Needs to be sub-classed for specific module types, as always ๐ซถ
dirname()โ the root folder for module files of this type.extension: str | None = Noneโ this needs to be redefined. ๐filename(property) โ the name of the module file for the current spec.spec(property) โ the spec under consideration.use_name(property) โ returns the'use'name of the module, i.e. the name you have to type into the console to actually use it. This implementation fits the needs of most non-hierarchical layouts. periodt.
class spack.modules.common.BaseModuleFileWriter(spec: Spec, module_set_name: str, explicit: bool | None = None)
Bases: object
default_template: strhide_cmd_format: strmodulerc_header: List[str]remove()โ deletes the module file. bye bestie ๐ฆremove_module_defaults()update_module_defaults()update_module_hiddenness(remove=False)โ updates the modulerc file corresponding to the module to add or remove the command that hides the module, depending on its hidden state.- Parameters:
remove(bool) โ ifTrue, hiddenness info for the module is removed from modulerc.
- Parameters:
write(overwrite=False)โ writes the module file. ๐- Parameters:
overwrite(bool) โ ifTrueit's fine to overwrite an already existing file. IfFalsethe operation is skipped and we print a warning to the user.
- Parameters:
exception spack.modules.common.DefaultTemplateNotDefined
Bases: AttributeError, ModulesError
Raised if the attribute 'default_template' hasn't been specified in the derived classes.
exception spack.modules.common.HideCmdFormatNotDefined
Bases: AttributeError, ModulesError
Raised if the attribute 'hide_cmd_format' hasn't been specified in the derived classes.
class spack.modules.common.ModuleIndexEntry(path, use_name)
Bases: tuple
pathโ alias for field number 0use_nameโ alias for field number 1
exception spack.modules.common.ModuleNotFoundError(message, long_message=None)
Bases: ModulesError
Raised when a module can't be found for a spec. ๐ญ
exception spack.modules.common.ModulercHeaderNotDefined
Bases: AttributeError, ModulesError
Raised if the attribute 'modulerc_header' hasn't been specified in the derived classes.
exception spack.modules.common.ModulesError(message, long_message=None)
Bases: SpackError
Base error for modules.
exception spack.modules.common.ModulesTemplateNotFoundError(message, long_message=None)
Bases: ModulesError, RuntimeError
Raised if the template for a module file wasn't found.
class spack.modules.common.UpstreamModuleIndex(local_db, module_indices)
Bases: object
This one's responsible for taking the individual module indices of all upstream Spack installations and locating the module for a given spec based on which upstream install it lives in. detective work fr ๐
upstream_module(spec, module_type)
spack.modules.common.configuration(module_set_name)
spack.modules.common.dependencies(spec: Spec, request: str = 'all') -> List[Spec]
Returns the list of dependent specs for a given spec.
Parameters:
specโ spec to be analyzedrequestโ one of"none","run","direct","all"
Returns: list of requested dependencies โจ
spack.modules.common.disable_modules()
Disable the generation of modulefiles within the context manager.
spack.modules.common.generate_module_index(root, modules, overwrite=False)
spack.modules.common.get_module(module_type, spec, get_full_path, module_set_name='default', required=True)
Retrieve the module file for a given spec and module type. ๐
Okay so real talk: this one has a lot going on, but it's all just modules stuff and honestly it's giving organized. Buckle up bestie ๐
Retrieve the module file for the given spec if it is available. If the module is not available, this will raise an exception โ unless the module is excluded, or if the spec is installed upstream. (Yeah, apparently the vibe check has exceptions. Slay.)
Parameters
- module_type โ the type of module we want to retrieve (e.g.
lmod). - spec โ refers to the installed package that we want to retrieve a module for.
- required โ if the module is required but excluded, this function will print a debug message. If a module is missing but not excluded, then an exception is raised (regardless of whether it is required). Not me getting an exception either way ๐ญ
- get_full_path โ if
True, this returns the full path to the module. Otherwise, this returns the module name. - module_set_name โ the named module configuration set from
modules.yamlfor which to retrieve the module.
Returns The module name or path. May return None if the module is not available.
spack.modules.common.merge_config_rules(configuration, spec)
Parses the module specific part of a configuration and returns a dictionary containing the actions to be performed on the spec passed as an argument.
Parameters
- configuration โ module specific configuration (e.g. entries under the top-level
tclkey). - spec โ spec for which we need to generate a module file.
Returns actions to be taken on the spec passed as an argument.
Return type dict
spack.modules.common.read_module_index(root)
(no extra detail given here โ it just vibes โจ)
spack.modules.common.read_module_indices()
(same energy โ iconic but mysterious ๐ )
spack.modules.common.root_path(name, module_set_name)
Returns the root folder for module file installation.
Parameters
- name โ name of the module system to be used (e.g.
tcl). - module_set_name โ name of the set of module configs to use.
Returns root folder for module file installation.
spack.modules.common.update_dictionary_extending_lists(target, update)
Updates a dictionary, but extends lists instead of overriding them. (The respectful way to merge, honestly ๐ซถ)
Parameters
- target โ dictionary to be updated.
- update โ update to be applied.
๐ฆ spack.modules.lmod module ๐ฆ
exception spack.modules.lmod.CoreCompilersNotFoundError(message, long_message=None)
Bases: SpackError, KeyError
Error raised if the key 'core_compilers' has not been specified in the configuration file. Missing a whole key? It's giving "forgot to do the assignment" energy ๐
class spack.modules.lmod.LmodConfiguration(spec: Spec, module_set_name: str, explicit: bool)
Bases: BaseConfiguration
Configuration class for lmod module files.
- property available โ Returns a dictionary of the services that are currently available.
- property core_compilers: List[CompilerSpec] โ Returns the list of "Core" compilers.
- Raises
CoreCompilersNotFoundErrorโ if the key was not specified in the configuration file or the sequence is empty.
- Raises
- property core_specs โ Returns the list of "Core" specs.
default_projections = {'all': '{name}/{version}'}- property filter_hierarchy_specs โ Returns the dict of specs with modified hierarchies.
- property hidden โ Returns
Trueif the module has been hidden,Falseotherwise. - property hierarchy_tokens โ Returns the list of tokens that are part of the modulefile hierarchy.
'compiler'is always present. - property missing โ Returns the list of tokens that are not available.
- property provides โ Returns a dictionary mapping all the services provided by this spec to the spec itself.
- property requires โ Returns a dictionary mapping all the requirements of this spec to the actual provider.
'compiler'is always present among the requirements.
class spack.modules.lmod.LmodContext(configuration)
Bases: BaseContext
Context class for lmod module files.
- property conditionally_unlocked_paths โ Returns the list of paths that are unlocked conditionally. Each item in the list is a tuple with the structure
(condition, path). context_properties= (verbatim, don't touch ๐ โโ๏ธ)
context_properties = ['has_modulepath_modifications',
'has_conditional_modifications', 'name_part', 'version_part',
'provides', 'missing', 'unlocked_paths',
'conditionally_unlocked_paths', 'spec', 'timestamp', 'category',
'short_description', 'long_description', 'configure_options',
'environment_modifications', 'has_manpath_modifications',
'conflicts', 'autoload', 'verbose']- property has_conditional_modifications โ
Trueif this module modifiesMODULEPATHconditionally to the presence of other services in the environment,Falseotherwise. - property has_modulepath_modifications โ
Trueif this module modifiesMODULEPATH,Falseotherwise. - property missing โ Returns a list of missing services.
- property name_part โ Name of this provider.
- property provides โ Returns the dictionary of provided services.
- property unlocked_paths โ Returns the list of paths that are unlocked unconditionally.
- property version_part โ Version of this provider.
class spack.modules.lmod.LmodFileLayout(configuration)
Bases: BaseFileLayout
File layout for lmod module files.
- property arch_dirname โ Returns the root folder for THIS architecture.
- property available_path_parts โ List of path parts that are currently available. Needed to construct the file name.
extension: str | None = 'lua'โ file extension ofluamodule files.- property filename โ Returns the filename for the current module file.
- property modulerc โ Returns the
modulercfile associated with current module file.
token_to_path(name, value)
Transforms a hierarchy token into the corresponding path part.
Parameters
- name (
str) โ name of the service in the hierarchy. - value โ actual provider of the service.
Returns part of the path associated with the service.
Return type str
- property unlocked_paths โ Returns a dictionary mapping conditions to a list of unlocked paths. The paths that are unconditionally unlocked are under the key
'None'. The other keys represent the list of services you need loaded to unlock the corresponding paths.
class spack.modules.lmod.LmodModulefileWriter(spec: Spec, module_set_name: str, explicit: bool | None = None)
Bases: BaseModuleFileWriter
Writer class for lmod module files.
default_template: str = 'modules/modulefile.lua'hide_cmd_format: str = 'hide_version("%s")'modulerc_header: List[str] = []
exception spack.modules.lmod.NonVirtualInHierarchyError(message, long_message=None)
Bases: SpackError, TypeError
Error raised if non-virtual specs are used as hierarchy tokens in the lmod section of 'modules.yaml'. Bestie, hierarchy tokens need to be virtual, no cap ๐ซ
spack.modules.lmod.configuration(module_set_name: str) -> dict
(returns the config dict, that's it, that's the tweet โจ)
spack.modules.lmod.guess_core_compilers(name, store=False) -> List[CompilerSpec]
Guesses the list of core compilers installed in the system. Detective mode ๐
Parameters
- store (
bool) โ ifTruewrites the core compilers to themodules.yamlconfiguration file.
Returns List of found core compilers.
spack.modules.lmod.make_configuration(spec: Spec, module_set_name: str, explicit: bool | None = None) -> BaseConfiguration
Returns the lmod configuration for spec.
spack.modules.lmod.make_context(spec: Spec, module_set_name: str, explicit: bool | None = None) -> BaseContext
Returns the context information for spec.
spack.modules.lmod.make_layout(spec: Spec, module_set_name: str, explicit: bool | None = None) -> BaseFileLayout
Returns the layout information for spec.
๐ spack.modules.tcl module ๐
This module implements the classes necessary to generate Tcl non-hierarchical modules. Yes, non-hierarchical โ she said what she said ๐
class spack.modules.tcl.TclConfiguration(spec: Spec, module_set_name: str, explicit: bool)
Bases: BaseConfiguration
Configuration class for tcl module files.
class spack.modules.tcl.TclContext(configuration)
Bases: BaseContext
Context class for tcl module files.
context_properties= (again, verbatim ๐ธ)
context_properties = ['prerequisites', 'spec', 'timestamp',
'category', 'short_description', 'long_description',
'configure_options', 'environment_modifications',
'has_manpath_modifications', 'conflicts', 'autoload', 'verbose']- property prerequisites โ List of modules that needs to be loaded automatically.
class spack.modules.tcl.TclFileLayout(configuration)
Bases: BaseFileLayout
File layout for tcl module files.
- property modulerc โ Returns the
modulercfile associated with current module file.
class spack.modules.tcl.TclModulefileWriter(spec: Spec, module_set_name: str, explicit: bool | None = None)
Bases: BaseModuleFileWriter
Writer class for tcl module files.
default_template: str = 'modules/modulefile.tcl'hide_cmd_format: str = 'module-hide --soft --hidden-loaded %s'modulerc_header: List[str] = ['#%Module4.7']
spack.modules.tcl.configuration(module_set_name: str) -> dict
(the config, returned, slay โจ)
spack.modules.tcl.make_configuration(spec: Spec, module_set_name: str, explicit: bool | None = None) -> BaseConfiguration
Returns the tcl configuration for spec.
And that's the modules machinery, bestie โ every class, every property, every default, all accounted for ๐ซถ we stay thorough ๐
spack.modules.tcl.make_context(spec: Spec, module_set_name: str, explicit: bool | None = None) -> BaseContextโ hands you back the context info for spec. bestie, it's giving "here's everything you need to know about this spec's vibes" โจspack.modules.tcl.make_layout(spec: Spec, module_set_name: str, explicit: bool | None = None) -> BaseFileLayoutโ same deal but for the layout info of spec. the file-placement of it all ๐
๐ณ spack.oci package
๐ธ Submodules
๐ฆ spack.oci.image module
class spack.oci.image.Digest(*, algorithm: str, digest: str)โ Bases:object
the way this one literally represents a digest in the format <algorithm>:<digest> ๐ซถ and ngl, currently it only supports sha256 digests. no cap.
algorithmdigestclassmethod from_sha256(digest: str) -> Digestclassmethod from_string(string: str) -> Digest
class spack.oci.image.ImageReference(*, domain: str, name: str, tag: str = 'latest', digest: Digest | None = None)โ Bases:object
a parsed image of the form domain/name:tag[@digest]. the digest is optional, and domain and tag get automatically filled out with defaults when parsed from string โ so convenient of it ๐
blob_url(digest: str | Digest) -> strdigestdomainendpoint(path: str = '') -> strclassmethod from_string(string) -> ImageReferencemanifest_url() -> strnametagtags_url() -> struploads_url(digest: Digest | None = None) -> strwith_digest(digest: str | Digest) -> ImageReferencewith_tag(tag: str) -> ImageReference
spack.oci.image.default_config(architecture: str, os: str)โ default config moment ๐spack.oci.image.default_index_tag = 'index.spack'โ the Default OCI index tag, iconic, don't touch it ๐spack.oci.image.default_manifest()โ default manifest, yes ma'amspack.oci.image.default_tag(spec: Spec) -> strโ returns a valid, default image tag for a spec. literally tailor-made for your spec ๐ฆspack.oci.image.ensure_sha256_checksum(oci_blob: str)โ validates that the reference is of the formatsha256:<checksum>. returns the checksum if valid, and raisesValueErrorotherwise โ so catch it or perish ๐ญspack.oci.image.tag_is_spec(tag: str) -> boolโ checks if a tag is likely aSpec. a little vibe check, if you will ๐
๐ spack.oci.oci module
class spack.oci.oci.Blob(compressed_digest, uncompressed_digest, size)โ Bases:NamedTuplecompressed_digest: Digestโ alias for field number 0size: intโ alias for field number 2uncompressed_digest: Digestโ alias for field number 1
spack.oci.oci.all_content_type = ['application/vnd.oci.image.manifest.v1+json', 'application/vnd.docker.distribution.manifest.v2+json', 'application/vnd.oci.image.index.v1+json', 'application/vnd.docker.distribution.manifest.list.v2+json']โ All OCI manifest / index content types ๐ the whole squad is herespack.oci.oci.blob_exists(ref: ImageReference, digest: Digest, _urlopen: Callable[[...], HTTPResponse] | None = None) -> boolโ checks if a blob exists in an OCI registry. is it there or is it not, the suspense fr fr ๐spack.oci.oci.copy_missing_layers(src: ImageReference, dst: ImageReference, architecture: str, _urlopen: Callable[[...], HTTPResponse] | None = None) -> Tuple[dict, dict]โ copy image layers from src to dst for given architecture.
Parameters
- src โ the source image reference.
- dst โ the destination image reference.
- architecture โ the architecture (when referencing an index).
Returns โ Tuple of manifest and config of the base image. ๐ซถ
spack.oci.oci.copy_missing_layers_with_retry(*args, **kwargs)โ same ascopy_missing_layers, but with retry wrapper. it's giving "try again, bestie" ๐spack.oci.oci.create_tarball(spec: Spec, tarfile_path)โ tarball time โจspack.oci.oci.get_manifest_and_config(ref: ImageReference, architecture='amd64', recurse=3, _urlopen: Callable[[...], HTTPResponse] | None = None) -> Tuple[dict, dict]โ recursively fetches manifest and config for a given image reference with a given architecture.
Parameters
- ref โ the image reference.
- architecture โ the architecture (when referencing an index).
- recurse โ how many levels of index to recurse into.
Returns โ a tuple of (manifest, config). nesting doll situation ๐ช
spack.oci.oci.get_manifest_and_config_with_retry(*args, **kwargs)โ same asget_manifest_and_config, but with retry wrapper ๐spack.oci.oci.image_from_mirror(mirror: Mirror) -> ImageReferenceโ given an OCI based mirror, extracts the URL and image name from it. detective work ๐spack.oci.oci.index_content_type = ['application/vnd.oci.image.index.v1+json', 'application/vnd.docker.distribution.manifest.list.v2+json']โ OCI index content types (including docker type) ๐ณspack.oci.oci.list_tags(ref: ImageReference, _urlopen: Callable[[...], HTTPResponse] | None = None) -> List[str]โ retrieves the list of tags associated with an image, handling pagination. so many tags, so little time ๐ญspack.oci.oci.make_stage(url: str, digest: Digest, keep: bool = False, _urlopen: Callable[[...], HTTPResponse] | None = None) -> Stageโ makes a stage, obviously ๐ธspack.oci.oci.manifest_content_type = ['application/vnd.oci.image.manifest.v1+json', 'application/vnd.docker.distribution.manifest.v2+json']โ OCI manifest content types (including docker type) ๐spack.oci.oci.upload_blob(ref: ImageReference, file: str, digest: Digest, force: bool = False, small_file_size: int = 0, _urlopen: Callable[[...], HTTPResponse] | None = None) -> boolโ uploads a blob to an OCI registry.
ok so important tea: we only do monolithic uploads, even though it's very simple to do chunked. observed problems with chunked uploads: (1) it's slow, many sequential requests, (2) some registries set an unknown max chunk size, and the spec doesn't say how to obtain it. like, rude.
Parameters
- ref โ the image reference.
- file โ the file to upload.
- digest โ the digest of the file.
- force โ whether to force upload the blob, even if it already exists.
- small_file_size โ for files at most this size, attempt to do a single POST request instead of POST + PUT. some registries do no support single requests, and others do not specify what size they support in single POST. for now this feature is disabled by default (0KB) ๐
Returns โ True if the blob was uploaded, False if it already existed. simple as ๐ซถ
spack.oci.oci.upload_blob_with_retry(*args, **kwargs)โ same asupload_blob, but with retry wrapper ๐spack.oci.oci.upload_manifest(ref: ImageReference, manifest: dict, tag: bool = True, _urlopen: Callable[[...], HTTPResponse] | None = None)โ uploads a manifest/index to a registry.
Parameters
- ref โ the image reference.
- manifest โ the manifest or index.
- tag โ when true, use the tag, otherwise use the digest. this is relevant for multi-arch images, where the tag is an index, referencing the manifests by digest. iykyk ๐ง
Returns โ the digest and size of the uploaded manifest. โจ
spack.oci.oci.upload_manifest_with_retry(*args, **kwargs)โ same asupload_manifest, but with retry wrapper ๐spack.oci.oci.with_query_param(url: str, param: str, value: str) -> strโ add a query parameter to a URL.
Parameters
- url โ the URL to add the parameter to.
- param โ the parameter name.
- value โ the parameter value.
Returns โ the URL with the parameter added. tiny little glow-up for your URL ๐
๐ spack.oci.opener module
all the logic for OCI fetching and authentication. the security girlie of the group chat ๐ก๏ธ
class spack.oci.opener.Challenge(scheme: str | None = None, params: List[Tuple[str, str]] | None = None)โ Bases:objectparamsscheme
class spack.oci.opener.OCIAuthHandler(credentials_provider: Callable[[str], UsernamePassword | None])โ Bases:BaseHandlerhttp_error_401(req: Request, fp, code, msg, headers)โ the "you shall not pass" handler ๐ซhttps_request(req: Request)obtain_bearer_token(registry: str, challenge: RealmServiceScope, timeout) -> strโ gets the bearer token, slay ๐ฅ
class spack.oci.opener.RealmServiceScope(realm, service, scope)โ Bases:NamedTuplerealm: strโ alias for field number 0scope: strโ alias for field number 2service: strโ alias for field number 1
class spack.oci.opener.State(*values)โ Bases:EnumAUTH_PARAM = 3AUTH_PARAM_LIST_START = 2AUTH_PARAM_OR_SCHEME = 5CHALLENGE = 1NEXT_IN_LIST = 4
class spack.oci.opener.TokenType(*values)โ Bases:TokenBaseANY = 7AUTH_PARAM = 1COMMA = 4EOF = 6EQUALS = 3SPACE = 5TOKEN = 2
class spack.oci.opener.UsernamePassword(username, password)โ Bases:NamedTuplepassword: strโ alias for field number 1username: strโ alias for field number 0
spack.oci.opener.create_opener()โ creates an opener that can handle OCI authentication. go little opener, go ๐ฃspack.oci.opener.credentials_from_mirrors(domain: str, *, mirrors: Iterable[Mirror] | None = None) -> UsernamePassword | Noneโ filters out OCI registry credentials from a list of mirrors. separating the useful from the noise ๐spack.oci.opener.default_retry(f, retries: int = 5, sleep=None)โ the default try-again-ery,5retries by default ๐spack.oci.opener.ensure_status(request: Request, response: HTTPResponse, status: int)โ raises an error if the response status is not the expected one. no quiet failures allowed bestie ๐คspack.oci.opener.get_bearer_challenge(challenges: List[Challenge]) -> RealmServiceScope | Noneโ finds the bearer challenge in the list ๐ฏspack.oci.opener.parse_www_authenticate(input: str)โ very basic parsing ofwww-authenticateparsing (RFC7235 section 4.1). Notice: this omitstoken68support. yes that's a real caveat, don't come for me ๐spack.oci.opener.tokenize(input: str)โ tokenizes the input, as one does โ๏ธspack.oci.opener.urlopen: Callable[[...], HTTPResponse] = <function _urlopen.<locals>.dispatch_open>โ the opener that automatically uses OCI authentication based on mirror config. it just knows ๐ง
๐ป spack.operating_systems package
class spack.operating_systems.CrayBackendโ Bases:LinuxDistro
Compute Node Linux (CNL) is the operating system used for the Cray XC series super computers. it's a very stripped down version of GNU/Linux, like the minimalist aesthetic of operating systems ๐ any compilers found through this operating system will be used with modules. if updated, user must make sure that version and name are updated to indicate that OS has been upgraded (or downgraded) โ so like, keep the receipts, bestie ๐งพ
Still inside the previous class, the remaining method signatures are:
arguments_to_detect_version_fn(paths)detect_version(detect_version_args)make_compilers(compiler_id, paths)
class spack.operating_systems.CrayFrontendโ Bases:LinuxDistro
Represents OS that runs on login and service nodes of the Cray platform. It acts like a regular Linux without Cray-specific modules and compiler wrappers. ๐ซถ
compiler_search_paths(property) โ Calls the default function but unloads Cray's programming environments first.
This prevents from detecting Cray compiler wrappers and avoids possible false detections. ๐
class spack.operating_systems.FreeBSDOsโ Bases:OperatingSystemclass spack.operating_systems.LinuxDistroโ Bases:OperatingSystem
This class will represent the autodetected operating system for a Linux System. Since there are many different flavors of Linux, this class will attempt to encompass them all through autodetection using the python module platform and the method platform.dist(). ๐งโจ
class spack.operating_systems.MacOsโ Bases:OperatingSystem
This class represents the macOS operating system. This will be auto detected using the python platform.mac_ver. The macOS platform will be represented using the major version operating system name, i.e el capitan, yosemite...etc. ๐๐ป
class spack.operating_systems.OperatingSystem(name, version)โ Bases:object
Base class for all the Operating Systems. ๐
On a multiple architecture machine, the architecture spec field can be set to build a package against any target and operating system that is present on the platform. On Cray platforms or any other architecture that has different front and back end environments, the operating system will determine the method of compiler detection.
There are two different types of compiler detection:
- Through the
$PATHenv variable (front-end detection) - Through the module system. (back-end detection)
Depending on which operating system is specified, the compiler will be detected using one of those methods. ๐
For platforms such as linux and darwin, the operating system is autodetected.
to_dict()
class spack.operating_systems.WindowsOsโ Bases:OperatingSystem
This class represents the Windows operating system. This will be auto detected using the python platform.win32_ver() once we have a python setup that runs natively. The Windows platform will be represented using the major version operating system number, e.g. 10. ๐ช
compiler_search_paths(property)msvc_paths(property)oneapi_root(property)vs_install_paths(property)
๐งฉ Submodules
spack.operating_systems.cray_backend module
class spack.operating_systems.cray_backend.CrayBackendโ Bases:LinuxDistro
Compute Node Linux (CNL) is the operating system used for the Cray XC series super computers. It is a very stripped down version of GNU/Linux. Any compilers found through this operating system will be used with modules. If updated, user must make sure that version and name are updated to indicate that OS has been upgraded (or downgraded). ๐ญ๐ฅ
arguments_to_detect_version_fn(paths)detect_version(detect_version_args)make_compilers(compiler_id, paths)
spack.operating_systems.cray_backend.read_cle_release_file()
Read the CLE release file and return a dict with its attributes. ๐
This file is present on newer versions of Cray.
The release file looks something like this:
RELEASE=6.0.UP07
BUILD=6.0.7424
...The dictionary we produce looks like this:
{
"RELEASE": "6.0.UP07",
"BUILD": "6.0.7424",
...
}Returns: dictionary of release attributes
Return type: dict
spack.operating_systems.cray_backend.read_clerelease_file()
Read the CLE release file and return the Cray OS version. ๐
This file is present on older versions of Cray.
The release file looks something like this:
5.2.UP04Returns: the Cray OS version
Return type: str
spack.operating_systems.cray_frontend module
class spack.operating_systems.cray_frontend.CrayFrontendโ Bases:LinuxDistro
Represents OS that runs on login and service nodes of the Cray platform. It acts as a regular Linux without Cray-specific modules and compiler wrappers. ๐
compiler_search_paths(property) โ Calls the default function but unloads Cray's programming environments first.
This prevents from detecting Cray compiler wrappers and avoids possible false detections. โจ
spack.operating_systems.cray_frontend.unload_programming_environment()
Context manager that unloads Cray Programming Environments. ๐ซถ
spack.operating_systems.freebsd module
class spack.operating_systems.freebsd.FreeBSDOsโ Bases:OperatingSystem
spack.operating_systems.linux_distro module
class spack.operating_systems.linux_distro.LinuxDistroโ Bases:OperatingSystem
This class will represent the autodetected operating system for a Linux System. Since there are many different flavors of Linux, this class will attempt to encompass them all through autodetection using the python module platform and the method platform.dist(). ๐ง
spack.operating_systems.linux_distro.kernel_version()
Return the kernel version as a Version object. Note that the kernel version is distinct from OS and/or distribution versions. For instance:
>>> distro.id()
'centos'
>>> distro.version()
'7'
>>> platform.release()
'5.10.84+'spack.operating_systems.mac_os module
class spack.operating_systems.mac_os.MacOsโ Bases:OperatingSystem
This class represents the macOS operating system. This will be auto detected using the python platform.mac_ver. The macOS platform will be represented using the major version operating system name, i.e el capitan, yosemite...etc. ๐ธ
spack.operating_systems.mac_os.macos_cltools_version()
Find the last installed version of the CommandLineTools. ๐ ๏ธ
The CLT version might only affect the build if it's selected as the macOS SDK path.
spack.operating_systems.mac_os.macos_sdk_path()
Return path to the active macOS SDK. ๐
spack.operating_systems.mac_os.macos_sdk_version()
Return the version of the active macOS SDK.
The SDK version usually corresponds to the installed Xcode version and can affect how some packages (especially those that use the GUI) can fail. This information should somehow be embedded into the future "compilers are dependencies" feature. ๐
The macOS deployment target cannot be greater than the SDK version, but usually it can be at least a few versions less.
spack.operating_systems.mac_os.macos_version()
Get the current macOS version as a version object.
This has three mechanisms for determining the macOS version, which is used for spack identification (the os in the spec's arch) and indirectly for setting the value of MACOSX_DEPLOYMENT_TARGET, which affects the minos value of the LC_BUILD_VERSION macho header. Mixing minos values can lead to lots of linker warnings, and using a consistent version (pinned to the major OS version) allows distribution across clients that might be slightly behind. ๐ญ
The version determination is made with three mechanisms in decreasing priority:
- The
MACOSX_DEPLOYMENT_TARGETvariable overrides the actual operating system version, just like the value can be used to build for older macOS targets on newer systems. Spack currently will truncate this value when building packages, but at least the major version will be the same. - The system
sw_verscommand reports the actual operating system version. - The Python
platform.mac_verfunction is a fallback if the operating system identification fails, because some Python versions and/or installations report the OS on which Python was built rather than the one on which it is running.
spack.operating_systems.windows_os module
class spack.operating_systems.windows_os.WindowsOsโ Bases:OperatingSystem
This class represents the Windows operating system. This will be auto detected using the python platform.win32_ver() once we have a python setup that runs natively. The Windows platform will be represented using the major version operating system number, e.g. 10. ๐ช
compiler_search_paths(property)msvc_paths(property)oneapi_root(property)vs_install_paths(property)
spack.operating_systems.windows_os.windows_version()
Windows version as a Version object. ๐ฅ
spack.platforms package
class spack.platforms.Crayโ Bases:Platformclassmethod craype_type_and_version()classmethod detect()
Detect whether this system requires CrayPE module support. ๐ง
Systems with newer CrayPE (21.10 for EX systems, future work for CS and XC systems) have compilers and MPI wrappers that can be used directly by path. These systems are considered linux platforms.
For systems running an older CrayPE, we detect the Cray platform based on the availability through module of the Cray programming environment. If this environment is available, we can use it to find compilers, target modules, etc. If the Cray programming environment is not available via modules, then we will treat it as a standard linux system, as the Cray compiler wrappers and other components of the Cray programming environment are irrelevant without module support. ๐
priority: int | None = 10setup_platform_environment(pkg, env)โ Change the linker to default dynamic to be more similar to linux/standard linker behavior. โญ
class spack.platforms.Darwinโ Bases:Platformbinary_formats = ['macho']โ the binary formats this platform speaks; the relocation logic uses this ๐classmethod detect()โ returnsTrueif the host platform is detected to be this exactPlatformclass, andFalseotherwise. Derived classes are the ones responsible for implementing it โ so yeah, base class says "not my job, bestie" ๐ซถpriority: int | None = 89โ the priority number, lowkey important for detection ordering ๐setup_platform_environment(pkg, env)โ pins down the deployment target based on the target OS version.
The MACOSX_DEPLOYMENT_TARGET environment variable gives you a default -mmacosx-version-min argument for GCC and Clang compilers, plus it sets the default value of CMAKE_OSX_DEPLOYMENT_TARGET for CMake-based build systems. Normally the deployment target default is just the major version (11, 10.16, ...) for CMake and Clang โ but some GCC versions go and specify a minor component too (11.3), which leads to numerous link warnings about inconsistent or incompatible target versions ๐ญ. Setting the env var keeps versions consistent for an install toolchain target, even when the host macOS version changes. Slay.
TODO: it may be necessary to add SYSTEM_VERSION_COMPAT for older versions of the macosx developer tools; see โ https://github.com/spack/spack/pull/26290 for discussion. ๐ง
๐ง class spack.platforms.FreeBSD
Bases: Platform
classmethod detect()โ returnsTrueif the host platform is detected to be thisPlatformclass,Falseotherwise. Once again, derived classes have to actually implement this โจpriority: int | None = 102
๐ง class spack.platforms.Linux
Bases: Platform
classmethod detect()โTrueif the host platform matches thisPlatformclass,Falseotherwise. Derived classes do the implementing, bestie ๐ซถpriority: int | None = 90
๐ class spack.platforms.Platform(name)
Bases: object
Platform is an abstract class that gets extended by subclasses. ๐
So if you wanna add a new type of platform (like cray_xe, for example), you create a subclass and set all the class attributes like priority, front_target, back_target, front_os, back_os. That's the whole vibe.
Platform also has a priority class attribute, and here's the tea: a lower number means higher priority. These numbers are arbitrarily set and can be changed, though usually there's not much need unless a new platform gets added and the user wants that one detected first ๐
Targets get created inside the platform subclasses. Most architectures (like linux and darwin) only have one target family (x86_64), but in the case of Cray machines there's both a frontend and a backend processor. The user can specify which targets are present on the front-end and back-end architecture. ๐
Depending on the platform, operating systems are either autodetected or set. The user can set the frontend and backend operating setting via the class attributes front_os and back_os. The operating system is the one responsible for compiler detection. Periodt.
add_operating_system(name, os_class)โ drops theoperating_systemclass object into theplatform.operating_sysdictionary ๐add_target(name, target)โ used by the platform-specific subclass to list available targets. Raises an error if the platform specifies a name that's reserved by spack as an alias ๐จback_end: str | None = Noneback_os: str | None = Nonebinary_formats = ['elf']โ binary formats used on this platform; used by the relocation logicdefault: str | None = Nonedefault_os: str | None = Noneclassmethod detect()โTrueif the host platform is detected to be thisPlatformclass,Falseotherwise. Derived classes are responsible for implementing it, no cap ๐ซถfront_end: str | None = Nonefront_os: str | None = Noneoperating_system(name)priority: int | None = Nonereserved_oss = ['default_os', 'frontend', 'fe', 'backend', 'be']reserved_targets = ['default_target', 'frontend', 'fe', 'backend', 'be']setup_platform_environment(pkg, env)โ a subclass can override this method if it needs any platform-specific build environment modifications ๐ ๏ธtarget(name)โ this is a getter method for the target dictionary that handles defaulting based on the values provided bydefault, front-end, and back-end. A subclass can overwrite it if it wants to offer further aliasing options โจ
๐งช class spack.platforms.Test(name=None)
Bases: Platform
back_end: str | None = 'core2'back_os: str | None = 'debian6'default: str | None = 'core2'default_os: str | None = 'debian6'classmethod detect()โTrueif the host platform is detected to be thisPlatformclass,Falseotherwise. Derived classes implement it, as always ๐front_end: str | None = 'x86_64'front_os: str | None = 'redhat6'priority: int | None = 1000000โ yes, one million. it's a test platform, she's not trying to be detected ๐
๐ช class spack.platforms.Windows
Bases: Platform
classmethod detect()โTrueif the host platform is detected to be thisPlatformclass,Falseotherwise. Derived classes are responsible for implementing it ๐ซถpriority: int | None = 101
spack.platforms.by_name(name)โ returns a platform object that corresponds to the given name, orNoneif there's no match.- Parameters:
name(str) -- name of the platform ๐ท๏ธ
- Parameters:
spack.platforms.host()โ the current platform used by Spack. It may be swapped by theuse_platformcontext manager ๐spack.platforms.prevent_cray_detection()โ a context manager that prevents detection of the Cray platform ๐ฆspack.platforms.reset()โ the result of the host search is memoized, so if it needs to be recomputed we have to clear the cache โ which is literally what this function does ๐งน
๐ธ Submodules
๐ฆ spack.platforms.cray module
class spack.platforms.cray.Cray
Bases: Platform
classmethod craype_type_and_version()classmethod detect()โ detects whether this system requires CrayPE module support.
Systems with newer CrayPE (21.10 for EX systems, future work for CS and XC systems) have compilers and MPI wrappers that can be used directly by path. These systems are considered linux platforms. ๐ง
For systems running an older CrayPE, we detect the Cray platform based on the availability, through modules, of the Cray programming environment. If this environment is available, we can use it to find compilers, target modules, etc. If the Cray programming environment isn't available via modules, then we treat it as a standard linux system โ because the Cray compiler wrappers and other components of the Cray programming environment are irrelevant without module support. Simple as that ๐
priority: int | None = 10setup_platform_environment(pkg, env)โ switches the linker to default dynamic so it behaves more like a linux/standard linker ๐ ๏ธ
spack.platforms.cray.slingshot_network()
๐ spack.platforms.darwin module
class spack.platforms.darwin.Darwin
Bases: Platform
binary_formats = ['macho']โ binary formats used on this platform; used by the relocation logicclassmethod detect()โTrueif the host platform is detected to be thisPlatformclass,Falseotherwise. Derived classes are responsible for implementing it ๐ซถpriority: int | None = 89setup_platform_environment(pkg, env)โ specifies the deployment target based on the target OS version.
The MACOSX_DEPLOYMENT_TARGET environment variable provides a default -mmacosx-version-min argument for GCC and Clang compilers, and also sets the default value of CMAKE_OSX_DEPLOYMENT_TARGET for CMake-based build systems. The default value for the deployment target is usually the major version (11, 10.16, ...) for CMake and Clang, but some versions of GCC specify a minor component too (11.3), leading to numerous link warnings about inconsistent or incompatible target versions. Setting the environment variable ensures consistent versions for an install toolchain target, even when the host macOS version changes. Obsessed with consistency tbh โจ
TODO: it may be necessary to add SYSTEM_VERSION_COMPAT for older versions of the macosx developer tools; see โ https://github.com/spack/spack/pull/26290 for discussion. ๐
๐ง spack.platforms.freebsd module
class spack.platforms.freebsd.FreeBSD
Bases: Platform
classmethod detect()โTrueif the host platform is detected to be thisPlatformclass,Falseotherwise. Derived classes do the implementing ๐priority: int | None = 102
๐ง spack.platforms.linux module
class spack.platforms.linux.Linux
Bases: Platform
classmethod detect()โTrueif the host platform is detected to be thisPlatformclass,Falseotherwise. Derived classes are responsible for implementing it ๐ซถpriority: int | None = 90
๐งช spack.platforms.test module
class spack.platforms.test.Test(name=None)
Bases: Platform
back_end: str | None = 'core2'back_os: str | None = 'debian6'default: str | None = 'core2'default_os: str | None = 'debian6'classmethod detect()โ returnsTrueif the host platform is detected to be the currentPlatformclass, andFalseotherwise. Derived classes are the ones responsible for implementing this method, bestie, so don't come for the base class ๐front_end: str | None = 'x86_64'front_os: str | None = 'redhat6'priority: int | None = 1000000
๐ช spack.platforms.windows module
class spack.platforms.windows.Windows
Bases: Platform
classmethod detect()โ returnsTrueif the host platform is detected to be the currentPlatformclass,Falseotherwise. Same deal as always: derived classes gotta implement this one themselves, no cap ๐ซถpriority: int | None = 101
๐ spack.reporters package
class spack.reporters.CDash(configuration: CDashConfiguration)
Bases: Reporter
Generates reports of spec installations for CDash. It's giving observability era โจ
To use this reporter, you pass the --cdash-upload-url argument to spack install:
spack install --cdash-upload-url=\
https://mydomain.com/cdash/submit.php?project=Spack <spec>In this example, results get uploaded to the Spack project on the CDash instance hosted at https://mydomain.com/cdash. Iconic.
build_report(report_dir, specs)build_report_for_package(report_dir, package, duration)concretization_report(report_dir, msg)extract_standalone_test_data(package, phases, report_data)โ extracts the stand-alone test outputs for the package ๐finalize_report()initialize_report(report_dir)report_build_name(pkg_name)report_test_data(report_dir, package, phases, report_data)โ generates and uploads the test report(s) for the package ๐คsuccessโ set toFalseif any error occurs while building the CDash report. So like, if it breaks, this flips toFalse, bestie ๐test_report(report_dir, specs)โ generates reports for each package in each spectest_report_for_package(report_dir, package, duration)test_skipped_report(report_dir: str, spec: Spec, reason: str | None = None)โ explicitly reports a spec as being skipped (e.g. in CI).
Examples of when this happens: the installation failed, or the package is known to have broken tests.
Parameters:
report_dirโ directory where the report is to be writtenspecโ the spec being testedreasonโ optional reason the test is being skipped
upload(filename)
class spack.reporters.CDashConfiguration(upload_url, packages, build, site, buildstamp, track)
Bases: tuple
buildโ alias for field number 2buildstampโ alias for field number 4packagesโ alias for field number 1siteโ alias for field number 3trackโ alias for field number 5upload_urlโ alias for field number 0
class spack.reporters.JUnit
Bases: Reporter
Generates reports of spec installations for JUnit ๐งช
build_report(filename, specs)concretization_report(filename, msg)test_report(filename, specs)
class spack.reporters.Reporter
Bases: object
The base class for report writers โ the blueprint all the others are built from, periodt ๐
build_report(filename: str, specs: List[Dict[str, Any]])concretization_report(filename: str, msg: str)test_report(filename: str, specs: List[Dict[str, Any]])
๐งฉ Submodules
spack.reporters.base module
class spack.reporters.base.Reporter โ bases: object. The base class for report writers.
build_report(filename: str, specs: List[Dict[str, Any]])concretization_report(filename: str, msg: str)test_report(filename: str, specs: List[Dict[str, Any]])
spack.reporters.cdash module
class spack.reporters.cdash.CDash(configuration: CDashConfiguration) โ bases: Reporter. Generates reports of spec installations for CDash.
To use this reporter, pass the --cdash-upload-url argument to spack install:
spack install --cdash-upload-url=\
https://mydomain.com/cdash/submit.php?project=Spack <spec>In this example, results get uploaded to the Spack project on the CDash instance hosted at https://mydomain.com/cdash.
buildIds: Dict[str, str]build_report(report_dir, specs)build_report_for_package(report_dir, package, duration)concretization_report(report_dir, msg)extract_standalone_test_data(package, phases, report_data)โ extracts the stand-alone test outputs for the packagefinalize_report()initialize_report(report_dir)report_build_name(pkg_name)report_test_data(report_dir, package, phases, report_data)โ generates and uploads the test report(s) for the packagesuccessโ set toFalseif any error occurs when building the CDash reporttest_report(report_dir, specs)โ generates reports for each package in each spectest_report_for_package(report_dir, package, duration)test_skipped_report(report_dir: str, spec: Spec, reason: str | None = None)โ explicitly reports a spec as being skipped (e.g., CI).
Examples are the installation failed, or the package is known to have broken tests.
Parameters:
report_dirโ directory where the report is to be writtenspecโ spec being testedreasonโ optional reason the test is being skipped
upload(filename)
class spack.reporters.cdash.CDashConfiguration(upload_url, packages, build, site, buildstamp, track) โ bases: tuple
buildโ alias for field number 2buildstampโ alias for field number 4packagesโ alias for field number 1siteโ alias for field number 3trackโ alias for field number 5upload_urlโ alias for field number 0
spack.reporters.cdash.build_stamp(track, timestamp) โ that's the whole signature, slay ๐ฆ
spack.reporters.extract module
spack.reporters.extract.add_part_output(part, line)spack.reporters.extract.elapsed(current, previous)spack.reporters.extract.expected_failure(line)spack.reporters.extract.extract_test_parts(default_name, outputs)spack.reporters.extract.new_part()spack.reporters.extract.part_name(source)spack.reporters.extract.process_part_end(part, curr_time, last_time)spack.reporters.extract.skip(line)spack.reporters.extract.status(line)spack.reporters.extract.timestamp(time_string)
spack.reporters.junit module
class spack.reporters.junit.JUnit โ bases: Reporter. Generates reports of spec installations for JUnit.
build_report(filename, specs)concretization_report(filename, msg)test_report(filename, specs)
๐ spack.schema package
This module contains jsonschema files for all of Spack's YAML formats. Every single one of them, no skips ๐
๐งฉ Submodules
spack.schema.bootstrap module
Schema for the bootstrap.yaml configuration file.
spack.schema.bootstrap.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'bootstrap': {'properties': {'enable': {'type':
'boolean'}, 'root': {'type': 'string'}, 'sources': {'items':
{'additionalProperties': False, 'properties': {'metadata': {'type':
'string'}, 'name': {'type': 'string'}}, 'required': ['name',
'metadata'], 'type': 'object'}, 'type': 'array'}, 'trusted':
{'patternProperties': {'\\w[\\w-]*': {'type': 'boolean'}}, 'type':
'object'}}, 'type': 'object'}}, 'title': 'Spack bootstrap configuration
file schema', 'type': 'object'}That's the full schema with metadata, bestie ๐ธ
spack.schema.buildcache_spec module
Schema for a buildcache spec.yaml file.
properties: Dict[str, Any] = {
# `buildinfo` is no longer needed as of Spack 0.21
"buildinfo": {"type": "object"},
"spec": {
"type": "object",
"additionalProperties": True,
"items": spack.schema.spec.properties,
},
"binary_cache_checksum": {
"type": "object",
"properties": {"hash_algorithm": {"type": "string"}, "hash": {"type": "string"}},
},
"buildcache_layout_version": {"type": "number"},
}
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack buildcache specfile schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}spack.schema.cdash module
Schema for the cdash.yaml configuration file.
properties: Dict[str, Any] = {
"cdash": {
"type": "object",
"additionalProperties": False,
# "required": ["build-group", "url", "project", "site"],
"required": ["build-group"],
"patternProperties": {
r"build-group": {"type": "string"},
r"url": {"type": "string"},
r"project": {"type": "string"},
r"site": {"type": "string"},
},
}
}
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack cdash configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}spack.schema.cdash.properties: Dict[str, Any] = {...} โ properties for inclusion in other schemas:
spack.schema.cdash.properties: Dict[str, Any] = {'cdash':
{'additionalProperties': False, 'patternProperties': {'build-group':
{'type': 'string'}, 'project': {'type': 'string'}, 'site': {'type':
'string'}, 'url': {'type': 'string'}}, 'required': ['build-group'],
'type': 'object'}}spack.schema.cdash.schema = {...} โ the full schema with metadata:
spack.schema.cdash.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'cdash': {'additionalProperties': False,
'patternProperties': {'build-group': {'type': 'string'}, 'project':
{'type': 'string'}, 'site': {'type': 'string'}, 'url': {'type':
'string'}}, 'required': ['build-group'], 'type': 'object'}}, 'title':
'Spack cdash configuration file schema', 'type': 'object'}๐ฆ spack.schema.ci module
okay so this module is literally the schema for the gitlab-ci.yaml configuration file โจ like it's the vibe-check rulebook that says what you're allowed to put in your CI config and what is so not the move. let's go through it piece by piece, no cap ๐
๐ the script schema
first up, the rules for script fields โ and bestie, it's a list of lists and/or strings, which is similar to what is allowed in the gitlab schema:
# Schema for script fields
# List of lists and/or strings
# This is similar to what is allowed in
# the gitlab schema
script_schema = {
"type": "array",
"items": {"anyOf": [{"type": "string"}, {"type": "array", "items": {"type": "string"}}]},
}so an array, and each item is either a plain string or an array of strings. nesting rights, we love to see it ๐ซถ
๐ผ๏ธ the image schema
then the CI image โ and it's a oneOf situation, meaning you can hand it a string or an object:
# Schema for CI image
image_schema = {
"oneOf": [
{"type": "string"},
{
"type": "object",
"properties": {
"name": {"type": "string"},
"entrypoint": {"type": "array", "items": {"type": "string"}},
},
},
]
}inside the object version you get name (a string) and entrypoint (an array of strings). pick your fighter ๐
โจ attributes, aka the extra stuff
now the attributes schema. note the comment: additional attributes are allow โ yes, that's the original wording, we're keeping it exactly, don't come for me ๐ญ โ and will be forwarded directly to the CI target YAML for each job:
# Additional attributes are allow
# and will be forwarded directly to the
# CI target YAML for each job.
attributes_schema = {
"type": "object",
"additionalProperties": True,
"properties": {
"image": image_schema,
"tags": {"type": "array", "items": {"type": "string"}},
"variables": {
"type": "object",
"patternProperties": {r"[\w\d\-_\.]+": {"type": "string"}},
},
"before_script": script_schema,
"script": script_schema,
"after_script": script_schema,
},
}so: image (the schema from above), tags (array of strings), variables (an object where the keys match r"[\w\d\-_\.]+" and the values are strings โ regex key check, she's thorough ๐
), plus before_script, script and after_script, all using script_schema. and additionalProperties: True means extras just sail on through ๐ฅ
๐ the submapping schema
then we get the submapping situation:
submapping_schema = {
"type": "object",
"additionalProperties": False,
"required": ["submapping"],
"properties": {
"match_behavior": {"type": "string", "enum": ["first", "merge"], "default": "first"},
"submapping": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": False,
"required": ["match"],
"properties": {
"match": {"type": "array", "items": {"type": "string"}},
"build-job": attributes_schema,
"build-job-remove": attributes_schema,
},
},
},
},
}additionalProperties is False here, so no sneaky extras โ and submapping is literally required, not optional, periodt. match_behavior is a string that's either "first" or "merge", defaulting to "first". each submapping entry also requires match (array of strings) and can carry build-job / build-job-remove using the attributes schema above ๐
๐ท๏ธ named attributes schema
now the big oneOf โ the named attributes. every branch has additionalProperties: False and pairs each job with its -remove twin:
named_attributes_schema = {
"oneOf": [
{
"type": "object",
"additionalProperties": False,
"properties": {"noop-job": attributes_schema, "noop-job-remove": attributes_schema},
},
{
"type": "object",
"additionalProperties": False,
"properties": {"build-job": attributes_schema, "build-job-remove": attributes_schema},
},
{
"type": "object",
"additionalProperties": False,
"properties": {"copy-job": attributes_schema, "copy-job-remove": attributes_schema},
},
{
"type": "object",
"additionalProperties": False,
"properties": {
"reindex-job": attributes_schema,
"reindex-job-remove": attributes_schema,
},
},
{
"type": "object",
"additionalProperties": False,
"properties": {
"signing-job": attributes_schema,
"signing-job-remove": attributes_schema,
},
},
{
"type": "object",
"additionalProperties": False,
"properties": {
"cleanup-job": attributes_schema,
"cleanup-job-remove": attributes_schema,
},
},
{
"type": "object",
"additionalProperties": False,
"properties": {"any-job": attributes_schema, "any-job-remove": attributes_schema},
},
]
}so the job families are noop-job, build-job, copy-job, reindex-job, signing-job, cleanup-job and any-job โ each one with a -remove counterpart. seven branches, all iconic, all doing their own thing ๐ง
๐งต pipeline-gen schema
then pipeline-gen is just an array, and each item is either a submapping schema or a named attributes schema:
pipeline_gen_schema = {
"type": "array",
"items": {"oneOf": [submapping_schema, named_attributes_schema]},
}mix and match, bestie, we contain multitudes ๐
๐งฉ core shared properties
here's where the shared bits get merged together with union_dicts:
core_shared_properties = union_dicts(
{
"pipeline-gen": pipeline_gen_schema,
"rebuild-index": {"type": "boolean"},
"broken-specs-url": {"type": "string"},
"broken-tests-packages": {"type": "array", "items": {"type": "string"}},
"target": {"type": "string", "enum": ["gitlab"], "default": "gitlab"},
}
)so you've got pipeline-gen, rebuild-index (boolean), broken-specs-url (string), broken-tests-packages (array of strings), and target โ which is a string, only allowed to be "gitlab", and defaults to "gitlab". one option, but she's confident about it ๐
๐ง ci properties (the deprecated era)
โ ๏ธ okay this next one has a # TODO: Remove in Spack 0.23 on it, so this is very much on its way out โ clock is ticking, don't build your whole personality on it ๐
# TODO: Remove in Spack 0.23
ci_properties = {
"anyOf": [
{
"type": "object",
"additionalProperties": False,
# "required": ["mappings"],
"properties": union_dicts(
core_shared_properties, {"enable-artifacts-buildcache": {"type": "boolean"}}
),
},
{
"type": "object",
"additionalProperties": False,
# "required": ["mappings"],
"properties": union_dicts(
core_shared_properties, {"temporary-storage-url-prefix": {"type": "string"}}
),
},
]
}two anyOf branches, both additionalProperties: False, both with that # "required": ["mappings"], line commented out (we see you, commented-out requirement ๐). branch one adds enable-artifacts-buildcache (boolean) on top of the core shared properties; branch two adds temporary-storage-url-prefix (string). either or, bestie โจ
๐ฆ the properties dict
then the properties that get included in other schemas โ note the #: docstring-style comment, very demure:
#: Properties for inclusion in other schemas
properties: Dict[str, Any] = {
"ci": {
"oneOf": [
# TODO: Replace with core-shared-properties in Spack 0.23
ci_properties,
# Allow legacy format under `ci` for `config update ci`
spack.schema.gitlab_ci.gitlab_ci_properties,
]
}
}so ci is a oneOf between the ci_properties above and spack.schema.gitlab_ci.gitlab_ci_properties. and yeah, another # TODO: Replace with core-shared-properties in Spack 0.23 โ the legacy format is still hanging around so that config update ci keeps working ๐ซถ
๐๏ธ the full schema
and here's the whole thing with its metadata:
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack CI configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}draft-07, titled Spack CI configuration file schema, type object, additionalProperties: False (no chaos allowed, we're structured here), and properties from above. immaculate ๐
๐ง the update function
last but absolutely not least, the update function that migrates things:
def update(data):
import llnl.util.tty as tty
import spack.ci
import spack.environment as ev
# Warn if deprecated section is still in the environment
ci_env = ev.active_environment()
if ci_env:
env_config = ci_env.manifest[ev.TOP_LEVEL_KEY]
if "gitlab-ci" in env_config:
tty.die("Error: `gitlab-ci` section detected with `ci`, these are not compatible")
# Detect if the ci section is using the new pipeline-gen
# If it is, assume it has already been converted
return spack.ci.translate_deprecated_config(data)walking through it: imports llnl.util.tty as tty, spack.ci, and spack.environment as ev. then grabs the active environment with ev.active_environment(), and if there is one, pulls ci_env.manifest[ev.TOP_LEVEL_KEY] and checks whether "gitlab-ci" is sitting in there โ if it is, it calls tty.die("Error: gitlab-cisection detected withci, these are not compatible"). so if you've got both gitlab-ci and ci in your environment, it's not compatible and you're getting shut down ๐ (the comment says warn but the code says die, and honestly? same energy ๐ญ). then it hands everything to spack.ci.translate_deprecated_config(data) and returns whatever comes back.
and that's the whole module โ schema rules, job families, the deprecated stuff on its way out in 0.23, all of it. next module's right around the corner and we're literally just getting started ๐
...and listen, we're about to hit a chunk of the manpage that is literally just the raw schema definition โ like, spack.schema.ci.properties is being spelled out in full dict form, and it is so long the manpage itself had to break it into 259 parts. Part 153 is basically mid-dict. So we're gonna dump it verbatim (because facts are sacred ๐
) and then I'll walk you through what you're actually looking at, because the nesting is giving โจ ancient cursed JSON artifact โจ.
spack.schema.ci.properties: Dict[str, Any] = {'ci': {'oneOf': [{'anyOf':
[{'additionalProperties': False, 'properties': {'broken-specs-url':
{'type': 'string'}, 'broken-tests-packages': {'items': {'type':
'string'}, 'type': 'array'}, 'enable-artifacts-buildcache': {'type':
'boolean'}, 'pipeline-gen': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'submapping': {'items': {'additionalProperties': False, 'properties':
{'build-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'build-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'match':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['match'],
'type': 'object'}, 'type': 'array'}}, 'required': ['submapping'],
'type': 'object'}, {'oneOf': [{'additionalProperties': False,
'properties': {'noop-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'noop-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type':
'object'}, {'additionalProperties': False, 'properties': {'build-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'build-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'copy-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'copy-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'reindex-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'reindex-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'signing-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'signing-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'cleanup-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':๐ฆ okay so what is ANY of this
Bestie, this is the spack.schema.ci.properties value, like, fully spelled out. It's a Dict[str, Any] and the manpage is showing you every last nested brace, no cap. Here's the vibe check on the structure, top to bottom:
'ci'is the top-level key, and its value is a'oneOf'list ๐ธ- The first thing in that
'oneOf'is an'anyOf'list, and the first entry there is a dict with'additionalProperties': Falseand a'properties'block containing:'broken-specs-url'โ'type': 'string''broken-tests-packages'โ'type': 'array','items': {'type': 'string'}'enable-artifacts-buildcache'โ'type': 'boolean''pipeline-gen'โ'type': 'array', whose'items'are a'oneOf'๐ฅ
๐ the 'pipeline-gen' branches
First branch of that 'oneOf': 'additionalProperties': False, and a 'properties' block with exactly two keys:
'match_behavior'โ'type': 'string','default': 'first','enum': ['first', 'merge']'submapping'โ'type': 'array', and its'items'are objects with'additionalProperties': Falseand a'properties'block holding:'build-job''build-job-remove''match'โ'type': 'array','items': {'type': 'string'}
and that submapping object carries 'required': ['match'] โจ โ and the 'submapping' key itself carries 'required': ['submapping'] up a level. Both 'type': 'object'. Periodt.
๐ the job template (it is literally copy-pasted, and honestly? iconic)
Every *-job and *-job-remove entry has 'additionalProperties': True and the exact same 'properties' shape:
'after_script'โ'type': 'array','items'are'anyOf'a'string'or an array of strings'before_script'โ same deal: array, items'anyOf'string or array of strings'image'โ'oneOf'either a'string', or an object with'entrypoint'('type': 'array','items': {'type': 'string'}) and'name'('type': 'string')'script'โ'type': 'array', items'anyOf'string or array of strings'tags'โ'type': 'array','items': {'type': 'string'}'variables'โ'type': 'object'with'patternProperties'matching[\w\d\-_\.]+, each mapping to'type': 'string'
And yes โ the -remove twins are byte-for-byte the same schema. Not me noticing that they duplicated the whole thing instead of using a $ref ๐ญ slay for explicitness tho.
๐ and then it just... keeps going
After the noop-job / noop-job-remove pair (which lives inside a 'oneOf' under 'additionalProperties': False), you get more sibling objects, each 'additionalProperties': False, each wrapping one job pair and its 'type': 'object':
build-job/build-job-removecopy-job/copy-job-removereindex-job/reindex-job-removesigning-job/signing-job-removecleanup-job/ ...
...and the expression literally cuts off mid-'script': {'items': {'anyOf': in this part. Like, we are genuinely inside an unclosed anyOf array right now ๐ซถ the manpage is holding the brace open and everything. Body is a 'oneOf', we're all just living in it.
fr fr: I didn't change a single character of the schema above โ every'type', every'required', every'patternProperties', every[\w\d\-_\.]+is exactly as the page ships it. The drama is free; the data is not. ๐
Whole 'ci' key is 'oneOf' โ 'anyOf' โ 'additionalProperties': False objects, and we're still deep inside the 'pipeline-gen' 'oneOf' when this part runs out of page. Hold that thought, bestie ๐
okay so real talk โ this next chunk is literally the raw schema dump, straight up JSON-schema-looking dictionaries, and I am not about to paraphrase one byte of it because every single key in here is load-bearing. So what I'm gonna do is give you the whole thing verbatim in one big block ๐ซถ and then tell you the little bits to actually pay attention to, because there's lore in here.
Here's the exact text, no edits, no vibes-based paraphrasing:
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'cleanup-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'any-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'any-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'}]}]}, 'type': 'array'},
'rebuild-index': {'type': 'boolean'}, 'target': {'default': 'gitlab',
'enum': ['gitlab'], 'type': 'string'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'broken-specs-url':
{'type': 'string'}, 'broken-tests-packages': {'items': {'type':
'string'}, 'type': 'array'}, 'pipeline-gen': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'submapping': {'items': {'additionalProperties': False, 'properties':
{'build-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'build-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'match':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['match'],
'type': 'object'}, 'type': 'array'}}, 'required': ['submapping'],
'type': 'object'}, {'oneOf': [{'additionalProperties': False,
'properties': {'noop-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'noop-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type':
'object'}, {'additionalProperties': False, 'properties': {'build-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'build-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'copy-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'copy-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'reindex-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'reindex-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':okay so what is this even ๐ญ bestie I hear you. This is schema โ the shape-of-legal-config data structure, spelled out in plain dicts. Every 'type' says what kind of value is allowed, 'items' says what goes inside an array, 'properties' lists the allowed keys, 'required' lists the keys that literally must be there, and 'additionalProperties': False means no extra keys, periodt. When you see 'additionalProperties': True, that's the schema saying "yeah whatever, anything goes in this bit" ๐
The little vocabulary you'll keep seeing over and over in that block, so your eyes stop glazing over:
'anyOf'โ the value can match any one of the listed schemas, like a lil choose-your-own-adventure ๐'oneOf'โ one of the listed schemas, e.g. it's either a plain string or an object withentrypointandname'patternProperties'with'[\\w\\d\\-_\\.]+'โ the key names are pattern-matched against that regex, and the value is a string'enum'โ the only allowed values, no improvising'default'โ what you get if you don't say anything
And the actual named things floating in this chunk, because they're the whole point ๐:
after_script,before_script,scriptโ the items areanyOfa'string'or an'array'of strings, and the whole thing is an'array'imageโ'oneOf'a'string'or an object withentrypoint('items':'string','type':'array') andname('type':'string')tagsโ array of stringsvariablesโ object whose keys match'[\\w\\d\\-_\\.]+'and whose values are strings- the job names themselves:
cleanup-job,cleanup-job-remove,any-job,any-job-remove,build-job,build-job-remove,noop-job,noop-job-remove,copy-job,copy-job-remove,reindex-job,reindex-job-remove๐ match_behaviorโ'default': 'first','enum': ['first', 'merge']submappingโ array of objects, and'required': ['submapping']- inner entries need
'required': ['match']withmatchbeing an array of strings broken-specs-url('type': 'string'),broken-tests-packages(array of strings)rebuild-indexโ a'boolean'๐ฆtargetโ'default': 'gitlab','enum': ['gitlab'],'type': 'string'
heads up bestie:'additionalProperties': Falseis a strict gate โ if you sneak a key in that isn't in'properties', that's a schema violation, not a fun surprise ๐ TheTrueones are the chill ones.
and yeah, 'enum': ['gitlab'] really does have exactly one option in it. iconic of it honestly. not me getting emotionally attached to a validator ๐ it's giving "the schema has opinions and the opinions are final."
there's more of this dump coming, so don't unmute yet โ we're still in the middle of the structure and your girl is not skipping a single } ๐ซถโจ
okay so we're still inside the job object and it literally will not stop ๐ here's the next slice, and it's all about image, script, tags and variables:
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}},so what's it actually saying, bestie? ๐ซถ
imageโ it's aoneOf, so pick one vibe: either a plain string, or an object withname(a string) andentrypoint(an array of strings). That's it, that's the whole look โจscriptโ an array where every item is either a string or an array of strings. nested arrays, very demuretagsโ an array of strings ๐ท๏ธvariablesโ an object where the keys match[\\w\\d\\-_\\.]+and every value is a'type': 'string'๐
๐ the -job / -job-remove twins ๐
okay so this is where it gets cute: every job key comes in two flavours โ the normal one and a -remove one, so you can yeet a job out of a merged config. iconic behaviour honestly ๐
signing-job
{'additionalProperties': False, 'properties': {'signing-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},- the outer object is
'additionalProperties': Falseโ so no sneaky extra keys, the schema said what it said ๐ซ signing-jobitself is'additionalProperties': Trueโ inside it you can have extra stuff, she's flexible ๐โโ๏ธ- it carries
after_script,before_script,image,script,tags,variablesโ each shaped exactly like the ones above (strings or arrays-of-strings for the scripts, theoneOfstring/object deal forimage, array-of-strings fortags, the pattern-matched object forvariables) ๐ธ - notice there's no
'required'list on this one โ nothing is mandatory, bestie ๐๏ธ
signing-job-remove
'signing-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},the -remove twin is a carbon copy of the shape โ same additionalProperties: True, same after_script, before_script, image, script, tags, variables โ but the key is signing-job-remove, so likeโฆ it's the one that says "actually, take this job away" ๐ซ literally the same outfit, different name tag.
cleanup-job & cleanup-job-remove
and then โ plot twist โ the exact same two-scoop situation again, just with cleanup:
{'additionalProperties': False, 'properties': {'cleanup-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'cleanup-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},no cap, the way this is literally copy-paste energy ๐ญ same additionalProperties: True, same six fields, same oneOf on image, same regex-y variables. the only thing that changes is the key: cleanup-job, and cleanup-job-remove.
any-job & any-job-remove
third time's the charm, and she's calling it any-job because it's giving main character energy that applies everywhere ๐
{'additionalProperties': False, 'properties': {'any-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'any-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'}]}]}, 'type': 'array'},yep. same six fields. we been knew. ๐
but look at the tail โ after all that, the whole thing closes out with 'type': 'array'} โ because this entire block is one item in an array of anyOf alternatives.
the three keys that live at this level ๐๏ธ
'rebuild-index': {'type': 'boolean'}, 'target': {'default': 'gitlab',
'enum': ['gitlab'], 'type': 'string'}, 'temporary-storage-url-prefix':
{'type': 'string'}}, 'type': 'object'}]}, {'anyOf':rebuild-indexโ'type': 'boolean'โtargetโ a string with'default': 'gitlab'and'enum': ['gitlab'], so the only legal value rn is literallygitlabโ she's committed to the bit ๐temporary-storage-url-prefixโ'type': 'string', a prefix, for temporary storage URLs, as advertised ๐๏ธ
then 'type': 'object'} closes the object, and }]} says "that was the alternatives array, done", and we're immediately onto {'anyOf': โ another set of alternatives. she never stops.
๐ฅ the mappings era ๐ฅ
okay bestie, next anyOf branch, and this is the one with mappings in it. first variant:
[{'additionalProperties': False, 'properties': {'after_script':
{'items': {'type': 'string'}, 'type': 'array'}, 'before_script':
{'items': {'type': 'string'}, 'type': 'array'}, 'bootstrap': {'items':
{'anyOf': [{'type': 'string'}, {'additionalProperties': False,
'properties': {'compiler-agnostic': {'default': False, 'type':
'boolean'}, 'name': {'type': 'string'}}, 'required': ['name'], 'type':
'object'}]}, 'type': 'array'}, 'broken-specs-url': {'type': 'string'},
'broken-tests-packages': {'items': {'type': 'string'}, 'type': 'array'},
'enable-artifacts-buildcache': {'type': 'boolean'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'script': {'items': {'type': 'string'},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'},
'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type':
'string'}}, 'type': 'object'}}, 'required': ['mappings'], 'type':
'object'}, {'additionalProperties': False, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'bootstrap': {'items': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'compiler-agnostic':
{'default': False, 'type': 'boolean'}, 'name': {'type': 'string'}},
'required': ['name'], 'type': 'object'}]}, 'type': 'array'},
'broken-specs-url': {'type': 'string'}, 'broken-tests-packages':
{'items': {'type': 'string'}, 'type': 'array'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':okay let's actually read her, because there is so much here and it's all useful ๐
top-level keys in this variant:
after_scriptโ array of stringsbefore_scriptโ array of stringsbootstrapโ array of items that are either a plain string or an object withcompiler-agnostic(boolean,'default': False) andname(string), where'required': ['name']โ so if you go the object route,nameis non-negotiable, no cap ๐จbroken-specs-urlโ stringbroken-tests-packagesโ array of stringsenable-artifacts-buildcacheโ boolean (this one only shows up in this variant, clock that ๐)imageโ the sameoneOf: string, or object withentrypoint(array of strings) +name(string)mappingsโ array of objects, each'additionalProperties': False, each with'required': ['match']match_behaviorโ string,'default': 'first','enum': ['first', 'merge']๐ฏrebuild-indexโ booleanscriptโ array of stringsservice-job-attributesโ object,'additionalProperties': True,'required': ['tags']signing-job-attributesโ object,'additionalProperties': True,'required': ['tags']tagsโ array of stringsvariablesโ the pattern-matched object again- and the whole variant is
'required': ['mappings']โ you must bringmappingsto this party ๐
inside each entry of mappings:
matchโ array of strings, and it's'required': ['match']so yeah, must have itremove-attributesโ object,'additionalProperties': False,'required': ['tags'], and its only described key istags(array of strings) ๐ท๏ธrunner-attributesโ object,'additionalProperties': True,'required': ['tags'], containingafter_script,before_script,image,script,tags,variablesโ note these scripts are plain{items: {type: string}}arrays here, not the nested anyOf situation ๐
and the three *-attributes siblings โ runner-attributes, service-job-attributes, signing-job-attributes โ are all the same deal: additionalProperties: True, required: ['tags'], and the same after_script / before_script / image / script / tags / variables set.
and then she immediately starts the second variant of the same thing โ 'additionalProperties': False, same after_script, before_script, bootstrap (again with compiler-agnostic defaulting to False and 'required': ['name']), broken-specs-url, broken-tests-packages, image, mappings, match_behavior ('default': 'first', 'enum': ['first', 'merge']), rebuild-index, script, and we're cutting off right as she says 'service-job-attributes': โ because this section is long and we're only halfway through the vibe ๐ญ
fr fr though: these two variants are near-twins. the visible difference so far is that enable-artifacts-buildcache appears in the first one and not in this second one. don't go assuming anything the schema didn't write down โ I'm just telling you what's on the page ๐
โฆand we're still deep inside the schema bestie, no we did not escape yet ๐ picking up right where we left off โ the second half of the JSON is giving more of the same but make it signing:
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'script': {'items': {'type': 'string'},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'},
'temporary-storage-url-prefix': {'type': 'string'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['mappings'], 'type': 'object'}]}]}}Properties for inclusion in other schemas
okay so let's do a vibe check on what that blob is actually saying, because it's literally just the same little job recipe copy-pasted twice with a glow-up ๐
โจ the job-attributes situation โจ
Both blocks are {'type': 'object'} with 'additionalProperties': True โ meaning yes babes, extra keys are allowed, live ur truth ๐ฆ. Both demand 'required': ['tags'], so tags is non-negotiable, periodt.
The properties in each one:
after_scriptโ{'items': {'type': 'string'}, 'type': 'array'}โ a lil array, and every item in it is a string ๐before_scriptโ{'items': {'type': 'string'}, 'type': 'array'}โ same exact energy, just happens before โจimageโ{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}โ this one's the main character fr, it'll accept either a plain string or an object with anentrypoint(array of strings) and aname(string) ๐scriptโ{'items': {'type': 'string'}, 'type': 'array'}โ array of strings, no cap ๐tagsโ{'items': {'type': 'string'}, 'type': 'array'}โ array of strings, and remember: required ๐คvariablesโ{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}โ an object where the keys have to match that pattern and the values are strings. That regex is[\\w\\d\\-_\\.]+exactly as written, double backslashes included, because it's living inside a JSON string, iykyk ๐ง
So the second block, signing-job-attributes, is lowkey the exact same shape โ after_script, before_script, image (that same oneOf situation with entrypoint and name), script, tags, variables with that same [\\w\\d\\-_\\.]+ pattern โ 'required': ['tags'], 'type': 'object', 'additionalProperties': True. Ate and left no crumbs ๐
๐ the outer layer ๐
Then the wrapper closes out with:
tagsโ{'items': {'type': 'string'}, 'type': 'array'}temporary-storage-url-prefixโ{'type': 'string'}โ just a lil string bestie, nothing dramatic ๐ธvariablesโ{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}โ same pattern-keyed string map as above
โฆand the whole thing goes 'required': ['mappings'], 'type': 'object', then a bunch of closing brackets }]}]}} that are doing the absolute most ๐
friendly reminder: mappings is required at this level โ that's not a vibe, that's a rule ๐ซถ
And that's the whole chunk: Properties for inclusion in other schemas โ iconic, reused everywhere, we love a shared definition โญ
okay bestie, buckle up, because this next bit is the actual ci schema and it is a lot of nested dict ๐ญ but don't panic โ it's literally just the shape-checker for your ci.yaml, and once you see the pattern it's lowkey satisfying. Like, it's giving "same outfit, different colorway" for every job type ๐
spack.schema.ci.schema = {'$schema': 'http://json-schema.org/draft-07/schema#', 'additionalProperties': False, 'properties': {'ci': {'oneOf': [{'anyOf': [{'additionalProperties': False, 'properties': {'broken-specs-url': {'type': 'string'}, 'broken-tests-packages': {'items': {'type': 'string'}, 'type': 'array'}, 'enable-artifacts-buildcache': {'type': 'boolean'}, 'pipeline-gen': {'items': {'oneOf': [{'additionalProperties': False, 'properties': {'match_behavior': {'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'}, 'submapping': {'items': {'additionalProperties': False, 'properties': {'build-job': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'build-job-remove': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'match': {'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['match'], 'type': 'object'}, 'type': 'array'}}, 'required': ['submapping'], 'type': 'object'}, {'oneOf': [{'additionalProperties': False, 'properties': {'noop-job': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'noop-job-remove': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}, {'additionalProperties': False, 'properties': {'build-job': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'build-job-remove': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}, {'additionalProperties': False, 'properties': {'copy-job': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'copy-job-remove': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}, {'additionalProperties': False, 'properties': {'reindex-job': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'reindex-job-remove': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}, {'additionalProperties': False, 'properties': {'signing-job': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'signing-job-remove': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}, {'additionalProperties': False, 'properties': {'cleanup-job': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':okay so real talk, this whole thing is one big vibe check for your config ๐ซถ here's what's actually in there, no gatekeeping:
$schemais'http://json-schema.org/draft-07/schema#'โ draft-07, periodt. That's the JSON Schema dialect this is written in โจadditionalProperties: Falseshows up at the top and on each named job container, meaning those objects are locked down โ no random extra keys sneaking in ๐- the whole thing hangs off
'ci'with a'oneOf', and the first branch is an'anyOf'listing the top-level goodies:'broken-specs-url'(a string),'broken-tests-packages'(an array of strings),'enable-artifacts-buildcache'(a boolean), and'pipeline-gen'(an array) ๐ธ
then inside 'pipeline-gen''s items there's another 'oneOf', and bestie this is where it gets juicy:
- one branch has
'match_behavior'โ default'first', enum['first', 'merge'], type string โ plus'submapping', and it requires['submapping']๐- each
'submapping'item is an object requiring['match'], where'match'is an array of strings, and it also knows about'build-job'and'build-job-remove'๐
- each
- the other branch is a
'oneOf'across the job pairs:'noop-job'/'noop-job-remove','build-job'/'build-job-remove','copy-job'/'copy-job-remove','reindex-job'/'reindex-job-remove','signing-job'/'signing-job-remove', and'cleanup-job'๐
and the way every single job object is literally the same six keys cloned over and over, iykyk ๐
after_scriptandbefore_scriptโ arrays whose items areanyOfa string or an array of stringsimageโoneOfa plain string or an object with'entrypoint'(array of strings) and'name'(string)scriptโ same array-of-string-or-array-of-strings action as the scripts abovetagsโ array of stringsvariablesโ an object whose keys match the pattern[\\w\\d\\-_\\.]+and whose values are strings ๐
each job object is marked 'additionalProperties': True (so those can take extra stuff), and is typed 'object'. no cap, it's repetitive, but it's clean ๐ง
and yeah โ we're literally mid-object right at 'image': {'oneOf': [{'type': 'string'}, {'properties': when this chunk runs out of page, so hold that thought, the braces are still open ๐๐
okay bestie we are deep in the schema mines now โ๏ธ๐ this chunk picks up mid-object, still inside the pipeline-gen schema, and it's giving very much "the same job shape copy-pasted with different names" energy. Watch the pattern, it's lowkey iconic:
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'cleanup-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'any-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'any-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'}]}]}, 'type': 'array'},
'rebuild-index': {'type': 'boolean'}, 'target': {'default': 'gitlab',
'enum': ['gitlab'], 'type': 'string'}}, 'type': 'object'},so real talk โ every single job block in here is built from the exact same six keys, no cap:
after_scriptโ array of either strings or arrays of stringsanyOf: [{type: string}, {items: {type: string}, type: array}]before_scriptโ sameanyOfdeal, same arrayimageโoneOfa plain string or an object withentrypoint(array of strings) andname(string)scriptโ that sameanyOfstring-or-string-array situationtagsโ array of stringsvariablesโ object whose keys match[\\w\\d\\-_\\.]+and whose values are strings
and each job type object has 'additionalProperties': True, meaning it's the โจanything goesโจ one. The *-remove variants are literally the same body, just named <thing>-remove ๐ซถ. And after all that we slide into rebuild-index ('type': 'boolean') and target ('default': 'gitlab', 'enum': ['gitlab'], 'type': 'string' โ so yeah, only gitlab is allowed, periodt).
next chunk hops to a whole new anyOf branch and the submapping machinery ๐ธ:
{'additionalProperties': False, 'properties': {'broken-specs-url':
{'type': 'string'}, 'broken-tests-packages': {'items': {'type':
'string'}, 'type': 'array'}, 'pipeline-gen': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'submapping': {'items': {'additionalProperties': False, 'properties':
{'build-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'build-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'match':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['match'],
'type': 'object'}, 'type': 'array'}}, 'required': ['submapping'],
'type': 'object'}, {'oneOf': [{'additionalProperties': False,key tea to sip here โ:
broken-specs-urlโ just a'type': 'string'broken-tests-packagesโ'items': {'type': 'string'}, 'type': 'array'match_behaviorโ'default': 'first','enum': ['first', 'merge'],'type': 'string'(so: exactly two choices, bestie, no inventing a third ๐)matchโ array of strings, and it's'required': ['match']inside that submapping entry, so you cannot skip itsubmappingโ array of objects, and the outer object has'required': ['submapping']โ also mandatory, fr fr- the whole containing branch is
'additionalProperties': False, meaning no sneaky extra keys allowed ๐
and then the submapping item is itself an object stuffed with build-job + build-job-remove, both 'additionalProperties': True, both rocking the six-key job shape from before. We keep rolling:
'properties': {'noop-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'noop-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type':
'object'}, {'additionalProperties': False, 'properties': {'build-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'build-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},okay so that's the noop-job / noop-job-remove pair living inside a second oneOf branch (the one that's 'additionalProperties': False), and then yet another 'additionalProperties': False branch with build-job + build-job-remove doing the exact same six-key routine ๐ฅ. Yes it's repetitive. Yes it's on purpose. That's the whole vibe of a schema, bestie โ consistency ate ๐
last chunk of this part, and then we literally get cut off mid-key ๐ญ:
{'additionalProperties': False, 'properties': {'copy-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'copy-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'reindex-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'reindex-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':so we've now met copy-job, copy-job-remove, reindex-job, and reindex-job-remove โ every one of them 'additionalProperties': True and every one of them with that identical after_script / before_script / image / script / tags / variables glow-up โจ๐ฆ and we stop smack in the middle of reindex-job-remove's after_script, mid-anyOf, mid-items. Iconic cliffhanger honestly ๐๐ท โ the next part picks it right back up.
ok so we're picking up mid-blob in the GitLab CI schema and honestly? it's giving "JSON that ate the whole group chat" ๐ but we're gonna read it together, item by item, because every single key here actually means something. first up: the tail end of a job definition โ script stuff, then tags and variables on the end ๐
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}},like, look at that logic โ before_script and script accept either a plain string or an array of strings (nested anyOf, slay), image can be a bare string or an object with entrypoint (array of strings) and name, tags is just an array of strings, and variables is an object where every key matches [\\w\\d\\-_\\.]+ and every value is a string. periodt. ๐
๐ฆ signing-job & signing-job-remove
now the named job slots. signing-job and signing-job-remove are both 'additionalProperties': True objects (so extra keys are allowed, no cap), each with the same core set: after_script, before_script, image, script, tags, variables โจ
{'additionalProperties': False, 'properties': {'signing-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'signing-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},and the object wrapping them is 'additionalProperties': False โ so inside that object, only the keys it lists are allowed. main character energy only, no random extras ๐
๐ง cleanup-job & cleanup-job-remove
same pattern, different job name. cleanup-job and cleanup-job-remove, both 'additionalProperties': True, both with after_script / before_script / image / script / tags / variables ๐ธ
{'additionalProperties': False, 'properties': {'cleanup-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'cleanup-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},bestie the way this is literally the same shape repeated with a -remove twinโฆ it's giving copy-paste but make it schema ๐
๐ฅ any-job & any-job-remove
and here's the wildcard pair โ any-job and any-job-remove. again 'additionalProperties': True, again the same six keys. highkey the vibe check is: whatever job you throw at it, it wants after_script, before_script, image, script, tags, variables ๐
{'additionalProperties': False, 'properties': {'any-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'any-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'}]}]}, 'type': 'array'},see that ending? 'type': 'object'}]}]}, 'type': 'array'}, โ we're closing out an array of these objects, and then the sibling keys come in ๐
โญ rebuild-index, target, temporary-storage-url-prefix
three more keys sliding in right after the array closes:
rebuild-indexโ typebooleanโtargetโ typestring, default'gitlab', and itsenumis literally just['gitlab'](soโฆ one option, and it's gitlab, iconic tbh)temporary-storage-url-prefixโ typestring
'rebuild-index': {'type': 'boolean'}, 'target': {'default': 'gitlab',
'enum': ['gitlab'], 'type': 'string'}, 'temporary-storage-url-prefix':
{'type': 'string'}}, 'type': 'object'}]}, {'anyOf':and then {'anyOf': opens the next big branch, bestie. buckle up ๐ข
๐ the anyOf branches begin
this next chunk is the first alternative in that anyOf. it's an 'additionalProperties': False object whose properties are: after_script, before_script, bootstrap, broken-specs-url, broken-tests-packages, enable-artifacts-buildcache, image, mappings, match_behavior, rebuild-index, script, service-job-attributes, signing-job-attributes, tags, variables.
some deets worth pointing at:
bootstrapis an array whose items are either a string or an object withcompiler-agnostic(boolean, defaultFalse) andname(string), wherenameis required ๐ซถbroken-tests-packagesis an array of stringsenable-artifacts-buildcacheis a booleanmappingsis a required array of objects, each withmatch(array of strings, required),remove-attributes(object withtagsarray of strings,tagsrequired), andrunner-attributes(additionalProperties: True, withtagsrequired)match_behaviorโ default'first',enum['first', 'merge']service-job-attributesandsigning-job-attributesare bothadditionalProperties: Truewithtagsrequired
[{'additionalProperties': False, 'properties': {'after_script':
{'items': {'type': 'string'}, 'type': 'array'}, 'before_script':
{'items': {'type': 'string'}, 'type': 'array'}, 'bootstrap': {'items':
{'anyOf': [{'type': 'string'}, {'additionalProperties': False,
'properties': {'compiler-agnostic': {'default': False, 'type':
'boolean'}, 'name': {'type': 'string'}}, 'required': ['name'], 'type':
'object'}]}, 'type': 'array'}, 'broken-specs-url': {'type': 'string'},
'broken-tests-packages': {'items': {'type': 'string'}, 'type': 'array'},
'enable-artifacts-buildcache': {'type': 'boolean'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':ok and notice the objects inside mappings are 'additionalProperties': False โ so match, remove-attributes, runner-attributes are the whole menu there, and match itself is required (see 'required': ['match']) ๐ฏ
๐ service-job-attributes & signing-job-attributes bodies
here's the guts of those two attribute sets, plus the object's own tags and variables, and the big 'required': ['mappings'] at the end. fr fr, mappings is not optional in that branch ๐
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'script': {'items': {'type': 'string'},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'},
'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type':
'string'}}, 'type': 'object'}}, 'required': ['mappings'], 'type':
'object'}, {'additionalProperties': False, 'properties':see how tags is required inside both service-job-attributes and signing-job-attributes? 'required': ['tags'], both times. no escaping it, bestie ๐
๐ branch two: same energy, no enable-artifacts-buildcache
and the second anyOf branch starts here. 'additionalProperties': False, properties: after_script, before_script, bootstrap (same string-or-object anyOf, same compiler-agnostic default False, same 'required': ['name']), broken-specs-url, broken-tests-packages, image, mappings (same match / remove-attributes / runner-attributes structure, same 'required': ['tags'] and 'required': ['match']), and it closes right at match_behavior โ default 'first', enum ['first', 'merge'] ๐ธ
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'bootstrap': {'items': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'compiler-agnostic':
{'default': False, 'type': 'boolean'}, 'name': {'type': 'string'}},
'required': ['name'], 'type': 'object'}]}, 'type': 'array'},
'broken-specs-url': {'type': 'string'}, 'broken-tests-packages':
{'items': {'type': 'string'}, 'type': 'array'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},and that's where we pause โ trailing comma and all, because the schema keeps going in the next part ๐ don't touch that dial, bestie ๐
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'script': {'items': {'type': 'string'},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'},
'temporary-storage-url-prefix': {'type': 'string'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['mappings'], 'type': 'object'}]}]}}, 'title':
'Spack CI configuration file schema', 'type': 'object'}Full schema with metadata ๐
spack.schema.ci.update(data)okay bestie that's the CI schema all wrapped up โ spack.schema.ci.update(data) is the little helper that patches the juicy metadata on top, and yes it literally has "update" in the name because that's what it does, slay.
๐ธ spack.schema.compilers module ๐ธ
Schema for compilers.yaml configuration file. This is the blueprint for telling Spack which compilers exist on your machine and how to call them โ very main character of your toolchain, honestly.
properties: Dict[str, Any] = {
"compilers": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": False,
"properties": {
"compiler": {
"type": "object",
"additionalProperties": False,
"required": ["paths", "spec", "modules", "operating_system"],
"properties": {
"paths": {
"type": "object",
"required": ["cc", "cxx", "f77", "fc"],
"additionalProperties": False,
"properties": {
"cc": {"anyOf": [{"type": "string"}, {"type": "null"}]},
"cxx": {"anyOf": [{"type": "string"}, {"type": "null"}]},
"f77": {"anyOf": [{"type": "string"}, {"type": "null"}]},
"fc": {"anyOf": [{"type": "string"}, {"type": "null"}]},
},
},
"flags": {
"type": "object",
"additionalProperties": False,
"properties": {
"cflags": {"anyOf": [{"type": "string"}, {"type": "null"}]},
"cxxflags": {"anyOf": [{"type": "string"}, {"type": "null"}]},
"fflags": {"anyOf": [{"type": "string"}, {"type": "null"}]},
"cppflags": {"anyOf": [{"type": "string"}, {"type": "null"}]},
"ldflags": {"anyOf": [{"type": "string"}, {"type": "null"}]},
"ldlibs": {"anyOf": [{"type": "string"}, {"type": "null"}]},
},
},
"spec": {"type": "string"},
"operating_system": {"type": "string"},
"target": {"type": "string"},
"alias": {"anyOf": [{"type": "string"}, {"type": "null"}]},
"modules": {
"anyOf": [{"type": "string"}, {"type": "null"}, {"type": "array"}]
},
"implicit_rpaths": {
"anyOf": [
{"type": "array", "items": {"type": "string"}},
{"type": "boolean"},
]
},
"environment": spack.schema.environment.definition,
"extra_rpaths": {
"type": "array",
"default": [],
"items": {"type": "string"},
},
},
}
},
},
}
}
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack compiler configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}so yeah โ properties holds the actual meat (the compilers array and all its compiler guts), and schema is that plus the $schema line, the title "Spack compiler configuration file schema", "type": "object", "additionalProperties": False, and "properties": properties bolted on. It's giving "same outfit, extra accessories" ๐
Notice how paths is required to have all four of cc, cxx, f77, fc โ no skipping f77, bestie, Fortran is coming to the party whether you like it or not ๐ญ
๐ฆ spack.schema.compilers attributes ๐ฆ
spack.schema.compilers.properties: Dict[str, Any] โ Properties for inclusion in other schemas โจ (i.e. this bad boy gets imported and reused elsewhere, iconic reuse behavior)
{'compilers':
{'items': {'additionalProperties': False, 'properties': {'compiler':
{'additionalProperties': False, 'properties': {'alias': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'environment':
{'additionalProperties': False, 'default': {}, 'properties':
{'append_path': {'patternProperties': {'\w[\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\w[\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\w[\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\w[\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'extra_rpaths': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}, 'flags': {'additionalProperties': False, 'properties':
{'cflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]},
'cppflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]},
'cxxflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'fflags':
{'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'ldflags': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'ldlibs': {'anyOf': [{'type':
'string'}, {'type': 'null'}]}}, 'type': 'object'}, 'implicit_rpaths':
{'anyOf': [{'items': {'type': 'string'}, 'type': 'array'}, {'type':
'boolean'}]}, 'modules': {'anyOf': [{'type': 'string'}, {'type':
'null'}, {'type': 'array'}]}, 'operating_system': {'type': 'string'},
'paths': {'additionalProperties': False, 'properties': {'cc': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'cxx': {'anyOf': [{'type':
'string'}, {'type': 'null'}]}, 'f77': {'anyOf': [{'type': 'string'},
{'type': 'null'}]}, 'fc': {'anyOf': [{'type': 'string'}, {'type':
'null'}]}}, 'required': ['cc', 'cxx', 'f77', 'fc'], 'type': 'object'},
'spec': {'type': 'string'}, 'target': {'type': 'string'}}, 'required':
['paths', 'spec', 'modules', 'operating_system'], 'type': 'object'}},
'type': 'object'}, 'type': 'array'}}spack.schema.compilers.schema โ Full schema with metadata ๐
{'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'compilers': {'items': {'additionalProperties':
False, 'properties': {'compiler': {'additionalProperties': False,
'properties': {'alias': {'anyOf': [{'type': 'string'}, {'type':
'null'}]}, 'environment': {'additionalProperties': False, 'default': {},
'properties': {'append_path': {'patternProperties': {'\w[\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'prepend_path': {'patternProperties': {'\w[\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\w[\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\w[\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'extra_rpaths': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}, 'flags': {'additionalProperties': False, 'properties':
{'cflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]},
'cppflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]},
'cxxflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'fflags':
{'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'ldflags': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'ldlibs': {'anyOf': [{'type':
'string'}, {'type': 'null'}]}}, 'type': 'object'}, 'implicit_rpaths':
{'anyOf': [{'items': {'type': 'string'}, 'type': 'array'}, {'type':
'boolean'}]}, 'modules': {'anyOf': [{'type': 'string'}, {'type':
'null'}, {'type': 'array'}]}, 'operating_system': {'type': 'string'},
'paths': {'additionalProperties': False, 'properties': {'cc': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'cxx': {'anyOf': [{'type':
'string'}, {'type': 'null'}]}, 'f77': {'anyOf': [{'type': 'string'},
{'type': 'null'}]}, 'fc': {'anyOf': [{'type': 'string'}, {'type':
'null'}]}}, 'required': ['cc', 'cxx', 'f77', 'fc'], 'type': 'object'},
'spec': {'type': 'string'}, 'target': {'type': 'string'}}, 'required':
['paths', 'spec', 'modules', 'operating_system'], 'type': 'object'}},
'type': 'object'}, 'type': 'array'}}, 'title': 'Spack compiler
configuration file schema', 'type': 'object'}ngl these two are basically twins โ samecompilersarray, samecompilerobject withalias,environment,extra_rpaths,flags,implicit_rpaths,modules,operating_system,paths,spec,target, samerequired: ['paths', 'spec', 'modules', 'operating_system']โ the only flexschemahas is the$schemaURL and thetitle. That's it. That's the difference. ๐ซถ
quick recap of the environment sub-object since it shows up in both, because it's lowkey the most extra part: append_path, prepend_path, remove_path, and set all take a patternProperties map of '\w[\w-]*' keys pointing at either a 'string' or a 'number', and unset is an array defaulting to [] whose items can be a 'string' or a 'number'. 'default': {} on the whole thing, 'type': 'object'. periodt โจ
๐ง spack.schema.concretizer module ๐ง
Schema for concretizer.yaml configuration file. This one starts with the little building block:
LIST_OF_SPECS = {"type": "array", "items": {"type": "string"}}look at her, one line, does exactly what it says on the tin โ an array of strings โ and she's reusable, which is very generous of her ๐
properties: Dict[str, Any] = {
"concretizer": {
"type": "object",
"additionalProperties": False,
"properties": {
"reuse": {
"oneOf": [
{"type": "boolean"},
{"type": "string", "enum": ["dependencies"]},
{
"type": "object",
"properties": {
"roots": {"type": "boolean"},
"include": LIST_OF_SPECS,
"exclude": LIST_OF_SPECS,
"from": {
"type": "array",
"items": {
"type": "object",
"properties": {
"type": {
"type": "string",
"enum": ["local", "buildcache", "external"],
},
"include": LIST_OF_SPECS,
"exclude": LIST_OF_SPECS,
},
},
},
},
},
]
},
"enable_node_namespace": {"type": "boolean"},
"targets": {
"type": "object",
"properties": {
"host_compatible": {"type": "boolean"},
"granularity": {"type": "string", "enum": ["generic", "microarchitectures"]},
},
},
"unify": {
"oneOf": [{"type": "boolean"}, {"type": "string", "enum": ["when_possible"]}]
},
"duplicates": {
"type": "object",
"properties": {
"strategy": {"type": "string", "enum": ["none", "minimal", "full"]}
},
},
"os_compatible": {"type": "object", "additionalProperties": {"type": "array"}},
},
}
}
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack concretizer configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}okay so that's the whole concretizer schema shape โ every knob from reuse (which can be a bool, the string dependencies, or a whole object with roots / include / exclude / from) down to targets with its host_compatible and granularity (generic or microarchitectures), unify (bool or when_possible), duplicates.strategy (none, minimal, full), and os_compatible. Literally every branch, bestie ๐
And here's the same thing rendered out with its metadata โ this is the actual spack.schema.concretizer.schema dict, one long line, no cap:
spack.schema.concretizer.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'concretizer': {'additionalProperties': False,
'properties': {'duplicates': {'properties': {'strategy': {'enum':
['none', 'minimal', 'full'], 'type': 'string'}}, 'type': 'object'},
'enable_node_namespace': {'type': 'boolean'}, 'os_compatible':
{'additionalProperties': {'type': 'array'}, 'type': 'object'}, 'reuse':
{'oneOf': [{'type': 'boolean'}, {'enum': ['dependencies'], 'type':
'string'}, {'properties': {'exclude': {'items': {'type': 'string'},
'type': 'array'}, 'from': {'items': {'properties': {'exclude': {'items':
{'type': 'string'}, 'type': 'array'}, 'include': {'items': {'type':
'string'}, 'type': 'array'}, 'type': {'enum': ['local', 'buildcache',
'external'], 'type': 'string'}}, 'type': 'object'}, 'type': 'array'},
'include': {'items': {'type': 'string'}, 'type': 'array'}, 'roots':
{'type': 'boolean'}}, 'type': 'object'}]}, 'targets': {'properties':
{'granularity': {'enum': ['generic', 'microarchitectures'], 'type':
'string'}, 'host_compatible': {'type': 'boolean'}}, 'type': 'object'},
'unify': {'oneOf': [{'type': 'boolean'}, {'enum': ['when_possible'],
'type': 'string'}]}}, 'type': 'object'}}, 'title': 'Spack concretizer
configuration file schema', 'type': 'object'}That trailing line โ Full schema with metadata โ is just the caption saying "yep, this is the full schema, metadata and all." ๐ธ
โจ spack.schema.config module
Schema for config.yaml configuration file. This is the big one, the main character of config-land ๐
properties: Dict[str, Any] = {
"config": {
"type": "object",
"default": {},
"properties": {
"flags": {
"type": "object",
"properties": {
"keep_werror": {"type": "string", "enum": ["all", "specific", "none"]}
},
},
"shared_linking": {
"anyOf": [
{"type": "string", "enum": ["rpath", "runpath"]},
{
"type": "object",
"properties": {
"type": {"type": "string", "enum": ["rpath", "runpath"]},
"bind": {"type": "boolean"},
},
},
]
},
"install_tree": {
"anyOf": [
{
"type": "object",
"properties": union_dicts(
{"root": {"type": "string"}},
{
"padded_length": {
"oneOf": [
{"type": "integer", "minimum": 0},
{"type": "boolean"},
]
}
},
spack.schema.projections.properties,
),
},
{"type": "string"}, # deprecated
]
},
"install_hash_length": {"type": "integer", "minimum": 1},
"install_path_scheme": {"type": "string"}, # deprecated
"build_stage": {
"oneOf": [{"type": "string"}, {"type": "array", "items": {"type": "string"}}]
},
"stage_name": {"type": "string"},
"develop_stage_link": {"type": "string"},
"test_stage": {"type": "string"},
"extensions": {"type": "array", "items": {"type": "string"}},
"template_dirs": {"type": "array", "items": {"type": "string"}},
"license_dir": {"type": "string"},
"source_cache": {"type": "string"},
"misc_cache": {"type": "string"},
"environments_root": {"type": "string"},
"connect_timeout": {"type": "integer", "minimum": 0},
"verify_ssl": {"type": "boolean"},
"ssl_certs": {"type": "string"},
"suppress_gpg_warnings": {"type": "boolean"},
"install_missing_compilers": {"type": "boolean"},
"debug": {"type": "boolean"},
"checksum": {"type": "boolean"},
"deprecated": {"type": "boolean"},
"locks": {"type": "boolean"},
"dirty": {"type": "boolean"},
"build_language": {"type": "string"},
"build_jobs": {"type": "integer", "minimum": 1},
"ccache": {"type": "boolean"},
"concretizer": {"type": "string", "enum": ["original", "clingo"]},
"db_lock_timeout": {"type": "integer", "minimum": 1},
"package_lock_timeout": {
"anyOf": [{"type": "integer", "minimum": 1}, {"type": "null"}]
},
"allow_sgid": {"type": "boolean"},
"install_status": {"type": "boolean"},
"binary_index_root": {"type": "string"},
"url_fetch_method": {"type": "string", "enum": ["urllib", "curl"]},
"additional_external_search_paths": {"type": "array", "items": {"type": "string"}},
"binary_index_ttl": {"type": "integer", "minimum": 0},
"aliases": {"type": "object", "patternProperties": {r"\w[\w-]*": {"type": "string"}}},
},
"deprecatedProperties": {
"properties": ["terminal_title"],
"message": "config:terminal_title has been replaced by "
"install_status and is ignored",
"error": False,
},
}
}
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack core configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}Like, look at all of that. flags.keep_werror (all, specific, none), shared_linking (either the string rpath/runpath or an object with type + bind), install_tree (object or the deprecated string form), install_hash_length (integer, minimum 1), install_path_scheme (deprecated), build_stage, stage_name, develop_stage_link, test_stage, extensions, template_dirs, license_dir, source_cache, misc_cache, environments_root, connect_timeout, verify_ssl, ssl_certs, suppress_gpg_warnings, install_missing_compilers, debug, checksum, deprecated, locks, dirty, build_language, build_jobs (integer, minimum 1), ccache, concretizer (original or clingo), db_lock_timeout, package_lock_timeout (integer min 1 or null), allow_sgid, install_status, binary_index_root, url_fetch_method (urllib or curl), additional_external_search_paths, binary_index_ttl, and aliases keyed by \w[\w-]*. Obsessed ๐ญโจ
Also note the deprecatedProperties block: terminal_title is deprecated, its message is literally "config:terminal_title has been replaced by install_status and is ignored", and "error": False. So it's a soft nudge, not a hard fail ๐ซถ
And then there's the update function, which fixes up deprecated stuff in place:
def update(data):
"""Update the data in place to remove deprecated properties.
Args:
data (dict): dictionary to be updated
Returns:
True if data was changed, False otherwise
"""
# currently deprecated properties are
# install_tree: <string>
# install_path_scheme: <string>
# updated: install_tree: {root: <string>,
# projections: <projections_dict}
# root replaces install_tree, projections replace install_path_scheme
changed = False
install_tree = data.get("install_tree", None)
if isinstance(install_tree, str):
# deprecated short-form install tree
# add value as `root` in updated install_tree
data["install_tree"] = {"root": install_tree}
changed = True
install_path_scheme = data.pop("install_path_scheme", None)
if install_path_scheme:
projections_data = {"projections": {"all": install_path_scheme}}
# update projections with install_scheme
# whether install_tree was updated or not
# we merge the yaml to ensure we don't invalidate other projections
update_data = data.get("install_tree", {})
update_data = spack.config.merge_yaml(update_data, projections_data)
data["install_tree"] = update_data
changed = True
use_curl = data.pop("use_curl", None)
if use_curl is not None:
data["url_fetch_method"] = "curl" if use_curl else "urllib"
changed = TrueSo bestie, quick rundown of what it does ๐
โ it docstrings that it updates the data in place to remove deprecated properties, takes data (a dict), and returns True if data was changed, False otherwise. The comments spell out the deprecation tea: the old ones are install_tree: <string> and install_path_scheme: <string>, and the updated form is install_tree: {root: <string>, projections: <projections_dict>} โ root replaces install_tree, projections replace install_path_scheme.
Then the actual moves: it grabs install_tree (default None), and if it's a str โ that deprecated short form โ it rewrites data["install_tree"] = {"root": install_tree} and sets changed = True. Then it pops install_path_scheme; if it's truthy it builds projections_data = {"projections": {"all": install_path_scheme}}, merges that into whatever data.get("install_tree", {}) currently is via spack.config.merge_yaml (so we don't invalidate other projections, okurrr), assigns the merged result back to data["install_tree"], and sets changed = True. Finally it pops use_curl; if it's not None, it sets data["url_fetch_method"] = "curl" if use_curl else "urllib" and flips changed = True. Slay ๐
shared_linking = data.get("shared_linking", None)
if isinstance(shared_linking, str):
# deprecated short-form shared_linking: rpath/runpath
# add value as `type` in updated shared_linking
data["shared_linking"] = {"type": shared_linking, "bind": False}
changed = True
return changedokay so that's the tail end of it ๐
โ the part that catches you still using the deprecated short-form shared_linking (rpath/runpath), folds your value into the type key of the new-style dict, sets bind to False, and flips changed to True so the caller knows something moved. Then it hands changed back. Slay.
๐ spack.schema.config โ the rest of it
spack.schema.config.properties: Dict[str, Any] โ literally just the slice of the config schema that gets pulled into other schemas (that's the whole point of pulling it out, bestie):
spack.schema.config.properties: Dict[str, Any] = {'config': {'default':
{}, 'deprecatedProperties': {'error': False, 'message':
'config:terminal_title has been replaced by install_status and is
ignored', 'properties': ['terminal_title']}, 'properties':
{'additional_external_search_paths': {'items': {'type': 'string'},
'type': 'array'}, 'aliases': {'patternProperties': {'\\w[\\w-]*':
{'type': 'string'}}, 'type': 'object'}, 'allow_sgid': {'type':
'boolean'}, 'binary_index_root': {'type': 'string'}, 'binary_index_ttl':
{'minimum': 0, 'type': 'integer'}, 'build_jobs': {'minimum': 1, 'type':
'integer'}, 'build_language': {'type': 'string'}, 'build_stage':
{'oneOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'ccache': {'type': 'boolean'}, 'checksum': {'type':
'boolean'}, 'concretizer': {'enum': ['original', 'clingo'], 'type':
'string'}, 'connect_timeout': {'minimum': 0, 'type': 'integer'},
'db_lock_timeout': {'minimum': 1, 'type': 'integer'}, 'debug': {'type':
'boolean'}, 'deprecated': {'type': 'boolean'}, 'develop_stage_link':
{'type': 'string'}, 'dirty': {'type': 'boolean'}, 'environments_root':
{'type': 'string'}, 'extensions': {'items': {'type': 'string'}, 'type':
'array'}, 'flags': {'properties': {'keep_werror': {'enum': ['all',
'specific', 'none'], 'type': 'string'}}, 'type': 'object'},
'install_hash_length': {'minimum': 1, 'type': 'integer'},
'install_missing_compilers': {'type': 'boolean'}, 'install_path_scheme':
{'type': 'string'}, 'install_status': {'type': 'boolean'},
'install_tree': {'anyOf': [{'properties': {'padded_length': {'oneOf':
[{'minimum': 0, 'type': 'integer'}, {'type': 'boolean'}]},
'projections': {'patternProperties': {'all|\\w[\\w-]*': {'type':
'string'}}, 'type': 'object'}, 'root': {'type': 'string'}}, 'type':
'object'}, {'type': 'string'}]}, 'license_dir': {'type': 'string'},
'locks': {'type': 'boolean'}, 'misc_cache': {'type': 'string'},
'package_lock_timeout': {'anyOf': [{'minimum': 1, 'type': 'integer'},
{'type': 'null'}]}, 'shared_linking': {'anyOf': [{'enum': ['rpath',
'runpath'], 'type': 'string'}, {'properties': {'bind': {'type':
'boolean'}, 'type': {'enum': ['rpath', 'runpath'], 'type': 'string'}},
'type': 'object'}]}, 'source_cache': {'type': 'string'}, 'ssl_certs':
{'type': 'string'}, 'stage_name': {'type': 'string'},
'suppress_gpg_warnings': {'type': 'boolean'}, 'template_dirs': {'items':
{'type': 'string'}, 'type': 'array'}, 'test_stage': {'type': 'string'},
'url_fetch_method': {'enum': ['urllib', 'curl'], 'type': 'string'},
'verify_ssl': {'type': 'boolean'}}, 'type': 'object'}}Properties for inclusion in other schemas ๐ซถ
A couple of things worth clocking in there, since they're literally in the dict:
deprecatedPropertiesforconfighas'error': Falseand the message'config:terminal_title has been replaced by install_status and is ignored', with'properties': ['terminal_title']โ soterminal_titleis the deprecated one, and it gets ignored, not blown up.shared_linkingisanyOfa plain string enum of['rpath', 'runpath']or an object withbind(boolean) andtype(enum['rpath', 'runpath']). Both forms, iconic behaviour.install_treeisanyOfan object withpadded_length(oneOfinteger โฅ0or boolean),projections(patternProperties onall|\w[\w-]*) androot, or just a string.
spack.schema.config.schema โ the full thing with metadata, $schema set to http://json-schema.org/draft-07/schema#, additionalProperties False, and 'title': 'Spack core configuration file schema':
spack.schema.config.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'config': {'default {}, 'deprecatedProperties':
{'error': False, 'message': 'config:terminal_title has been replaced by
install_status and is ignored', 'properties': ['terminal_title']},
'properties': {'additional_external_search_paths': {'items': {'type':
'string'}, 'type': 'array'}, 'aliases': {'patternProperties':
{'\\w[\\w-]*': {'type': 'string'}}, 'type': 'object'}, 'allow_sgid':
{'type': 'boolean'}, 'binary_index_root': {'type': 'string'},
'binary_index_ttl': {'minimum': 0, 'type': 'integer'}, 'build_jobs':
{'minimum': 1, 'type': 'integer'}, 'build_language': {'type': 'string'},
'build_stage': {'oneOf': [{'type': 'string'}, {'items': {'type':
'string'}, 'type': 'array'}]}, 'ccache': {'type': 'boolean'},
'checksum': {'type': 'boolean'}, 'concretizer': {'enum': ['original',
'clingo'], 'type': 'string'}, 'connect_timeout': {'minimum': 0, 'type':
'integer'}, 'db_lock_timeout': {'minimum': 1, 'type': 'integer'},
'debug': {'type': 'boolean'}, 'deprecated': {'type': 'boolean'},
'develop_stage_link': {'type': 'string'}, 'dirty': {'type': 'boolean'},
'environments_root': {'type': 'string'}, 'extensions': {'items':
{'type': 'string'}, 'type': 'array'}, 'flags': {'properties':
{'keep_werror': {'enum': ['all', 'specific', 'none'], 'type':
'string'}}, 'type': 'object'}, 'install_hash_length': {'minimum': 1,
'type': 'integer'}, 'install_missing_compilers': {'type': 'boolean'},
'install_path_scheme': {'type': 'string'}, 'install_status': {'type':
'boolean'}, 'install_tree': {'anyOf': [{'properties': {'padded_length':
{'oneOf': [{'minimum': 0, 'type': 'integer'}, {'type': 'boolean'}]},
'projections': {'patternProperties': {'all|\\w[\\w-]*': {'type':
'string'}}, 'type': 'object'}, 'root': {'type': 'string'}}, 'type':
'object'}, {'type': 'string'}]}, 'license_dir': {'type': 'string'},
'locks': {'type': 'boolean'}, 'misc_cache': {'type': 'string'},
'package_lock_timeout': {'anyOf': [{'minimum': 1, 'type': 'integer'},
{'type': 'null'}]}, 'shared_linking': {'anyOf': [{'enum': ['rpath',
'runpath'], 'type': 'string'}, {'properties': {'bind': {'type':
'boolean'}, 'type': {'enum': ['rpath', 'runpath'], 'type': 'string'}},
'type': 'object'}]}, 'source_cache': {'type': 'string'}, 'ssl_certs':
{'type': 'string'}, 'stage_name': {'type': 'string'},
'suppress_gpg_warnings': {'type': 'boolean'}, 'template_dirs': {'items':
{'type': 'string'}, 'type': 'array'}, 'test_stage': {'type': 'string'},
'url_fetch_method': {'enum': ['urllib', 'curl'], 'type': 'string'},
'verify_ssl': {'type': 'boolean'}}, 'type': 'object'}}, 'title': 'Spack
core configuration file schema', 'type': 'object'}Full schema with metadata โจ
bestie, if you're copy-pasting that: keep the escaping exactly as written โ'\\w[\\w-]*','all|\\w[\\w-]*'โ because those backslashes are doing real work, no cap.
spack.schema.config.update(data) โ updates the data in place to strip out deprecated properties.
data(dict) โ the dictionary to be updated ๐
Returns โ True if data was changed, False otherwise.
๐ฆ spack.schema.container module
Schema for the container subsection of Spack environments. Big main-character energy, so here's the whole thing:
spack.schema.container.container_schema:
{'additionalProperties': False, 'properties': {'depfile': {'default':
False, 'type': 'boolean'}, 'docker': {'additionalProperties': False,
'default': {}, 'type': 'object'}, 'format': {'enum': ['docker',
'singularity'], 'type': 'string'}, 'images': {'anyOf':
[{'additionalProperties': False, 'properties': {'os': {'type':
'string'}, 'spack': {'anyOf': [{'type': 'string'}, {'additional_properties':
False, 'properties': {'ref': {'type': 'string'}, 'resolve_sha':
{'default': False, 'type': 'boolean'}, 'url': {'type': 'string'},
'verify': {'default': False, 'type': 'boolean'}}, 'type': 'object'}]}},
'required': ['os', 'spack'], 'type': 'object'}, {'additionalProperties':
False, 'properties': {'build': {'type': 'string'}, 'final': {'type':
'string'}}, 'required': ['build', 'final'], 'type': 'object'}]},
'labels': {'type': 'object'}, 'os_packages': {'additionalProperties':
False, 'properties': {'build': {'items': {'type': 'string'}, 'type':
'array'}, 'command': {'enum': ['apt', 'yum', 'zypper', 'apk',
'yum_amazon'], 'type': 'string'}, 'final': {'items': {'type': 'string'},
'type': 'array'}, 'update': {'type': 'boolean'}}, 'type': 'object'},
'singularity': {'additionalProperties': False, 'default': {},
'properties': {'help': {'type': 'string'}, 'runscript': {'type':
'string'}, 'startscript': {'type': 'string'}, 'test': {'type':
'string'}}, 'type': 'object'}, 'strip': {'default': True, 'type':
'boolean'}, 'template': {'default': None, 'type': 'string'}}, 'type':
'object'}Schema for the container attribute included in Spack environments ๐
the little details that ate, for the record: format is enum ['docker', 'singularity'], strip defaults to True, template defaults to None, depfile defaults to False, docker and singularity both default to {}, and os_packages.command is enum ['apt', 'yum', 'zypper', 'apk', 'yum_amazon']. periodt ๐
๐ spack.schema.cray_manifest module
okay this one is basically a document, not a schema-of-config:
Schema for Cray descriptive manifest: this describes a set of installed packages on the system and also specifies dependency relationships between them (so this provides more information than external entries in packages configuration).
๐ซถ important vibe check: This does not specify a configuration โ it is an input format that is consumed and transformed into Spack DB records. so don't go treating it like config, bestie.
๐ spack.schema.database_index module
Schema for the database index.json file. two literals live here, and they're both giving structured-data realness:
properties: Dict[str, Any] = {
"database": {
"type": "object",
"required": ["installs", "version"],
"additionalProperties": False,
"properties": {
"installs": {
"type": "object",
"patternProperties": {
r"^[\w\d]{32}$": {
"type": "object",
"properties": {
"spec": spack.schema.spec.properties,
"path": {"oneOf": [{"type": "string"}, {"type": "null"}]},
"installed": {"type": "boolean"},
"ref_count": {"type": "integer", "minimum": 0},
"explicit": {"type": "boolean"},
"installation_time": {"type": "number"},
},
}
},
},
"version": {"type": "string"},
},
}
}
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack spec schema",
"type": "object",
"required": ["database"],
"additionalProperties": False,
"properties": properties,
}a few things to clock, since they're all right there โจ:
databaserequires["installs", "version"]and setsadditionalPropertiestoFalseโ no surprise keys sneaking in, we don't do that here.- the
installskeys are matched byr"^[\w\d]{32}$"โ a raw string, so the backslashes are literal, iykyk. - each install entry can carry
spec(pulled straight fromspack.schema.spec.properties),path(oneOfstring ornull),installed(boolean),ref_count(integer,minimum0),explicit(boolean), andinstallation_time(number). - and the full
schemadict throws"title": "Spack spec schema"on there, plusrequired: ["database"]andadditionalProperties: False.
the way that #: Full schema with metadata comment is just sitting there, being iconic and doing absolutely nothing at runtime ๐ง๐
okay so this next chunk is a whole parade of schema modules, and honestly? it's giving JSON schema but make it fashion ๐ โจ we're just gonna walk through each one, bestie, no cap.
here's the full schema with metadata for spack.schema.database_index.schema:
spack.schema.database_index.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'database': {'additionalProperties': False,
'properties': {'installs': {'patternProperties': {'^[\\w\\d]{32}$':
{'properties': {'explicit': {'type': 'boolean'}, 'installation_time':
{'type': 'number'}, 'installed': {'type': 'boolean'}, 'path': {'oneOf':
[{'type': 'string'}, {'type': 'null'}]}, 'ref_count': {'minimum': 0,
'type': 'integer'}, 'spec': {'spec': {'additionalProperties': False,
'properties': {'_meta': {'properties': {'version': {'type': 'number'}},
'type': 'object'}, 'nodes': {'items': {'additionalProperties': False,
'properties': {'arch': {'additionalProperties': False, 'properties':
{'platform': {}, 'platform_os': {}, 'target': {'oneOf': [{'type':
'string'}, {'additionalProperties': False, 'properties': {'features':
{'items': {'type': 'string'}, 'type': 'array'}, 'generation': {'type':
'integer'}, 'name': {'type': 'string'}, 'parents': {'items': {'type':
'string'}, 'type': 'array'}, 'vendor': {'type': 'string'}}, 'required':
['name', 'vendor', 'features', 'generation', 'parents'], 'type':
'object'}]}}, 'type': 'object'}, 'build_hash': {'type': 'string'},
'build_spec': {'additionalProperties': False, 'properties': {'hash':
{'type': 'string'}, 'name': {'type': 'string'}}, 'required': ['name',
'hash'], 'type': 'object'}, 'compiler': {'additionalProperties': False,
'properties': {'name': {'type': 'string'}, 'version': {'type':
'string'}}, 'type': 'object'}, 'dependencies': {'patternProperties':
{'\\w[\\w-]*': {'properties': {'hash': {'type': 'string'}, 'type':
{'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'}},
'type': 'object'}, 'develop': {'anyOf': [{'type': 'boolean'}, {'type':
'string'}]}, 'full_hash': {'type': 'string'}, 'hash': {'type':
'string'}, 'name': {'type': 'string'}, 'namespace': {'type': 'string'},
'package_hash': {'type': 'string'}, 'parameters':
{'additionalProperties': True, 'properties': {'cflags': {'items':
{'type': 'string'}, 'type': 'array'}, 'cppflags': {'items': {'type':
'string'}, 'type': 'array'}, 'cxxflags': {'items': {'type': 'string'},
'type': 'array'}, 'fflags': {'items': {'type': 'string'}, 'type':
'array'}, 'ldflags': {'items': {'type': 'string'}, 'type': 'array'},
'ldlib': {'items': {'type': 'string'}, 'type': 'array'}, 'patches':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['cflags',
'cppflags', 'cxxflags', 'fflags', 'ldflags', 'ldlibs'], 'type':
'object'}, 'patches': {'items': {}, 'type': 'array'}, 'version':
{'oneOf': [{'type': 'string'}, {'type': 'number'}]}}, 'required':
['version', 'arch', 'compiler', 'namespace', 'parameters'], 'type':
'object'}, 'type': 'array'}}, 'required': ['_meta', 'nodes'], 'type':
'object'}}}, 'type': 'object'}}, 'type': 'object'}, 'version': {'type':
'string'}}, 'required': ['installs', 'version'], 'type': 'object'}},
'required': ['database'], 'title': 'Spack spec schema', 'type':
'object'}Full schema with metadata โ that's the whole assignment, periodt. ๐
๐งฉ spack.schema.definitions module
Schema for definitions โ literally the blueprint for definitions, bestie. iykyk.
properties: Dict[str, Any] = {
"definitions": {
"type": "array",
"default": [],
"items": {
"type": "object",
"properties": {"when": {"type": "string"}},
"patternProperties": {r"^(?!when$)\w*": spec_list_schema},
},
}
}
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack definitions configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}Full schema with metadata โ another one, we love consistency. โจ
spack.schema.definitions.properties: Dict[str, Any]
Properties for inclusion in other schemas โ she's a reusable queen. ๐
spack.schema.definitions.properties: Dict[str, Any] = {'definitions':
{'default': [], 'items': {'patternProperties': {'^(?!when$)\\w*':
{'default': [], 'items': {'anyOf': [{'additionalProperties': False,
'properties': {'exclude': {'items': {'type': 'string'}, 'type':
'array'}, 'matrix': {'items': {'items': {'type': 'string'}, 'type':
'array'}, 'type': 'array'}}, 'type': 'object'}, {'type': 'string'},
{'type': 'null'}]}, 'type': 'array'}}, 'properties': {'when': {'type':
'string'}}, 'type': 'object'}, 'type': 'array'}}spack.schema.definitions.schema
Full schema with metadata โ iconic. no notes. ๐
spack.schema.definitions.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'definitions': {'default': [], 'items':
{'patternProperties': {'^(?!when$)\\w*': {'default': [], 'items':
{'anyOf': [{'additionalProperties': False, 'properties': {'exclude':
{'items': {'type': 'string'}, 'type': 'array'}, 'matrix': {'items':
{'items': {'type': 'string'}, 'type': 'array'}, 'type': 'array'}},
'type': 'object'}, {'type': 'string'}, {'type': 'null'}]}, 'type':
'array'}}, 'properties': {'when': {'type': 'string'}}, 'type':
'object'}, 'type': 'array'}}, 'title': 'Spack definitions configuration
file schema', 'type': 'object'}๐ ๏ธ spack.schema.develop module
no description in the original for this one, but here's the schema and the update signature โ short and sweet.
spack.schema.develop.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'develop': {'additionalProperties': False,
'default': {}, 'patternProperties': {'\\w[\\w-]*':
{'additionalProperties': False, 'properties': {'path': {'type':
'string'}, 'spec': {'type': 'string'}}, 'type': 'object'}}, 'type':
'object'}}, 'title': 'Spack repository configuration file schema',
'type': 'object'}Full schema with metadata โ yes, that's the develop schema, bestie.
spack.schema.develop.update(data)โ the update hook signature. no extra description in this part, fr fr.
๐ฟ spack.schema.env module
Schema for env.yaml configuration file. โ the environment file's whole personality, fr fr. ๐
from .spec_list import spec_list_schema
#: Top level key in a manifest file
TOP_LEVEL_KEY = "spack"
properties: Dict[str, Any] = {
"spack": {
"type": "object",
"default": {},
"additionalProperties": False,
"properties": union_dicts(
# Include deprecated "gitlab-ci" section
spack.schema.gitlab_ci.properties,
# merged configuration scope schemas
spack.schema.merged.properties,
# extra environment schema properties
{
"include": {"type": "array", "default": [], "items": {"type": "string"}},
"specs": spec_list_schema,
"include_concrete": {"type": "array", "default": [], "items": {"type": "string"}},
},
),
}
}
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack environment file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}
def update(data):
"""Update the data in place to remove deprecated properties.
Args:
data (dict): dictionary to be updated
Returns:
True if data was changed, False otherwise
"""
import spack.ci
if "gitlab-ci" in data:
data["ci"] = data.pop("gitlab-ci")
if "ci" in data:
return spack.ci.translate_deprecated_config(data["ci"])
# There are not currently any deprecated attributes in this section
# that have not been removed
return Falsespack.schema.env.TOP_LEVEL_KEY = 'spack'
Top level key in a manifest file โ literally the main character key, no cap. ๐
spack.schema.env.update(data)
Update the data in place to remove deprecated properties. โ she's doing cleanup, slay. ๐งน
Parameters
data(dict) โ dictionary to be updated
Returns
Trueif data was changed,Falseotherwise
๐ spack.schema.environment module
Schema for environment modifications. Meant for inclusion in other schemas. โ lowkey important. ๐
spack.schema.environment.parse(config_obj)
Returns an EnvironmentModifications object containing the modifications parsed from input. โ she's parsing, bestie. ๐ง
Parameters
config_objโ a configuration dictionary conforming to the schema definition for environment modifications
๐ฆ spack.schema.gitlab_ci module
Schema for gitlab-ci.yaml configuration file. โ CI but make it cute. ๐
image_schema = {
"oneOf": [
{"type": "string"},
{
"type": "object",
"properties": {
"name": {"type": "string"},
"entrypoint": {"type": "array", "items": {"type": "string"}},
},
},
]
}
runner_attributes_schema_items = {
"image": image_schema,
"tags": {"type": "array", "items": {"type": "string"}},
"variables": {"type": "object", "patternProperties": {r"[\w\d\-_\.]+": {"type": "string"}}},
"before_script": {"type": "array", "items": {"type": "string"}},
"script": {"type": "array", "items": {"type": "string"}},
"after_script": {"type": "array", "items": {"type": "string"}},
}
runner_selector_schema = {
"type": "object",
"additionalProperties": True,
"required": ["tags"],
"properties": runner_attributes_schema_items,
}
remove_attributes_schema = {
"type": "object",
"additionalProperties": False,
"required": ["tags"],
"properties": {"tags": {"type": "array", "items": {"type": "string"}}},
}and yes, that's the exact remove_attributes_schema โ periodt. ๐
โฆand we're back inside core_shared_properties, where the schema keeps unioning things like it's collecting stickers ๐
core_shared_properties = union_dicts(
runner_attributes_schema_items,
{
"bootstrap": {
"type": "array",
"items": {
"anyOf": [
{"type": "string"},
{
"type": "object",
"additionalProperties": False,
"required": ["name"],
"properties": {
"name": {"type": "string"},
"compiler-agnostic": {"type": "boolean", "default": False},
},
},
]
},
},
"match_behavior": {"type": "string", "enum": ["first", "merge"], "default": "first"},
"mappings": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": False,
"required": ["match"],
"properties": {
"match": {"type": "array", "items": {"type": "string"}},
"remove-attributes": remove_attributes_schema,
"runner-attributes": runner_selector_schema,
},
},
},
"service-job-attributes": runner_selector_schema,
"signing-job-attributes": runner_selector_schema,
"rebuild-index": {"type": "boolean"},
"broken-specs-url": {"type": "string"},
"broken-tests-packages": {"type": "array", "items": {"type": "string"}},
},
)
gitlab_ci_properties = {
"anyOf": [
{
"type": "object",
"additionalProperties": False,
"required": ["mappings"],
"properties": union_dicts(
core_shared_properties, {"enable-artifacts-buildcache": {"type": "boolean"}}
),
},
{
"type": "object",
"additionalProperties": False,
"required": ["mappings"],
"properties": union_dicts(
core_shared_properties, {"temporary-storage-url-prefix": {"type": "string"}}
),
},
]
}
#: Properties for inclusion in other schemas
properties: Dict[str, Any] = {"gitlab-ci": gitlab_ci_properties}
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack gitlab-ci configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}okay so bestie, vibe check on what just happened: core_shared_properties is the base pile of gitlab-ci keys โจ
bootstrapโ an array, and each item is either a plain string or an object with a requirednameand an optionalcompiler-agnosticboolean that defaults toFalse. Very "pick your fighter" ๐match_behaviorโ a string, and the only legal vibes are"first"or"merge", defaulting to"first".mappingsโ an array of objects, each one required to havematch(an array of strings), plus optionalremove-attributesandrunner-attributes(those are theremove_attributes_schemaandrunner_selector_schemavariables, no cap).service-job-attributesandsigning-job-attributesโ both just point atrunner_selector_schema, iconic reuse ๐ซถrebuild-indexโ boolean, that's it, that's the whole personality.broken-specs-urlโ a string.broken-tests-packagesโ an array of strings.
then gitlab_ci_properties is an anyOf over two shapes, and BOTH of them:
- are
"type": "object" - have
"additionalProperties": False(no sneaky extra keys, periodt ๐) - require
"mappings" - and spread in
core_shared_propertiesviaunion_dicts(...)
the difference is literally one key each:
- shape one adds
enable-artifacts-buildcacheโ boolean - shape two adds
temporary-storage-url-prefixโ string
then properties: Dict[str, Any] = {"gitlab-ci": gitlab_ci_properties} wraps the whole thing up under the key gitlab-ci, and the full schema adds the metadata:
"$schema": "http://json-schema.org/draft-07/schema#""title": "Spack gitlab-ci configuration file schema""type": "object""additionalProperties": False"properties": properties
now for the fully expanded, every-brace-spelled-out version โ this is what the dict actually looks like once everything's been unioned and serialized, and yes it's giving "wall of text" but she's beautiful ๐ญ
spack.schema.gitlab_ci.properties: Dict[str, Any] = {'gitlab-ci':
{'anyOf': [{'additionalProperties': False, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'bootstrap': {'items': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'compiler-agnostic':
{'default': False, 'type': 'boolean'}, 'name': {'type': 'string'}},
'required': ['name'], 'type': 'object'}]}, 'type': 'array'},
'broken-specs-url': {'type': 'string'}, 'broken-tests-packages':
{'items': {'type': 'string'}, 'type': 'array'},
'enable-artifacts-buildcache': {'type': 'boolean'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'script': {'items': {'type': 'string'},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'},
'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type':
'string'}}, 'type': 'object'}}, 'required': ['mappings'], 'type':
'object'}, {'additionalProperties': False, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'bootstrap': {'items': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'compiler-agnostic':
{'default': False, 'type': 'boolean'}, 'name': {'type': 'string'}},
'required': ['name'], 'type': 'object'}]}, 'type': 'array'},
'broken-specs-url': {'type': 'string'}, 'broken-tests-packages':
{'items': {'type': 'string'}, 'type': 'array'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'script': {'items': {'type': 'string'},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'},
'temporary-storage-url-prefix': {'type': 'string'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['mappings'], 'type': 'object'}]}}
Properties for inclusion in other schemaslet me walk you through the extra keys that showed up in the fully-expanded version, because the expanded form reveals stuff the compact one hid ๐
after_scriptโ{'items': {'type': 'string'}, 'type': 'array'}โ array of strings โจbefore_scriptโ same deal, array of strings.imageโ'oneOf'on either a plainstring, or an object withentrypoint(array of strings) andname(string). two moods, both valid ๐scriptโ array of strings, obviously.tagsโ array of strings, and insiderunner-attributes,service-job-attributesandsigning-job-attributesit is required no cap.variablesโ an object whosepatternPropertieskey is[\\w\\d\\-_\\.]+, and every value there is a string. so like, word chars, digits, dash, underscore, dot โ that's the whole name alphabet, periodt ๐ธ
and the nested runner-attributes / service-job-attributes / signing-job-attributes objects are 'additionalProperties': True (unlike their parent mappings entries which are 'additionalProperties': False), and they carry after_script, before_script, image, script, tags, variables, with tags required ๐ซถ
meanwhile remove-attributes is 'additionalProperties': False with tags required and only tags allowed โ she's strict, she's small, she's the moment ๐
and the very last line, the docstring note:
Properties for inclusion in other schemasthat's it for this slice โ there's still more schema goodness coming, so don't you dare close the tab ๐ญ๐ฅ
okay so the tail end of this schema is literally the whole thing spelled out for you, bestie โ no summaries, no vibes-only version, the actual keys. ๐ Grab a snack, it's a big one:
spack.schema.gitlab_ci.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'gitlab-ci': {'anyOf': [{'additionalProperties':
False, 'properties': {'after_script': {'items': {'type': 'string'},
'type': 'array'}, 'before_script': {'items': {'type': 'string'}, 'type':
'array'}, 'bootstrap': {'items': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'compiler-agnostic':
{'default': False, 'type': 'boolean'}, 'name': {'type': 'string'}},
'required': ['name'], 'type': 'object'}]}, 'type': 'array'},
'broken-specs-url': {'type': 'string'}, 'broken-tests-packages':
{'items': {'type': 'string'}, 'type': 'array'},
'enable-artifacts-buildcache': {'type': 'boolean'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items': {'type':
'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'},
{'properties': {'entrypoint': {'items': {'type': 'string'}, 'type':
'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script':
{'items': {'type': 'string'}, 'type': 'array'}, 'tags': {'items':
{'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items': {'type':
'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'},
{'properties': {'entrypoint': {'items': {'type': 'string'}, 'type':
'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script':
{'items': {'type': 'string'}, 'type': 'array'}, 'tags': {'items':
{'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'script': {'items': {'type': 'string'}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}, 'tags': {'items':
{'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['mappings'], 'type': 'object'},
{'additionalProperties': False, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items': {'type':
'string'}, 'type': 'array'}, 'bootstrap': {'items': {'anyOf': [{'type':
'string'}, {'additionalProperties': False, 'properties':
{'compiler-agnostic': {'default': False, 'type': 'boolean'}, 'name':
{'type': 'string'}}, 'required': ['name'], 'type': 'object'}]}, 'type':
'array'}, 'broken-specs-url': {'type': 'string'},
'broken-tests-packages': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'mappings': {'items':
{'additionalProperties': False, 'properties': {'match': {'items':
{'type': 'string'}, 'type': 'array'}, 'remove-attributes':
{'additionalProperties': False, 'properties': {'tags': {'items': {'type':
'string'}, 'type': 'array'}}, 'required': ['tags'], 'type': 'object'},
'runner-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'script': {'items': {'type': 'string'}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items': {'type':
'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'},
{'properties': {'entrypoint': {'items': {'type': 'string'}, 'type':
'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script':
{'items': {'type': 'string'}, 'type': 'array'}, 'tags': {'items':
{'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'script': {'items': {'type': 'string'}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}, 'tags': {'items':
{'type': 'string'}, 'type': 'array'}, 'temporary-storage-url-prefix':
{'type': 'string'}, 'variables': {'patternProperties':
{'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'required':
['mappings'], 'type': 'object'}]}}, 'title': 'Spack gitlab-ci
configuration file schema', 'type': 'object'}Full schema with metadata โจ (that's literally the one-liner, she's not being shady, that's just what it says ๐ซถ)
๐ spack.schema.merged module
okay so this one is the merge era module โ schema for configuration merged into one file. it literally just smooshes all your fave schemas together with union_dicts and calls it a day. iconic behavior honestly ๐
properties: Dict[str, Any] = union_dicts(
spack.schema.bootstrap.properties,
spack.schema.cdash.properties,
spack.schema.compilers.properties,
spack.schema.concretizer.properties,
spack.schema.config.properties,
spack.schema.container.properties,
spack.schema.ci.properties,
spack.schema.definitions.properties,
spack.schema.develop.properties,
spack.schema.mirrors.properties,
spack.schema.modules.properties,
spack.schema.packages.properties,
spack.schema.repos.properties,
spack.schema.upstreams.properties,
spack.schema.view.properties,
)
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack merged configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}the way this works: every one of those properties dicts gets union'd into one giant properties dict, and that becomes the properties key of the merged schema. additionalProperties is False so random extra keys are a no-no, and the title is "Spack merged configuration file schema" โ periodt. ๐
no cap, that's the whole module โ fifteen schemas walk into a union_dicts() call and only one schema walks out. ๐ฆ
okay so we're deep in the schema dump now, and this chunk is the actual payload โ spack.schema.merged.properties itself, and it's typed Dict[str, Any] bestie, meaning "it's a dict, don't ask more questions" ๐ญ๐
here's the vibe check on what's chillin' in this part of the dict, before you read the whole thing:
bootstrapโ a whole little object withenable('boolean'),root('string'),sources(an array of objects withmetadata+name, both strings, both required, andadditionalProperties: Falseso no sneaky extras), andtrusted, which usespatternPropertieswith'\\w[\\w-]*'mapping to'boolean'๐ฅcdashโadditionalProperties: False,patternPropertiesforbuild-group,project,site, andurl(all'string'), andbuild-groupis the one thing markedrequiredโจciโ this is the drama section, it's aoneOfcontaininganyOfs, and it carriesbroken-specs-url(string),broken-tests-packages(array of strings),enable-artifacts-buildcache(boolean), andpipeline-gen๐
and inside pipeline-gen we have items โ a oneOf โ objects with match_behavior (which has 'default': 'first' and 'enum': ['first', 'merge'] ๐ณ) plus submapping, where submapping is an array of objects that each require match, and can hold build-job, build-job-remove, noop-job, noop-job-remove, copy-job, copy-job-remove, reindex-job, reindex-job-remove, signing-job, and signing-job-remove ๐ซถ
and every single one of those job objects has literally the same shape: after_script, before_script, image, script, tags, variables, with additionalProperties: True on the -job variants so you can add more stuff to 'em. the -remove variants are the same deal. match is items: {'type': 'string'} with 'type': 'array', and required: ['match']. submapping is required: ['submapping']. image is a oneOf between a plain string and an object with entrypoint (array of strings) + name (string) ๐ธ
kk here's the whole thing verbatim, no cap, every character accounted for ๐ง
spack.schema.merged.properties: Dict[str, Any] = {'bootstrap':
{'properties': {'enable': {'type': 'boolean'}, 'root': {'type':
'string'}, 'sources': {'items': {'additionalProperties': False,
'properties': {'metadata': {'type': 'string'}, 'name': {'type':
'string'}}, 'required': ['name', 'metadata'], 'type': 'object'}, 'type':
'array'}, 'trusted': {'patternProperties': {'\\w[\\w-]*': {'type':
'boolean'}}, 'type': 'object'}}, 'type': 'object'}, 'cdash':
{'additionalProperties': False, 'patternProperties': {'build-group':
{'type': 'string'}, 'project': {'type': 'string'}, 'site': {'type':
'string'}, 'url': {'type': 'string'}}, 'required': ['build-group'],
'type': 'object'}, 'ci': {'oneOf': [{'anyOf': [{'additionalProperties':
False, 'properties': {'broken-specs-url': {'type': 'string'},
'broken-tests-packages': {'items': {'type': 'string'}, 'type': 'array'},
'enable-artifacts-buildcache': {'type': 'boolean'}, 'pipeline-gen':
{'items': {'oneOf': [{'additionalProperties': False, 'properties':
{'match_behavior': {'default': 'first', 'enum': ['first', 'merge'],
'type': 'string'}, 'submapping': {'items': {'additionalProperties':
False, 'properties': {'build-job': {'additionalProperties': True,
'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'build-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'match':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['match'],
'type': 'object'}, 'type': 'array'}}, 'required': ['submapping'],
'type': 'object'}, {'oneOf': [{'additionalProperties': False,
'properties': {'noop-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'noop-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type':
'object'}, {'additionalProperties': False, 'properties': {'build-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'build-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'copy-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'copy-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'reindex-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'reindex-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'signing-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'signing-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':and yeah โ we literally stop mid-object at 'variables': ๐ the dict is still going, so hold that thought, bestie, because the next chunk picks it up right there ๐
okay so we're picking up mid-schema and honestly this chunk is the thicc one ๐
โ it's the guts of pipeline-gen, the thing that tells Spack how to lay out its CI jobs. Every job type gets the same little DNA: after_script, before_script, image, script, tags, variables. Watched. No notes. Here's the raw thing, character for character, no cap:
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'cleanup-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'cleanup-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'any-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'any-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'}]}]}, 'type': 'array'},
'rebuild-index': {'type': 'boolean'}, 'target': {'default': 'gitlab',
'enum': ['gitlab'], 'type': 'string'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'broken-specs-url':
{'type': 'string'}, 'broken-tests-packages': {'items': {'type':
'string'}, 'type': 'array'}, 'pipeline-gen': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'submapping': {'items': {'additionalProperties': False, 'properties':
{'build-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'build-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'match':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['match'],
'type': 'object'}, 'type': 'array'}}, 'required': ['submapping'],
'type': 'object'}, {'oneOf': [{'additionalProperties': False,
'properties': {'noop-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'noop-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type':
'object'}, {'additionalProperties': False, 'properties': {'build-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'build-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'copy-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'copy-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'reindex-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':๐ง okay but what did we just read
Bestie, it's a JSON Schema. Not vibes-based. It's the literal contract for what you're allowed to put in the pipeline-gen config, and it's checking everything. Let me translate:
variablesis an object. Its keys have to match the pattern[\w\d\-_\.]+โ so word chars, digits, dashes, underscores, and dots โ and every value must be a string ({'type': 'string'}). No sneaky ints, no booleans, no lists. Strings only. Periodt.after_script,before_script, andscriptare arrays, and each item isanyOfa plain string or an array of strings. So you can nest, and the schema will not clutch its pearls about it.imageisoneOfa plain string or an object withentrypoint(array of strings) andname(string). Very "pick your fighter" of it.tagsis just an array of strings, gorge.- Each job object has
'additionalProperties': True(so extra keys slide right through ๐ซถ) while the outer wrapper has'additionalProperties': Falseโ meaning at that level, only the keys it names are allowed. That's the whole bit.
๐ the job roster, in order
The big oneOf list โ every entry is a little wrapper object that can hold exactly one job key:
cleanup-job(pluscleanup-job-removealongside it in the same object)any-job(plusany-job-remove)noop-job(plusnoop-job-remove)build-job(plusbuild-job-remove)copy-job(pluscopy-job-remove)reindex-jobโ and this one's still open when our chunk ends ๐ญ
And at that first outer level you also get:
rebuild-indexโ{'type': 'boolean'}. True or false, no in-between.targetโ{'default': 'gitlab', 'enum': ['gitlab'], 'type': 'string'}. Default isgitlab, and the only blessed value isgitlab. The enum is doing a vibe check and passing exactly one thing.
๐ and then the second top-level shape
Right after that, a whole other object shows up with 'additionalProperties': False, and its named keys are:
broken-specs-urlโ plain string.broken-tests-packagesโ array of strings.pipeline-genโ an array whose items areoneOfโฆ a bunch of things, and this is wheresubmappinglives.
For the submapping variant: the object only allows match_behavior, submapping, and the job keys. match_behavior has {'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'} โ so first or merge, defaulting to first. Iconic duo.
Inside submapping, each array item allows build-job, build-job-remove, match, and it's 'required': ['match'] โ so match (an array of strings) is mandatory, no exceptions, don't @ me. Then submapping itself is also required on the parent: 'required': ['submapping'].
๐ the same DNA, over and over
Every single job block that follows โ noop-job, noop-job-remove, build-job, build-job-remove, copy-job, copy-job-remove, reindex-job โ is literally a rerun of the exact same six properties with the exact same types. It's giving copy-paste but, like, schema copy-paste, which is the legal kind. Variables still string-only, still matching [\w\d\-_\.]+, forever and always ๐
And that's exactly where this chunk runs out โ the reindex-job block is still open, image is mid-oneOf, and the next part picks up the second half of that line like nothing happened. Stay tuned bestie โญ
okay so we're deep in the schema mines right now, bestie โ like, deep deep. no heading to repeat, no intro needed, just vibes and validation rules. this next stretch is literally the same job-shaped blob copy-pasted for every job slot the config knows about, plus their -remove twins (yes, every job gets a lil delete-button sibling, we love symmetry ๐
). dropping it here verbatim because every bracket is sacred:
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'reindex-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'signing-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'signing-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'cleanup-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'cleanup-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'any-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'any-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'}]}]}, 'type': 'array'},
'rebuild-index': {'type': 'boolean'}, 'target': {'default': 'gitlab',
'enum': ['gitlab'], 'type': 'string'}, 'temporary-storage-url-prefix':
{'type': 'string'}}, 'type': 'object'}]}, {'anyOf':
[{'additionalProperties': False, 'properties': {'after_script':
{'items': {'type': 'string'}, 'type': 'array'}, 'before_script':
{'items': {'type': 'string'}, 'type': 'array'}, 'bootstrap': {'items':
{'anyOf': [{'type': 'string'}, {'additionalProperties': False,
'properties': {'compiler-agnostic': {'default': False, 'type':
'boolean'}, 'name': {'type': 'string'}}, 'required': ['name'], 'type':
'object'}]}, 'type': 'array'}, 'broken-specs-url': {'type': 'string'},
'broken-tests-packages': {'items': {'type': 'string'}, 'type': 'array'},
'enable-artifacts-buildcache': {'type': 'boolean'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'script': {'items': {'type': 'string'},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'},
'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type':
'string'}}, 'type': 'object'}}, 'required': ['mappings'], 'type':
'object'}, {'additionalProperties': False, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'bootstrap': {'items': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'compiler-agnostic':
{'default': False, 'type': 'boolean'}, 'name': {'type': 'string'}},
'required': ['name'], 'type': 'object'}]}, 'type': 'array'},
'broken-specs-url': {'type': 'string'}, 'broken-tests-packages':
{'items': {'type': 'string'}, 'type': 'array'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':okay. take a breath. that was a lot of brackets and I read every one of them so you don't have to ๐ซถ
here's the tea on what you just scrolled past, because it's genuinely not that scary:
- the job shapes โ the blobs near the top are job schemas named
reindex-job,reindex-job-remove,signing-job,signing-job-remove,cleanup-job,cleanup-job-remove,any-job, andany-job-remove. every single one of them allows the same handful of keys:after_script,before_script,image,script,tags,variables. and each of those job objects is marked'additionalProperties': True, so extra keys are welcome, no gatekeeping. - their
-removetwins โ same keys, same'additionalProperties': True, just the delete-flavoured version of each job. iconic of them to keep the pairs matching โจ - what those shared keys accept โ
after_script,before_script, andscriptare arrays whose items are either a plain string or an array of strings (that's the'anyOf'situation, lowkey the most repeated snippet in this whole page).tagsis an array of strings.variablesmaps keys matching[\\w\\d\\-_\\.]+to strings. - images โ
imageisoneOfa plain string or an object withentrypoint(an array of strings) andname(a string). pick your fighter ๐ - the three stray keys โ after the jobs array closes you get
rebuild-index(a boolean),target(a string,'default': 'gitlab', and the only legal value in the enum isgitlabbestie, it's very committed), andtemporary-storage-url-prefix(a string). - the second branch of the
anyOf</har> โ new object,'additionalProperties': False(so this one is strict, no surprise keys), and its keys areafter_script,before_script,bootstrap,broken-specs-url,broken-tests-packages,enable-artifacts-buildcache,image,mappings,match_behavior,rebuild-index,script,service-job-attributes,signing-job-attributes,tags,variables. bootstrapโ an array of items that are either strings or objects withname(string) pluscompiler-agnostic(boolean,'default': False), andnameis required ('required': ['name']).mappingsโ an array of strict objects. each one needsmatch(an array of strings, and yes'required': ['match']). then there'sremove-attributes: strict, withtags(array of strings) and'required': ['tags']. plusrunner-attributes(that one's'additionalProperties': True) holding the usualafter_script,before_script,image,script,tags,variablesโ with'required': ['tags'].match_behaviorโ string,'default': 'first', and the enum is exactly['first', 'merge']. choose wisely, no third option exists ๐service-job-attributesandsigning-job-attributesโ both'additionalProperties': True, both with theafter_script/before_script/image/script/tags/variablesset, both requiringtags.- the required bit โ that whole object insists on
'required': ['mappings']. no mappings, no entry, periodt. - and then the third block starts โ
'additionalProperties': Falseagain, same cast of keys returning, and it literally cuts off mid-imageat'image': {'oneOf': [{'type':because that's where this chunk ends ๐ซ
nothing here is dangerous, it's just a lot of validation. next part picks up right where that truncated oneOf left off, so don't let the dangling bracket stress you out ๐
Okay so we're still deep in the schema dump, and honestly? It's giving "artisanal JSON hand-crafted by someone who fears nothing" ๐ . Same vibe, same facts, zero details dropped โ here's the next chunk verbatim, then I'll walk you through the cute bits:
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'script': {'items': {'type': 'string'},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'},
'temporary-storage-url-prefix': {'type': 'string'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['mappings'], 'type': 'object'}]}]},
'compilers': {'items': {'additionalProperties': False, 'properties':
{'compiler': {'additionalProperties': False, 'properties': {'alias':
{'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'environment':
{'additionalProperties': False, 'default': {}, 'properties':
{'append_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'extra_rpaths': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}, 'flags': {'additionalProperties': False, 'properties':
{'cflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]},
'cppflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]},
'cxxflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'fflags':
{'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'ldflags': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'ldlibs': {'anyOf': [{'type':
'string'}, {'type': 'null'}]}}, 'type': 'object'}, 'implicit_rpaths':
{'anyOf': [{'items': {'type': 'string'}, 'type': 'array'}, {'type':
'boolean'}]}, 'modules': {'anyOf': [{'type': 'string'}, {'type':
'null'}, {'type': 'array'}]}, 'operating_system': {'type': 'string'},
'paths': {'additionalProperties': False, 'properties': {'cc': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'cxx': {'anyOf': [{'type':
'string'}, {'type': 'null'}]}, 'f77': {'anyOf': [{'type': 'string'},
{'type': 'null'}]}, 'fc': {'anyOf': [{'type': 'string'}, {'type':
'null'}]}}, 'required': ['cc', 'cxx', 'f77', 'fc'], 'type': 'object'},
'spec': {'type': 'string'}, 'target': {'type': 'string'}}, 'required':
['paths', 'spec', 'modules', 'operating_system'], 'type': 'object'}},
'type': 'object'}, 'type': 'array'}, 'concretizer':
{'additionalProperties': False, 'properties': {'duplicates':
{'properties': {'strategy': {'enum': ['none', 'minimal', 'full'],
'type': 'string'}}, 'type': 'object'}, 'enable_node_namespace': {'type':
'boolean'}, 'os_compatible': {'additionalProperties': {'type': 'array'},
'type': 'object'}, 'reuse': {'oneOf': [{'type': 'boolean'}, {'enum':
['dependencies'], 'type': 'string'}, {'properties': {'exclude':
{'items': {'type': 'string'}, 'type': 'array'}, 'from': {'items':
{'properties': {'exclude': {'items': {'type': 'string'}, 'type':
'array'}, 'include': {'items': {'type': 'string'}, 'type': 'array'},
'type': {'enum': ['local', 'buildcache', 'external'], 'type':
'string'}}, 'type': 'object'}, 'type': 'array'}, 'include': {'items':
{'type': 'string'}, 'type': 'array'}, 'roots': {'type': 'boolean'}},
'type': 'object'}]}, 'targets': {'properties': {'granularity': {'enum':
['generic', 'microarchitectures'], 'type': 'string'}, 'host_compatible':
{'type': 'boolean'}}, 'type': 'object'}, 'unify': {'oneOf': [{'type':
'boolean'}, {'enum': ['when_possible'], 'type': 'string'}]}}, 'type':
'object'}, 'config': {'default': {}, 'deprecatedProperties': {'error':
False, 'message': 'config:terminal_title has been replaced by
install_status and is ignored', 'properties': ['terminal_title']},
'properties': {'additional_external_search_paths': {'items': {'type':
'string'}, 'type': 'array'}, 'aliases': {'patternProperties':
{'\\w[\\w-]*': {'type': 'string'}}, 'type': 'object'}, 'allow_sgid':
{'type': 'boolean'}, 'binary_index_root': {'type': 'string'},
'binary_index_ttl': {'minimum': 0, 'type': 'integer'}, 'build_jobs':
{'minimum': 1, 'type': 'integer'}, 'build_language': {'type': 'string'},
'build_stage': {'oneOf': [{'type': 'string'}, {'items': {'type':
'string'}, 'type': 'array'}]}, 'ccache': {'type': 'boolean'},
'checksum': {'type': 'boolean'}, 'concretizer': {'enum': ['original',
'clingo'], 'type': 'string'}, 'connect_timeout': {'minimum': 0, 'type':
'integer'}, 'db_lock_timeout': {'minimum': 1, 'type': 'integer'},
'debug': {'type': 'boolean'}, 'deprecated': {'type': 'boolean'},
'develop_stage_link': {'type': 'string'}, 'dirty': {'type': 'boolean'},
'environments_root': {'type': 'string'}, 'extensions': {'items':
{'type': 'string'}, 'type': 'array'}, 'flags': {'properties':
{'keep_werror': {'enum': ['all', 'specific', 'none'], 'type':
'string'}}, 'type': 'object'}, 'install_hash_length': {'minimum': 1,
'type': 'integer'}, 'install_missing_compilers': {'type': 'boolean'},
'install_path_scheme': {'type': 'string'}, 'install_status': {'type':
'boolean'}, 'install_tree': {'anyOf': [{'properties': {'padded_length':
{'oneOf': [{'minimum': 0, 'type': 'integer'}, {'type': 'boolean'}]},
'projections': {'patternProperties': {'all|\\w[\\w-]*': {'type':
'string'}}, 'type': 'object'}, 'root': {'type': 'string'}}, 'type':
'object'}, {'type': 'string'}]}, 'license_dir': {'type': 'string'},
'locks': {'type': 'boolean'}, 'misc_cache': {'type': 'string'},
'package_lock_timeout': {'anyOf': [{'minimum': 1, 'type': 'integer'},
{'type': 'null'}]}, 'shared_linking': {'anyOf': [{'enum': ['rpath',
'runpath'], 'type': 'string'}, {'properties': {'bind': {'type':
'boolean'}, 'type': {'enum': ['rpath', 'runpath'], 'type': 'string'}},
'type': 'object'}]}, 'source_cache': {'type': 'string'}, 'ssl_certs':
{'type': 'string'}, 'stage_name': {'type': 'string'},
'suppress_gpg_warnings': {'type': 'boolean'}, 'template_dirs': {'items':
{'type': 'string'}, 'type': 'array'}, 'test_stage': {'type': 'string'},
'url_fetch_method': {'enum': ['urllib', 'curl'], 'type': 'string'},
'verify_ssl': {'type': 'boolean'}}, 'type': 'object'}, 'container':
{'additionalProperties': False, 'properties': {'depfile': {'default':
False, 'type': 'boolean'}, 'docker': {'additionalProperties': False,
'default': {}, 'type': 'object'}, 'format': {'enum': ['docker',
'singularity'], 'type': 'string'}, 'images': {'anyOf':
[{'additionalProperties': False, 'properties': {'os': {'type':
'string'}, 'spack': {'anyOf': [{'type': 'string'},
{'additional_properties': False, 'properties': {'ref': {'type':
'string'}, 'resolve_sha': {'default': False, 'type': 'boolean'}, 'url':
{'type': 'string'}, 'verify': {'default': False, 'type': 'boolean'}},
'type': 'object'}]}}, 'required': ['os', 'spack'], 'type': 'object'},
{'additionalProperties': False, 'properties': {'build': {'type':
'string'}, 'final': {'type': 'string'}}, 'required': ['build', 'final'],
'type': 'object'}]}, 'labels': {'type': 'object'}, 'os_packages':
{'additionalProperties': False, 'properties': {'build': {'items':
{'type': 'string'}, 'type': 'array'}, 'command': {'enum': ['apt', 'yum',
'zypper', 'apk', 'yum_amazon'], 'type': 'string'}, 'final': {'items':
{'type': 'string'}, 'type': 'array'}, 'update': {'type': 'boolean'}},
'type': 'object'}, 'singularity': {'additionalProperties': False,
'default': {}, 'properties': {'help': {'type': 'string'}, 'runscript':
{'type': 'string'}, 'startscript': {'type': 'string'}, 'test': {'type':
'string'}}, 'type': 'object'}, 'strip': {'default': True, 'type':
'boolean'}, 'template': {'default': None, 'type': 'string'}}, 'type':
'object'}, 'definitions': {'default': [], 'items': {'patternProperties':
{'^(?!when$)\\w*': {'default': [], 'items': {'anyOf':
[{'additionalProperties': False, 'properties': {'exclude': {'items':
{'type': 'string'}, 'type': 'array'}, 'matrix': {'items': {'items':
{'type': 'string'}, 'type': 'array'}, 'type': 'array'}}, 'type':
'object'}, {'type': 'string'}, {'type': 'null'}]}, 'type': 'array'}},
'properties': {'when': {'type': 'string'}}, 'type': 'object'}, 'type':
'array'}, 'develop': {'additionalProperties': False, 'default': {},
'patternProperties': {'\\w[\\w-]*': {'additionalProperties': False,
'properties': {'path': {'type': 'string'}, 'spec': {'type': 'string'}},
'type': 'object'}}, 'type': 'object'}, 'mirrors':
{'additionalProperties': False, 'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'additionalProperties':
False, 'anyOf': [{'required': ['url']}, {'required': ['fetch']},
{'required': ['pull']}], 'properties': {'access_pair': {'items':
{'maxItems': 2, 'minItems': 2, 'type': ['string', 'null']}, 'type':
'array'}, 'access_token': {'type': ['string', 'null']}, 'autopush':
{'type': 'boolean'}, 'binary': {'type': 'boolean'}, 'endpoint_url':
{'type': ['string', 'null']}, 'fetch': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'access_pair': {'items':
{'maxItems': 2, 'minItems': 2, 'type': ['string', 'null']}, 'type':๐ซถ what you're actually looking at, bestie
So this whole chunk is the Spack configuration schema โ the shape of the config.yaml-adjacent stuff, described in JSON Schema. Let me point at the icons โจ:
match_behaviorโ has adefaultof'first', and the only allowedenumvalues are'first'and'merge'. Pick your fighter, bestie.rebuild-indexโ plain'type': 'boolean', no default listed here, don't @ me.scriptโ'items': {'type': 'string'}, 'type': 'array', i.e. an array of strings.service-job-attributesandsigning-job-attributesโ both areadditionalProperties: True, bothrequired: ['tags'], and both can carryafter_script,before_script,script,tags,variables, and animagethat'soneOfeither a string or an object withentrypoint(array of strings) andname(string). Iconic CI energy ๐ .variablesโpatternPropertieswhere the key regex is[\\w\\d\\-_\\.]+and the value is a string. Chef's kiss.temporary-storage-url-prefixโ just a'type': 'string', living its best life.- The whole thing has
'required': ['mappings'], so if you skipmappingsthe schema will simply Not Be Having It.
๐ง the compilers block, aka the whole personality
Lowkey the star of the show:
- Each entry has
additionalProperties: Falseand a nestedcompilerobject โ same deal,Falsefor extras. aliasโanyOfstring ornull.environmentโdefault: {}, with the full menu:append_path,prepend_path,remove_path,setโ allpatternPropertieson\\w[\\w-]*, valuesanyOfstring or number.unsetโdefault: [],itemsanyOfstring or number,typearray.
extra_rpathsโdefault: [], array of strings.flagsโadditionalProperties: False, withcflags,cppflags,cxxflags,fflags,ldflags,ldlibs, eachanyOfstring ornull.implicit_rpathsโanyOfan array of strings or a boolean.modulesโanyOfstring,null, or array.operating_systemโ string.pathsโadditionalProperties: False, requires all four ofcc,cxx,f77,fc(required: ['cc', 'cxx', 'f77', 'fc']), eachanyOfstring ornull.specandtargetโ both strings.- Top level of a compiler entry:
required: ['paths', 'spec', 'modules', 'operating_system']. So those four are non-negotiable ๐.
๐ฅ concretizer (say it with your chest)
duplicatesโstrategywithenum: ['none', 'minimal', 'full'].enable_node_namespaceโ boolean.os_compatibleโ object whose values are arrays.reuseโoneOf:- a boolean, or
- the string
'dependencies'(enum: ['dependencies']), or - an object with
exclude(array of strings),from(array of objects withexclude,include, andtypefromenum: ['local', 'buildcache', 'external']),include(array of strings), androots(boolean).
targetsโgranularityisenum: ['generic', 'microarchitectures'], plushost_compatibleboolean.unifyโoneOfboolean orenum: ['when_possible'].
All wrapped in additionalProperties: False, so no surprise keys, periodt.
๐ config โ the big one, and yes it has drama
First up, deprecatedProperties is yelling at you (lovingly):
'error': False,'message': 'config:terminal_title has been replaced by install_status and is ignored','properties': ['terminal_title']
Translation? terminal_title is deprecated, replaced by install_status, and it gets ignored โ not an error, just ignored. Noted ๐.
Then the properties list, and it is long: additional_external_search_paths (array of strings), aliases (patternProperties on \\w[\\w-]* โ string), allow_sgid (boolean), binary_index_root (string), binary_index_ttl (minimum: 0, integer), build_jobs (minimum: 1, integer), build_language (string), build_stage (oneOf string or array of strings), ccache (boolean), checksum (boolean), concretizer (enum: ['original', 'clingo'] โ yes, just those two), connect_timeout (minimum: 0, integer), db_lock_timeout (minimum: 1, integer), debug (boolean), deprecated (boolean), develop_stage_link (string), dirty (boolean), environments_root (string), extensions (array of strings), flags with keep_werror (enum: ['all', 'specific', 'none']), install_hash_length (minimum: 1, integer), install_missing_compilers (boolean), install_path_scheme (string), install_status (boolean), install_tree (anyOf object-or-string, where the object has padded_length as oneOf a minimum: 0 integer or a boolean, projections with patternProperties all|\\w[\\w-]* โ string, and root string), license_dir (string), locks (boolean), misc_cache (string), package_lock_timeout (anyOf minimum: 1 integer or null), shared_linking (anyOf enum: ['rpath', 'runpath'] string, or an object with bind boolean and type enum: ['rpath', 'runpath']), source_cache (string), ssl_certs (string), stage_name (string), suppress_gpg_warnings (boolean), template_dirs (array of strings), test_stage (string), url_fetch_method (enum: ['urllib', 'curl']), and verify_ssl (boolean). Whew. Obsessed with the completeness though ๐ซถ.
๐ container โ docker girlies rise
depfileโdefault: False, boolean.dockerโadditionalProperties: False,default: {}.formatโenum: ['docker', 'singularity'].imagesโanyOfan object requiring['os', 'spack'](wherespackisanyOfa string or an object withref,resolve_shadefaultFalse,url,verifydefaultFalse), or an object requiring['build', 'final'](both strings).labelsโ object.os_packagesโadditionalProperties: False, withbuild(array of strings),command(enum: ['apt', 'yum', 'zypper', 'apk', 'yum_amazon']),final(array of strings),update(boolean).singularityโadditionalProperties: False,default: {}, propertieshelp,runscript,startscript,test, all strings.stripโdefault: True, boolean.templateโdefault: None, string.
๐ฆ and the stragglers
definitionsโdefault: [], withpatternPropertiesmatching^(?!when$)\\w*. Each valuedefault: [], itemsanyOfan object (additionalProperties: False, withexcludearray-of-strings andmatrixas an array of arrays of strings), a string, ornull. Plus awhenstring at the top level.developโadditionalProperties: False,default: {}, withpatternProperties\\w[\\w-]*, each entryadditionalProperties: Falseand holdingpath(string) andspec(string).mirrorsโadditionalProperties: False,default: {},patternPropertieson\\w[\\w-]*, where each mirror isanyOfa string or an object. That object isadditionalProperties: Falseand wantsanyOfone of{'required': ['url']},{'required': ['fetch']}, or{'required': ['pull']}โ at least one of the three, bestie. Itspropertiesincludeaccess_pair(itemswithmaxItems: 2,minItems: 2,type: ['string', 'null']),access_token(type: ['string', 'null']),autopush(boolean),binary(boolean),endpoint_url(type: ['string', 'null']), andfetchโ which isanyOfa string or anotheradditionalProperties: Falseobject that starts the wholeaccess_pair,access_token, โฆ pattern over again, and the chunk cuts off right there mid-access_pair. Not me ending on a cliffhanger ๐ญ.
okay bestie so this chunk is pure schema-core ๐
โ we're still deep inside the modules section of the config schema, and it's giving "wall of JSON that ate and left no crumbs, but also ate my whole afternoon." no cap, just read it slow, it's tcl and lmod module config all the way down, with prefix_inspections and roots sitting in the middle like the main characters they are. ๐ซถ
word of warning before you scroll: this is verbatim, every quote and backslash exactly where the source put it, and it starts mid-structure because we're picking up from the last part. don't try to "fix" the indentation, don't try to balance the braces, that's not our job today ๐ญ
'array'}, 'access_token': {'type': ['string', 'null']}, 'endpoint_url':
{'type': ['string', 'null']}, 'profile': {'type': ['string', 'null']},
'url': {'type': 'string'}}, 'type': 'object'}]}, 'profile': {'type':
['string', 'null']}, 'push': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'access_pair': {'items':
{'maxItems': 2, 'minItems': 2, 'type': ['string', 'null']}, 'type':
'array'}, 'access_token': {'type': ['string', 'null']}, 'endpoint_url':
{'type': ['string', 'null']}, 'profile': {'type': ['string', 'null']},
'url': {'type': 'string'}}, 'type': 'object'}]}, 'signed': {'type':
'boolean'}, 'source': {'type': 'boolean'}, 'url': {'type': 'string'}},
'type': 'object'}]}}, 'type': 'object'}, 'modules':
{'additionalProperties': False, 'patternProperties':
{'^(?!prefix_inspections$)\\w[\\w-]*$': {'additionalProperties': False,
'default': {}, 'properties': {'arch_folder': {'type': 'boolean'},
'enable': {'default': [], 'items': {'enum': ['tcl', 'lmod'], 'type':
'string'}, 'type': 'array'}, 'lmod': {'allOf': [{'allOf':
[{'properties': {'all': {'additionalProperties': False, 'default': {},
'properties': {'autoload': {'enum': ['none', 'run', 'direct', 'all'],
'type': 'string'}, 'conflict': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'environment': {'additionalProperties':
False, 'default': {}, 'properties': {'append_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'filter': {'additionalProperties': False, 'default': {}, 'properties':
{'exclude_env_vars': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}, 'load': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'prerequisites': {'enum': ['none',
'run', 'direct', 'all'], 'type': 'string'}, 'suffixes':
{'patternProperties': {'\\w[\\w-]*': {'type': 'string'}}, 'type':
'object', 'validate_spec': True}, 'template': {'type': 'string'}},
'type': 'object'}, 'defaults': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'exclude': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'exclude_implicits': {'default':
False, 'type': 'boolean'}, 'hash_length': {'default': 7, 'minimum': 0,
'type': 'integer'}, 'hide_implicits': {'default': False, 'type':
'boolean'}, 'include': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'naming_scheme': {'type': 'string'}, 'projections':
{'patternProperties': {'all|\\w[\\w-]*': {'type': 'string'}}, 'type':
'object'}, 'verbose': {'default': False, 'type': 'boolean'}}},
{'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}, '^[\\^@%+~]':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}}, 'validate_spec': True}],
'default': {}, 'type': 'object'}, {'properties': {'core_compilers':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'core_specs': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}, 'filter_hierarchy_specs': {'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type':
'object'}, 'hierarchy': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}]}, 'prefix_inspections':
{'additionalProperties': False, 'patternProperties': {'^[\\w-]*':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type':
'object'}, 'roots': {'properties': {'lmod': {'type': 'string'}, 'tcl':
{'type': 'string'}}, 'type': 'object'}, 'tcl': {'allOf': [{'allOf':
[{'properties': {'all': {'additionalProperties': False, 'default': {},
'properties': {'autoload': {'enum': ['none', 'run', 'direct', 'all'],
'type': 'string'}, 'conflict': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'environment': {'additionalProperties':
False, 'default': {}, 'properties': {'append_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'filter': {'additionalProperties': False, 'default': {}, 'properties':
{'exclude_env_vars': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}, 'load': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'prerequisites': {'enum': ['none',
'run', 'direct', 'all'], 'type': 'string'}, 'suffixes':
{'patternProperties': {'\\w[\\w-]*': {'type': 'string'}}, 'type':
'object', 'validate_spec': True}, 'template': {'type': 'string'}},
'type': 'object'}, 'defaults': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'exclude': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'exclude_implicits': {'default':
False, 'type': 'boolean'}, 'hash_length': {'default': 7, 'minimum': 0,
'type': 'integer'}, 'hide_implicits': {'default': False, 'type':
'boolean'}, 'include': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'naming_scheme': {'type': 'string'}, 'projections':
{'patternProperties': {'all|\\w[\\w-]*': {'type': 'string'}}, 'type':
'object'}, 'verbose': {'default': False, 'type': 'boolean'}}},
{'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}, '^[\\^@%+~]':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,okay so quick vibe check on what you just scrolled past, because it's genuinely iconic once you clock the pattern ๐
- the
modulesblock only allows the keys matching^(?!prefix_inspections$)\\w[\\w-]*$โ that(?!prefix_inspections$)is a negative lookahead saying "anything word-shaped EXCEPT literalprefix_inspections", becauseprefix_inspectionsgets its own sibling definition right after. sneaky but slay ๐ enabletakes['tcl', 'lmod']and defaults to[]. so it's opt-in, no cap.hash_lengthdefaults to7with'minimum': 0โ yes bestie, zero is technically legal.verboseandhide_implicitsandexclude_implicitsare all booleans defaulting toFalse. humble defaults, honestly.autoloadandprerequisitesboth share that same enum:['none', 'run', 'direct', 'all']๐ฅ- inside
environmentyou've gotappend_path,prepend_path,remove_path,set(allpatternPropertieskeyed on\\w[\\w-]*, valuesstringornumber), plusunsetas an array of string-or-number. that's the whole env-manipulation roster, periodt. filteronly hasexclude_env_vars, an array of strings. small but mighty.- the
lmodandtclblocks both nest the sameall/defaults/exclude/exclude_implicits/hash_length/hide_implicits/include/naming_scheme/projections/verboseshape, and both carry'validate_spec': True. twins fr fr ๐ซถ - that big negative lookahead listing
hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|allis how per-compiler/per-spec module configs get matched without colliding with the reserved names. it's giving bouncer at the club. ๐ซ prefix_inspectionskeys match^[\\w-]*and map to arrays of strings (default[]), androotsis justlmod+tcl, both strings. tiny section, huge responsibility โจcore_compilers,core_specs, andhierarchyare all arrays of strings defaulting to[], andfilter_hierarchy_specskeys get the same reserved-name lookahead treatment. ๐ธ
and yeah โ the block above ends mid-line on 'environment': {'additionalProperties': False, because the source cuts there. not a typo on my end, bestie, we're literally mid-thought ๐ญ the next part picks it right back up, so don't close the tab ๐ฆ
fr fr don't hand-edit this schema trying to make it "prettier" โ the quoting, the escapes like \\w[\\w-]*, and that trailing comma situation are load-bearing. copy it exactly or your config validation will simply not validate, and then you'll be sad. ๐
โฆand the schema just keeps going, bestie โ no cap, we're still inside the same block. Here's the rest of it, character for character ๐ซถ
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}}, 'validate_spec': True}],
'default': {}, 'type': 'object'}, {}]}, 'use_view': {'anyOf': [{'type':
'string'}, {'type': 'boolean'}]}}, 'type': 'object'}}, 'properties':
{'prefix_inspections': {'additionalProperties': False,
'patternProperties': {'^[\\w-]*': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}}, 'type': 'object'}}, 'type': 'object'},
'packages': {'additionalProperties': False, 'default': {},
'patternProperties': {'(?!^all$)(^\\w[\\w-]*)': {'additionalProperties':
False, 'default': {}, 'deprecatedProperties': {'error': False,
'message': "setting 'compiler:', 'target:' or 'provider:' preferences in
a package-specific section of packages.yaml is deprecated, and will be
removed in v0.23.\n\n\tThese preferences will be ignored by Spack, and
can be set only in the 'all' section of the same file. You can
run:\n\n\t\t$ spack audit configs\n\n\tto get better diagnostics,
including files:lines where the deprecated attributes are used.\n\n\tUse
requirements to enforce conditions on specific packages:
https://spack.readthedocs.io/en/latest/packages_yaml.html#package-requirements\n",
'properties': ['target', 'compiler', 'providers']}, 'properties':
{'buildable': {'default': True, 'type': 'boolean'}, 'compiler': {},
'conflict': {'items': {'oneOf': [{'additionalProperties': False,
'properties': {'message': {'type': 'string'}, 'spec': {'type':
'string'}, 'when': {'type': 'string'}}, 'type': 'object'}, {'type':
'string'}]}, 'type': 'array'}, 'externals': {'items':
{'additionalProperties': True, 'properties': {'extra_attributes':
{'additionalProperties': True, 'properties': {'environment':
{'additionalProperties': False, 'default': {}, 'properties':
{'append_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}},
'type': 'object'}, 'modules': {'items': {'type': 'string'}, 'type':
'array'}, 'prefix': {'type': 'string'}, 'spec': {'type': 'string'}},
'required': ['spec'], 'type': 'object'}, 'type': 'array'},
'package_attributes': {'additionalProperties': False,
'patternProperties': {'\\w+': {}}, 'type': 'object'}, 'permissions':
{'additionalProperties': False, 'properties': {'group': {'type':
'string'}, 'read': {'enum': ['user', 'group', 'world'], 'type':
'string'}, 'write': {'enum': ['user', 'group', 'world'], 'type':
'string'}}, 'type': 'object'}, 'prefer': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'}, 'providers':
{}, 'require': {'oneOf': [{'items': {'oneOf': [{'additionalProperties':
False, 'properties': {'any_of': {'items': {'type': 'string'}, 'type':
'array'}, 'message': {'type': 'string'}, 'one_of': {'items': {'type':
'string'}, 'type': 'array'}, 'spec': {'type': 'string'}, 'when':
{'type': 'string'}}, 'type': 'object'}, {'type': 'string'}]}, 'type':
'array'}, {'type': 'string'}]}, 'target': {}, 'variants': {'oneOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'version': {'default': [], 'items': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}},
'properties': {'all': {'additionalProperties': False, 'default': {},
'deprecatedProperties': {'error': False, 'message': "setting version
preferences in the 'all' section of packages.yaml is deprecated and will
be removed in v0.23\n\n\tThese preferences will be ignored by Spack. You
can set them only in package-specific sections of the same file.\n",
'properties': ['version']}, 'properties': {'buildable': {'default':
True, 'type': 'boolean'}, 'compiler': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'conflict': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'},
'package_attributes': {'additionalProperties': False,
'patternProperties': {'\\w+': {}}, 'type': 'object'}, 'permissions':
{'additionalProperties': False, 'properties': {'group': {'type':
'string'}, 'read': {'enum': ['user', 'group', 'world'], 'type':
'string'}, 'write': {'enum': ['user', 'group', 'world'], 'type':
'string'}}, 'type': 'object'}, 'prefer': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'}, 'providers':
{'additionalProperties': False, 'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}}, 'type': 'object'}, 'require': {'oneOf': [{'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'any_of': {'items':
{'type': 'string'}, 'type': 'array'}, 'message': {'type': 'string'},
'one_of': {'items': {'type': 'string'}, 'type': 'array'}, 'spec':
{'type': 'string'}, 'when': {'type': 'string'}}, 'type': 'object'},
{'type': 'string'}]}, 'type': 'array'}, {'type': 'string'}]}, 'target':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'variants': {'oneOf': [{'type': 'string'}, {'items': {'type': 'string'},
'type': 'array'}]}, 'version': {}}, 'type': 'object'}}, 'type':
'object'}, 'repos': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}, 'upstreams': {'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'additionalProperties': False, 'default': {},
'properties': {'install_tree': {'type': 'string'}, 'modules':
{'properties': {'lmod': {'type': 'string'}, 'tcl': {'type': 'string'}},
'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}, 'view':
{'anyOf': [{'type': 'boolean'}, {'type': 'string'},
{'patternProperties': {'\\w+': {'additionalProperties': False,
'properties': {'exclude': {'items': {'type': 'string'}, 'type':
'array'}, 'link': {'pattern': '(roots|all|run)', 'type': 'string'},
'link_type': {'type': 'string'}, 'projections': {'patternProperties':
{'all|\\w[\\w-]*': {'type': 'string'}}, 'type': 'object'}, 'root':
{'type': 'string'}, 'select': {'items': {'type': 'string'}, 'type':
'array'}}, 'required': ['root']}}, 'type': 'object'}]}}Okay so real talk โ that was a lot, and if your eyes glazed over, that's so valid ๐ญ but like, let me point at the juicy bits so you know what you just scrolled past:
- The environment-modification shape you've been seeing everywhere (
append_path,prepend_path,remove_path,set,unset) makes another appearance, plusfilterwith itsexclude_env_vars(default[], array ofstring),load(default[], array ofstring),prerequisiteswith theenumof['none', 'run', 'direct', 'all'],suffixes,template, anduse_viewwhich isanyOfastringor aboolean. โจ prefix_inspectionsโ keys matching^[\\w-]*, each defaulting to[]as an array ofstring. ๐- The
packagessection, where things get spicy ๐ฅต โ it has adeprecatedPropertiesblock, and you should lowkey read thatmessage, because it's literally telling you: setting'compiler:','target:'or'provider:'preferences in a package-specific section ofpackages.yamlis deprecated, and will be removed in v0.23. Those preferences will be ignored by Spack and can be set only in the'all'section of the same file. It even tells you to run$ spack audit configsfor better diagnostics, including files:lines where the deprecated attributes are used, and points you at requirements for enforcing conditions on specific packages:https://spack.readthedocs.io/en/latest/packages_yaml.html#package-requirements. Iconic of the schema to just say that to you. ๐ซถ - And the
allsection has its own deprecation tea โ โ setting version preferences there is deprecated and will be removed in v0.23, those preferences will be ignored by Spack, and you can set them only in package-specific sections of the same file. The way the schema is out here reading us fr fr ๐ - You've also got
externals(withrequired: ['spec'], plusextra_attributes,modules,prefix, and anenvironmentblock),permissions(group, andread/writewithenum['user', 'group', 'world']),prefer,providers,require(withany_of/one_of/spec/when/messagegoodness),variants,buildabledefaulting toTrue,conflict,package_attributes,target, andcompiler. - Then
repos(default[], array ofstring),upstreams(withinstall_treeandmodulesโlmod/tcl), andview, which isanyOfaboolean, astring, or an object of named views where each hasexclude,linkmatching pattern(roots|all|run),link_type,projections,rootandselectโ withrequired: ['root'], no exceptions bestie. ๐ธ
And the line that closes it out, plain as day:
Properties for inclusion in other schemas ๐
That's the whole vibe of this chunk โ it's not a config file you write directly, it's schema scaffolding other schemas pull in. Very main character of it, honestly. ๐
get this โ 'match' is required inside each submapping, it's a type: 'array' of 'string's, and the whole submapping array is what's required on the outer object. Like, you literally cannot skip it. ๐ซถ
Then the 'oneOf' kicks in with the second branch โ the single-job shapes, each one with a regular key and a -remove twin:
- ๐
'noop-job'/'noop-job-remove'โ'additionalProperties': True, plus the same six properties:'after_script','before_script','image','script','tags','variables', all with the exact same types as above. - ๐
'build-job'/'build-job-remove'โ same deal. - ๐
'copy-job'/'copy-job-remove'โ same deal. - ๐
'reindex-job'/'reindex-job-remove'โ same deal. - ๐
'signing-job'/'signing-job-remove'โ same deal.
Now, bestie, I'm gonna show you the raw schema block itself because it's literally the source of truth and I refuse to paraphrase a single bracket of it. It's giving โจload-bearing punctuationโจ โ every ' and ] and } matters, so this is inside a code block, verbatim. Do not retype it by hand, I'm begging you ๐ญ
spack.schema.merged.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'bootstrap': {'properties': {'enable': {'type':
'boolean'}, 'root': {'type': 'string'}, 'sources': {'items':
{'additionalProperties': False, 'properties': {'metadata': {'type':
'string'}, 'name': {'type': 'string'}}, 'required': ['name',
'metadata'], 'type': 'object'}, 'type': 'array'}, 'trusted':
{'patternProperties': {'\\w[\\w-]*': {'type': 'boolean'}}, 'type':
'object'}}, 'type': 'object'}, 'cdash': {'additionalProperties': False,
'patternProperties': {'build-group': {'type': 'string'}, 'project':
{'type': 'string'}, 'site': {'type': 'string'}, 'url': {'type':
'string'}}, 'required': ['build-group'], 'type': 'object'}, 'ci':
{'oneOf': [{'anyOf': [{'additionalProperties': False, 'properties':
{'broken-specs-url': {'type': 'string'}, 'broken-tests-packages':
{'items': {'type': 'string'}, 'type': 'array'},
'enable-artifacts-buildcache': {'type': 'boolean'}, 'pipeline-gen':
{'items': {'oneOf': [{'additionalProperties': False, 'properties':
{'match_behavior': {'default': 'first', 'enum': ['first', 'merge'],
'type': 'string'}, 'submapping': {'items': {'additionalProperties':
False, 'properties': {'build-job': {'additionalProperties': True,
'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'build-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'match':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['match'],
'type': 'object'}, 'type': 'array'}}, 'required': ['submapping'],
'type': 'object'}, {'oneOf': [{'additionalProperties': False,
'properties': {'noop-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'noop-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type':
'object'}, {'additionalProperties': False, 'properties': {'build-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'build-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'copy-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'copy-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'reindex-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'reindex-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'signing-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'signing-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},So yeah, that's the block. Notice the pattern, bestie, because once you see it you can't unsee it ๐ โ every single job type is the same six keys:
'after_script'โ'type': 'array', items areanyOfa plain'string'or an array of strings. Both vibes accepted. ๐'before_script'โ identicalanyOfstring-or-array-of-strings situation.'image'โ'oneOf'a plain'string'or an object with'entrypoint'(array of strings) and'name'(string). Choose your fighter. ๐'script'โ'type': 'array', againanyOfstring or array-of-strings per item.'tags'โ'type': 'array','items': {'type': 'string'}.'variables'โ'type': 'object', and the'patternProperties'key is'[\\w\\d\\-_\\.]+'mapping to'type': 'string'. That regex is not a suggestion, it's the whole allowed-variable-name alphabet, so if your var is called something feral with a space in it, the schema will simply โจnotโจ.
โ ๏ธ Real talk though:'build-job','build-job-remove','noop-job','noop-job-remove','copy-job','copy-job-remove','reindex-job','reindex-job-remove','signing-job','signing-job-remove'โ these are two different branches of a'oneOf', and each branch pairs a job with its-removecounterpart. So you're describing a job either as "here's a build job and here's the thing that strips it back down" or as one of the other pairs. Not a free-for-all. The JSON schema will absolutely clock you ๐ซถ
Also worth clocking: the -remove variants all have 'additionalProperties': True too, so extra keys aren't being locked out at that level โ the schema is describing the keys it knows about, not forbidding the rest. That's the difference between "I validate this" and "I police everything," and this schema picked validate. ๐
And the top-level keys we've met so far in this properties blob: 'bootstrap', 'cdash', 'ci'. Each one is 'type': 'object', each one has its own little quirks โ 'cdash' is 'additionalProperties': False with 'required': ['build-group'], while 'bootstrap' has no required at all even though its 'sources' items demand both 'name' and 'metadata'. Sneaky. ๐
'type': 'object'}, {'oneOf': [{'additionalProperties': False, 'properties': {'noop-job': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\w\d\-_\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'noop-job-remove': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\w\d\-_\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}, {'additionalProperties': False, 'properties': {'build-job': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\w\d\-_\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'build-job-remove': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\w\d\-_\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}, {'additionalProperties': False, 'properties': {'copy-job': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\w\d\-_\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'copy-job-remove': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties': {'[\w\d\-_\.]+': {'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}, {'additionalProperties': False, 'properties': {'reindex-job': {'additionalProperties': True, 'properties': {'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
Now the prose. This is part 173, mid-schema. Sections already covered include BUILD SYSTEMS, DEVELOPER GUIDE, SPACK PACKAGE, and lots of table fragments. So I'm inside a JSON schema block (looks like it belongs to something like spack ci config / gitlab-ci schema from the docs). I should just continue with code blocks and commentary, no new headings (or maybe a soft heading? The instructions say if this part starts mid-section, carry on without repeating its heading). The "headings" covered list includes weird table fragments โ so I'm clearly in a code dump region. I'll keep it mostly code with light commentary and no big heading, maybe just a soft note. Adding an "###" heading might be odd; I'll skip headings and just do commentary lines between fences.
Commentary should be Gen Z, playful, but factual: this is JSON schema, describing job templates: cleanup-job, cleanup-job-remove, any-job, any-job-remove, match_behavior default first with enum first/merge, submapping array with match required, noop-job, noop-job-remove, build-job, build-job-remove, copy-job, copy-job-remove, reindex-job, rebuild-index boolean, target default gitlab enum gitlab, broken-specs-url, broken-tests-packages, pipeline-gen. These are facts I can restate โ they're in the original.
Careful: don't invent claims. I can describe the shape: jobs have after_script, before_script, image, script, tags, variables; -remove variants are additionalProperties: True too... actually both are additionalProperties: True. Hmm, both build-job and build-job-remove have 'additionalProperties': True. I can note that both are permissive. Fine.
Let me write.
Block 1 lines โ I'll reproduce exactly. The first line starts with 7 spaces. In a fenced code block that's preserved.
Write:
"okay bestie we are STILL in the pipeline-gen schema and no I will not apologise ๐
this is the part where the JSON schema lists every single CI job shape it knows about..."
Then fence.
Then commentary between blocks. Keep it fun but not adding false claims.
For the last cut-off, note "and it literally cuts off there mid-token ๐ญ that's the page boundary, not me".
Also mention \\w\\d\\-_\\. pattern for variables keys verbatim in backticks.
Let me assemble the final output.
Check: variable pattern appears as '[\\w\\d\\-_\\.]+' โ in a code block, verbatim, fine.
Now write the response with fences and commentary. Keep prose short-ish since content is huge.
Block 1 ends with 'enum': ['gitlab'], 'type': 'string'}}, 'type': 'object'},.
Block 2 ends with 'object'}, {'additionalProperties': False, 'properties': {'build-job':.
Block 3 to end.
Careful with block 2 start: {'additionalProperties': False, 'properties': {'broken-specs-url':.
Now assemble.okay bestie we're still in the pipeline-gen schema and yes it's still going ๐ญ this is the part where every single CI job shape gets spelled out one by one, and honestly it's giving "someone let the schema generator cook". it's a lot. it's iconic. it's also just JSON. stay with me ๐ซถ
first chunk โ the tail end of a job shape's variables/tags/script block, then the cleanup-job and cleanup-job-remove entries, then any-job and any-job-remove:
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'cleanup-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'cleanup-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'any-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'any-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'}]}]}, 'type': 'array'},
'rebuild-index': {'type': 'boolean'}, 'target': {'default': 'gitlab',
'enum': ['gitlab'], 'type': 'string'}}, 'type': 'object'},see how every job shape has the exact same lil loadout? after_script, before_script, image, script, tags, variables โ and variables keys all have to match '[\\w\\d\\-_\\.]+' and map to strings, periodt. also clock 'rebuild-index' is a boolean and 'target' defaults to 'gitlab' with 'enum': ['gitlab'], so it's gitlab or nothing bestie, no cap ๐ฅ
next chunk โ the sibling object with broken-specs-url, broken-tests-packages, and the pipeline-gen items, including match_behavior (default 'first', enum 'first' / 'merge') and the submapping array where match is required:
{'additionalProperties': False, 'properties': {'broken-specs-url':
{'type': 'string'}, 'broken-tests-packages': {'items': {'type':
'string'}, 'type': 'array'}, 'pipeline-gen': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'submapping': {'items': {'additionalProperties': False, 'properties':
{'build-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'build-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'match':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['match'],
'type': 'object'}, 'type': 'array'}}, 'required': ['submapping'],
'type': 'object'}, {'oneOf': [{'additionalProperties': False,
'properties': {'noop-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'noop-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type':
'object'}, {'additionalProperties': False, 'properties': {'build-job':okay the submapping bit is the main character here ngl ๐ each entry needs match (an array of strings) and that's the only required field in there, everything else is optional vibes. and match_behavior? 'first' unless you say 'merge'. adorable. also spot the noop-job / noop-job-remove pair doing absolutely nothing and doing it beautifully ๐
last chunk for this part โ build-job again, then build-job-remove, copy-job, copy-job-remove, and the start of reindex-job:
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'build-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'copy-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'copy-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'reindex-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},and it literally stops mid-object right there ๐ญ that's the page boundary, not me being chaotic, i promise. copy-job and copy-job-remove are just twins with the same six fields, build-job shows up again on its own, and reindex-job is just getting started โ it's giving cliffhanger energy and i am sat ๐
okay so we're deep in the oneOf / anyOf sauce now and this is where the schema gets juicy ๐
โ every branch down here is one full shape the generated CI config can take, and the vibe is very "same four fields, copy-pasted with love, twice, forever." we're picking up mid-object btw, so if the first line looks like it's closing something, that's because it is ๐ซถ
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'reindex-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},so like โ the four little besties that keep showing up are after_script, before_script, image, script, plus tags and variables tagging along ๐ and notice: image is oneOf a plain {'type': 'string'} or an {'type': 'object'} carrying entrypoint (an array of strings) and name (a string). script is an array whose items are anyOf a string or an array of strings โ both, either, gorgeous. variables is an object whose keys match [\\w\\d\\-_\\.]+ and whose values are strings, periodt.
and 'reindex-job-remove' gets 'additionalProperties': True โ the whole point of the -remove variants is that they're loose and chill, no cap โจ
next up: signing-job and its chaotic twin, signing-job-remove ๐ฅ same shape, same energy:
{'additionalProperties': False, 'properties': {'signing-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'signing-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},the wrapper branch is 'additionalProperties': False โ strict mode, nothing extra allowed โ while the job objects inside get 'additionalProperties': True. we love a permissive inner life ๐ฆ
cleanup-job / cleanup-job-remove? bestie it's the exact same six fields again. i'm not even mad, it's giving copy-paste slay ๐ธ
{'additionalProperties': False, 'properties': {'cleanup-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'cleanup-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},and then any-job / any-job-remove, i.e. the "whatever runner, it's fine" branch ๐ same vibes, same fields:
{'additionalProperties': False, 'properties': {'any-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'any-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}]}]}, 'type': 'array'},okay breathe ๐ฎโ๐จ that closing run of }} }} }} at the end is doing a lot โ that's the whole job-list wrapping itself up so it can become 'type': 'array'.
and then, right after all that therapy, we get the three chill siblings ๐ฅ
'rebuild-index': {'type': 'boolean'}, 'target': {'default': 'gitlab',
'enum': ['gitlab'], 'type': 'string'}, 'temporary-storage-url-prefix':
{'type': 'string'}}, 'type': 'object'}]}, {'anyOf':so, for the record: rebuild-index is a boolean, target defaults to 'gitlab' and its 'enum' contains exactly ['gitlab'] (so likeโฆ it's the only option, icon ๐), and temporary-storage-url-prefix is a plain string. then the schema closes that object and opens a whole new anyOf ๐ซถ
new anyOf, new me โจ this branch is the one with mappings, and look โ after_script / before_script here are just {'items': {'type': 'string'}, 'type': 'array'}, no anyOf nesting. glow-up fr fr.
[{'additionalProperties': False, 'properties': {'after_script':
{'items': {'type': 'string'}, 'type': 'array'}, 'before_script':
{'items': {'type': 'string'}, 'type': 'array'}, 'bootstrap': {'items':
{'anyOf': [{'type': 'string'}, {'additionalProperties': False,
'properties': {'compiler-agnostic': {'default': False, 'type':
'boolean'}, 'name': {'type': 'string'}}, 'required': ['name'], 'type':
'object'}]}, 'type': 'array'}, 'broken-specs-url': {'type': 'string'},
'broken-tests-packages': {'items': {'type': 'string'}, 'type': 'array'},
'enable-artifacts-buildcache': {'type': 'boolean'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,mini recap of the new cast: bootstrap is an array whose items are anyOf a string or an object that 'required': ['name'], 'type': 'object' and optionally takes compiler-agnostic (default False, a boolean). broken-specs-url is a string ๐, broken-tests-packages is an array of strings, enable-artifacts-buildcache is a boolean, and image is the same oneOf string-or-object situation we've been seeing all day. mappings items get 'additionalProperties': False โ strict again, bestie.
and here's the meat of the mappings ๐ read it slow:
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'script': {'items': {'type': 'string'},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'},
'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type':
'string'}}, 'type': 'object'}}, 'required': ['mappings'], 'type':
'object'}, {'additionalProperties': False, 'properties':let me count the requirements for you bestie, because they matter ๐
- each entry in
mappingsrequires['match']โmatchitself being an array of strings remove-attributesrequires['tags']and is'additionalProperties': Falserunner-attributesrequires['tags']and is'additionalProperties': Trueservice-job-attributesrequires['tags'],'additionalProperties': Truesigning-job-attributesrequires['tags'],'additionalProperties': Truematch_behaviordefaults to'first'and its'enum'is['first', 'merge']โ two whole options, we ate ๐ฅ- and the whole outer object requires
['mappings']
also s/o to script and tags and variables sitting there at the top level of the same object like the main characters they are ๐
aaaand the second branch starts, which is basically the first branch's twin minus enable-artifacts-buildcache, and then we get cut off mid-{'items': which is honestly so rude of the page break ๐ญ
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'bootstrap': {'items': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'compiler-agnostic':
{'default': False, 'type': 'boolean'}, 'name': {'type': 'string'}},
'required': ['name'], 'type': 'object'}]}, 'type': 'array'},
'broken-specs-url': {'type': 'string'}, 'broken-tests-packages':
{'items': {'type': 'string'}, 'type': 'array'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':we're literally hanging on 'before_script': {'items': โ the tension is unreal. to be continued, bestie ๐ง๐
okay so we are still DEEP in the schema mines โ๏ธ๐ โ this chunk picks up right in the middle of that job-attributes object (the one that ends with match and mappings), and the way it's giving "the JSON ate my entire afternoon" ๐ญ no shade to whoever wrote it, but somebody had to sit there and type out 'type': 'string' like four thousand times and honestly? respect. Here it is, character for character, bestie:
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items': {'type':
'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'},
{'properties': {'entrypoint': {'items': {'type': 'string'}, 'type':
'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script':
{'items': {'type': 'string'}, 'type': 'array'}, 'tags': {'items':
{'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties':
{'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}},
'required': ['tags'], 'type': 'object'}, 'signing-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items': {'type':
'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'},
{'properties': {'entrypoint': {'items': {'type': 'string'}, 'type':
'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script':
{'items': {'type': 'string'}, 'type': 'array'}, 'tags': {'items':
{'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties':
{'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}},
'required': ['tags'], 'type': 'object'}, 'tags': {'items': {'type':
'string'}, 'type': 'array'}, 'temporary-storage-url-prefix': {'type':
'string'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['mappings'], 'type':
'object'}]}]},vibe check on what you just scrolled past ๐:
imageis either a plain string or an object withentrypoint(an array of strings) andnameโจscript,tagsare arrays of strings, andvariablesmaps keys matching[\\w\\d\\-_\\.]+to stringsrequired: ['tags']โ tags are non-negotiable, periodt ๐match_behaviorhas'default': 'first'and'enum': ['first', 'merge'], andrebuild-indexis a plain booleanservice-job-attributesandsigning-job-attributesboth rock'additionalProperties': Trueand carryafter_script,before_script,image,script,tags,variables, withrequired: ['tags']๐ฆtemporary-storage-url-prefixis just a string, and the whole thing closes out with'required': ['mappings']
and then the schema simply KEEPS SCHEMING. not a single break. here's the rest of this part, no cap:
'compilers': {'items': {'additionalProperties': False, 'properties':
{'compiler': {'additionalProperties': False, 'properties': {'alias':
{'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'environment':
{'additionalProperties': False, 'default': {}, 'properties':
{'append_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'extra_rpaths': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}, 'flags': {'additionalProperties': False, 'properties':
{'cflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]},
'cppflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]},
'cxxflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'fflags':
{'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'ldflags': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'ldlibs': {'anyOf': [{'type':
'string'}, {'type': 'null'}]}}, 'type': 'object'}, 'implicit_rpaths':
{'anyOf': [{'items': {'type': 'string'}, 'type': 'array'}, {'type':
'boolean'}]}, 'modules': {'anyOf': [{'type': 'string'}, {'type':
'null'}, {'type': 'array'}]}, 'operating_system': {'type': 'string'},
'paths': {'additionalProperties': False, 'properties': {'cc': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'cxx': {'anyOf': [{'type':
'string'}, {'type': 'null'}]}, 'f77': {'anyOf': [{'type': 'string'},
{'type': 'null'}]}, 'fc': {'anyOf': [{'type': 'string'}, {'type':
'null'}]}}, 'required': ['cc', 'cxx', 'f77', 'fc'], 'type': 'object'},
'spec': {'type': 'string'}, 'target': {'type': 'string'}}, 'required':
['paths', 'spec', 'modules', 'operating_system'], 'type': 'object'}},
'type': 'object'}, 'type': 'array'}, 'concretizer':
{'additionalProperties': False, 'properties': {'duplicates':
{'properties': {'strategy': {'enum': ['none', 'minimal', 'full'],
'type': 'string'}}, 'type': 'object'}, 'enable_node_namespace': {'type':
'boolean'}, 'os_compatible': {'additionalProperties': {'type': 'array'},
'type': 'object'}, 'reuse': {'oneOf': [{'type': 'boolean'}, {'enum':
['dependencies'], 'type': 'string'}, {'properties': {'exclude':
{'items': {'type': 'string'}, 'type': 'array'}, 'from': {'items':
{'properties': {'exclude': {'items': {'type': 'string'}, 'type':
'array'}, 'include': {'items': {'type': 'string'}, 'type': 'array'},
'type': {'enum': ['local', 'buildcache', 'external'], 'type':
'string'}}, 'type': 'object'}, 'type': 'array'}, 'include': {'items':
{'type': 'string'}, 'type': 'array'}, 'roots': {'type': 'boolean'}},
'type': 'object'}]}, 'targets': {'properties': {'granularity': {'enum':
['generic', 'microarchitectures'], 'type': 'string'}, 'host_compatible':
{'type': 'boolean'}}, 'type': 'object'}, 'unify': {'oneOf': [{'type':
'boolean'}, {'enum': ['when_possible'], 'type': 'string'}]}}, 'type':
'object'}, 'config': {'default': {}, 'deprecatedProperties': {'error':
False, 'message': 'config:terminal_title has been replaced by
install_status and is ignored', 'properties': ['terminal_title']},
'properties': {'additional_external_search_paths': {'items': {'type':
'string'}, 'type': 'array'}, 'aliases': {'patternProperties':
{'\\w[\\w-]*': {'type': 'string'}}, 'type': 'object'}, 'allow_sgid':
{'type': 'boolean'}, 'binary_index_root': {'type': 'string'},
'binary_index_ttl': {'minimum': 0, 'type': 'integer'}, 'build_jobs':
{'minimum': 1, 'type': 'integer'}, 'build_language': {'type': 'string'},
'build_stage': {'oneOf': [{'type': 'string'}, {'items': {'type':
'string'}, 'type': 'array'}]}, 'ccache': {'type': 'boolean'},
'checksum': {'type': 'boolean'}, 'concretizer': {'enum': ['original',
'clingo'], 'type': 'string'}, 'connect_timeout': {'minimum': 0, 'type':
'integer'}, 'db_lock_timeout': {'minimum': 1, 'type': 'integer'},
'debug': {'type': 'boolean'}, 'deprecated': {'type': 'boolean'},
'develop_stage_link': {'type': 'string'}, 'dirty': {'type': 'boolean'},
'environments_root': {'type': 'string'}, 'extensions': {'items':
{'type': 'string'}, 'type': 'array'}, 'flags': {'properties':
{'keep_werror': {'enum': ['all', 'specific', 'none'], 'type': 'string'}},
'type': 'object'}, 'install_hash_length': {'minimum': 1,
'type': 'integer'}, 'install_missing_compilers': {'type': 'boolean'},
'install_path_scheme': {'type': 'string'}, 'install_status': {'type':
'boolean'}, 'install_tree': {'anyOf': [{'properties': {'padded_length':
{'oneOf': [{'minimum': 0, 'type': 'integer'}, {'type': 'boolean'}]},
'projections': {'patternProperties': {'all|\\w[\\w-]*': {'type':
'string'}}, 'type': 'object'}, 'root': {'type': 'string'}}, 'type':
'object'}, {'type': 'string'}]}, 'license_dir': {'type': 'string'},
'locks': {'type': 'boolean'}, 'misc_cache': {'type': 'string'},
'package_lock_timeout': {'anyOf': [{'minimum': 1, 'type': 'integer'},
{'type': 'null'}]}, 'shared_linking': {'anyOf': [{'enum': ['rpath',
'runpath'], 'type': 'string'}, {'properties': {'bind': {'type':
'boolean'}, 'type': {'enum': ['rpath', 'runpath'], 'type': 'string'}},
'type': 'object'}]}, 'source_cache': {'type': 'string'}, 'ssl_certs':
{'type': 'string'}, 'stage_name': {'type': 'string'},
'suppress_gpg_warnings': {'type': 'boolean'}, 'template_dirs': {'items':
{'type': 'string'}, 'type': 'array'}, 'test_stage': {'type': 'string'},
'url_fetch_method': {'enum': ['urllib', 'curl'], 'type': 'string'},
'verify_ssl': {'type': 'boolean'}}, 'type': 'object'}, 'container':
{'additionalProperties': False, 'properties': {'depfile': {'default':
False, 'type': 'boolean'}, 'docker': {'additionalProperties': False,
'default': {}, 'type': 'object'}, 'format': {'enum': ['docker',
'singularity'], 'type': 'string'}, 'images': {'anyOf':
[{'additionalProperties': False, 'properties': {'os': {'type':
'string'}, 'spack': {'anyOf': [{'type': 'string'},
{'additional_properties': False, 'properties': {'ref': {'type':
'string'}, 'resolve_sha': {'default': False, 'type': 'boolean'}, 'url':
{'type': 'string'}, 'verify': {'default': False, 'type': 'boolean'}},
'type': 'object'}]}}, 'required': ['os', 'spack'], 'type': 'object'},
{'additionalProperties': False, 'properties': {'build': {'type':
'string'}, 'final': {'type': 'string'}}, 'required': ['build', 'final'],
'type': 'object'}]}, 'labels': {'type': 'object'}, 'os_packages':
{'additionalProperties': False, 'properties': {'build': {'items':
{'type': 'string'}, 'type': 'array'}, 'command': {'enum': ['apt', 'yum',
'zypper', 'apk', 'yum_amazon'], 'type': 'string'}, 'final': {'items':
{'type': 'string'}, 'type': 'array'}, 'update': {'type': 'boolean'}},
'type': 'object'}, 'singularity': {'additionalProperties': False,
'default': {}, 'properties': {'help': {'type': 'string'}, 'runscript':
{'type': 'string'}, 'startscript': {'type': 'string'}, 'test': {'type':
'string'}}, 'type': 'object'}, 'strip': {'default': True, 'type':
'boolean'}, 'template': {'default': None, 'type': 'string'}}, 'type':
'object'}, 'definitions': {'default': [], 'items': {'patternProperties':
{'^(?!when$)\\w*': {'default': [], 'items': {'anyOf':
[{'additionalProperties': False, 'properties': {'exclude': {'items':
{'type': 'string'}, 'type': 'array'}, 'matrix': {'items': {'items':
{'type': 'string'}, 'type': 'array'}, 'type': 'array'}}, 'type':
'object'}, {'type': 'string'}, {'type': 'null'}]}, 'type': 'array'}},
'properties': {'when': {'type': 'string'}}, 'type': 'object'}, 'type':
'array'}, 'develop': {'additionalProperties': False, 'default': {},
'patternProperties': {'\\w[\\w-]*': {'additionalProperties': False,
'properties': {'path': {'type': 'string'}, 'spec': {'type': 'string'}},
'type': 'object'}}, 'type': 'object'}, 'mirrors':
{'additionalProperties': False, 'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'additionalProperties':
False, 'anyOf': [{'required': ['url']}, {'required': ['fetch']},
{'required': ['pull']}], 'properties': {'access_pair': {'items':
{'maxItems': 2, 'minItems': 2, 'type': ['string', 'null']}, 'type':
'array'}, 'access_token': {'type': ['string', 'null']}, 'autopush':
{'type': 'boolean'}, 'binary': {'type': 'boolean'}, 'endpoint_url':
{'type': ['string', 'null']}, 'fetch': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'access_pair': {'items':okay now let's actually read the thing, because there is SO much in here and some of it is lowkey iconic ๐ :
compilers๐ง โ an array of objects, each with acompilerkey. Inside:alias(string ornull),environment(default{}) withappend_path,prepend_path,remove_path,setandunset(default[]) โ the path-ish ones are patternProperties maps of\w[\w-]*to string-or-number ๐;extra_rpaths(default[]);flagswithcflags,cppflags,cxxflags,fflags,ldflags,ldlibs(each string ornull);implicit_rpaths(array of strings or a boolean);modules(string,nullor array);operating_system(string);paths, and this one's the drama โcc,cxx,f77,fcare all four required, each string ornull; thenspecandtarget(strings). And the big finish:'required': ['paths', 'spec', 'modules', 'operating_system']โ you literally cannot skip those, bestie ๐concretizerโจ โduplicates.strategyisnone,minimalorfull;enable_node_namespaceis a boolean;os_compatibleis an object of arrays;reusecan be a boolean, the stringdependencies, or an object withexclude(array of strings),from(array of objects carryingexclude,includeandtypeโlocal,buildcache,external),includeandroots(boolean);targetshasgranularityโgeneric/microarchitecturesplushost_compatible(boolean); andunifyis a boolean or the stringwhen_possibleconfig๐ โ the motherlode,default: {}. First the drama, so listen up:
๐terminal_titleis deprecated, fr fr โ the schema literally says'config:terminal_title has been replaced by install_status and is ignored'. it does not error ('error': False), it just quietly ignores you. tragic.
Then the roster: additional_external_search_paths, aliases, allow_sgid, binary_index_root, binary_index_ttl (min 0), build_jobs (min 1), build_language, build_stage (string or array), ccache, checksum, concretizer (original or clingo), connect_timeout (min 0), db_lock_timeout (min 1), debug, deprecated, develop_stage_link, dirty, environments_root, extensions, flags.keep_werror (all/specific/none), install_hash_length (min 1), install_missing_compilers, install_path_scheme, install_status, install_tree (object with padded_length [integer min 0 or boolean], projections, root โ or just a string), license_dir, locks, misc_cache, package_lock_timeout (integer min 1 or null), shared_linking (rpath/runpath or an object with bind + type), source_cache, ssl_certs, stage_name, suppress_gpg_warnings, template_dirs, test_stage, url_fetch_method (urllib or curl), verify_ssl ๐ฆ
container๐ โdepfiledefaultFalse,dockerdefault{},formatโdocker/singularity,images(an object withos+spack, wherespackis a string or an object withref,resolve_shadefaultFalse,url,verifydefaultFalse, and bothosandspackrequired โ or an object withbuild+final, both required),labels,os_packages(build,commandโapt/yum/zypper/apk/yum_amazon,final,update),singularitywithhelp,runscript,startscript,test;stripdefaultTrue;templatedefaultNonedefinitions๐คก โ default[], an array of objects whose keys match^(?!when$)\w*(read it: anything that is not literallywhen), each one an array of{exclude, matrix}objects, strings, ornull; and then awhen(string) of its owndevelop๐ โ default{}, keys matching\w[\w-]*, each withpathandspecmirrorsโญ โ default{}, keys matching\w[\w-]*, each either a plain string or an object that requires at least one ofurl,fetchorpull, withaccess_pair(array,minItems: 2,maxItems: 2, items string-or-null),access_token(string ornull),autopush(boolean),binary(boolean),endpoint_url(string ornull), andfetchโ a string or an object with its ownaccess_pair, whoseitemsโฆ
and THAT is exactly where this part's text runs out ๐ญ mid-'fetch', mid-'items', mid-everything. the schema is still scheming, bestie โ it literally cut off before the sentence could finish, so yeah, we ball ๐
And then we slide from the git-push mirror settings straight into the modules config, which is the part of the schema that ate everything else for breakfast ๐
{'maxItems': 2, 'minItems': 2, 'type': ['string', 'null']}, 'type':
'array'}, 'access_token': {'type': ['string', 'null']}, 'endpoint_url':
{'type': ['string', 'null']}, 'profile': {'type': ['string', 'null']},
'url': {'type': 'string'}}, 'type': 'object'}]}, 'signed': {'type':
'boolean'}, 'source': {'type': 'boolean'}, 'url': {'type': 'string'}},
'type': 'object'}]}}, 'type': 'object'}, 'modules':okay so quick recap on that bit bestie โ access_pair is a maxItems/minItems 2 array of string-or-null โจ, access_token / endpoint_url / profile are all ['string', 'null'], url is a plain string, then signed is a boolean, source is a boolean, another url that's a string, and push itself is an anyOf of either a string or that whole object. iconic. moving on ๐ซถ
๐ The modules block (yes, all of it) ๐
so modules is additionalProperties: False with patternProperties, and the first pattern key is this absolute legend:
{'additionalProperties': False, 'patternProperties':
{'^(?!prefix_inspections$)\\w[\\w-]*$': {'additionalProperties': False,
'default': {}, 'properties': {'arch_folder': {'type': 'boolean'},
'enable': {'default': [], 'items': {'enum': ['tcl', 'lmod'], 'type':
'string'}, 'type': 'array'}, 'lmod': {'allOf': [{'allOf':translation for the group chat ๐ฌ โ any key that matches ^(?!prefix_inspections$)\\w[\\w-]*$ (so: a word char then word chars or hyphens, EXCEPT the literal prefix_inspections) gets an object with additionalProperties: False, default: {}, and these properties:
arch_folderโ{'type': 'boolean'}โenableโ{'default': [], 'items': {'enum': ['tcl', 'lmod'], 'type': 'string'}, 'type': 'array'}โ so a list, empty by default, and each item has to betclorlmod๐lmodโ and here's where it gets unhinged (affectionate), it's anallOfof anallOf๐
๐ธ lmod โ the shared module-file properties ๐ธ
[{'properties': {'all': {'additionalProperties': False, 'default': {},
'properties': {'autoload': {'enum': ['none', 'run', 'direct', 'all'],
'type': 'string'}, 'conflict': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'environment': {'additionalProperties':
False, 'default': {}, 'properties': {'append_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},so under all (which is additionalProperties: False, default: {}) you get the classic module-file starter pack ๐:
autoloadโ{'enum': ['none', 'run', 'direct', 'all'], 'type': 'string'}conflictโ{'default': [], 'items': {'type': 'string'}, 'type': 'array'}environmentโadditionalProperties: False,default: {}, and inside it:append_pathโpatternPropertieson\\w[\\w-]*, each valueanyOfstringornumber,type: 'object'prepend_pathโ same deal, sameanyOfstring/number,type: 'object'remove_pathโ same again bestie,string/number,type: 'object'setโpatternPropertieson\\w[\\w-]*withanyOfstring/number,type: 'object'unsetโ{'default': [], 'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type': 'array'}โ note it's an array here, not an object, the way thatunsetjust has to be different ๐ญ
lowkey the pattern is always the same:\\w[\\w-]*keys, and the value is astringor anumber. it's giving consistency ๐
โจ lmod continued: filter, load, prerequisites, suffixes, template โจ
'filter': {'additionalProperties': False, 'default': {}, 'properties':
{'exclude_env_vars': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}, 'load': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'prerequisites': {'enum': ['none',
'run', 'direct', 'all'], 'type': 'string'}, 'suffixes':
{'patternProperties': {'\\w[\\w-]*': {'type': 'string'}}, 'type':
'object', 'validate_spec': True}, 'template': {'type': 'string'}},
'type': 'object'}, 'defaults': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'exclude': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'exclude_implicits': {'default':
False, 'type': 'boolean'}, 'hash_length': {'default': 7, 'minimum': 0,
'type': 'integer'}, 'hide_implicits': {'default': False, 'type':
'boolean'}, 'include': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'naming_scheme': {'type': 'string'}, 'projections':
{'patternProperties': {'all|\\w[\\w-]*': {'type': 'string'}}, 'type':
'object'}, 'verbose': {'default': False, 'type': 'boolean'}}},breaking it down like a playlist ๐ง:
filterโadditionalProperties: False,default: {}, and its one property isexclude_env_vars={'default': [], 'items': {'type': 'string'}, 'type': 'array'}loadโ{'default': [], 'items': {'type': 'string'}, 'type': 'array'}prerequisitesโ{'enum': ['none', 'run', 'direct', 'all'], 'type': 'string'}suffixesโpatternPropertieson\\w[\\w-]*where the value is{'type': 'string'},type: 'object', and it's marked'validate_spec': True๐templateโ{'type': 'string'}
and then the leveled-up versions of the same vibes:
defaultsโ{'default': [], 'items': {'type': 'string'}, 'type': 'array'}excludeโ{'default': [], 'items': {'type': 'string'}, 'type': 'array'}exclude_implicitsโ{'default': False, 'type': 'boolean'}hash_lengthโ{'default': 7, 'minimum': 0, 'type': 'integer'}โ yeah, seven, and no negatives bestie ๐ซhide_implicitsโ{'default': False, 'type': 'boolean'}includeโ{'default': [], 'items': {'type': 'string'}, 'type': 'array'}naming_schemeโ{'type': 'string'}projectionsโpatternPropertiesonall|\\w[\\w-]*, values are{'type': 'string'},type: 'object'โ noteallgets to be in the pattern here, main character energy ๐verboseโ{'default': False, 'type': 'boolean'}
๐ฆ the other pattern branch (per-compiler / per-spec rules) ๐ฆ
{'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],hmm wait, careful โ read it again, the actual text is:
'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}, '^[\\^@%+~]':so this second branch has the negative lookahead โ it matches (^\\w[\\w-]*) as long as it's not one of: hierarchy, core_specs, verbose, hash_length, defaults, filter_hierarchy_specs, hide, include, exclude, projections, naming_scheme, core_compilers, all ๐ซ (that's the reserved-words list, learn it, love it, respect it)
and inside it's the exact same crew again: autoload, conflict, environment (with append_path, prepend_path, remove_path, set, unset), filter (exclude_env_vars), load, prerequisites, suffixes (with validate_spec: True), template ๐
๐ and the ^[\^@%+~] branch ๐
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],(it's the same block, bestie, I'm not gonna make you read a different one ๐ )
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}}, 'validate_spec': True}],
'default': {}, 'type': 'object'}, {'properties': {'core_compilers':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'},so ^[\\^@%+~] โ keys starting with a caret, at-sign, percent, plus or tilde โ get the same full property set, and then that whole lmod allOf closes with 'validate_spec': True, 'default': {}, 'type': 'object' ๐ซ
๐ the second half of the lmod allOf ๐
'core_specs': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}, 'filter_hierarchy_specs': {'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type':
'object'}, 'hierarchy': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}]}, 'prefix_inspections':
{'additionalProperties': False, 'patternProperties': {'^[\\w-]*':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type':
'object'}, 'roots': {'properties': {'lmod': {'type': 'string'}, 'tcl':
'type': 'string'}}, 'type': 'object'}, 'tcl': {'allOf': [{'allOf':yes bestie, fresh properties unlocked ๐:
core_compilersโ{'default': [], 'items': {'type': 'string'}, 'type': 'array'}core_specsโ{'default': [], 'items': {'type': 'string'}, 'type': 'array'}filter_hierarchy_specsโpatternPropertieson that same big negative-lookahead pattern, each value{'default': [], 'items': {'type': 'string'}, 'type': 'array'},type: 'object'hierarchyโ{'default': [], 'items': {'type': 'string'}, 'type': 'array'}
then, outside the lmod nest:
prefix_inspectionsโadditionalProperties: False,patternPropertieson^[\\w-]*, values{'default': [], 'items': {'type': 'string'}, 'type': 'array'},type: 'object'rootsโ{'properties': {'lmod': {'type': 'string'}, 'tcl': {'type': 'string'}}, 'type': 'object'}โ just two little string paths, one for each flavor ๐ซถtclโ and here we go again, anotherallOfinside anallOf๐
๐ tcl: the encore ๐
[{'properties': {'all': {'additionalProperties': False, 'default': {},
'properties': {'autoload': {'enum': ['none', 'run', 'direct', 'all'],
'type': 'string'}, 'conflict': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'environment': {'additionalProperties':
False, 'default': {}, 'properties': {'append_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'filter': {'additionalProperties': False, 'default': {}, 'properties':
{'exclude_env_vars': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}, 'load': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'prerequisites': {'enum': ['none',
'run', 'direct', 'all'], 'type': 'string'}, 'suffixes':
{'patternProperties': {'\\w[\\w-]*': {'type': 'string'}}, 'type':
'object', 'validate_spec': True}, 'template': {'type': 'string'}},
'type': 'object'}, 'defaults': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'exclude': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'exclude_implicits': {'default':
False, 'type': 'boolean'}, 'hash_length': {'default': 7, 'minimum': 0,
'type': 'integer'}, 'hide_implicits': {'default': False, 'type':
'boolean'}, 'include': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'naming_scheme': {'type': 'string'}, 'projections':
{'patternProperties': {'all|\\w[\\w-]*': {'type': 'string'}}, 'type':
'object'}, 'verbose': {'default': False, 'type': 'boolean'}}},literally the exact same property set as lmod, no cap ๐งข โ all with autoload/conflict/environment/filter/load/prerequisites/suffixes/template, then defaults, exclude, exclude_implicits (default False), hash_length (default 7, minimum 0), hide_implicits (default False), include, naming_scheme, projections, verbose (default False). it's giving "copy my notes but change the name" ๐
โญ and the last stretch we can see โญ
{'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type':same negative-lookahead reserved-words situation as before (say it with me: hierarchy, core_specs, verbose, hash_length, defaults, filter_hierarchy_specs, hide, include, exclude, projections, naming_scheme, core_compilers, all), and it opens with autoload and conflictโฆ and that's exactly where this part cuts off ๐ซ
ngl the schema is basically the same six properties photocopied with different regexes guarding the door. but every single one of those keys matters, so we're reading them all. no skipping. periodt. ๐
stay tuned, bestie โ we're mid-conflict and there's more schema to survive ๐ญ๐ฆ
okay bestie we are still IN IT โ deep in the merged schema blob, and yes it's a lot of patternProperties and anyOf, but this is the part where the actual rules live so we're reading it. stay with me ๐ซถ
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}}, 'validate_spec': True}],
'default': {}, 'type': 'object'}, {}]}, 'use_view': {'anyOf': [{'type':
'string'}, {'type': 'boolean'}]}}, 'type': 'object'}}, 'properties':
{'prefix_inspections': {'additionalProperties': False,
'patternProperties': {'^[\\w-]*': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}}, 'type': 'object'}}, 'type': 'object'},
'packages': {'additionalProperties': False, 'default': {},
'patternProperties': {'(?!^all$)(^\\w[\\w-]*)': {'additionalProperties':
False, 'default': {}, 'deprecatedProperties': {'error': False,
'message': "setting 'compiler:', 'target:' or 'provider:' preferences in
a package-specific section of packages.yaml is deprecated, and will be
removed in v0.23.\n\n\tThese preferences will be ignored by Spack, and
can be set only in the 'all' section of the same file. You can
run:\n\n\t\t$ spack audit configs\n\n\tto get better diagnostics,
including files:lines where the deprecated attributes are used.\n\n\tUse
requirements to enforce conditions on specific packages:
https://spack.readthedocs.io/en/latest/packages_yaml.html#package-requirements\n",
'properties': ['target', 'compiler', 'providers']}, 'properties':
{'buildable': {'default': True, 'type': 'boolean'}, 'compiler': {},
'conflict': {'items': {'oneOf': [{'additionalProperties': False,
'properties': {'message': {'type': 'string'}, 'spec': {'type':
'string'}, 'when': {'type': 'string'}}, 'type': 'object'}, {'type':
'string'}]}, 'type': 'array'}, 'externals': {'items':
{'additionalProperties': True, 'properties': {'extra_attributes':
{'additionalProperties': True, 'properties': {'environment':
{'additionalProperties': False, 'default': {}, 'properties':
{'append_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}},
'type': 'object'}, 'modules': {'items': {'type': 'string'}, 'type':
'array'}, 'prefix': {'type': 'string'}, 'spec': {'type': 'string'}},
'required': ['spec'], 'type': 'object'}, 'type': 'array'},
'package_attributes': {'additionalProperties': False,
'patternProperties': {'\\w+': {}}, 'type': 'object'}, 'permissions':
{'additionalProperties': False, 'properties': {'group': {'type':
'string'}, 'read': {'enum': ['user', 'group', 'world'], 'type':
'string'}, 'write': {'enum': ['user', 'group', 'world'], 'type':
'string'}}, 'type': 'object'}, 'prefer': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'}, 'providers':
{}, 'require': {'oneOf': [{'items': {'oneOf': [{'additionalProperties':
False, 'properties': {'any_of': {'items': {'type': 'string'}, 'type':
'array'}, 'message': {'type': 'string'}, 'one_of': {'items': {'type':
'string'}, 'type': 'array'}, 'spec': {'type': 'string'}, 'when':
{'type': 'string'}}, 'type': 'object'}, {'type': 'string'}]}, 'type':
'array'}, {'type': 'string'}]}, 'target': {}, 'variants': {'oneOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'version': {'default': [], 'items': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}},
'properties': {'all': {'additionalProperties': False, 'default': {},
'deprecatedProperties': {'error': False, 'message': "setting version
preferences in the 'all' section of packages.yaml is deprecated and will
be removed in v0.23\n\n\tThese preferences will be ignored by Spack. You
can set them only in package-specific sections of the same file.\n",
'properties': ['version']}, 'properties': {'buildable': {'default':
True, 'type': 'boolean'}, 'compiler': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'conflict': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'},
'package_attributes': {'additionalProperties': False,
'patternProperties': {'\\w+': {}}, 'type': 'object'}, 'permissions':
{'additionalProperties': False, 'properties': {'group': {'type':
'string'}, 'read': {'enum': ['user', 'group', 'world'], 'type':
'string'}, 'write': {'enum': ['user', 'group', 'world'], 'type':
'string'}}, 'type': 'object'}, 'prefer': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'}, 'providers':
{'additionalProperties': False, 'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}}, 'type': 'object'}, 'require': {'oneOf': [{'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'any_of': {'items':
{'type': 'string'}, 'type': 'array'}, 'message': {'type': 'string'},
'one_of': {'items': {'type': 'string'}, 'type': 'array'}, 'spec':
{'type': 'string'}, 'when': {'type': 'string'}}, 'type': 'object'},
{'type': 'string'}]}, 'type': 'array'}, {'type': 'string'}]}, 'target':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'variants': {'oneOf': [{'type': 'string'}, {'items': {'type': 'string'},
'type': 'array'}]}, 'version': {}}, 'type': 'object'}}, 'type':
'object'}, 'repos': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}, 'upstreams': {'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'additionalProperties': False, 'default': {},
'properties': {'install_tree': {'type': 'string'}, 'modules':
{'properties': {'lmod': {'type': 'string'}, 'tcl': {'type': 'string'}},
'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}, 'view':
{'anyOf': [{'type': 'boolean'}, {'type': 'string'},
{'patternProperties': {'\\w+': {'additionalProperties': False,
'properties': {'exclude': {'items': {'type': 'string'}, 'type':
'array'}, 'link': {'pattern': '(roots|all|run)', 'type': 'string'},
'link_type': {'type': 'string'}, 'projections': {'patternProperties':
{'all|\\w[\\w-]*': {'type': 'string'}}, 'type': 'object'}, 'root':
{'type': 'string'}, 'select': {'items': {'type': 'string'}, 'type':
'array'}}, 'required': ['root']}}, 'type': 'object'}]}}, 'title': 'Spack
merged configuration file schema', 'type': 'object'}Full schema with metadata ๐ โ that's the one-liner on the attribute, fr.
okay two things from that blob that you should NOT sleep on, because Spack will literally tell you off for them. first: puttingcompiler:,target:orprovider:preferences in a package-specific section ofpackages.yamlis deprecated and will be removed in v0.23 โ Spack ignores those. they can only be set in the'all'section of the same file. second: setting version preferences in the'all'section is also deprecated and will be removed in v0.23, and those get ignored too โ versions only go in package-specific sections. the deprecation message literally points you at$ spack audit configsfor better diagnostics, including files:lines where the deprecated attributes are used. and it tells you to use requirements to enforce conditions on specific packages:https://spack.readthedocs.io/en/latest/packages_yaml.html#package-requirements. periodt. ๐ซถ
๐ช spack.schema.mirrors module
Schema for mirrors.yaml configuration file. โจ
the way this one is actual readable Python and not one giant stringโฆ we love to see it ๐ญ
#: Common properties for connection specification
connection = {
"url": {"type": "string"},
# todo: replace this with named keys "username" / "password" or "id" / "secret"
"access_pair": {
"type": "array",
"items": {"type": ["string", "null"], "minItems": 2, "maxItems": 2},
},
"access_token": {"type": ["string", "null"]},
"profile": {"type": ["string", "null"]},
"endpoint_url": {"type": ["string", "null"]},
}
#: Mirror connection inside pull/push keys
fetch_and_push = {
"anyOf": [
{"type": "string"},
{
"type": "object",
"additionalProperties": False,
"properties": {**connection}, # type: ignore
},
]
}
#: Mirror connection when no pull/push keys are set
mirror_entry = {
"type": "object",
"additionalProperties": False,
"anyOf": [{"required": ["url"]}, {"required": ["fetch"]}, {"required": ["pull"]}],
"properties": {
"source": {"type": "boolean"},
"binary": {"type": "boolean"},
"signed": {"type": "boolean"},
"fetch": fetch_and_push,
"push": fetch_and_push,
"autopush": {"type": "boolean"},
**connection, # type: ignore
},
}
#: Properties for inclusion in other schemas
properties: Dict[str, Any] = {
"mirrors": {
"type": "object",
"default": {},
"additionalProperties": False,
"patternProperties": {r"\w[\w-]*": {"anyOf": [{"type": "string"}, mirror_entry]}},
}
}
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack mirror configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}quick vibe check on what's happening in there, bestie ๐ง โ a mirrors entry is either a plain string or a mirror_entry object; a mirror_entry requires at least one of url, fetch or pull (that's the anyOf), allows source, binary, signed, fetch, push and autopush, and then spreads in the whole connection spec. fetch_and_push is either a string or an object that's just the connection fields. the $schema is draft-07 and the title is literally Spack mirror configuration file schema. iconic. ๐
spack.schema.mirrors.connection ๐
spack.schema.mirrors.connection = {'access_pair': {'items': {'maxItems':
2, 'minItems': 2, 'type': ['string', 'null']}, 'type': 'array'},
'access_token': {'type': ['string', 'null']}, 'endpoint_url': {'type':
['string', 'null']}, 'profile': {'type': ['string', 'null']}, 'url':
{'type': 'string'}}Common properties for connection specification โ ngl the cleanest little dict in this whole manpage ๐ธ
and yeah, there's that# todo:comment sitting right there in the source about replacing the stuff with named keys"username"/"password"or"id"/"secret"โ I'm just telling you it's there, I'm not out here predicting when it changes ๐
next up, the mirror-connection bits โ these are the config nuggets that live inside the pull / push keys, so like, the actual how-do-I-talk-to-this-remote part ๐
spack.schema.mirrors.fetch_and_push = {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'access_pair': {'items':
{'maxItems': 2, 'minItems': 2, 'type': ['string', 'null']}, 'type':
'array'}, 'access_token': {'type': ['string', 'null']}, 'endpoint_url':
{'type': ['string', 'null']}, 'profile': {'type': ['string', 'null']},
'url': {'type': 'string'}}, 'type': 'object'}]}Mirror connection inside pull/push keys.
okay and here's the whole mirror entry โ this is the big one, it's giving "everything you can possibly put in one mirror block" energy โจ
spack.schema.mirrors.mirror_entry = {'additionalProperties': False,
'anyOf': [{'required': ['url']}, {'required': ['fetch']}, {'required':
['pull']}], 'properties': {'access_pair': {'items': {'maxItems': 2,
'minItems': 2, 'type': ['string', 'null']}, 'type': 'array'},
'access_token': {'type': ['string', 'null']}, 'autopush': {'type':
'boolean'}, 'binary': {'type': 'boolean'}, 'endpoint_url': {'type':
['string', 'null']}, 'fetch': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'access_pair': {'items':
{'maxItems': 2, 'minItems': 2, 'type': ['string', 'null']}, 'type':
'array'}, 'access_token': {'type': ['string', 'null']}, 'endpoint_url':
{'type': ['string', 'null']}, 'profile': {'type': ['string', 'null']},
'url': {'type': 'string'}}, 'type': 'object'}]}, 'profile': {'type':
['string', 'null']}, 'push': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'access_pair': {'items':
{'maxItems': 2, 'minItems': 2, 'type': ['string', 'null']}, 'type':
'array'}, 'access_token': {'type': ['string', 'null']}, 'endpoint_url':
{'type': ['string', 'null']}, 'profile': {'type': ['string', 'null']},
'url': {'type': 'string'}}, 'type': 'object'}]}, 'signed': {'type':
'boolean'}, 'source': {'type': 'boolean'}, 'url': {'type': 'string'}},
'type': 'object'}Mirror connection when no pull/push keys are set. ๐ซถ
and this one is literally the same vibes but exposed as a properties blob so other schemas can yeet it into themselves ๐
spack.schema.mirrors.properties: Dict[str, Any] = {'mirrors':
{'additionalProperties': False, 'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'additionalProperties':
False, 'anyOf': [{'required': ['url']}, {'required': ['fetch']},
{'required': ['pull']}], 'properties': {'access_pair': {'items':
{'maxItems': 2, 'minItems': 2, 'type': ['string', 'null']}, 'type':
'array'}, 'access_token': {'type': ['string', 'null']}, 'autopush':
{'type': 'boolean'}, 'binary': {'type': 'boolean'}, 'endpoint_url':
{'type': ['string', 'null']}, 'fetch': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'access_pair': {'items':
{'maxItems': 2, 'minItems': 2, 'type': ['string', 'null']}, 'type':
'array'}, 'access_token': {'type': ['string', 'null']}, 'endpoint_url':
{'type': ['string', 'null']}, 'profile': {'type': ['string', 'null']},
'url': {'type': 'string'}}, 'type': 'object'}]}, 'profile': {'type':
['string', 'null']}, 'push': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'access_pair': {'items':
{'maxItems': 2, 'minItems': 2, 'type': ['string', 'null']}, 'type':
'array'}, 'access_token': {'type': ['string', 'null']}, 'endpoint_url':
{'type': ['string', 'null']}, 'profile': {'type': ['string', 'null']},
'url': {'type': 'string'}}, 'type': 'object'}]}, 'signed': {'type':
'boolean'}, 'source': {'type': 'boolean'}, 'url': {'type': 'string'}},
'type': 'object'}]}}, 'type': 'object'}}Properties for inclusion in other schemas. (iykyk โ it's the reusable piece ๐)
and finally the full schema with all the metadata wrapped around it, crown and all ๐
spack.schema.mirrors.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'mirrors': {'additionalProperties': False,
'default': {}, 'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'additionalProperties': False, 'anyOf': [{'required':
['url']}, {'required': ['fetch']}, {'required': ['pull']}],
'properties': {'access_pair': {'items': {'maxItems': 2, 'minItems': 2,
'type': ['string', 'null']}, 'type': 'array'}, 'access_token': {'type':
['string', 'null']}, 'autopush': {'type': 'boolean'}, 'binary': {'type':
'boolean'}, 'endpoint_url': {'type': ['string', 'null']}, 'fetch':
{'anyOf': [{'type': 'string'}, {'additionalProperties': False,
'properties': {'access_pair': {'items': {'maxItems': 2, 'minItems': 2,
'type': ['string', 'null']}, 'type': 'array'}, 'access_token': {'type':
['string', 'null']}, 'endpoint_url': {'type': ['string', 'null']},
'profile': {'type': ['string', 'null']}, 'url': {'type': 'string'}},
'type': 'object'}]}, 'profile': {'type': ['string', 'null']}, 'push':
{'anyOf': [{'type': 'string'}, {'additionalProperties': False,
'properties': {'access_pair': {'items': {'maxItems': 2, 'minItems': 2,
'type': ['string', 'null']}, 'type': 'array'}, 'access_token': {'type':
['string', 'null']}, 'endpoint_url': {'type': ['string', 'null']},
'profile': {'type': ['string', 'null']}, 'url': {'type': 'string'}},
'type': 'object'}]}, 'signed': {'type': 'boolean'}, 'source': {'type':
'boolean'}, 'url': {'type': 'string'}}, 'type': 'object'}]}}, 'type':
'object'}}, 'title': 'Spack mirror configuration file schema', 'type':
'object'}Full schema with metadata โญ
๐ spack.schema.modules module ๐
okay bestie, now we're in the modules.yaml schema and it's lowkey the glow-up era of Spack config ๐ธ this is the schema for the modules.yaml configuration file, periodt.
#: Matches a spec or a multi-valued variant but not another
#: valid keyword.
#:
#: THIS NEEDS TO BE UPDATED FOR EVERY NEW KEYWORD THAT
#: IS ADDED IMMEDIATELY BELOW THE MODULE TYPE ATTRIBUTE
spec_regex = (
r"(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|"
r"include|exclude|projections|naming_scheme|core_compilers|all)(^\w[\w-]*)"
)
#: Matches a valid name for a module set
valid_module_set_name = r"^(?!prefix_inspections$)\w[\w-]*$"
#: Matches an anonymous spec, i.e. a spec without a root name
anonymous_spec_regex = r"^[\^@%+~]"
#: Definitions for parts of module schema
array_of_strings = {"type": "array", "default": [], "items": {"type": "string"}}
dictionary_of_strings = {"type": "object", "patternProperties": {r"\w[\w-]*": {"type": "string"}}}
dependency_selection = {"type": "string", "enum": ["none", "run", "direct", "all"]}
module_file_configuration = {
"type": "object",
"default": {},
"additionalProperties": False,
"properties": {
"filter": {
"type": "object",
"default": {},
"additionalProperties": False,
"properties": {
"exclude_env_vars": {"type": "array", "default": [], "items": {"type": "string"}}
},
},
"template": {"type": "string"},
"autoload": dependency_selection,
"prerequisites": dependency_selection,
"conflict": array_of_strings,
"load": array_of_strings,
"suffixes": {
"type": "object",
"validate_spec": True,
"patternProperties": {r"\w[\w-]*": {"type": "string"}}, # key
},
"environment": spack.schema.environment.definition,
},
}
projections_scheme = spack.schema.projections.properties["projections"]
module_type_configuration = {
"type": "object",
"default": {},
"allOf": [
{
"properties": {
"verbose": {"type": "boolean", "default": False},
"hash_length": {"type": "integer", "minimum": 0, "default": 7},
"include": array_of_strings,
"exclude": array_of_strings,
"exclude_implicits": {"type": "boolean", "default": False},
"defaults": array_of_strings,
"hide_implicits": {"type": "boolean", "default": False},
"naming_scheme": {"type": "string"}, # Can we be more specific here?
"projections": projections_scheme,
"all": module_file_configuration,
}
},
{
"validate_spec": True,
"patternProperties": {
spec_regex: module_file_configuration,
anonymous_spec_regex: module_file_configuration,
},
},
],
}
module_config_properties = {
"use_view": {"anyOf": [{"type": "string"}, {"type": "boolean"}]},
"arch_folder": {"type": "boolean"},
"roots": {
"type": "object",
"properties": {"tcl": {"type": "string"}, "lmod": {"type": "string"}},
},
"enable": {
"type": "array",
"default": [],
"items": {"type": "string", "enum": ["tcl", "lmod"]},
},
"lmod": {
"allOf": [
# Base configuration
module_type_configuration,
{
"type": "object",
"properties": {
"core_compilers": array_of_strings,
"hierarchy": array_of_strings,
"core_specs": array_of_strings,
"filter_hierarchy_specs": {
"type": "object",
"patternProperties": {spec_regex: array_of_strings},
},
},
}, # Specific lmod extensions
]
},
"tcl": {
"allOf": [
# Base configuration
module_type_configuration,
{}, # Specific tcl extensions
]
},
"prefix_inspections": {
"type": "object",
"additionalProperties": False,
"patternProperties": {
# prefix-relative path to be inspected for existence
r"^[\w-]*": array_of_strings
},
},
}โ ๏ธ real talk: that spec_regex comment is fr fr shouting at you โ THIS NEEDS TO BE UPDATED FOR EVERY NEW KEYWORD THAT IS ADDED IMMEDIATELY BELOW THE MODULE TYPE ATTRIBUTE. Do not be the person who forgets, bestie ๐ญ
so yeah, that's the module schema in all its glory โ tcl and lmod each get the base module_type_configuration meal plus their own little side dishes, prefix_inspections maps a prefix-relative path to be inspected for existence onto an array of strings, and the whole thing is giving very organized, very slay ๐ฆ
# Properties for inclusion into other schemas (requires definitions)
properties: Dict[str, Any] = {
"modules": {
"type": "object",
"additionalProperties": False,
"properties": {
"prefix_inspections": {
"type": "object",
"additionalProperties": False,
"patternProperties": {
# prefix-relative path to be inspected for existence
r"^[\w-]*": array_of_strings
},
}
},
"patternProperties": {
valid_module_set_name: {
"type": "object",
"default": {},
"additionalProperties": False,
"properties": module_config_properties,
}
},
}
}
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack module file configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}okay bestie, that block you just scrolled past is the actual blueprint ๐
โ like, this is the properties dict that gets included into other schemas (and yeah, it requires definitions, so don't come at it empty-handed ๐). Notice how "modules" is a strict little object: "additionalProperties": False, because we do not do random surprise keys around here, bestie ๐. Inside it, "prefix_inspections" is also closed off, and its "patternProperties" are keyed by a regex โ r"^[\w-]*" โ which is the prefix-relative path to be inspected for existence ๐ธ. And every single one of those matches has to be an array_of_strings, periodt.
Then there's "patternProperties" at the modules level too, keyed by valid_module_set_name, and each of those is an object that defaults to {}, refuses extra properties, and whose "properties" is exactly module_config_properties โจ.
And the finale, the glow-up: #: Full schema with metadata โ the full schema dict that wraps it all up with "$schema" set to "http://json-schema.org/draft-07/schema#", a "title" of "Spack module file configuration file schema", "type": "object", "additionalProperties": False, and "properties": properties. Iconic ๐.
spack.schema.modules.anonymous_spec_regex = '^[\\^@%+~]'โ the ultimate vibe check for anonymous specs ๐ โ it matches an anonymous spec, i.e. a spec without a root name. If it kicks off with one of those sigils, it's out here with no name, main character energy only ๐ฅ. Keep the escaping exact, bestie: that's^[\\^@%+~]fr fr ๐ .spack.schema.modules.array_of_strings = {'default': [], 'items': {'type': 'string'}, 'type': 'array'}โ the definitions for parts of the module schema ๐ง, and honestly it's giving "list of just strings, no drama":'type': 'array','items': {'type': 'string'}, and'default': []so it starts empty and unbothered ๐ฆ.
okay bestie, so this is the part where the manpage literally just drops the whole schema dict on you โ like, no preamble, no warm-up, just spack.schema.modules.schema in all its glory ๐
and honestly? it's giving "raw data, no notes." so here it is, exactly as printed, backslashes and all:
spack.schema.modules.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'modules': {'additionalProperties': False,
'patternProperties': {'^(?!prefix_inspections$)\\w[\\w-]*$':
{'additionalProperties': False, 'default': {}, 'properties':
{'arch_folder': {'type': 'boolean'}, 'enable': {'default': [], 'items':
{'enum': ['tcl', 'lmod'], 'type': 'string'}, 'type': 'array'}, 'lmod':
{'allOf': [{'allOf': [{'properties': {'all': {'additionalProperties':
False, 'default': {}, 'properties': {'autoload': {'enum': ['none',
'run', 'direct', 'all'], 'type': 'string'}, 'conflict': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}, 'environment':
{'additionalProperties': False, 'default': {}, 'properties':
{'append_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'filter': {'additionalProperties': False, 'default': {}, 'properties':
{'exclude_env_vars': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}, 'load': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'prerequisites': {'enum': ['none',
'run', 'direct', 'all'], 'type': 'string'}, 'suffixes':
{'patternProperties': {'\\w[\\w-]*': {'type': 'string'}}, 'type':
'object', 'validate_spec': True}, 'template': {'type': 'string'}},
'type': 'object'}, 'defaults': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'exclude': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'exclude_implicits': {'default':
False, 'type': 'boolean'}, 'hash_length': {'default': 7, 'minimum': 0,
'type': 'integer'}, 'hide_implicits': {'default': False, 'type':
'boolean'}, 'include': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'naming_scheme': {'type': 'string'}, 'projections':
{'patternProperties': {'all|\\w[\\w-]*': {'type': 'string'}}, 'type':
'object'}, 'verbose': {'default': False, 'type': 'boolean'}}},
{'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}, '^[\\^@%+~]':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}}, 'validate_spec': True}],
'default': {}, 'type': 'object'}, {'properties': {'core_compilers':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'core_specs': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}, 'filter_hierarchy_specs': {'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type':
'object'}, 'hierarchy': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}]}, 'prefix_inspections':
{'additionalProperties': False, 'patternProperties': {'^[\\w-]*':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type':
'object'}, 'roots': {'properties': {'lmod': {'type': 'string'}, 'tcl':
{'type': 'string'}}, 'type': 'object'}, 'tcl': {'allOf': [{'allOf':
[{'properties': {'all': {'additionalProperties': False, 'default': {},
'properties': {'autoload': {'enum': ['none', 'run', 'direct', 'all'],
'type': 'string'}, 'conflict': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'environment': {'additionalProperties':
False, 'default': {}, 'properties': {'append_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'filter': {'additionalProperties': False, 'default': {}, 'properties':
{'exclude_env_vars': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}, 'load': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'prerequisites': {'enum': ['none',
'run', 'direct', 'all'], 'type': 'string'}, 'suffixes':
{'patternProperties': {'\\w[\\w-]*': {'type': 'string'}}, 'type':
'object', 'validate_spec': True}, 'template': {'type': 'string'}},
'type': 'object'}, 'defaults': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'exclude': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'exclude_implicits': {'default':
False, 'type': 'boolean'}, 'hash_length': {'default': 7, 'minimum': 0,
'type': 'integer'}, 'hide_implicits': {'default': False, 'type':
'boolean'}, 'include': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'naming_scheme': {'type': 'string'}, 'projections':
{'patternProperties': {'all|\\w[\\w-]*': {'type': 'string'}}, 'type':
'object'}, 'verbose': {'default': False, 'type': 'boolean'}}},
{'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':okaay so if you actually read it instead of letting your eyes glaze over ๐ซถ, you'll notice the same lil squad keeps showing up in every block: autoload, conflict, environment, filter, load, prerequisites, suffixes, template โ like a friend group that goes everywhere together ๐ญ and then the outer layer has all, defaults, exclude, exclude_implicits, hash_length, hide_implicits, include, naming_scheme, projections, verbose doing its own thing.
bestie note: we literally cut off mid-token there on 'type' โ that's not me being sloppy, that's just where this part of the page ends ๐ the next chunk picks the dict right back up, no cap.
and fr, the negation patterns are doing the absolute most โ '(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)' is a whole mood โจ and '^[\\^@%+~]' is lowkey iconic for reasons the next part will absolutely get into ๐ธ
okay bestie, we're still deep in the schema mines and I need you to trust me โ this next chunk is the tail end of the modules schema, like the part that actually describes what lives inside your module files. stay with me, it's giving "homework but make it fashion" ๐
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}}, 'validate_spec': True}],
'default': {}, 'type': 'object'}, {}]}, 'use_view': {'anyOf': [{'type':
'string'}, {'type': 'boolean'}]}}, 'type': 'object'}}, 'properties':
{'prefix_inspections': {'additionalProperties': False,
'patternProperties': {'^[\\w-]*': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}}, 'type': 'object'}}, 'type': 'object'}},
'title': 'Spack module file configuration file schema', 'type':
'object'}so what's actually in there, bestie? a filter block with exclude_env_vars (array of strings, defaults to empty), load (array of strings, default []), prerequisites which is an enum of exactly none, run, direct, or all, a suffixes object whose keys match \\w[\\w-]* and whose values are strings (and yes, validate_spec is True in there), a template string, use_view which is anyOf a string or a boolean, and prefix_inspections with keys matching ^[\\w-]* mapping to arrays of strings. the whole thing is titled 'Spack module file configuration file schema' and, obviously, 'type': 'object'. iconic, no notes ๐
โจ the two little helpers โจ
spack.schema.modules.spec_regexโ Matches a spec or a multi-valued variant but not another valid keyword. and then the source is SCREAMING at you in all caps, so I'm gonna let it scream:
(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)and the all-caps note, verbatim, because it is so serious: THIS NEEDS TO BE UPDATED FOR EVERY NEW KEYWORD THAT IS ADDED IMMEDIATELY BELOW THE MODULE TYPE ATTRIBUTE ๐จ like bestie they are not playing around, if you add a keyword and forget this regex the whole vibe collapses fr fr
spack.schema.modules.valid_module_set_nameโ Matches a valid name for a module set. here's the pattern:
'^(?!prefix_inspections$)\\w[\\w-]*$'it literally says "anything that's a word-ish name, except don't you dare call your module set prefix_inspections" ๐ญ respect the reserved word, periodt.
๐ธ spack.schema.packages module ๐ธ
okay this next one is the big one โ Schema for packages.yaml configuration files. and it is a whole mood board of Python dicts. I'm gonna walk you through it chunk by chunk because dumping it in one go is giving "wall of text jumpscare" ๐ญ
first up, permissions and variants:
permissions = {
"type": "object",
"additionalProperties": False,
"properties": {
"read": {"type": "string", "enum": ["user", "group", "world"]},
"write": {"type": "string", "enum": ["user", "group", "world"]},
"group": {"type": "string"},
},
}
variants = {"oneOf": [{"type": "string"}, {"type": "array", "items": {"type": "string"}}]}so permissions has read and write, both locked to the enum user, group, or world, plus a plain-string group. and variants is either a single string or an array of strings โ pick your fighter ๐ฆ
next, requirements โ this one has lore:
requirements = {
"oneOf": [
# 'require' can be a list of requirement_groups.
# each requirement group is a list of one or more
# specs. Either at least one or exactly one spec
# in the group must be satisfied (depending on
# whether you use "any_of" or "one_of",
# repectively)
{
"type": "array",
"items": {
"oneOf": [
{
"type": "object",
"additionalProperties": False,
"properties": {
"one_of": {"type": "array", "items": {"type": "string"}},
"any_of": {"type": "array", "items": {"type": "string"}},
"spec": {"type": "string"},
"message": {"type": "string"},
"when": {"type": "string"},
},
},
{"type": "string"},
]
},
},
# Shorthand for a single requirement group with
# one member
{"type": "string"},
]
}so require is either a list of requirement groups, or one plain string as shorthand for a single group with one member. inside a group you can use one_of (exactly one member must be satisfied), any_of (at least one must be satisfied), or spec, plus optional message and when. and yes bestie, the comment in the source says "repectively" โ that's not me, that's in the code, we love a typo that ships ๐
then prefer_and_conflict, which is the chill cousin:
prefer_and_conflict = {
"type": "array",
"items": {
"oneOf": [
{
"type": "object",
"additionalProperties": False,
"properties": {
"spec": {"type": "string"},
"message": {"type": "string"},
"when": {"type": "string"},
},
},
{"type": "string"},
]
},
}an array where each item is either an object with spec / message / when, or just a plain string. simple, clean, slay โจ
and then โ not a drill โ permissions shows up again, identical, copy-pasted in the source:
permissions = {
"type": "object",
"additionalProperties": False,
"properties": {
"read": {"type": "string", "enum": ["user", "group", "world"]},
"write": {"type": "string", "enum": ["user", "group", "world"]},
"group": {"type": "string"},
},
}
package_attributes = {
"type": "object",
"additionalProperties": False,
"patternProperties": {r"\w+": {}},
}
REQUIREMENT_URL = "https://spack.readthedocs.io/en/latest/packages_yaml.html#package-requirements"that's right, permissions twice. I'm not gaslighting you, it's literally in the file twice ๐ package_attributes is an object whose keys match \w+ and whose values are unconstrained ({}), and REQUIREMENT_URL is that docs link, which we will be seeing again very soon โ foreshadowing ๐ง
and now the main event: properties: Dict[str, Any]. deep breath. this is the shape of packages.yaml itself.
#: Properties for inclusion in other schemas
properties: Dict[str, Any] = {
"packages": {
"type": "object",
"default": {},
"additionalProperties": False,
"properties": {
"all": { # package name
"type": "object",
"default": {},
"additionalProperties": False,
"properties": {
"require": requirements,
"prefer": prefer_and_conflict,
"conflict": prefer_and_conflict,
"version": {}, # Here only to warn users on ignored properties
"target": {
"type": "array",
"default": [],
# target names
"items": {"type": "string"},
},
"compiler": {
"type": "array",
"default": [],
"items": {"type": "string"},
}, # compiler specs
"buildable": {"type": "boolean", "default": True},
"permissions": permissions,
# If 'get_full_repo' is promoted to a Package-level
# attribute, it could be useful to set it here
"package_attributes": package_attributes,
"providers": {
"type": "object",
"default": {},
"additionalProperties": False,
"patternProperties": {
r"\w[\w-]*": {
"type": "array",
"default": [],
"items": {"type": "string"},
}
},
},
"variants": variants,
},
"deprecatedProperties": {
"properties": ["version"],
"message": "setting version preferences in the 'all' section of packages.yaml "
"is deprecated and will be removed in v0.23\n\n\tThese preferences "
"will be ignored by Spack. You can set them only in package-specific sections "
"of the same file.\n",
"error": False,
},
}
},so the all section (the one that applies to every package) can carry require, prefer, conflict, version, target (array of target names, default []), compiler (array of compiler specs, default []), buildable (boolean, default True), permissions, package_attributes, providers (object whose keys match \w[\w-]* and whose values are arrays of strings, default []), and variants.
BUT ๐จ โ version in the all section is deprecated: "setting version preferences in the 'all' section of packages.yaml is deprecated and will be removed in v0.23", those preferences will be ignored by Spack, and you can only set them in package-specific sections of the same file. note "error": False, so it warns rather than hard-fails. love a gentle warning, honestly ๐ซถ
and the second half of properties โ the per-package pattern:
"patternProperties": {
r"(?!^all$)(^\w[\w-]*)": { # package name
"type": "object",
"default": {},
"additionalProperties": False,
"properties": {
"require": requirements,
"prefer": prefer_and_conflict,
"conflict": prefer_and_conflict,
"version": {
"type": "array",
"default": [],
# version strings
"items": {"anyOf": [{"type": "string"}, {"type": "number"}]},
},
"target": {}, # Here only to warn users on ignored properties
"compiler": {}, # Here only to warn users on ignored properties
"buildable": {"type": "boolean", "default": True},
"permissions": permissions,
# If 'get_full_repo' is promoted to a Package-level
# attribute, it could be useful to set it here
"package_attributes": package_attributes,
"providers": {}, # Here only to warn users on ignored properties
"variants": variants,
"externals": {
"type": "array",
"items": {
"type": "object",
"properties": {
"spec": {"type": "string"},
"prefix": {"type": "string"},
"modules": {"type": "array", "items": {"type": "string"}},
"extra_attributes": {
"type": "object",
"additionalProperties": True,
"properties": {
"environment": spack.schema.environment.definition
},
},
},
"additionalProperties": True,
"required": ["spec"],
},
},
},
"deprecatedProperties": {
"properties": ["target", "compiler", "providers"],
"message": "setting 'compiler:', 'target:' or 'provider:' preferences in "
"a package-specific section of packages.yaml is deprecated, and will be "
"removed in v0.23.\n\n\tThese preferences will be ignored by Spack, and "
"can be set only in the 'all' section of the same file. "
"You can run:\n\n\t\t$ spack audit configs\n\n\tto get better diagnostics, "
"including files:lines where the deprecated attributes are used.\n\n"
"\tUse requirements to enforce conditions on specific packages: "
f"{REQUIREMENT_URL}\n",
"error": False,
},
}
},
}
}the key pattern is r"(?!^all$)(^\w[\w-]*)" โ any word-ish package name except all, because all already got its own section above, and we are not doubling up ๐ค
in a package-specific section you get require, prefer, conflict, version (array of version strings โ strings or numbers, default []), target, compiler, buildable (boolean, default True), permissions, package_attributes, providers, variants, and externals.
externals is an array of objects where spec is required โ
, prefix is a string, modules is an array of strings, and extra_attributes is an object with additionalProperties: True that can hold environment (which is spack.schema.environment.definition). also additionalProperties: True at the external level, so extras are allowed. generous queen ๐
AND the deprecation tea โ โ target, compiler, and providers in a package-specific section are deprecated and removed in v0.23, they will be ignored by Spack, and they can only be set in the all section of the same file. the message literally tells you to run:
$ spack audit configsfor better diagnostics, "including files:lines where the deprecated attributes are used". and then it points you at REQUIREMENT_URL (that https://spack.readthedocs.io/en/latest/packages_yaml.html#package-requirements link from earlier โ told you it'd come back ๐ง) to enforce conditions on specific packages. also "error": False here too, so it's a warning, not a vibe-killer.
okay that's the full dict, and bestie there is so much more coming โ don't you dare close the tab ๐ญ๐
ok so this next bit is the actual schema โ the bouncer at the door of your packages.yaml ๐
like it's straight-up JSON Schema, draft-07, and it does not play about extra keys:
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack package configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}
def update(data):
changed = False
for key in data:
version = data[key].get("version")
if not version or all(isinstance(v, str) for v in version):
continue
data[key]["version"] = [str(v) for v in version]
changed = True
return changedbestie look at that update function for one sec ๐ โ it walks every key in your data, grabs the "version" field, and if there's nothing there or literally every entry is already a string, it just continues like "nothing to see here โจ". But if any version is, say, a number, it rewrites the whole list as [str(v) for v in version] and flips changed to True. Then it hands you back that boolean. that's it, that's the whole function, no cap ๐ซถ
also note "additionalProperties": False up top โ so yeah, make up your own keys at your own risk, the schema said what it said ๐ญ
๐ spack.schema.packages.properties ๐
and here's the big one, the full spack.schema.packages.properties: Dict[str, Any] dump. it's giving wall-of-text, it's giving "I memorized the whole thing," but it's the real deal and you're gonna want it:
spack.schema.packages.properties: Dict[str, Any] = {'packages':
{'additionalProperties': False, 'default': {}, 'patternProperties':
{'(?!^all$)(^\\w[\\w-]*)': {'additionalProperties': False, 'default':
{}, 'deprecatedProperties': {'error': False, 'message': "setting
'compiler:', 'target:' or 'provider:' preferences in a package-specific
section of packages.yaml is deprecated, and will be removed in
v0.23.\n\n\tThese preferences will be ignored by Spack, and can be set
only in the 'all' section of the same file. You can run:\n\n\t\t$ spack
audit configs\n\n\tto get better diagnostics, including files:lines
where the deprecated attributes are used.\n\n\tUse requirements to
enforce conditions on specific packages:
https://spack.readthedocs.io/en/latest/packages_yaml.html#package-requirements\n",
'properties': ['target', 'compiler', 'providers']}, 'properties':
{'buildable': {'default': True, 'type': 'boolean'}, 'compiler': {},
'conflict': {'items': {'oneOf': [{'additionalProperties': False,
'properties': {'message': {'type': 'string'}, 'spec': {'type':
'string'}, 'when': {'type': 'string'}}, 'type': 'object'}, {'type':
'string'}]}, 'type': 'array'}, 'externals': {'items':
{'additionalProperties': True, 'properties': {'extra_attributes':
{'additionalProperties': True, 'properties': {'environment':
{'additionalProperties': False, 'default': {}, 'properties':
{'append_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}},
'type': 'object'}, 'modules': {'items': {'type': 'string'}, 'type':
'array'}, 'prefix': {'type': 'string'}, 'spec': {'type': 'string'}},
'required': ['spec'], 'type': 'object'}, 'type': 'array'},
'package_attributes': {'additionalProperties': False,
'patternProperties': {'\\w+': {}}, 'type': 'object'}, 'permissions':
{'additionalProperties': False, 'properties': {'group': {'type':
'string'}, 'read': {'enum': ['user', 'group', 'world'], 'type':
'string'}, 'write': {'enum': ['user', 'group', 'world'], 'type':
'string'}}, 'type': 'object'}, 'prefer': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'}, 'providers':
{}, 'require': {'oneOf': [{'items': {'oneOf': [{'additionalProperties':
False, 'properties': {'any_of': {'items': {'type': 'string'}, 'type':
'array'}, 'message': {'type': 'string'}, 'one_of': {'items': {'type':
'string'}, 'type': 'array'}, 'spec': {'type': 'string'}, 'when':
{'type': 'string'}}, 'type': 'object'}, {'type': 'string'}]}, 'type':
'array'}, {'type': 'string'}]}, 'target': {}, 'variants': {'oneOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'version': {'default': [], 'items': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}},
'properties': {'all': {'additionalProperties': False, 'default': {},
'deprecatedProperties': {'error': False, 'message': "setting version
preferences in the 'all' section of packages.yaml is deprecated and will
be removed in v0.23\n\n\tThese preferences will be ignored by Spack. You
can set them only in package-specific sections of the same file.\n",
'properties': ['version']}, 'properties': {'buildable': {'default':
True, 'type': 'boolean'}, 'compiler': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'conflict': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'},
'package_attributes': {'additionalProperties': False,
'patternProperties': {'\\w+': {}}, 'type': 'object'}, 'permissions':
{'additionalProperties': False, 'properties': {'group': {'type':
'string'}, 'read': {'enum': ['user', 'group', 'world'], 'type':
'string'}, 'write': {'enum': ['user', 'group', 'world'], 'type':
'string'}}, 'type': 'object'}, 'prefer': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'}, 'providers':
{'additionalProperties': False, 'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}}, 'type': 'object'}, 'require': {'oneOf': [{'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'any_of': {'items':
{'type': 'string'}, 'type': 'array'}, 'message': {'type': 'string'},
'one_of': {'items': {'type': 'string'}, 'type': 'array'}, 'spec':
{'type': 'string'}, 'when': {'type': 'string'}}, 'type': 'object'},
{'type': 'string'}]}, 'type': 'array'}, {'type': 'string'}]}, 'target':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'variants': {'oneOf': [{'type': 'string'}, {'items': {'type': 'string'},
'type': 'array'}]}, 'version': {}}, 'type': 'object'}}, 'type':
'object'}}okay breathe ๐ฌ๏ธ โ I know. let me walk you through the spicy parts so you're not just staring at brackets for the rest of the night:
- the
allsection is special โ see that'(?!^all$)(^\\w[\\w-]*)'? that's a negative lookahead, bestie. it matches any\w[\\w-]*word key except literalall. soallgets its own dedicatedpropertiesblock instead of being swept up with everything else. icon behavior, honestly ๐ - deprecated property #1 โ inside a package-specific section, setting
compiler:,target:orprovider:preferences is deprecated and will be removed inv0.23. and it's not just a warning you can sleep on: these preferences will be ignored by Spack, and can be set only in theallsection of the same file. the message literally tells you to run$ spack audit configsfor better diagnostics, including file:line info on where the deprecated attributes are used, and points you at requirements for enforcing conditions on specific packages:https://spack.readthedocs.io/en/latest/packages_yaml.html#package-requirements. also note'error': Falseโ so it's a deprecation, not a hard fail ๐ - deprecated property #2 โ flip side, in the
allsection: setting version preferences is deprecated and will be removed inv0.23. These preferences will be ignored by Spack. you can set them only in package-specific sections of the same file. same'error': Falseenergy. so like โallhandles compiler/target/providers, package sections handle versions. don't mix 'em up ๐ - the shared squad โ both sections carry
buildable(defaultTrue, boolean),conflict,package_attributes(additionalProperties: False,patternProperties: {'\\w+': {}}),permissions(withgroup, andread/writerestricted to the enum'user','group','world'),prefer,require, andvariants(string or array of strings,oneOfstyle โจ) requireis the drama queen โoneOfa plain string, or an array whose items are either strings or objects carryingany_of,one_of,message,specandwhen. so much range ๐ญexternals</braces> โ each item requiresspec;prefixandmodules(array of strings) are in there too, plusextra_attributeswhich isadditionalProperties: Trueand can carryenvironmentwithappend_path,prepend_path,remove_path,setandunset. the path ones are objects matching\\w[\\w-]*keys whose values areanyOfstring-or-number;unsetis an array withdefault: []where each item is string-or-number. genuinely a whole ecosystem in one key ๐- sneaky detail โ in the package-specific block,
versionhas'default': []and itemsanyOfstring-or-number. in theallblock,versionis just{}โ wide open in terms of schema, because Spack's gonna ignore it there anyway (see deprecated property #2 ๐ซ ). andcompileris{}in the package block but a proper string array withdefault: []inall. yep. that's the whole point.
so the moral of the story: put your versions in the package sections, put your compiler/target/providers in all, and if you're getting yelled at, spack audit configs is your bestie ๐
and the last line of this whole thing, plain as anything:
Properties for inclusion in other schemas
which โ lowkey iconic closer, ngl. that's the one-sentence summary of the entire dict we just read. these properties don't live alone, they get pulled into other schemas elsewhere. $ref-adjacent behavior, iykyk ๐ฆ and like, obviously there's more coming, so don't go anywhere ๐
...okay so this next blob is the packages schema in all its glory, and bestie, it is a lot โ it's giving "final boss of JSON schemas" ๐. Strap in, we're reading it verbatim because every character here is load-bearing:
spack.schema.packages.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'packages': {'additionalProperties': False,
'default': {}, 'patternProperties': {'(?!^all$)(^\\w[\\w-]*)':
{'additionalProperties': False, 'default': {}, 'deprecatedProperties':
{'error': False, 'message': "setting 'compiler:', 'target:' or
'provider:' preferences in a package-specific section of packages.yaml
is deprecated, and will be removed in v0.23.\n\n\tThese preferences will
be ignored by Spack, and can be set only in the 'all' section of the
same file. You can run:\n\n\t\t$ spack audit configs\n\n\tto get better
diagnostics, including files:lines where the deprecated attributes are
used.\n\n\tUse requirements to enforce conditions on specific packages:
https://spack.readthedocs.io/en/latest/packages_yaml.html#package-requirements\n",
'properties': ['target', 'compiler', 'providers']}, 'properties':
{'buildable': {'default': True, 'type': 'boolean'}, 'compiler': {},
'conflict': {'items': {'oneOf': [{'additionalProperties': False,
'properties': {'message': {'type': 'string'}, 'spec': {'type':
'string'}, 'when': {'type': 'string'}}, 'type': 'object'}, {'type':
'string'}]}, 'type': 'array'}, 'externals': {'items':
{'additionalProperties': True, 'properties': {'extra_attributes':
{'additionalProperties': True, 'properties': {'environment':
{'additionalProperties': False, 'default': {}, 'properties':
{'append_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}},
'type': 'object'}, 'modules': {'items': {'type': 'string'}, 'type':
'array'}, 'prefix': {'type': 'string'}, 'spec': {'type': 'string'}},
'required': ['spec'], 'type': 'object'}, 'type': 'array'},
'package_attributes': {'additionalProperties': False,
'patternProperties': {'\\w+': {}}, 'type': 'object'}, 'permissions':
{'additionalProperties': False, 'properties': {'group': {'type':
'string'}, 'read': {'enum': ['user', 'group', 'world'], 'type':
'string'}, 'write': {'enum': ['user', 'group', 'world'], 'type':
'string'}}, 'type': 'object'}, 'prefer': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'}, 'providers':
{}, 'require': {'oneOf': [{'items': {'oneOf': [{'additionalProperties':
False, 'properties': {'any_of': {'items': {'type': 'string'}, 'type':
'array'}, 'message': {'type': 'string'}, 'one_of': {'items': {'type':
'string'}, 'type': 'array'}, 'spec': {'type': 'string'}, 'when':
{'type': 'string'}}, 'type': 'object'}, {'type': 'string'}]}, 'type':
'array'}, {'type': 'string'}]}, 'target': {}, 'variants': {'oneOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'version': {'default': [], 'items': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}},
'properties': {'all': {'additionalProperties': False, 'default': {},
'deprecatedProperties': {'error': False, 'message': "setting version
preferences in the 'all' section of packages.yaml is deprecated and will
be removed in v0.23\n\n\tThese preferences will be ignored by Spack. You
can set them only in package-specific sections of the same file.\n",
'properties': ['version']}, 'properties': {'buildable': {'default':
True, 'type': 'boolean'}, 'compiler': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'conflict': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'},
'package_attributes': {'additionalProperties': False,
'patternProperties': {'\\w+': {}}, 'type': 'object'}, 'permissions':
{'additionalProperties': False, 'properties': {'group': {'type':
'string'}, 'read': {'enum': ['user', 'group', 'world'], 'type':
'string'}, 'write': {'enum': ['user', 'group', 'world'], 'type':
'string'}}, 'type': 'object'}, 'prefer': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'}, 'providers':
{'additionalProperties': False, 'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}}, 'type': 'object'}, 'require': {'oneOf': [{'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'any_of': {'items':
{'type': 'string'}, 'type': 'array'}, 'message': {'type': 'string'},
'one_of': {'items': {'type': 'string'}, 'type': 'array'}, 'spec':
{'type': 'string'}, 'when': {'type': 'string'}}, 'type': 'object'},
{'type': 'string'}]}, 'type': 'array'}, {'type': 'string'}]}, 'target':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'variants': {'oneOf': [{'type': 'string'}, {'items': {'type': 'string'},
'type': 'array'}]}, 'version': {}}, 'type': 'object'}}, 'type':
'object'}}, 'title': 'Spack package configuration file schema', 'type':
'object'}Full schema with metadata โจ
Next up, one (1) function signature, no description attached โ so I'm just gonna hand it to you raw, no notes:
spack.schema.packages.update(data)๐ฆ spack.schema.projections module (it's giving "view projection config" energy)
Schema for projections.yaml configuration file. Here's the literal source, bestie:
properties: Dict[str, Any] = {
"projections": {"type": "object", "patternProperties": {r"all|\w[\w-]*": {"type": "string"}}}
}
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack view projection configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}spack.schema.projections.properties: Dict[str, Any] = {'projections': {'patternProperties': {'all|\\w[\\w-]*': {'type': 'string'}}, 'type': 'object'}}
Properties for inclusion in other schemas ๐ซถ
spack.schema.projections.schema = {'$schema': 'http://json-schema.org/draft-07/schema#', 'additionalProperties': False, 'properties': {'projections': {'patternProperties': {'all|\\w[\\w-]*': {'type': 'string'}}, 'type': 'object'}}, 'title': 'Spack view projection configuration file schema', 'type': 'object'}
Full schema with metadata ๐
๐ spack.schema.repos module (short and sweet, unlike the last one)
Schema for repos.yaml configuration file. Look at this little cutie:
properties: Dict[str, Any] = {
"repos": {"type": "array", "default": [], "items": {"type": "string"}}
}
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack repository configuration file schema",
"type": "object",
"additionalProperties": False,
"properties": properties,
}spack.schema.repos.properties: Dict[str, Any] = {'repos': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}
Properties for inclusion in other schemas ๐
spack.schema.repos.schema = {'$schema': 'http://json-schema.org/draft-07/schema#', 'additionalProperties': False, 'properties': {'repos': {'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'Spack repository configuration file schema', 'type': 'object'}
Full schema with metadata โญ
๐ธ spack.schema.spec module
Schema for a spec found in spec descriptor or database index.json files.
ok so the original literally leaves a TODO here and I'm not gonna pretend otherwise โ it says, quote: "TODO: This needs to be updated? Especially the hashes under properties." ๐ญ Respect the honesty tbh.
target = {
"oneOf": [
{"type": "string"},
{
"type": "object",
"additionalProperties": False,
"required": ["name", "vendor", "features", "generation", "parents"],
"properties": {
"name": {"type": "string"},
"vendor": {"type": "string"},
"features": {"type": "array", "items": {"type": "string"}},
"generation": {"type": "integer"},
"parents": {"type": "array", "items": {"type": "string"}},
},
},
]
}
arch = {
"type": "object",
"additionalProperties": False,
"properties": {"platform": {}, "platform_os": {}, "target": target},
}
dependencies = {
"type": "object",
"patternProperties": {
r"\w[\w-]*": { # package name
"type": "object",
"properties": {
"hash": {"type": "string"},
"type": {"type": "array", "items": {"type": "string"}},
},
}
},
}
build_spec = {
"type": "object",
"additionalProperties": False,
"required": ["name", "hash"],
"properties": {"name": {"type": "string"}, "hash": {"type": "string"}},
}Every key, every required list, every additionalProperties: False โ no cap, it's all above, untouched ๐ฅ
Okay so this is the actual properties dict for the spec schema ๐
and honestly it's a whole mood โ nested dicts inside nested dicts, a #: comment about 0.18, the works. Do not retype any of it from memory, bestie, copy it character for character:
#: Properties for inclusion in other schemas
properties: Dict[str, Any] = {
"spec": {
"type": "object",
"additionalProperties": False,
"required": ["_meta", "nodes"],
"properties": {
"_meta": {"type": "object", "properties": {"version": {"type": "number"}}},
"nodes": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": False,
"required": ["version", "arch", "compiler", "namespace", "parameters"],
"properties": {
"name": {"type": "string"},
"hash": {"type": "string"},
"package_hash": {"type": "string"},
# these hashes were used on some specs prior to 0.18
"full_hash": {"type": "string"},
"build_hash": {"type": "string"},
"version": {"oneOf": [{"type": "string"}, {"type": "number"}]},
"arch": arch,
"compiler": {
"type": "object",
"additionalProperties": False,
"properties": {
"name": {"type": "string"},
"version": {"type": "string"},
},
},
"develop": {"anyOf": [{"type": "boolean"}, {"type": "string"}]},
"namespace": {"type": "string"},
"parameters": {
"type": "object",
"required": [
"cflags",
"cppflags",
"cxxflags",
"fflags",
"ldflags",
"ldlibs",
],
"additionalProperties": True,
"properties": {
"patches": {"type": "array", "items": {"type": "string"}},
"cflags": {"type": "array", "items": {"type": "string"}},
"cppflags": {"type": "array", "items": {"type": "string"}},
"cxxflags": {"type": "array", "items": {"type": "string"}},
"fflags": {"type": "array", "items": {"type": "string"}},
"ldflags": {"type": "array", "items": {"type": "string"}},
"ldlib": {"type": "array", "items": {"type": "string"}},
},
},
"patches": {"type": "array", "items": {}},
"dependencies": dependencies,
"build_spec": build_spec,
},
},
},
},
}
}Literally the definition of nested โ a spec whose nodes are an array whose items are objects whose parameters are objects whoseโฆ you get it. ๐ช The #: comment on full_hash/build_hash is doing lore duty: those hashes were used on some specs prior to 0.18.
And the docstring attached to all that? Just:
Properties for inclusion in other schemas
Short. Iconic. No notes. โจ
And next up is the full schema with the metadata bolted on top โ same energy, one more layer of wrapper, patternProperties instead of properties at the root:
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack spec schema",
"type": "object",
"additionalProperties": False,
"patternProperties": properties,
}Its docstring, meanwhile, is serving:
Full schema with metadata
Bet. ๐
๐ spack.schema.spec.properties (as rendered, bestie)
So when the docs dump the attribute, it looks like this โ yes it's the same info, yes it's printed as one glorious wall. Keep it as-is:
spack.schema.spec.properties: Dict[str, Any] = {'spec':
{'additionalProperties': False, 'properties': {'_meta': {'properties':
{'version': {'type': 'number'}}, 'type': 'object'}, 'nodes': {'items':
{'additionalProperties': False, 'properties': {'arch':
{'additionalProperties': False, 'properties': {'platform': {},
'platform_os': {}, 'target': {'oneOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'features': {'items':
{'type': 'string'}, 'type': 'array'}, 'generation': {'type': 'integer'},
'name': {'type': 'string'}, 'parents': {'items': {'type': 'string'},
'type': 'array'}, 'vendor': {'type': 'string'}}, 'required': ['name',
'vendor', 'features', 'generation', 'parents'], 'type': 'object'}]}},
'type': 'object'}, 'build_hash': {'type': 'string'}, 'build_spec':
{'additionalProperties': False, 'properties': {'hash': {'type':
'string'}, 'name': {'type': 'string'}}, 'required': ['name', 'hash'],
'type': 'object'}, 'compiler': {'additionalProperties': False,
'properties': {'name': {'type': 'string'}, 'version': {'type':
'string'}}, 'type': 'object'}, 'dependencies': {'patternProperties':
{'\\w[\\w-]*': {'properties': {'hash': {'type': 'string'}, 'type':
{'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'}},
'type': 'object'}, 'develop': {'anyOf': [{'type': 'boolean'}, {'type':
'string'}]}, 'full_hash': {'type': 'string'}, 'hash': {'type':
'string'}, 'name': {'type': 'string'}, 'namespace': {'type': 'string'},
'package_hash': {'type': 'string'}, 'parameters':
{'additionalProperties': True, 'properties': {'cflags': {'items':
{'type': 'string'}, 'type': 'array'}, 'cppflags': {'items': {'type':
'string'}, 'type': 'array'}, 'cxxflags': {'items': {'type': 'string'},
'type': 'array'}, 'fflags': {'items': {'type': 'string'}, 'type':
'array'}, 'ldflags': {'items': {'type': 'string'}, 'type': 'array'},
'ldlib': {'items': {'type': 'string'}, 'type': 'array'}, 'patches':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['cflags',
'cppflags', 'cxxflags', 'fflags', 'ldflags', 'ldlibs'], 'type':
'object'}, 'patches': {'items': {}, 'type': 'array'}, 'version':
{'oneOf': [{'type': 'string'}, {'type': 'number'}]}}, 'required':
['version', 'arch', 'compiler', 'namespace', 'parameters'], 'type':
'object'}, 'type': 'array'}}, 'required': ['_meta', 'nodes'], 'type':
'object'}}Obsessed with the detail thatparameters.requiredlistsldlibswhile thepropertiesblock spells the keyldlib๐ซถ โ that's exactly what the page says, and we are not editing it. Copy literal, always.
And its trailing docstring line, sitting all the way over there:
Properties for inclusion in other schemas
Yes, same docstring as before. Twin energy. ๐ฏ
โญ spack.schema.spec.schema (as rendered)
And here's the rendered full schema attribute โ note the 'title': 'Spack spec schema' and the '$schema': 'http://json-schema.org/draft-07/schema#' living way at the end of the dump:
spack.schema.spec.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'patternProperties': {'spec': {'additionalProperties': False,
'properties': {'_meta': {'properties': {'version': {'type': 'number'}},
'type': 'object'}, 'nodes': {'items': {'additionalProperties': False,
'properties': {'arch': {'additionalProperties': False, 'properties':
{'platform': {}, 'platform_os': {}, 'target': {'oneOf': [{'type':
'string'}, {'additionalProperties': False, 'properties': {'features':
{'items': {'type': 'string'}, 'type': 'array'}, 'generation': {'type':
'integer'}, 'name': {'type': 'string'}, 'parents': {'items': {'type':
'string'}, 'type': 'array'}, 'vendor': {'type': 'string'}}, 'required':
['name', 'vendor', 'features', 'generation', 'parents'], 'type':
'object'}]}}, 'type': 'object'}, 'build_hash': {'type': 'string'},
'build_spec': {'additionalProperties': False, 'properties': {'hash':
{'type': 'string'}, 'name': {'type': 'string'}}, 'required': ['name',
'hash'], 'type': 'object'}, 'compiler': {'additionalProperties': False,
'properties': {'name': {'type': 'string'}, 'version': {'type':
'string'}}, 'type': 'object'}, 'dependencies': {'patternProperties':
{'\\w[\\w-]*': {'properties': {'hash': {'type': 'string'}, 'type':
{'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'}},
'type': 'object'}, 'develop': {'anyOf': [{'type': 'boolean'}, {'type':
'string'}]}, 'full_hash': {'type': 'string'}, 'hash': {'type':
'string'}, 'name': {'type': 'string'}, 'namespace': {'type': 'string'},
'package_hash': {'type': 'string'}, 'parameters':
{'additionalProperties': True, 'properties': {'cflags': {'items':
{'type': 'string'}, 'type': 'array'}, 'cppflags': {'items': {'type':
'string'}, 'type': 'array'}, 'cxxflags': {'items': {'type': 'string'},
'type': 'array'}, 'fflags': {'items': {'type': 'string'}, 'type':
'array'}, 'ldflags': {'items': {'type': 'string'}, 'type': 'array'},
'ldlib': {'items': {'type': 'string'}, 'type': 'array'}, 'patches':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['cflags',
'cppflags', 'cxxflags', 'fflags', 'ldflags', 'ldlibs'], 'type':
'object'}, 'patches': {'items': {}, 'type': 'array'}, 'version':
{'oneOf': [{'type': 'string'}, {'type': 'number'}]}}, 'required':
['version', 'arch', 'compiler', 'namespace', 'parameters'], 'type':
'object'}, 'type': 'array'}}, 'required': ['_meta', 'nodes'], 'type':
'object'}}, 'title': 'Spack spec schema', 'type': 'object'}Docstring: Full schema with metadata ๐
๐ฆ spack.schema.spec_list module
That's it. That's the entry. No body, no code, no docstring โ just the module name and vibes. Sometimes the most powerful thing you can do is say nothing. ๐
๐ธ spack.schema.upstreams module
Now we're on upstreams, and the properties here is honestly so much chiller than the spec one โ one level, default: {}, a couple of modules keys:
#: Properties for inclusion in other schemas
properties: Dict[str, Any] = {'upstreams':
{'default': {}, 'patternProperties': {'\\w[\\w-]*':
{'additionalProperties': False, 'default': {}, 'properties':
{'install_tree': {'type': 'string'}, 'modules': {'properties': {'lmod':
{'type': 'string'}, 'tcl': {'type': 'string'}}, 'type': 'object'}},
'type': 'object'}}, 'type': 'object'}}Docstring, once again: Properties for inclusion in other schemas โจ (they really said "reuse the line, it ate the first time").
And the full schema with metadata for it:
schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'upstreams': {'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'additionalProperties': False, 'default': {},
'properties': {'install_tree': {'type': 'string'}, 'modules':
{'properties': {'lmod': {'type': 'string'}, 'tcl': {'type': 'string'}},
'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}}, 'title':
'Spack core configuration file schema', 'type': 'object'}Heads up bestie ๐ โ thetitleon this one is'Spack core configuration file schema', not anything upstream-themed. That's what it says. Don't "fix" it.
Its docstring: Full schema with metadata. Consistent queen. ๐
๐ spack.schema.view module
Last one in this chunk, and the docstring is literally just:
Schema for view
Then there's a line pulling the projections scheme straight out of the projections schema:
projections_scheme = spack.schema.projections.properties["projections"]Then the properties dict โ and this one has range, because view can be a boolean, a string, or an object, and inside that object link has a pattern of (roots|all|run):
#: Properties for inclusion in other schemas
properties: Dict[str, Any] = {
"view": {
"anyOf": [
{"type": "boolean"},
{"type": "string"},
{
"type": "object",
"patternProperties": {
r"\w+": {
"required": ["root"],
"additionalProperties": False,
"properties": {
"root": {"type": "string"},
"link": {"type": "string", "pattern": "(roots|all|run)"},
"link_type": {"type": "string"},
"select": {"type": "array", "items": {"type": "string"}},
"exclude": {"type": "array", "items": {"type": "string"}},
"projections": projections_scheme,
},
}
},
},
]
}
}The way view gives you three whole shapes to pick from ๐ซถ โ boolean, string, or the big object where the only required key is root, and everything else (link, link_type, select, exclude, projections) is optional and additionalProperties is False. And note r"\w+" is a raw string in there โ keep the backslash, keep the r. ๐ฅ
okay bestie we're picking right back up mid-schema, no re-intro, let's GO ๐
#: Full schema with metadata
schema = {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Spack view configuration file schema",
"properties": properties,
}spack.schema.view.properties โ typed as Dict[str, Any] โ Properties for inclusion in other schemas:
spack.schema.view.properties: Dict[str, Any] = {'view': {'anyOf':
[{'type': 'boolean'}, {'type': 'string'}, {'patternProperties': {'\\w+':
{'additionalProperties': False, 'properties': {'exclude': {'items':
{'type': 'string'}, 'type': 'array'}, 'link': {'pattern':
'(roots|all|run)', 'type': 'string'}, 'link_type': {'type': 'string'},
'projections': {'patternProperties': {'all|\\w[\\w-]*': {'type':
'string'}}, 'type': 'object'}, 'root': {'type': 'string'}, 'select':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['root']}},
'type': 'object'}]}}spack.schema.view.schema โ Full schema with metadata โจ:
spack.schema.view.schema = {'$schema':
'http://json-schema.org/draft-07/schema#', 'properties': {'view':
{'anyOf': [{'type': 'boolean'}, {'type': 'string'},
{'patternProperties': {'\\w+': {'additionalProperties': False,
'properties': {'exclude': {'items': {'type': 'string'}, 'type':
'array'}, 'link': {'pattern': '(roots|all|run)', 'type': 'string'},
'link_type': {'type': 'string'}, 'projections': {'patternProperties':
{'all|\\w[\\w-]*': {'type': 'string'}}, 'type': 'object'}, 'root':
{'type': 'string'}, 'select': {'items': {'type': 'string'}, 'type':
'array'}}, 'required': ['root']}}, 'type': 'object'}]}}, 'title': 'Spack
view configuration file schema'}๐งฉ spack.solver package
Submodules
โจ spack.solver.asp module โ this is the whole concretizer brain, buckle up ๐ง ๐
class spack.solver.asp.CompilerParser(configuration) โ Bases: object
Parses configuration files, and builds a list of possible compilers for the solve. Basically it's the little matchmaker going "who's available for this build ๐".
add_compiler_from_concrete_spec(spec: Spec) -> Noneโ account for compilers that are coming from concrete specs, through reuse.- Parameters:
specโ concrete spec to be reused
- Parameters:
possible_compilers() -> List[KnownCompiler]โ no blurb here, it's just giving "here's your list, bestie".with_input_specs(input_specs: List[Spec]) -> CompilerParserโ accounts for input specs when building the list of possible compilers.- Parameters:
input_specsโ specs to be concretized
- Parameters:
class spack.solver.asp.ConcreteSpecsByHash โ Bases: Mapping
A Mapping containing concrete specs keyed by DAG hash. And the glow-up: the mapping is ensured to be consistent โ if a spec in the mapping has a dependency with hash X, it is ensured to be the same object in memory as the spec keyed by X. No duplicate impostor specs, periodt ๐
add(spec: Spec) -> boolโ adds a new concrete spec to the mapping. ReturnsTrueif the spec was just added,Falseif the spec was already in the mapping.- Calling this function marks the spec as added explicitly.
- Parameters:
specโ spec to be added - Raises
ValueErrorโ if the spec is not concrete
explicit_items() -> Iterator[Tuple[str, Spec]]โ iterate on items that have been added explicitly, and not just as a dependency of other nodes. That's main character energy only, no plus-ones ๐ซถ
spack.solver.asp.DEFAULT_OUTPUT_CONFIGURATION = (False, False, None, False) โ default output configuration for a solve. That's the blueprint, don't touch it ๐
class spack.solver.asp.DeclaredVersion(version: str, idx: int, origin: Provenance) โ Bases: NamedTuple
Data class to contain information on declared versions used in the solve. Tiny, but it's carrying a whole backstory ๐
idx: intโ unique index assigned to this versionorigin: Provenanceโ provenance of the versionversion: strโ string representation of the version
class spack.solver.asp.ErrorHandler(model) โ Bases: object
The friend who shows up when the solve goes off the rails and explains exactly what happened, with receipts.
get_cause_tree(cause: Tuple[str, str]) -> List[str]โ get the cause tree associated with the given cause.- Parameters:
causeโ The root cause of the tree (final condition) - Returns: A list of strings describing the causes, formatted to display tree structure.
- Parameters:
handle_error(msg, *args)โ handle an error state derived by the solver.message(errors) -> strโ hands you the whole sob story as a string.multiple_values_error(attribute, pkg)โ "you gave it how many values??" ๐ญno_value_error(attribute, pkg)โ "you gave it zero??" ๐raise_if_errors()โ yup, if anything's wrong it's gonna raise. No silent sweep-under-the-rug here.
exception spack.solver.asp.InternalConcretizerError(msg) โ Bases: UnsatisfiableSpecError
Errors that indicate a bug in Spack. Like, not your fault bestie, that one's on us ๐
class spack.solver.asp.KnownCompiler(spec: spack.spec.Spec, os: str, target: str, available: bool, compiler_obj: spack.compiler.Compiler | None) โ Bases: NamedTuple
Data class to collect information on compilers. It's giving "compiler rรฉsumรฉ" ๐
available: boolโ alias for field number 3compiler_obj: Compiler | Noneโ alias for field number 4os: strโ alias for field number 1spec: Specโ alias for field number 0target: strโ alias for field number 2
class spack.solver.asp.OutputConfiguration(timers, stats, out, setup_only) โ Bases: tuple
Data class that contains configuration on what a clingo solve should output. The "how much yapping do you want from the solver" settings ๐ฃ๏ธ
Parameters:
timers(bool) โ print out coarse timers for different solve phases.stats(bool) โ whether to output Clingo's internal solver statistics.outโ optional output stream for the generated ASP program.setup_only(bool) โ ifTrue, stop after setup and don't solve (defaultFalse).
Fields:
outโ alias for field number 2setup_onlyโ alias for field number 3statsโ alias for field number 1timersโ alias for field number 0
class spack.solver.asp.ProblemInstanceBuilder โ Bases: object
Provides an interface to construct a problem instance. Once all the facts and rules have been added, the problem instance can be retrieved with:
>>> builder = ProblemInstanceBuilder()
>>> ...
>>> problem_instance = builder.value()The problem instance can be added directly to the "control" structure of clingo. Chef's kiss of an API ๐งโ๐ณ๐
append(rule: str) -> Noneโ tack a rule onto the pile.fact(atom: AspFunction) -> Noneโ drop a fact, no notes.h1(header: str) -> Noneโ big header energy.h2(header: str) -> Noneโ slightly smaller header energy.newline()โ literally just a newline, she's doing her best ๐ธtitle(header: str, char: str) -> Noneโ a title, but make it fancy with your chosenchar.value() -> strโ the final reveal, the whole problem instance as a string โจ
class spack.solver.asp.Provenance(*values) โ Bases: IntEnum
Enumeration of the possible provenances of a version. Where'd this version come from? The receipts ๐
DEV_SPEC = 2โ a dev spec, living on the edge.EXTERNAL = 3โ external package, not built by us.INSTALLED = 7โ already installed, been there done that.PACKAGES_YAML = 4โ straight out ofpackages.yaml.PACKAGE_PY = 6โ came from apackage.py.PACKAGE_REQUIREMENT = 5โ a package requirement said so.RUNTIME = 8โ runtime provenance.SPEC = 1โ it's just in the spec, bestie.
class spack.solver.asp.PyclingoDriver(cores=True) โ Bases: object
solve(setup, specs, reuse=None, output=None, control=None, allow_deprecated=False)โ set up the input and solve for dependencies of specs. The main event ๐ฌ
Parameters:
setup(SpackSolverSetup) โ An object to set up the ASP problem.specs(list) โ List ofSpecobjects to solve for.reuse(Noneorlist) โ list of concrete specs that can be reusedoutput(NoneorOutputConfiguration) โ configuration object to set the output of this solve.control(clingo.Control) โ configuration for the solver. IfNone, default values will be usedallow_deprecatedโ ifTrue, allow deprecated versions in the solve
Returns: A tuple of the solve result, the timer for the different phases of the solve, and the internal statistics from clingo.
class spack.solver.asp.RequirementKind(*values) โ Bases: Enum
Purpose / provenance of a requirement. Three flavors, all valid ๐
DEFAULT = 1โ default requirement expressed under the'all'attribute ofpackages.yamlPACKAGE = 3โ requirement expressed on a specific packageVIRTUAL = 2โ requirement expressed on a virtual package
class spack.solver.asp.RequirementParser(configuration) โ Bases: object
Parses requirements from package.py files and configuration, and returns rules. She reads everything and takes notes ๐
reject_requirement_constraint(pkg_name: str, *, constraint: Spec, kind: RequirementKind) -> boolโ returnsTrueif a requirement constraint should be rejected. Sorry babes, that one's not making the cut ๐ซrules(pkg: PackageBase) -> List[RequirementRule]โ all the rules for this package.rules_from_conflict(pkg: PackageBase) -> List[RequirementRule]โ rules sourced from conflicts.rules_from_package_py(pkg) -> List[RequirementRule]โ rules sourced frompackage.py.rules_from_prefer(pkg: PackageBase) -> List[RequirementRule]โ rules sourced from preferences.rules_from_require(pkg: PackageBase) -> List[RequirementRule]โ rules sourced fromrequire.rules_from_virtual(virtual_str: str) -> List[RequirementRule]โ rules sourced from a virtual string.
class spack.solver.asp.RequirementRule(pkg_name: str, policy: str, requirements: List[spack.spec.Spec], condition: spack.spec.Spec, kind: RequirementKind, message: str | None) โ Bases: NamedTuple
Data class to collect information on a requirement. One neat little bundle of "you must do this" ๐
condition: Specโ alias for field number 3kind: RequirementKindโ alias for field number 4message: str | Noneโ alias for field number 5pkg_name: strโ alias for field number 0policy: strโ alias for field number 1requirements: List[Spec]โ alias for field number 2
class spack.solver.asp.Result(specs, asp=None) โ Bases: object
Result of an ASP solve. The grand finale of a solve ๐
format_core(core)โ format an unsatisfiable core for human readability.- Returns: a list of strings, where each string is the human readable representation of a single fact in the core, including a newline.
- Modeled after
traceback.format_stack. Respect the ancestors ๐
format_cores()โ gives you a list of facts for each core ๐งฉ every separate core gets its own empty line between them, and bestie, the cores are not minimized here โ no shrinking, no trimming, you get the whole thing ๐format_minimal_cores()โ same cute deal: a list of facts for each core, separate cores separated by an empty line โจ (but this one's the minimal version, so it's giving smaller-and-tidier energy ๐)static format_unsolved(unsolved_specs)โ whips up a message that gives you the info on unsolved user specs, and for each one it shows the associated candidate spec from the solver (if there even is one ๐ yes, that "if" is doing real work here)minimal_cores()โ returns a list of subset-minimal unsatisfiable cores ๐ฅ subset-minimal, so no extra fluff, only the essential dramaminimize_core(core)โ returns a subset-minimal subset of the core ๐ like, clingo cores may literally be thousands of lines when two facts are sufficient to ensure unsatisfiability โ which is so extra of them โ so this algorithm reduces the core down to only those essential facts. icon behavior, honestly ๐raise_if_unsat()โ raises an appropriate error if the result is unsatisfiable ๐จ the error is aSolverError, and it includes the minimized cores resulting from the solve, formatted to be human-readable ๐ซถ- property
specsโ list of concretized specs satisfying the initial abstract request โจ - property
specs_by_inputโ yes bestie, it's a property, and that's the whole description iykyk ๐ธ - property
unsolved_specsโ list of tuples pairing abstract input specs that were not solved with their associated candidate spec from the solver (if the solve completed) ๐ญ the "if the solve completed" is so ominous, lowkey
๐ class spack.solver.asp.ReusableSpecsSelector(configuration: Configuration)
Bases: object
This one selects specs that can be reused during concretization ๐ it's giving recycling-but-make-it-solver-energy ๐
reusable_specs(specs: List[Spec]) -> List[Spec]โ hands you back the reusable ones, as a list of specs ๐
๐ฆ class spack.solver.asp.ReuseStrategy(*values)
Bases: Enum
The strategy menu, no cap:
DEPENDENCIES = 2NONE = 3ROOTS = 1
(and yes, the numbers are in that order in the source, not ascending, don't come for me ๐)
๐ง class spack.solver.asp.RuntimePropertyRecorder(setup)
Bases: object
Okay so an object of this class gets injected in callbacks to compilers โ the whole point is to let them declare properties of the runtimes they support and of the runtimes they provide, and to add runtime dependencies to the nodes using said compiler ๐ซถ
The usage is literally this: first, a runtime package name or the wildcard "*" gets passed as an argument to __call__, to set which kind of package we're talking about. Then we can call one method with a directive-like API ๐
Examples
>>> pkg = RuntimePropertyRecorder(setup)
>>> # Every package compiled with %gcc has a link dependency on 'gcc-runtime'
>>> pkg("*").depends_on(
... "gcc-runtime",
... when="%gcc",
... type="link",
... description="If any package uses %gcc, it depends on gcc-runtime"
... )
>>> # The version of gcc-runtime is the same as the %gcc used to "compile" it
>>> pkg("gcc-runtime").requires("@=9.4.0", when="%gcc@=9.4.0")Methods, let's go:
consume_facts()โ consumes the facts collected by this object, and emits rules and facts for the runtimes โจ (yes, eats the facts, iconic)depends_on(dependency_str: str, *, when: str, type: str, description: str, languages: List[str] | None = None) -> Noneโ injects conditional dependencies on packages ๐ conditional dependencies can be either "real" packages or virtual dependencies, fyi
Parameters
dependency_strโ the dependency spec to injectwhenโ anonymous condition to be met on a package to have the dependencytypeโ dependency typelanguagesโ languages needed by the package for the dependency to be considereddescriptionโ human-readable description of the rule for adding the dependency
propagate(constraint_str: str, *, when: str)โ yep, that's the signature, that's all the page gives us, don't shoot the messenger ๐requires(impose: str, *, when: str)โ injects conditional requirements on a given package ๐
Parameters
imposeโ constraint to be imposedwhenโ condition triggering the constraint
reset()โ resets the current state ๐ธ clean slate, we love a fresh start
๐ class spack.solver.asp.Solver
Bases: object
This is THE main external interface class for solving ๐ฅ it manages solver configuration and preferences all in one place, sets up the solve, and passes the setup method to the driver as well. main character energy, fr fr ๐
solve(specs, out=None, timers=False, stats=False, tests=False, setup_only=False, allow_deprecated=False)
Parameters
specs(list) โ list ofSpecobjects to solve for ๐งฉoutโ optionally write the generate ASP program to a file-like object (yes, the page says "generate", we're keeping it exactly as-is ๐ญ)timers(bool) โ print out coarse timers for different solve phases โฑ๏ธstats(bool) โ print out detailed stats from clingo ๐tests(bool or tuple) โ ifTrue, concretize test dependencies for all packages. If a tuple of package names, concretize test dependencies for named packages (defaults toFalse: do not concretize test dependencies) ๐งชsetup_only(bool) โ ifTrue, stop after setup and don't solve (defaultFalse) ๐allow_deprecated(bool) โ allow deprecated version in the solve โ ๏ธ
solve_in_rounds(specs, out=None, timers=False, stats=False, tests=False, allow_deprecated=False)โ solves for a stable model of specs in multiple rounds ๐ this relaxes the assumption ofsolvethat everything must be consistent and solvable in a single round โ each round tries to maximize the reuse of specs from previous rounds, which is such a smart little move ๐
The function is a generator that yields the result of each round โจ
Parameters
specs(list) โ list of specs to solveoutโ optionally write the generate ASP program to a file-like objecttimers(bool) โ print timing if set toTrueโฑ๏ธstats(bool) โ print internal statistics if set toTrue๐tests(bool) โ add test dependencies to the solve ๐งชallow_deprecated(bool) โ allow deprecated version in the solve โ ๏ธ
๐ exception spack.solver.asp.SolverError(provided, conflicts)
Bases: InternalConcretizerError
For cases where the solver is unable to produce a solution ๐ญ such cases are unexpected because we allow for solutions with errors โ so for example, user specs that are over-constrained should still get a solution. So if you're seeing this, something genuinely wild happened ๐
๐ class spack.solver.asp.SpackSolverSetup(tests: bool = False)
Bases: object
Class to set up and run a Spack concretization solve ๐ฅ the whole backstage crew, honestly. Here's everything it does:
collect_virtual_constraints()โ defines versions for constraints on virtuals ๐ฆ must be called beforedefine_version_constraints()โ order matters, bestie, don't skip the line ๐compiler_facts()โ facts about available compilers ๐งconcrete_specs()โ emits facts for reusable specs โจcondition(required_spec: ~spack.spec.Spec, imposed_spec: ~spack.spec.Spec | None = None, name: str | None = None, msg: str | None = None, transform_required: ~typing.Callable[[~spack.spec.Spec, ~typing.List[~spack.solver.core.AspFunction]], ~typing.List[~spack.solver.core.AspFunction]] | None = None, transform_imposed: ~typing.Callable[[~spack.spec.Spec, ~typing.List[~spack.solver.core.AspFunction]], ~typing.List[~spack.solver.core.AspFunction]] | None = <function remove_node>)โ generates facts for a dependency or virtual provider condition ๐ (yes that default value is literally<function remove_node>, it's a whole function in the signature, obsessed)
Parameters
required_specโ the constraints that triggers this conditionimposed_specโ the constraints that are imposed when this condition is triggerednameโ name forrequired_spec(required ifrequired_specis anonymous, ignored if not)msgโ description of the conditiontransform_requiredโ transformation applied to facts from the required spec. Defaults to leave facts as they are.transform_imposedโ transformation applied to facts from the imposed spec. Defaults to removing"node"and"virtual_node"facts.
Returns โ id of the condition created by this function
Return type โ int
config_compatible_os()โ facts about compatible os's specified in configs ๐งconflict_rules(pkg)โ that's the whole line, bestie, it's giving mystery ๐ต๏ธโโ๏ธdefine_ad_hoc_versions_from_specs(specs, origin, *, allow_deprecated: bool, require_checksum: bool)โ add concrete versions to possible versions from lists of CLI/dev specs ๐define_compiler_version_constraints()โ another one-liner, we respect the brevity ๐ซถdefine_concrete_input_specs(specs, possible)โ yep, that's the signature ๐ธdefine_package_versions_and_validate_preferences(possible_pkgs: Set[str], *, require_checksum: bool, allow_deprecated: bool)โ declares any versions in specs not declared in packages ๐define_runtime_constraints()โ defines the constraints to be imposed on the runtimes ๐งdefine_target_constraints()โ another signature-only situation, don't @ me ๐define_variant_values()โ validates variant values from the command line โ and also adds valid variant values from the command line to the possible values for a variant, which is lowkey doing two jobs at once, we love a multitasker ๐define_version_constraints()โ defines whatversion_satisfies(...)means in ASP logic ๐ง periodteffect_rules()โ flushes all the effect rules collected so far, and clears the cache ๐ฝโจ (sorry, flushing just sounds like that)
okay continuing the spack.solver.asp setup energy ๐
โ here's the rest of the method squad, no skips, no cap:
emit_facts_from_requirement_rules(rules: List[RequirementRule])โ generates the facts that enforce requirements. therulesparam is literally just "the rules we want facts emitted for" โจexternal_packages()โ drops facts on external packages, pulled frompackages.yamlplus the implicit externals ๐ซถimpose(condition_id, imposed_spec, node=True, name=None, body=False)โ that's the whole signature, bestie, docs don't spill more tea on itinternal_errors()โ signature only, we moveliteral_specs(specs)โ signature only ๐os_defaults(specs)โ signature onlypackage_dependencies_rules(pkg)โ translatesdepends_ondirectives into ASP logic. that's the whole job, icon behavior ๐ฅpackage_languages(pkg)โ signature onlypackage_provider_rules(pkg)โ signature onlypackage_requirement_rules(pkg)โ signature onlypkg_class(pkg_name: str) -> Type[PackageBase]โ signature only, takes a package name, hands back the class type ๐ฆpkg_rules(pkg, tests)โ signature onlypkg_version_rules(pkg)โ outputs the declared versions of a package. it usesself.declared_versionsso that we also include any versions that pop up from a spec ๐ธplatform_defaults()โ signature onlypreferred_variants(pkg_name)โ facts on concretization preferences, as read frompackages.yaml. lowkey the vibe-check layer of concretization ๐provider_defaults()โ signature onlyprovider_requirements()โ signature onlyregister_concrete_spec(spec, possible)โ signature onlysetup(specs: List[Spec], *, reuse: List[Spec] | None = None, allow_deprecated: bool = False) -> strโ generates an ASP program with the relevant constraints for the given specs. it calls methods on the solve driver to set up the problem using facts and rules from all the possible dependencies of the input specs, plus the constraints from the specs themselves. the params:specsโ list of Specs to solve ๐reuseโ list of concrete specs that can be reusedallow_deprecatedโ ifTrue, adds deprecated versions into the solve โ ๏ธ
spec_clauses(spec: Spec, *, body: bool = False, transitive: bool = True, expand_hashes: bool = False, concrete_build_deps=False, required_from: str | None = None) -> List[AspFunction]โ wraps a call to_spec_clauses()in atry/exceptblock with better error handling. arguments are as for_spec_clauses()exceptrequired_from. andrequired_from? that's the name of the package that caused this call ๐งspec_versions(spec)โ returns a list of clauses expressing the spec's version constraints โญtarget_defaults(specs)โ adds facts about targets and target compatibility ๐target_preferences()โ signature onlytarget_ranges(spec, single_target_fn)โ signature onlytrigger_rules()โ flushes all the trigger rules collected so far, and clears the cache ๐งนvalidate_and_define_versions_from_requirements(*, allow_deprecated: bool, require_checksum: bool)โ if package requirements mention concrete versions that aren't mentioned anywhere else, then we gotta collect those and mark them as possible versions. if they're abstract and statically have no match, then we throw an error ๐ญ. this function assumes all possible versions are already registered inself.possible_versions.variant_rules(pkg)โ signature onlyvirtual_preferences(pkg_name, func)โ callsfunc(vspec, provider, i)for each of the package's provider prefs ๐virtual_providers()โ signature only
๐งฑ class spack.solver.asp.SpecBuilder(specs, hash_lookup=None)
bases: object. it's the class with all the actions to rebuild a spec from ASP results โ the glow-up machine fr fr ๐ ๏ธ
build_specs(function_tuples)โ signature onlydepends_on(parent_node, dependency_node, type)โ signature onlydeprecated(node: NodeArgument, version: str) -> Noneโ signature onlyexternal_spec_selected(node, idx)โ means the external spec with indexidxhas been selected for this package โhash(node, h)โ signature onlyignored_attributes = re.compile(...)โ the regex for attributes that don't need actions b/c they aren't used to construct specs. verbatim, don't @ me:
ignored_attributes = re.compile('^.*_propagate$|^.*_satisfies$|^.*_set$|^compatible_libc$|^dependency_holds$|^external_conditions_hold$|^node_compiler$|^package_hash$|^root$|^track_dependencies$|^variant_default_value_from_cli$|^vir')- static
make_node(*, pkg: str) -> NodeArgumentโ given a package name, returns the string representation of the"min_dupe_id"node in the ASP encoding. thepkgparam = name of a package ๐ namespace(node, namespace)โ signature onlyno_flags(node, flag_type)โ signature onlynode(node)โ signature onlynode_compiler_version(node, compiler, version)โ signature onlynode_flag(node, flag_type, flag)โ signature onlynode_flag_compiler_default(node)โ signature onlynode_flag_source(node, flag_type, source)โ signature onlynode_os(node, os)โ signature onlynode_platform(node, platform)โ signature onlynode_target(node, target)โ signature onlyreorder_flags()โ orders compiler flags on specs in a predefined order. we order flags so that any node's flags take priority over those of its dependents. that is, the deepest node in the DAG's flags will appear last on the compile line, in the order they were specified. the solver decides which flags are on nodes; this routine imposes the order afterwards ๐ฏ- static
sort_fn(function_tuple)โ makes sure attributes are evaluated in the correct order, bestie:hashattributes are handled first, since they imply entire concrete specs- node attributes are handled next, since they instantiate nodes
external_spec_selectedattributes are handled last, so that external extensions can find the concrete specs they depend on โ because all nodes are fully constructed before we consider which ones are external ๐ฆ
variant_value(node, name, value)โ signature onlyversion(node, version)โ signature onlyvirtual_on_edge(parent_node, provider_node, virtual)โ signature only
๐ class spack.solver.asp.SpecFilter(factory: Callable[[], List[Spec]], is_usable: Callable[[Spec], bool], include: List[str], exclude: List[str])
bases: object. given a method to produce a list of specs, this class can filter them according to different criteria. the picky bestie of the solver world ๐
- static
from_buildcache(configuration, include, exclude) -> SpecFilterโ builds a filter that takes the specs from the configured buildcaches ๐ฆ - static
from_store(configuration, include, exclude) -> SpecFilterโ builds a filter that takes the specs from the current store ๐ช is_selected(s: Spec) -> boolโ signature onlyselected_specs() -> List[Spec]โ signature only
๐จ exception spack.solver.asp.UnsatisfiableSpecError(msg)
bases: UnsatisfiableSpecError. there was an issue with the spec that was requested (i.e. a user error) โ yeah that one's on you, bestie ๐ญ๐
๐ module-level goodies in spack.solver.asp
WITH_RUNTIME = Trueโ enable the addition of a runtime node ๐ฅall_compilers_in_config(configuration)โ signature onlyall_libcs() -> Set[Spec]โ returns a set of all libc specs targeted by any configured compiler. if there are none, falls back to the libc determined from the current Python process, if it's dynamically linked ๐build_criteria_names(costs, arg_tuples)โ constructs an ordered mapping from criteria names to costs ๐บ๏ธbuild_priority_offset = 200โ priority offset for"build"criteria (regular criteria shifted to higher priority for specs we have to build) ๐๏ธc_compiler_runs(compiler: Compiler) -> boolโ signature onlycheck_packages_exist(specs)โ makes sure all the packages mentioned in the specs actually exist โdefault_clingo_control()โ returns a control object with the default settings used in Spack ๐๏ธextend_flag_list(flag_list, new_flags)โ extends a list of flags, preserving order and precedence. addsnew_flagsat the end offlag_list. if any flags innew_flagsare already inflag_list, they get moved to the end so they take higher precedence on the compile line ๐fixed_priority_offset = 100โ priority offset of"fixed"criteria (those without build criteria) ๐high_fixed_priority_offset = 300โ high fixed priority offset for criteria that supersede all build criteria ๐issequence(obj)โ signature onlylibc_is_compatible(lhs: Spec, rhs: Spec) -> List[Spec]โ signature onlylistify(args)โ signature onlypackagize(pkg)โ signature onlyremove_node(spec: Spec, facts: List[AspFunction]) -> List[AspFunction]โ the transformation that removes all"node"and"virtual_node"from the input list of facts ๐งนspec_with_name(spec, name)โ context manager to temporarily set the name of a spec ๐ท๏ธspecify(spec)โ signature onlyusing_libc_compatibility() -> boolโ returnsTrueif we're currently using libc compatibility ๐
๐ spack.solver.core module
low-level wrappers around the clingo API, periodt. this is the plumbing-level bestie ๐ซถ
๐งฉ class spack.solver.core.AspFunction(name: str, args: Tuple[Any, ...] | None = None)
bases: AspObject. a term in the ASP logic program โจ
argsโ attributenameโ attributesymbol()โ returns a clingo symbol for this function ๐
๐งฉ class spack.solver.core.AspObject
bases: object. object representing a piece of ASP code ๐
๐งฉ class spack.solver.core.NodeArgument(id: str, pkg: str)
bases: NamedTuple. represents a node in the DAG ๐ณ
id: strโ alias for field number 0pkg: strโ alias for field number 1
๐ ๏ธ functions in spack.solver.core
ast_sym(node)โ signature onlyast_type(node)โ signature onlyclingo() -> ModuleTypeโ lazily imports the Python module for clingo and returns it ๐clingo_cffi() -> boolโ returnsTrueif clingo uses the CFFI interface ๐extract_args(model, predicate_name)โ extracts the arguments to predicates with the provided name from a model ๐
Okay so this one literally pulls out all the predicates with name predicate_name from the model and hands you back their intermediate representation. Very "give me the receipts" of the solver world. ๐
spack.solver.core.fn = <spack.solver.core._AspFunctionBuilder object>โ the global AspFunction builder, one shared builder for everybody, iconic behavior.spack.solver.core.intermediate_repr(sym)โ returns an intermediate representation of clingo models for Spack's spec builder.
Returns ๐
Currently, transforms symbols from clingo models either to strings or to NodeArgument objects.
So basically: this will turn a clingo.Symbol into a string or a NodeArgument, or a sequence of clingo.Symbol objects into a tuple of those objects. Fruit-roll-up of symbols, no cap. ๐
spack.solver.core.parse_files(*args, **kwargs)โ wrapper around clingoparse_files, dispatching the function according to clingo API version.spack.solver.core.parse_term(*args, **kwargs)โ wrapper around clingoparse_term, dispatching the function according to clingo API version.
๐ spack.solver.counter module
class spack.solver.counter.Counter(specs: List[Spec], tests: bool)
Bases: object
Computes the possible packages and the maximum number of duplicates allowed for each of them. It's giving "budget manager energy" but for packages. ๐งฎ
Parameters ๐ธ
specsโ abstract specs to concretizetestsโ ifTrue, add test dependencies to the list of possible packagesensure_cache_values() -> Noneโ ensure the cache values have been computedpossible_dependencies() -> Set[str]โ returns the list of possible dependenciespossible_packages_facts(gen: PyclingoDriver, fn) -> Noneโ emit facts associated with the possible packagespossible_virtuals() -> Set[str]โ returns the list of possible virtuals
class spack.solver.counter.FullDuplicatesCounter(specs, tests)
Bases: MinimalDuplicatesCounter
possible_packages_facts(gen, fn)โ emit facts associated with the possible packages
class spack.solver.counter.MinimalDuplicatesCounter(specs, tests)
Bases: NoDuplicatesCounter
possible_packages_facts(gen, fn)โ emit facts associated with the possible packages
class spack.solver.counter.NoDuplicatesCounter(specs: List[Spec], tests: bool)
Bases: Counter
possible_packages_facts(gen, fn)โ emit facts associated with the possible packages
๐ spack.util package
Subpackages ๐ฆ
spack.util.unparse package
spack.util.unparse.unparse(tree, py_ver_consistent=False)
Submodules ๐ฆ
spack.util.unparse.unparser module
Usage: unparse.py <path to source file>
class spack.util.unparse.unparser.Unparser(py_ver_consistent=False, _avoid_backslashes=False)
Bases: object
Methods in this class recursively traverse an AST and output source code for the abstract syntax; original formatting is disregarded. So yeah โ it does not care about your cute little line breaks. Your formatting is gone. Periodt. ๐
binop = {'Add': '+', 'BitAnd': '&', 'BitOr': '|', 'BitXor': '^', 'Div': '/', 'FloorDiv': '//', 'LShift': '<<', 'MatMult': '@', 'Mod': '%', 'Mult': '*', 'Pow': '**', 'RShift': '>>', 'Sub': '-'}binop_precedence = {'%': 12, '&': 9, '*': 12, '**': 14, '+': 11, '-': 11, '/': 12, '//': 12, '<<': 10, '>>': 10, '@': 12, '^': 8, '|': 7}binop_rassoc = frozenset({'**'})block()boolop_precedence = {'and': 4, 'or': 3}boolops = {'And': 'and', 'Or': 'or'}cmpops = {'Eq': '==', 'Gt': '>', 'GtE': '>=', 'In': 'in', 'Is': 'is', 'IsNot': 'is not', 'Lt': '<', 'LtE': '<=', 'NotEq': '!=', 'NotIn': 'not in'}delimit(start, end)โ a context manager for preparing the source for expressions. It adds start to the buffer and enters, after exit it adds end.delimit_if(start, end, condition)dispatch(tree)โ dispatcher function, dispatching tree type T to method_T.fill(text='')โ indent a piece of text, according to the current indentation levelget_precedence(node)items_view(traverser, items)โ traverse and separate the given items with a comma and append it to the buffer. If items is a single item sequence, a trailing comma will be added.require_parens(precedence, node)โ shortcut to adding precedence related parensset_precedence(precedence, *nodes)unop = {'Invert': '~', 'Not': 'not', 'UAdd': '+', 'USub': '-'}unop_precedence = {'+': 13, '-': 13, 'not': 5, '~': 13}visit(tree, output_file)โ traverse tree and write source code tooutput_file.
Now the whole visitor squad โ one method per AST node type, and honestly they're all exactly what they say on the tin. Obsessed with the commitment. โจ
visit_AnnAssign(node)visit_Assert(node)visit_Assign(node)visit_AsyncFor(node)visit_AsyncFunctionDef(node)visit_AsyncWith(node)visit_Attribute(node)visit_AugAssign(node)visit_Await(node)visit_BinOp(node)visit_BoolOp(node)visit_Break(node)visit_Bytes(node)visit_Call(node)visit_ClassDef(node)visit_Compare(node)visit_Constant(node)visit_Continue(node)visit_Delete(node)visit_Dict(node)visit_DictComp(node)visit_Ellipsis(node)visit_ExceptHandler(node)visit_Expr(tree)visit_Expression(tree)visit_ExtSlice(node)visit_For(node)visit_FormattedValue(node)visit_FunctionDef(node)visit_GeneratorExp(node)visit_Global(node)visit_If(node)visit_IfExp(node)visit_Import(node)visit_ImportFrom(node)visit_Index(node)visit_Interactive(tree)visit_JoinedStr(node)visit_Lambda(node)visit_List(node)visit_ListComp(node)visit_Match(node)visit_MatchAs(node)visit_MatchClass(node)visit_MatchMapping(node)visit_MatchOr(node)visit_MatchSequence(node)visit_MatchSingleton(node)visit_MatchStar(node)visit_MatchValue(node)visit_Module(tree)visit_Name(node)visit_NameConstant(node)visit_NamedExpr(tree)visit_Nonlocal(node)visit_Num(node)visit_ParamSpec(node)visit_Pass(node)visit_Raise(node)visit_Return(node)visit_Set(node)visit_SetComp(node)visit_Slice(node)visit_Starred(node)visit_Str(tree)visit_Subscript(node)visit_Try(node)visit_Tuple(node)visit_TypeAlias(node)visit_TypeVar(node)visit_TypeVarTuple(node)visit_UnaryOp(node)visit_While(node)visit_With(node)visit_Yield(node)visit_YieldFrom(node)visit_alias(node)visit_arg(node)visit_arguments(node)visit_comprehension(node)visit_keyword(node)visit_match_case(node)visit_withitem(node)write(text)โ append a piece of text to the current line.spack.util.unparse.unparser.interleave(inter, f, seq)โ callfon each item inseq, callinginter()in between.spack.util.unparse.unparser.is_simple_tuple(slice_value)spack.util.unparse.unparser.nullcontext()spack.util.unparse.unparser.pnext(precedence)
Submodules ๐ฆ
spack.util.archive module
class spack.util.archive.ChecksumWriter(fileobj, algorithm=<built-in function openssl_sha256>)
Bases: BufferedIOBase
Checksum writer computes a checksum while writing to a file. Multitasking icon โ writes and checksums in one go. ๐ฅ
close()โ flush and close the IO object.
This method has no effect if the file is already closed.
- property
closed fileno()โ return underlying file descriptor if one exists.
Raise OSError if the IO object does not use a file descriptor.
flush()โ flush write buffers, if applicable.
This is not implemented for read-only and non-blocking streams.
hexdigest()myfileobj = Nonepeek(n)read(size=-1)โ read and return up to n bytes.
If the size argument is omitted, None, or negative, read and return all data until EOF.
If the size argument is positive, and the underlying raw stream is not 'interactive', multiple raw reads may be issued to satisfy the byte count (unless EOF is reached first). However, for interactive raw streams (as well as sockets and pipes), at most one raw read will be issued, and a short result does not imply that EOF is imminent.
Return an empty bytes object on EOF.
Return None if the underlying raw stream was open in non-blocking mode and no data is available at the moment.
read1(size=-1)โ read and return up to size bytes, with at most oneread()call to the underlying raw stream.
Return an empty bytes object on EOF. A short result does not imply that EOF is imminent.
readable()โ return whether object was opened for reading.
If False, read() will raise OSError.
readline(size=-1)โ read and return a line from the stream.
If size is specified, at most size bytes will be read.
The line terminator is always b'n' for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.
rewind()seek(offset, whence=0)โ change the stream position to the given byte offset.- offset โ the stream position, relative to 'whence'.
- whence โ the relative position to seek from.
The offset is interpreted relative to the position indicated by whence. Values for whence are:
os.SEEK_SETor0โ start of stream (the default); offset should be zero or positiveos.SEEK_CURor1โ current stream position; offset may be negative
and the lastwhenceoption, for the record: -os.SEEK_ENDor2โ end of stream; offset is usually negative Returns the new absolute position.
seekable()โ Returns whether the object supports random access. If it'sFalse, thenseek(),tell()andtruncate()will raiseOSError. Heads up: this method may need to do a testseek()to figure that out.tell()โ Returns the current stream position, bestie. That's it.writable()โ Returns whether the object was opened for writing. IfFalse, thenwrite()will raiseOSError.write(data)โ Write bufferbto the IO stream. Returns the number of bytes written, which is always the length ofbin bytes. RaisesBlockingIOErrorif the buffer is full and the underlying raw stream cannot accept more data at the moment.
Okay, back to spack.util.archive for the rest of the functions โ no cap, these are the ones that actually build your tarballs.
๐๏ธ spack.util.archive.default_path_to_name
spack.util.archive.default_path_to_name(path: str) -> str โ Converts a path to a tarfile name, which uses posix path separators. Slash it up, periodt.
๐ฆ spack.util.archive.gzip_compressed_tarfile
spack.util.archive.gzip_compressed_tarfile(path) โ Creates a reproducible, gzip compressed tarfile, and keeps track of shasums of both the compressed and the uncompressed tarfile. Reproducibility is achieved by normalizing the gzip header (no file name and zero mtime). Obsessed with determinism, honestly.
Yields a tuple of the following:
tarfile.TarFile: tarfile objectChecksumWriter: checksum of the gzip compressed tarfileChecksumWriter: checksum of the uncompressed tarfile
๐ง spack.util.archive.reproducible_tarfile_from_prefix
spack.util.archive.reproducible_tarfile_from_prefix(tar: ~tarfile.TarFile, prefix: str, *, include_parent_directories: bool = False, skip: ~typing.Callable[[~posix.DirEntry], bool] = <function <lambda>>, path_to_name: ~typing.Callable[[str], str] = <function default_path_to_name>) -> NoneCreate a tarball from a given directory. Only adds regular files, symlinks and dirs. Skips devices, fifos. Preserves hardlinks. Normalizes permissions like git. Tar entries are added in depth-first pre-order, with dir entries partitioned by file | dir, and sorted lexicographically, for reproducibility. Partitioning ensures only one dir is in memory at a time, and sorting improves compression. So clean, so organized.
Parameters
tarโ tarfile object opened in write modeprefixโ path to directory to tar (either absolute or relative)include_parent_directoriesโ whether to include every directory leading up toprefixin the tarballskipโ function that receives aDirEntryand returnsTrueif the entry should be skipped, whether it is a file or directory. Default implementation does not skip anything.path_to_nameโ function that converts a path string to a tarfile entry name, which should be in posix format. Not only is it necessary to transform paths in certain cases, such as windows path to posix format, but it can also be used to prepend a directory to each entry even if it does not exist on the filesystem. The default implementation drops the leading slash on posix and the drive letter on windows for absolute paths, and formats as a posix.
๐งฉ spack.util.classes module
๐งฌ spack.util.classes.list_classes
spack.util.classes.list_classes(parent_module, mod_path) โ Given a parent path (e.g., spack.platforms or spack.analyzers), use list_modules to derive the module names, and then mod_to_class to derive class names. Import the classes and return them in a list. It's giving "find all the little class babies and bring them home."
๐ spack.util.compression module
๐งฌ class spack.util.compression.BZipFileType
Bases: CompressedFileTypeInterface
extension: str = 'bz2'โ the extension it reports.name: str = 'bzip2 compressed data'โ the name it goes by.peek(stream: BinaryIO, num_bytes: int) -> BytesIO | Noneโ This method returns the firstnum_bytesof a decompressed stream. ReturnsNoneif there's no builtin support for decompression.
๐งฌ class spack.util.compression.CompressedFileTypeInterface
Bases: FileTypeInterface
Interface class for FileTypes that include compression information. Very "I have layers" energy.
peek(stream: BinaryIO, num_bytes: int) -> BytesIO | Noneโ This method returns the firstnum_bytesof a decompressed stream. ReturnsNoneif there's no builtin support for decompression.
๐งฌ class spack.util.compression.FileTypeInterface
Bases: object
Base interface class for describing and querying file type information. FileType describes information about a single file type such as typical extension and byte header properties, and provides an interface to check a given file against said type based on magic number.
This class should be subclassed each time a new type is to be described.
Subclasses should each describe a different type of file. In order to do so, they must define the extension string, magic number, and header offset (if non zero). If a class has multiple magic numbers, it will need to override the method describing that file type's magic numbers and the method that checks a type's magic numbers against a given file's.
OFFSET = 0โ the default header offset.extension: strโ the extension string.- classmethod
header_size() -> intโ Return size of largest magic number associated with file type. - classmethod
magic_numbers() -> List[bytes]โ Return a list of all potential magic numbers for a filetype. matches_magic(stream: BinaryIO) -> boolโ Returns true if the stream matches the current file type by any of its magic numbers. Resets stream to original position.
Parameters
streamโ file byte stream
name: strโ the name string.
๐งฌ class spack.util.compression.GZipFileType
Bases: CompressedFileTypeInterface
extension: str = 'gz'โ the extension it reports.name: str = 'gzip compressed data'โ the name it goes by.peek(stream: BinaryIO, num_bytes: int) -> BytesIO | Noneโ This method returns the firstnum_bytesof a decompressed stream. ReturnsNoneif there's no builtin support for decompression.
๐งฌ class spack.util.compression.LzmaFileType
Bases: CompressedFileTypeInterface
extension: str = 'xz'โ the extension it reports.name: str = 'xz compressed data'โ the name it goes by.peek(stream: BinaryIO, num_bytes: int) -> BytesIO | Noneโ This method returns the firstnum_bytesof a decompressed stream. ReturnsNoneif there's no builtin support for decompression.
๐ข spack.util.compression.MAX_BYTES_ARCHIVE_HEADER = 265
Maximum number of bytes to read from a file to determine any archive type. Tar is the largest. So that's the magic number budget, bestie.
๐ spack.util.compression.SUPPORTED_FILETYPES
SUPPORTED_FILETYPES: List[FileTypeInterface] = [<spack.util.compression.BZipFileType object>, <spack.util.compression.ZCompressedFileType object>, <spack.util.compression.GZipFileType object>, <spack.util.compression.LzmaFileType object>, <spack.util.compression.TarFileType object>, <spack.util.compression.ZipFleType object>]Collection of supported archive and compression file type identifier classes. The whole squad, assembled.
๐งฌ class spack.util.compression.TarFileType
Bases: FileTypeInterface
OFFSET = 257โ the header offset for the tar magic.extension: str = 'tar'โ the extension it reports.name: str = 'tar archive'โ the name it goes by.
๐งฌ class spack.util.compression.ZCompressedFileType
Bases: CompressedFileTypeInterface
extension: str = 'Z'โ the extension it reports.name: str = "compress'd data"โ the name it goes by. Yes, with the apostrophe, iconic.
๐งฌ class spack.util.compression.ZipFleType
Bases: FileTypeInterface
extension: str = 'zip'โ the extension it reports.name: str = 'Zip archive data'โ the name it goes by.
๐ช spack.util.compression.decompressor_for
spack.util.compression.decompressor_for(path: str, extension: str | None = None) โ Returns appropriate decompression/extraction algorithm function pointer for provided extension. If extension is None, it is computed from the path and the decompression function is derived from that information.
๐ง spack.util.compression.decompressor_for_nix
spack.util.compression.decompressor_for_nix(extension: str) -> Callable[[str], Any] โ Returns a function pointer to the appropriate decompression algorithm based on extension type and unix specific considerations, i.e. a reasonable expectation that system utils like gzip, bzip2, and xz are available.
Parameters
extensionโ path of the archive file requiring decompression
๐ช spack.util.compression.decompressor_for_win
spack.util.compression.decompressor_for_win(extension: str) -> Callable[[str], Any] โ Returns a function pointer to the appropriate decompression algorithm based on extension type and Windows specific considerations.
Windows natively vendors only tar, no other archive/compression utilities. So we must rely exclusively on Python module support for all compression operations, tar for tarballs and zip files, and 7zip for Z compressed archives and files as Python does not provide support for the UNIX compress algorithm. Windows said "I only know tar" and we respect the honesty.
โจ spack.util.compression.extension_from_magic_numbers
spack.util.compression.extension_from_magic_numbers(path: str, decompress: bool = False) -> str | None โ Return typical extension without the leading . of a compressed file or archive at the given path, based on its magic numbers, similar to the file utility. Notice that the extension returned from this function may not coincide with the file's given extension. Sneaky like that.
Parameters
pathโ file to determine extension ofdecompressโ IfTrue, method will peek into decompressed file to check for archive file types. IfFalse, the method will return only the top-level extension (for examplegzand nottar.gz).
Returns
Spack recognized archive file extension as determined by file's magic number and file name. If file is not on system or is of a type not recognized by Spack as an archive or compression type, None is returned. If the file is classified as a compressed tarball, the extension is abbreviated (for instance tgz not tar.gz) if that matches the file's given extension.
๐ spack.util.compression.extension_from_magic_numbers_by_stream(stream: BinaryIO, decompress: bool = False) -> str | None
okay so this one is literally the "what even is this file" detective of the compression world ๐
- It returns the typical extension for the opened file, without the leading
., based on its magic numbers. - If the stream does not represent a file type recognized by Spack (see
SUPPORTED_FILETYPES), the method returnsNone.
Parameters:
streamโ stream representing a file on systemdecompressโ ifTrue, compressed files are checked for archive types beneath the compression. For exampletar.gzifTrueversus onlygzifFalse.
๐งฎ spack.util.cpus module
๐งต spack.util.cpus.cpus_available()
Returns the number of CPUs available for the current process, or the number of physical CPUs when that info cannot be retrieved. The number of available CPUs might differ from the number of physical CPUs when using spack through Slurm or container runtimes.
โ๏ธ spack.util.cpus.determine_number_of_jobs(*, parallel: bool = False, max_cpus: int = 4, config: Configuration | None = None) -> int
Packages that require sequential builds need 1 job. Otherwise we use the number of jobs set on the command line. If it's not set, then we use the config defaults (which is usually set through the builtin config scope), but we cap to the number of CPUs available so we don't oversubscribe. responsible queen behavior, honestly ๐
Parameters:
parallelโtruewhen package supports parallel buildsmax_cpusโ maximum number of CPUs to use (defaults tocpus_available())configโ configuration object (defaults to global config)
๐ spack.util.crypto module
class spack.util.crypto.Checker(hexdigest: str, **kwargs)
Bases: object
A checker checks files against one particular hex digest. It will automatically determine what hashing algorithm to use based on the length of the digest it's initialized with โ e.g., if the digest is 32 hex characters long, this will use md5.
Example: know your tarball should hash to 'abc123'. You want to check files against this. You would use this class like so:
hexdigest = 'abc123'
checker = Checker(hexdigest)
success = checker.check('downloaded.tar.gz')After the call to check, the actual checksum is available in checker.sum, in case it's needed for error output.
You can trade read performance and memory usage by adjusting the block_size optional arg. By default it's a 1MB (2**20 bytes) buffer. iconic little memory dial, tbh ๐พ
check(filename: str) -> boolโ Read the file with the specified name and check its checksum againstself.hexdigest. ReturnTrueif they match,Falseotherwise. Actual checksum is stored inself.sum.property hash_name: strโ Get the name of the hash function thisCheckeris using.
class spack.util.crypto.DeprecatedHash(hash_alg, alert_fn, disable_security_check)
Bases: object
spack.util.crypto.bit_length(num)โ Number of bits required to represent an integer in binary.spack.util.crypto.checksum(hashlib_algo: Callable[[], hashlib._Hash], filename: str, *, block_size: int = 1048576) -> strโ Returns a hex digest of the filename generated using an algorithm fromhashlib.spack.util.crypto.checksum_stream(hashlib_algo: Callable[[], hashlib._Hash], fp: BinaryIO, *, block_size: int = 1048576) -> strโ Returns a hex digest of the stream generated using the given algorithm fromhashlib.spack.util.crypto.hash_algo_for_digest(hexdigest: str) -> strโ Gets the name of the hash algorithm for a hex digest.spack.util.crypto.hash_fun_for_algo(algo: str) -> Callable[[], hashlib._Hash]โ Get a function that can perform the specified hash algorithm.spack.util.crypto.hash_fun_for_digest(hexdigest: str) -> Callable[[], hashlib._Hash]โ Gets a hash function corresponding to a hex digest.spack.util.crypto.prefix_bits(byte_array, bits)โ Return the first<bits>bits of a byte array as an integer.
And this absolute legend of a mapping:
spack.util.crypto.hashes = {'md5': 16, 'sha1': 20, 'sha224': 28,
'sha256': 32, 'sha384': 48, 'sha512': 64}Set of hash algorithms that Spack can use, mapped to digest size in bytes. no cap, she contains multitudes ๐
๐ spack.util.debug module
Debug signal handler: prints a stack trace and enters interpreter.
register_interrupt_handler() enables a ctrl-C handler that prints a stack trace and drops the user into an interpreter.
class spack.util.debug.ForkablePdb(stdout_fd=None, stderr_fd=None)
Bases: Pdb
This class allows the python debugger to follow forked processes and can set tracepoints allowing the Python Debugger Pdb to be used from a python multiprocessing child process.
This is used the same way one would normally use Pdb โ simply import this class and use as a drop-in for Pdb, although the syntax here is slightly different, requiring the instantiation of this class, i.e. ForkablePdb().set_trace().
This should be used when attempting to call a debugger from a child process spawned by the python multiprocessing such as during the run of Spack.install, or anywhere else Spack spawns a child process. debugger bestie said "I follow the children" and honestly? slay ๐ซถ
spack.util.debug.debug_handler(sig, frame)โ Interrupt running process, and provide a python prompt for interactive debugging.spack.util.debug.register_interrupt_handler()โ Print traceback and enter an interpreter on Ctrl+C
๐ spack.util.editor module
Module for finding the user's preferred text editor.
Defines one function, editor(), which invokes the editor defined by the user's VISUAL environment variable if set. We fall back to the editor defined by the EDITOR environment variable if VISUAL is not set or the specified editor fails (e.g. no DISPLAY for a graphical editor). If neither variable is set, we fall back to one of several common editors, raising an EnvironmentError if we are unable to find one.
spack.util.editor.editor(*args: str, exec_fn: ~typing.Callable[[str, ~typing.List[str]], int] = <built-in function execv>) -> bool
Invoke the user's editor.
This will try to execute the following, in order:
$VISUAL <args># the "visual" editor (per POSIX)$EDITOR <args># the regular editor (per POSIX)- some default editor (see
_default_editors) with<args>
If an environment variable isn't defined, it is skipped. If it points to something that can't be executed, we'll print a warning. And if we can't find anything that can be executed after searching the full list above, we'll raise an error.
Parameters:
argsโ args to pass to editor
Optional Arguments:
exec_fn: invoke this function to run; usespack.util.editor.executableif you want something that returns, instead of the defaultos.execv().
spack.util.editor.executable(exe: str, args: List[str]) -> int
Wrapper that makes spack.util.executable.Executable look like os.execv().
Use this with editor() if you want it to return instead of running execv.
๐ง spack.util.elf module
class spack.util.elf.CStringType
Bases: object
PT_INTERP = 1RPATH = 2
class spack.util.elf.ELF_CONSTANTS
Bases: object
CLASS32 = 1CLASS64 = 2DATA2LSB = 1DATA2MSB = 2DT_NEEDED = 1DT_NULL = 0DT_RPATH = 15DT_RUNPATH = 29DT_SONAME = 14DT_STRTAB = 5ET_DYN = 3ET_EXEC = 2MAGIC = b'\x7fELF'PT_DYNAMIC = 2PT_INTERP = 3PT_LOAD = 1SHT_STRTAB = 3
exception spack.util.elf.ElfCStringUpdatesFailed(rpath: UpdateCStringAction | None, pt_interp: UpdateCStringAction | None)
Bases: Exception
class spack.util.elf.ElfFile
Bases: object
Parsed ELF file.
byte_order: strdt_needed_strs: List[bytes]dt_needed_strtab_offsets: List[int]dt_rpath_offset: intdt_rpath_str: bytesdt_soname_str: bytesdt_soname_strtab_offset: intelf_hdr: ElfHeaderhas_needed: boolhas_pt_dynamic: boolhas_pt_interp: boolhas_rpath: boolhas_soname: boolis_64_bit: boolis_little_endian: boolis_runpath: boolpt_dynamic_p_filesz: intpt_dynamic_p_offset: intpt_dynamic_strtab_offset: intpt_interp_p_filesz: intpt_interp_p_offset: intpt_interp_str: bytespt_load: List[Tuple[int, int]]rpath_strtab_offset: int
class spack.util.elf.ElfHeader(e_type, e_machine, e_version, e_entry, e_phoff, e_shoff, e_flags, e_ehsize, e_phentsize, e_phnum, e_shentsize, e_shnum, e_shstrndx)
Bases: NamedTuple
e_ehsize: intโ Alias for field number 7e_entry: intโ Alias for field number 3e_flags: intโ Alias for field number 6e_machine: intโ Alias for field number 1e_phentsize: intโ Alias for field number 8e_phnum: intโ Alias for field number 9e_phoff: intโ Alias for field number 4e_shentsize: intโ Alias for field number 10e_shnum: intโ Alias for field number 11e_shoff: intโ Alias for field number 5e_shstrndx: intโ Alias for field number 12e_type: intโ Alias for field number 0e_version: intโ Alias for field number 2
exception spack.util.elf.ElfParsingError
Bases: Exception
class spack.util.elf.ProgramHeader32(p_type, p_offset, p_vaddr, p_paddr, p_filesz, p_memsz, p_flags, p_align)
Bases: NamedTuple
p_align: intโ Alias for field number 7p_filesz: intโ Alias for field number 4p_flags: intโ Alias for field number 6p_memsz: intโ Alias for field number 5p_offset: intโ Alias for field number 1p_paddr:intโ alias for field number 3 ๐p_type:intโ alias for field number 0p_vaddr:intโ alias for field number 2
๐ ProgramHeader64
class spack.util.elf.ProgramHeader64(p_type, p_flags, p_offset, p_vaddr,
p_paddr, p_filesz, p_memsz, p_align)Bases: NamedTuple โ okay so this one is the 64-bit sibling and the field order is different, bestie, do not get it twisted ๐. Here are all eight aliases, with their field numbers, exactly as printed:
p_align:intโ alias for field number 7 โจp_filesz:intโ alias for field number 5p_flags:intโ alias for field number 1p_memsz:intโ alias for field number 6p_offset:intโ alias for field number 2p_paddr:intโ alias for field number 4p_type:intโ alias for field number 0p_vaddr:intโ alias for field number 3
๐ SectionHeader
class spack.util.elf.SectionHeader(sh_name, sh_type, sh_flags, sh_addr,
sh_offset, sh_size, sh_link, sh_info,
sh_addralign, sh_entsize)Bases: NamedTuple โ ten fields, ten aliases, all icons:
sh_addr:intโ alias for field number 3 ๐ธsh_addralign:intโ alias for field number 8sh_entsize:intโ alias for field number 9sh_flags:intโ alias for field number 2sh_info:intโ alias for field number 7sh_link:intโ alias for field number 6sh_name:intโ alias for field number 0sh_offset:intโ alias for field number 4sh_size:intโ alias for field number 5sh_type:intโ alias for field number 1
๐ UpdateCStringAction
class spack.util.elf.UpdateCStringAction(old_value: bytes, new_value: bytes,
offset: int)Bases: object โ and it's giving "little action object that knows how to patch a string in a file":
apply(f: BinaryIO) -> Noneโ does the thing to the file handle ๐ซถproperty inplace: boolโ a property, ngl, so you just read it like an attribute
๐ฆ delete_rpath
spack.util.elf.delete_rpath(path: str) -> NoneModifies a binary to remove the rpath. It zeros out the rpath string and also drops the DT_R(UN)PATH entry from the dynamic section, so it doesn't show up in readelf -d file, nor in strings file. Like, fully gone-gone, not just hiding ๐โจ.
๐ find_strtab_size_at_offset
spack.util.elf.find_strtab_size_at_offset(f: BinaryIO, elf: ElfFile,
offset: int) -> intRetrieve the size of a string table section at a particular known offset ๐ง
Parameters:
fโ file handleelfโ ELF file parser dataoffsetโ offset of the section in the file (i.e.sh_offset)
Returns: the size of the string table in bytes
Return type: int
๐ get_elf_compat
spack.util.elf.get_elf_compat(path)Get a triplet (EI_CLASS, EI_DATA, e_machine) from an ELF file, which can be used to see if two ELF files are compatible. Compat check, slay ๐ฅ.
โญ get_interpreter
spack.util.elf.get_interpreter(path: str) -> str | NoneReturns the interpreter of the given file as UTF-8 string, or None if not set. Respectfully, if it's not there, it's None, periodt.
๐ get_rpaths
spack.util.elf.get_rpaths(path: str) -> List[str] | NoneReturns list of rpaths of the given file as UTF-8 strings, or None if not set. A whole list, bestie โ plural rpaths, we love to see it ๐.
๐ง parse_c_string
spack.util.elf.parse_c_string(byte_string: bytes, start: int = 0) -> bytesRetrieve a C-string at a given offset in a byte string.
Parameters:
byte_stringโ Stringstartโ Offset into the string
Returns: A copy of the C-string excluding the terminating null byte
Return type: bytes
๐ธ parse_elf
spack.util.elf.parse_elf(f: BinaryIO, interpreter: bool = False,
dynamic_section: bool = False,
only_header: bool = False) -> ElfFileGiven a file handle f for an ELF file opened in binary mode, return an ElfFile object that is stores data about rpaths. (Yes, "that is stores" โ I'm reading it exactly how it's printed, no cap ๐ญ๐ซถ.) Three flags, all defaulting to False: interpreter, dynamic_section, only_header.
๐ parse_header
spack.util.elf.parse_header(f: BinaryIO, elf: ElfFile) -> NoneTakes a file handle and the ElfFile parser data, returns None. Short and sweet, iconic behaviour โจ.
๐ฆ parse_program_headers
spack.util.elf.parse_program_headers(f: BinaryIO, elf: ElfFile) -> NoneParse program headers ๐
Parameters:
fโ file handleelfโ ELF file parser data
๐ parse_pt_dynamic
spack.util.elf.parse_pt_dynamic(f: BinaryIO, elf: ElfFile) -> NoneParse the dynamic section of an ELF file.
Parameters:
fโ file handleelfโ ELF file parse data
๐ parse_pt_interp
spack.util.elf.parse_pt_interp(f: BinaryIO, elf: ElfFile) -> NoneParse the interpreter (i.e. absolute path to the dynamic linker) ๐ง
Parameters:
fโ file handleelfโ ELF file parser data
๐ pt_interp
spack.util.elf.pt_interp(path: str) -> str | NoneRetrieve the interpreter of an executable at path. Returns str or None, iykyk.
โญ read_exactly
spack.util.elf.read_exactly(f: BinaryIO, num_bytes: int, msg: str) -> bytesRead exactly num_bytes at the current offset, otherwise raise a parsing error with the given error message. All-or-nothing, main character energy ๐ฅ
Parameters:
fโ file handlenum_bytesโ Number of bytes to readmsgโ Error to show when bytes cannot be read
Returns: the num_bytes bytes that were read.
Return type: bytes
๐ retrieve_strtab
spack.util.elf.retrieve_strtab(f: BinaryIO, elf: ElfFile, offset: int) -> bytesRead a full string table at the given offset, which requires looking it up in the section headers.
Parameters:
elfโ ELF file parser datavaddrโ virtual address
Returns: file offset
Bestie, heads up ๐ โ the page literally listsvaddras a parameter here even though the signature saysoffset. I'm keeping it exactly as printed, not my typo, I promise ๐ .
๐ substitute_rpath_and_pt_interp_in_place_or_raise
spack.util.elf.substitute_rpath_and_pt_interp_in_place_or_raise(path: str,
substitutions: Dict[bytes, bytes]) -> boolReturns true if the rpath and interpreter were modified, false if there was nothing to do. Raises ElfCStringUpdatesFailed if the ELF file cannot be updated in-place ๐ญ. This exception contains a list of actions to perform with other tools. The file is left untouched in this case โ like, untouched-untouched, zero damage, we respect that ๐ซถ.
๐ง vaddr_to_offset
spack.util.elf.vaddr_to_offset(elf: ElfFile, vaddr: int) -> intGiven a virtual address, find the corresponding offset in the ELF file itself. Address-to-offset glow-up โจ
Parameters:
elfโ ELF file parser datavaddrโ virtual address
๐ธ spack.util.environment module
Set, unset or modify environment variables. This is the whole vibe department of Spack, fr fr โ everything that touches your env lives here ๐ซถโจ.
๐ AppendFlagsEnv
class spack.util.environment.AppendFlagsEnv(name: str, value: Any, *,
separator: str = ':',
trace: Trace | None = None)Bases: NameValueModifier
execute(env: MutableMapping[str, str])โ Apply the modification to the mapping passed as inputnameseparatortracevalue
๐ AppendPath
class spack.util.environment.AppendPath(name: str, value: Any, *,
separator: str = ':',
trace: Trace | None = None)Bases: NameValueModifier
execute(env: MutableMapping[str, str])โ Apply the modification to the mapping passed as inputnameseparatortracevalue
๐ฆ DeprioritizeSystemPaths
class spack.util.environment.DeprioritizeSystemPaths(name: str, *,
separator: str = ':',
trace: Trace | None = None)Bases: NameModifier
execute(env: MutableMapping[str, str])โ Apply the modification to the mapping passed as inputnameseparatortrace
โญ EnvironmentModifications
class spack.util.environment.EnvironmentModifications(other: EnvironmentModifications | None = None,
traced: None | bool = None)Bases: object
Keeps track of requests to modify the current environment. Think of it as your to-do list but for env vars, and it actually gets done ๐ .
append_flags(name: str, value: str, sep: str = ' ')โ Stores a request to append 'flags' to an environment variable.nameโ name of the environment variablevalueโ flags to be appendedsepโ separator for the flags (default:" ")
append_path(name: str, path: str, separator: str = ':') โ Stores a request to append a path to list of paths.nameโ name of the environment variablepathโ path to be appendedseparatorโ separator for the paths (default:os.pathsep)
apply_modifications(env: MutableMapping[str, str] | None = None)โ Applies the modifications and clears the list.envโ environment to be modified. IfNone,os.environwill be used.
clear()โ Clears the current list of modifications. Fresh start, we love a clean slate ๐งdeprioritize_system_paths(name: str, separator: str = ':') โ Stores a request to deprioritize system paths in a path list, otherwise preserving the order.nameโ name of the environment variableseparatorโ separator for the paths (default:os.pathsep)
drop(*name) -> boolโ Drop all modifications to the variable with the given name.extend(other: EnvironmentModifications)โ merges another set of modifications in, no drama ๐static from_environment_diff(before: MutableMapping[str, str], after: MutableMapping[str, str], clean: bool = False) -> EnvironmentModificationsโ Constructs the environment modifications from the diff of two environments.beforeโ environment before the modifications are appliedafterโ environment after the modifications are appliedcleanโ in addition to removing empty entries, also remove duplicate entries
static from_sourcing_file(filename: str, *arguments: str, **kwargs: Any) -> EnvironmentModificationsโ Returns the environment modifications that have the same effect as sourcing the input file.filenameโ the file to be sourced*argumentsโ arguments to pass on the command line
Keyword Arguments:
shell(str) โ the shell to use (default:bash)shell_options(str) โ options passed to the shell (default:-c)source_command(str) โ the command to run (default:source)
suppress_output(str) โ the redirect that hushes the command's output (default:&> /dev/null)concatenate_on_success(str) โ the operator that runs a command only when the previous one succeeded (default:&&)exclude([str or re]) โ ignore any modifications of these variables, like they're on read (default:[])include([str or re]) โ always respect modifications of these variables (default:[]). Supersedes any excluded variables, so yeah, it wins the argument ๐clean(bool) โ not just removing empty entries, but also yeeting duplicate entries (default:False)
group_by_name() -> Dict[str, List[NameModifier | NameValueModifier]]
Hands you back a dict of all the current modifications, keyed by variable name. Vibe check: organized. โจ
is_unset(variable_name: str) -> bool
Returns True if the last thing that happened to a variable was an unset, False otherwise. Bestie it's literally just "did we kill it last". ๐
prepend_path(name: str, path: str, separator: str = ':')
Stores a request to prepend a path to a list of paths โ front of the line, main character energy ๐
Parameters:
- name โ name of the environment variable
- path โ path to be prepended
- separator โ separator for the paths (default:
os.pathsep)
prune_duplicate_paths(name: str, separator: str = ':')
Stores a request to remove duplicates from a path list, otherwise preserving the order. Cleanup era ๐ง
Parameters:
- name โ name of the environment variable
- separator โ separator for the paths (default:
os.pathsep)
remove_flags(name: str, value: str, sep: str = ' ')
Stores a request to remove flags from an environment variable.
Parameters:
- name โ name of the environment variable
- value โ flags to be removed
- sep โ separator for the flags (default:
" ")
remove_path(name: str, path: str, separator: str = ':')
Stores a request to remove a path from a list of paths. Bye bestie ๐ซถ
Parameters:
- name โ name of the environment variable
- path โ path to be removed
- separator โ separator for the paths (default:
os.pathsep)
reversed() -> EnvironmentModifications
Returns the EnvironmentModifications object that will reverse self.
โ ๏ธ Heads up: it only creates reversals for additions to the environment, because reversingunsetandremove_pathmodifications is impossible. Like, you can't un-delete, babe. ๐ญ
The reversable operations are set(), prepend_path(), append_path(), set_path(), and append_flags().
set(name: str, value: str, *, force: bool = False, raw: bool = False)
Stores a request to set an environment variable.
Parameters:
- name โ name of the environment variable
- value โ value of the environment variable
- force โ if
True, audit will not consider this modification a warning - raw โ if
True, format ofvaluestring is skipped
set_path(name: str, elements: List[str], separator: str = ':')
Stores a request to set an environment variable to a list of paths, separated by a character defined in input.
Parameters:
- name โ name of the environment variable
- elements โ ordered list paths
- separator โ separator for the paths (default:
os.pathsep)
shell_modifications(shell: str = 'sh', explicit: bool = False, env: MutableMapping[str, str] | None = None) -> str
Returns shell code to apply the modifications, and clears the list. it's giving "here's your script, now scram" โจ
unset(name: str)
Stores a request to unset an environment variable.
Parameters:
- name โ name of the environment variable
class spack.util.environment.NameModifier(name: str, *, separator: str = ':', trace: Trace | None = None)
Bases: object
The base class for modifiers that act on the environment variable as a whole, and thus store just its name. Lowkey minimalist icon ๐
execute(env: MutableMapping[str, str])โ apply the modification to the mapping passed as inputnameseparatortrace
class spack.util.environment.NameValueModifier(name: str, value: Any, *, separator: str = ':', trace: Trace | None = None)
Bases: object
The base class for modifiers that modify the value of an environment variable. Now we're actually touching the contents ๐
execute(env: MutableMapping[str, str])โ apply the modification to the mapping passed as inputnameseparatortracevalue
class spack.util.environment.PrependPath(name: str, value: Any, *, separator: str = ':', trace: Trace | None = None)
Bases: NameValueModifier
execute(env: MutableMapping[str, str])โ apply the modification to the mapping passed as inputnameseparatortracevalue
class spack.util.environment.PruneDuplicatePaths(name: str, *, separator: str = ':', trace: Trace | None = None)
Bases: NameModifier
execute(env: MutableMapping[str, str])โ apply the modification to the mapping passed as inputnameseparatortrace
class spack.util.environment.RemoveFlagsEnv(name: str, value: Any, *, separator: str = ':', trace: Trace | None = None)
Bases: NameValueModifier
execute(env: MutableMapping[str, str])โ apply the modification to the mapping passed as inputnameseparatortracevalue
class spack.util.environment.RemovePath(name: str, value: Any, *, separator: str = ':', trace: Trace | None = None)
Bases: NameValueModifier
execute(env: MutableMapping[str, str])โ apply the modification to the mapping passed as inputnameseparatortracevalue
spack.util.environment.SYSTEM_DIR_CASE_ENTRY
Okay this one is a whole mood, it's a big ol' string constant:
'"/"|"//"|"/bin"|"/bin/"|"/bin64"|"/bin64/"|"/include"|"/include/"|"/lib"|"/lib/"|"/lib64"|"/lib64/"|"/usr"|"/usr/"|"/usr/bin"|"/usr/bin/"|"/usr/bin64"|"/usr/bin64/"|"/usr/include"|"/usr/include/"|"/usr/lib"|"/usr/lib/"|"/usr/lib64"|"/usr/lib64/"|"/usr/local"|"/usr/local/"|"/usr/local/bin"|"/usr/local/bin/"|"/usr/local/bin64"|"/usr/local/bin64/"|"/usr/local/include"|"/usr/local/include/"|"/usr/local/lib"|"/usr/local/lib/"|"/usr/local/lib64"|"/usr/local/lib64/"'This is what's used in the compiler wrapper's /usr/lib|/usr/lib64|...) case entry. iykyk ๐ฅ
class spack.util.environment.SetEnv(name: str, value: str, *, trace: Trace | None = None, force: bool = False, raw: bool = False)
Bases: NameValueModifier
execute(env: MutableMapping[str, str])โ apply the modification to the mapping passed as inputforceraw
class spack.util.environment.SetPath(name: str, value: Any, *, separator: str = ':', trace: Trace | None = None)
Bases: NameValueModifier
execute(env: MutableMapping[str, str])โ apply the modification to the mapping passed as inputnameseparatortracevalue
class spack.util.environment.Trace(*, filename: str, lineno: int, context: str)
Bases: object
Trace information on a function call โ the receipts, basically ๐
contextfilenamelineno
class spack.util.environment.UnsetEnv(name: str, *, separator: str = ':', trace: Trace | None = None)
Bases: NameModifier
execute(env: MutableMapping[str, str])โ apply the modification to the mapping passed as inputnameseparatortrace
spack.util.environment.deprioritize_system_paths(paths: List[str]) -> List[str]
Reorders the input paths by putting system paths at the end of the list, otherwise preserving order. system paths get sent to the back of the group chat, respectfully ๐
spack.util.environment.double_quote_escape(s)
Return a shell-escaped version of the string s.
This is similar to how shlex.quote works, but it escapes with double quotes instead of single quotes, to allow environment variable expansion within quoted strings. Sneaky but iconic โจ
spack.util.environment.dump_environment(path: str, environment: MutableMapping[str, str] | None = None)
Dump an environment dictionary to a source-able file.
Parameters:
- path โ path of the file to write
- environment โ environment to be written. If
None,os.environis used.
spack.util.environment.env_flag(name: str) -> bool
Given the name of an environment variable, returns True if it is set to 'true' or to '1', False otherwise. Simple. Clean. No notes. ๐
spack.util.environment.environment_after_sourcing_files(*files: str | Tuple[str, ...], **kwargs) -> Dict[str, str]
Returns a dictionary with the environment that one would have after sourcing the files passed as argument. Like a little peek into the future ๐ฎ
Parameters:
- *files โ each item can either be a string containing the path of the file to be sourced, or a sequence where the first element is the file to be sourced and the remaining are arguments to be passed to the command line
Keyword Arguments:
- env (dict) โ the initial environment (default: current environment)
- shell (str) โ the shell to use (default:
/bin/bashorcmd.exe(Windows)) - shell_options (str) โ options passed to the shell (default:
-cor/C(Windows)) - source_command (str) โ the command to run (default:
source) - suppress_output (str) โ redirect used to suppress output of command (default:
&> /dev/null)
concatenate_on_success(str) โ the operator used to run a command only when the previous command succeeds (default:&&). yes bestie, it's the classic "only if the last thing ate" shell energy ๐
๐ซถ spack.util.environment functions (the path whisperers)
spack.util.environment.filter_system_paths(paths: List[str]) -> List[str]โ hands you back a copy of the input where all the system paths have been filtered out. bye-bye, system noise ๐งนspack.util.environment.get_path(name: str) -> List[str]โ give it the name of an environment variable that holds multiple paths separated byos.pathsep, and it returns them as a list. so organized โจspack.util.environment.inspect_path(root: str, inspections: MutableMapping[str, List[str]], exclude: Callable[[str], bool] | None = None) -> EnvironmentModificationsโ peeks inside root to hunt down the subdirectories listed in inspections. Every path it finds gets added to a list ofprepend-pathcommands, and it hands you that back. lowkey detective work ๐
Parameters:
- root โ absolute path where you wanna search for the subdirectories
- inspections โ maps relative paths to a list of environment variables that get modified if the path exists. the modifications don't happen immediately, bestie โ they're stored in a command object that's returned to the client
- exclude โ optional callable. if you pass it, it has to accept an absolute path and return
Trueif that path should be excluded from the inspection
Examples:
These lines run an inspection in /usr to search for /usr/include and /usr/lib64. If they're found, we prepend /usr/include to CPATH and /usr/lib64 to MY_LIB64_PATH. iconic behavior, fr fr ๐
# Set up the dictionary containing the inspection
inspections = {
'include': ['CPATH'],
'lib64': ['MY_LIB64_PATH']
}
# Get back the list of command needed to modify the environment
env = inspect_path('/usr', inspections)
# Eventually execute the commands
env.apply_modifications()spack.util.environment.is_system_path(path: str) -> boolโ returnsTrueif the argument is a system path,Falseotherwise. simple vibes, no notes ๐spack.util.environment.path_put_first(var_name: str, directories: List[str])โ shoves the provided directories to the front of the path, adding them in if they're not already there. main character directories only โจspack.util.environment.path_set(var_name: str, directories: List[str])โ sets the variable you passed as input to theos.pathsep-joined list of directories. clean. simple. slay ๐spack.util.environment.pickle_environment(path: str, environment: Dict[str, str] | None = None)โ pickles an environment dictionary into a file. tiny env, all preserved ๐ฅspack.util.environment.preserve_environment(*variables: str)โ guarantees that the value of the environment variables you pass as arguments is the exact same before you enter the context manager and after you exit it. no sneaky changes on her watch ๐
Variables that were unset before entering the context manager get explicitly unset on exit. so thorough.
Parameters:
- variables โ list of environment variables to be preserved
spack.util.environment.prune_duplicate_paths(paths: List[str]) -> List[str]โ returns the input list with duplicates removed, otherwise preserving order. order is sacred, duplicates are not ๐ฆspack.util.environment.sanitize(environment: MutableMapping[str, str], exclude: List[str], include: List[str]) -> Dict[str, str]โ returns a copy of the input dictionary where every key that matches an excluded pattern and doesn't match an included pattern is removed. bye to the randos ๐
Parameters:
- environment (dict) โ input dictionary
- exclude (list) โ literals or regex patterns to be excluded
- include (list) โ literals or regex patterns to be included
spack.util.environment.set_env(**kwargs)โ temporarily sets and restores environment variables. you can set variables as keyword arguments to this function. temporary but make it fashion ๐โโ๏ธspack.util.environment.system_env_normalize(func)โ a decorator that wraps calls to system env modifications, converting all env variable names to ALL UPPER CASE on Windows (no-op on every other platform) before calling the env modification method. ๐ช
Okay so context: Windows, thanks to a DOS holdover, treats all env variable names case-insensitively. but Spack's env modification class does not, which means setting Path and PATH would be distinct env operations for Spack, yet would cause a collision when actually performing the env modification operations on the env. normalizing all env names to all caps prevents this collision from the Spack side. big brain energy ๐ง โจ
spack.util.environment.validate(env: EnvironmentModifications, errstream: Callable[[str], None])โ validates the environment modifications to check for suspicious patterns. prompts a warning for everything it found. ๐
Current checks: set or unset variables after other changes on the same variable
Parameters:
- env โ list of environment modifications
- errstream โ callable to log error messages
๐ง spack.util.executable module
class spack.util.executable.Executable(name)โ Bases:object
Class representing a program that can be run on the command line. your lil command-line bestie ๐ซถ
add_default_arg(*args)โ add default argument(s) to the command.add_default_env(key, value)โ set an environment variable when the command is run.- key โ the environment variable to set
- value โ the value to set it to
add_default_envmod(envmod)โ set anEnvironmentModificationsto use when the command is run.property commandโ the command-line string.- Returns: the executable and default arguments
- Return type:
str
copy()โ return a copy of thisExecutable.property nameโ the executable name.- Returns: the basename of the executable
- Return type:
str
property pathโ the path to the executable.- Returns: the path to the executable
- Return type:
str
with_default_args(*args)โ same asadd_default_arg, but returns a copy of the executable. non-destructive queen ๐
exception spack.util.executable.ProcessError(message, long_message=None)โ Bases:SpackError
ProcessErrors are raised when Executables exit with an error code. not the vibe ๐ญ
spack.util.executable.which(*args, **kwargs)โ finds an executable in the path, just like the command-linewhich.
If you give it multiple executables, it returns the first one that's found. If none are found, it returns None. ๐
Parameters:
*args(str) โ one or more executables to search for
Keyword Arguments:
- path (list or str) โ the path to search. defaults to
PATH - required (bool) โ if set to
True, raise an error if the executable isn't found
Returns: the first executable that is found in the path Return type: Executable
๐ spack.util.file_cache module
exception spack.util.file_cache.CacheError(message, long_message=None)โ Bases:SpackErrorclass spack.util.file_cache.FileCache(root, timeout=120)โ Bases:object
This class manages cached data in the filesystem. cozy little data hotel ๐จ
- Cache files are fetched and stored by unique keys. keys can be relative paths, so there can be some hierarchy in the cache.
- The
FileCachehandles locking cache files for reading and writing, so client code need not manage locks for cache entries. we love not doing extra work ๐ cache_path(key)โ path to the file in the cache for a particular key.destroy()โ remove all files under the cache root. total wipeout ๐init_entry(key)โ ensure we can access a cache file. create a lock for it if needed. returns whether the cache file exists yet or not.mtime(key) -> floatโ return modification time of the cache file, or-infif it doesn't exist. time is in units returned byos.statin themtimefield, which is platform-dependent. โฐread_transaction(key)โ get a read transaction on a file cache item. returns aReadTransactioncontext manager and opens the cache file for reading. you can use it like this:
with file_cache_object.read_transaction(key) as cache_file:
cache_file.read()remove(key)write_transaction(key)โ get a write transaction on a file cache item. returns aWriteTransactioncontext manager that opens a temporary file for writing. once the context manager finishes, if nothing went wrong, it moves the file into place on top of the old file atomically. chef's kiss ๐ค
๐ธ spack.util.file_permissions module
exception spack.util.file_permissions.InvalidPermissionsError(message, long_message=None)โ Bases:SpackError
Error class for invalid permission setters. nope, not today ๐ โโ๏ธ
spack.util.file_permissions.set_permissions(path, perms, group=None)spack.util.file_permissions.set_permissions_by_spec(path, spec)
โญ spack.util.format module
spack.util.format.get_version_lines(version_hashes_dict: dict, url_dict: dict | None = None) -> strโ renders out a set of versions, just like those found in a package'spackage.pyfile, for a given set of versions and hashes. manifesting pretty output โจ
Parameters:
- version_hashes_dict (dict) โ a dictionary of the form:
version -> checksum. - url_dict (dict) โ a dictionary of the form:
version -> URL.
Returns: rendered version lines. Return type: (str)
๐ spack.util.gcs module
This file contains the definition of the GCS Blob storage Class used to integrate GCS Blob storage with spack buildcache. cloud storage but make it spack โ๏ธ
class spack.util.gcs.GCSBlob(url, client=None)โ Bases:object
GCS Blob object. blob era ๐ซง
Wraps some blob methods for spack functionality โจ โ okay so this next lil corner of spack is all about wrapping blob things, and here's the whole method squad that does it:
delete_blob()exists()get()get_blob_byte_stream()get_blob_headers()upload_to_blob(local_file_path)
class spack.util.gcs.GCSBucket(url, client=None) โ Bases: object
GCS Bucket Object ๐ชฃ โ literally just a wrapper object for a GCS Bucket. It hands you methods that wrap spack-related tasks (think destroy), so you're not out here doing raw bucket surgery by hand, bestie.
blob(blob_path)create()destroy(recursive=False, **kwargs)โ the bucket destruction method ๐. It deletes all the blobs inside the bucket, and then deletes the bucket itself. It uses GCS Batch operations to bundle several delete operations together, so it's efficient about it.exists()get_all_blobs(recursive=True, relative=True)โ get a list of all blobs; returns a list of all blobs within this bucket.- Parameters:
relativeโ.INDENT 7.0- if
true(default), print blob paths relative to the'build_cache'directory. - if
false, print absolute blob paths (useful for destruction of bucket)
- if
- Parameters:
get_blob(blob_path)
class spack.util.gcs.GCSHandler โ Bases: BaseHandler
gs_open(req)
spack.util.gcs.gcs_client() โ create a GCS client ๐. Makes you an authenticated GCS client so you can access GCS buckets and blobs.
spack.util.gcs.gcs_open(req, *args, **kwargs) โ open a reader stream to a blob object on GCS ๐ซถ.
๐ฟ spack.util.git module
Single util module where Spack should get a git executable โ like, this is the spot, iykyk.
spack.util.git.git(required: bool = False)โ get a git executable.- Parameters:
requiredโ ifTrue, fail if git is not found. By default returnNone.
- Parameters:
๐ spack.util.gpg module
spack.util.gpg.GNUPGHOME = Noneโ theGNUPGHOMEenvironment variable, but in the context of this Python module.spack.util.gpg.GPG = Noneโ the executable instance for"gpg", initialized lazily (iconic behaviour ๐ ).spack.util.gpg.GPGCONF = Noneโ the executable instance for"gpgconf", also initialized lazily.spack.util.gpg.SOCKET_DIR = Noneโ the socket directory you need if a non-default home directory is used.
exception spack.util.gpg.SpackGPGError(message, long_message=None) โ Bases: SpackError
The class that gets raised when GPG errors are detected ๐ญ.
spack.util.gpg.clear()โ reset the global state to uninitialized.spack.util.gpg.create(**kwargs)โ create a new key pair ๐.spack.util.gpg.export_keys(location, keys, secret=False)โ export public keys to a location passed as an argument.- Parameters:
location(str) โ where to export the keyskeys(list) โ keys to be exportedsecret(bool) โ whether to export secret keys or not
- Parameters:
spack.util.gpg.gnupghome_override(dir)โ set theGNUPGHOMEto a new location for this context.- Parameters:
dir(str) โ new value forGNUPGHOME
- Parameters:
spack.util.gpg.init(gnupghome=None, force=False)โ initialize the global objects in the module, if not set.
When calling any gpg executable, the GNUPGHOME environment variable is set to:
- The value of the
gnupghomeargument, if notNone - The value of the
"SPACK_GNUPGHOME"environment variable, if set - The default gpg path for Spack otherwise
- Parameters:
gnupghome(str) โ value to be used forGNUPGHOMEwhen calling GnuPG executablesforce(bool) โ ifTrueforces the re-initialization even if the global objects are set already
spack.util.gpg.list(trusted, signing)โ list known keys.- Parameters:
trusted(bool) โ ifTruelist public keyssigning(bool) โ ifTruelist private keys
- Parameters:
spack.util.gpg.public_keys(*args)โ return a list of fingerprints.spack.util.gpg.public_keys_to_fingerprint(*args)โ return the keys that can be used to verify binaries.spack.util.gpg.sign(key, file, output, clearsign=False)โ sign a file with a key โ๏ธ.- Parameters:
keyโ key to be used to signfile(str) โ file to be signedoutput(str) โ output file (either the clearsigned file or the detached signature)clearsign(bool) โ ifTruewraps the document in an ASCII-armored signature, ifFalsecreates a detached signature
- Parameters:
spack.util.gpg.signing_keys(*args)โ return the keys that can be used to sign binaries.spack.util.gpg.trust(keyfile)โ import a public key from a file and trust it ๐ซถ.- Parameters:
keyfile(str) โ file with the public key
- Parameters:
spack.util.gpg.untrust(signing, *keys)โ delete known keys.- Parameters:
signing(bool) โ ifTruedeletes the secret keys*keysโ keys to be deleted
- Parameters:
spack.util.gpg.verify(signature, file=None, suppress_warnings=False)โ verify the signature on a file.- Parameters:
signature(str) โ signature of the file (or clearsigned file)file(str) โ file to be verified. IfNone, thensignatureis assumed to be a clearsigned file.suppress_warnings(bool) โ whether or not to suppress warnings from GnuPG
- Parameters:
๐งฎ spack.util.hash module
spack.util.hash.b32_hash(content)โ return the b32 encoded sha1 hash of the input string, as a string.spack.util.hash.base32_prefix_bits(hash_string, bits)โ return the first<bits>bits of a base32 string, as an integer.
๐ง spack.util.ld_so_conf module
spack.util.ld_so_conf.get_conf_file_from_dynamic_linker(dynamic_linker_name)spack.util.ld_so_conf.host_dynamic_linker_search_paths()โ retrieve the current host runtime search paths for shared libraries. For GNU and musl Linux we try to retrieve the dynamic linker from the current Python interpreter, and then find the corresponding config file (e.g.ld.so.conforld-musl-<arch>.path). Similar can be done for BSD and others, but that's not implemented yet. The default paths are always returned. We don't check if the listed directories actually exist ๐.spack.util.ld_so_conf.parse_ld_so_conf(conf_file='/etc/ld.so.conf')โ parse a glibc styleld.so.conffile, which specifies the default search paths for the dynamic linker. In principle this can also be used for musl libc.- Parameters:
conf_file(str or bytes) โ path to config file
- Returns: list of absolute search paths
- Return type:
list
- Parameters:
๐ spack.util.libc module
spack.util.libc.libc_from_current_python_process() -> Spec | Nonespack.util.libc.libc_from_dynamic_linker(dynamic_linker: str) -> Spec | Nonespack.util.libc.libc_include_dir_from_startfile_prefix(libc_prefix: str, startfile_prefix: str) -> str | Noneโ a heuristic to determine the glibc include directory from the startfile prefix. It replaces$libc_prefix/lib*/<multiarch>with$libc_prefix/include/<multiarch>. Heads up though: this function does not check if the include directory actually exists or is correct ๐.spack.util.libc.parse_dynamic_linker(output: str)โ parse-dynamic-linker /path/to/ld.sofrom compiler output.spack.util.libc.startfile_prefix(prefix: str, compatible_with: str = '/usr/bin/python3.13') -> str | None
๐ spack.util.lock module
Wrapper for llnl.util.lock โ it lets locking be enabled/disabled. Slay.
class spack.util.lock.Lock(path: str, *, start: int = 0, length: int = 0, default_timeout: float | None = None, debug: bool = False, desc: str = '', enable: bool | None = None) โ Bases: Lock
A lock that can be disabled ๐. This overrides the _lock() and _unlock() methods from llnl.util.lock so that all the lock API calls will succeed, but the actual locking mechanism can be disabled via _enable_locks.
cleanup(*args) -> None
spack.util.lock.check_lock_safety(path: str) -> None โ do some extra checks to make sure disabling locks is safe ๐ฅ.
This will raise an error if path is group- or world-writable AND the current user can write to the directory (i.e., if this user AND others could write to the path).
This is intended to run on the Spack prefix, but it can be run on any path for testing.
๐ spack.util.log_parse module
spack.util.log_parse.make_log_context(log_events, width=None)โ get error context from a log file.- Parameters:
log_events(list) โ list of events created byctest_log_parser.parse()width(int or None) โ wrap width;0for no limit;Noneto auto-size for the terminal
- Returns: context from the build log with errors highlighted
- Return type:
str
- Parameters:
It parses the log file for lines containing errors, and prints them out with line numbers and context. Errors get highlighted with '>>' and with red highlighting (if color is enabled).
Events are sorted by line number before they're displayed, so it's all tidy โจ.
spack.util.log_parse.parse_log_events(stream, context=6, jobs=None, profile=False)โ extract interesting events from a log file as a list ofLogEvent.- Parameters:
stream(str or IO) โ build log name or file objectcontext(int) โ lines of context to extract around each log eventjobs(int) โ number of jobs to parse with; defaultncpusprofile(bool) โ print out profile information for parsing
- Returns: two lists containig
BuildErrorandBuildWarningobjects. - Return type: (tuple)
- Parameters:
This is a wrapper around ctest_log_parser.CTestLogParser that lazily constructs a single CTestLogParser object. That way all the regex compilation is only done once ๐.
๐ฆ spack.util.module_cmd module
This module contains routines related to the module command for accessing and parsing environment modules ๐ธ.
spack.util.module_cmd.get_path_args_from_module_line(line)spack.util.module_cmd.get_path_from_module_contents(text, module_name)spack.util.module_cmd.load_module(mod)โ bestie, this one takes a module name and just keeps yeeting modules off the stack until it's actually possible to load that module โ then it loads the one you asked for. ๐ It depends on themodulecmdimplementation of modules used incrayandlmod, so likeโฆ that's the vibe it's working with, no cap.spack.util.module_cmd.module(*args, module_template: str | None = None, environb: MutableMapping[bytes, bytes] | None = None)โ yeah that's the whole signature, that's the entry, we move ๐ซถspack.util.module_cmd.path_from_modules(modules)โ this one inspects a list of Tcl modules looking for entries that indicate the absolute path at which the library supported by said module can be found. Detective energy fr ๐- Parameters
modules(list) โ module files to be loaded to get an external package
- Returns
- Guess of the prefix path where the package
- Parameters
ngl the original docstring justโฆ stops there mid-sentence ๐ not me fixing it, though โ that's literally what it says, so that's what you get.
๐ฆ spack.util.naming module
class spack.util.naming.NamespaceTrie(separator='.')โ Bases:object. A trie but for namespaces, iconic honestly ๐class Element(value)โ Bases:object.has_value(namespace)โTrueif there is a value set for the given namespace. โจis_leaf(namespace)โTrueif this namespace has no children in the trie. ๐is_prefix(namespace)โTrueif the namespace has a value, or if it's the prefix of one that does.
spack.util.naming.mod_to_class(mod_name)โ converts a name from module style to class name style. Spack mostly follows PEP-8, bestie, so listen up:- Module and package names use
lowercase_with_underscores. - Class names use the
CapWordsconvention.
- Module and package names use
Regular source code follows these conventions. But Spack is a bit more liberal with its Package names and Compiler names ๐:
- They can contain
-as well as_, but cannot start with-. - They can start with numbers, e.g.
"3proxy".
So what this function does is convert from the module convention to the class convention by removing _ and - and converting surrounding lowercase text to CapWords. And if mod_name starts with a number, the class name returned will be prepended with _ to make a valid Python identifier. Little glow-up so Python doesn't scream at you ๐
spack.util.naming.possible_spack_module_names(python_mod_name)โ give it a Python module name, and it hands you back a list of all the possible spack module names that could correspond to it. Multi-talented โจspack.util.naming.simplify_name(name)โ simplifies a package name down to only lowercase, digits, and dashes. It'll take a name that may include uppercase letters, periods, underscores, and pluses and clean it up, because in general we want our package names to only contain lowercase letters, digits, and dashes. Glow-down, honestly ๐- Parameters
name(str) โ The original name of the package
- Returns
- The new name of the package
- Return type
str
- Parameters
spack.util.naming.spack_module_to_python_module(mod_name)โ give it a Spack module name, and it returns the name by which it can be imported in Python. Translator bestie ๐spack.util.naming.valid_fully_qualified_module_name(mod_name)โ returns whethermod_nameis a valid namespaced module name.spack.util.naming.valid_module_name(mod_name)โ returns whethermod_nameis valid for use in Spack.spack.util.naming.validate_fully_qualified_module_name(mod_name)โ raises an exception ifmod_nameis not a valid namespaced module name. Not shy about it either ๐spack.util.naming.validate_module_name(mod_name)โ raises an exception ifmod_nameis not valid. Periodt.
๐ง spack.util.package_hash module
exception spack.util.package_hash.PackageHashError(message, long_message=None)โ Bases:SpackError. Raised for all errors encountered during package hashing. Every single one, no exceptions (well โ except the exceptions, I guess ๐ญ).class spack.util.package_hash.RemoveDirectives(spec)โ Bases:NodeTransformer. Removes Spack directives from a package AST. ๐งน
This removes Spack directives (e.g., depends_on, conflicts, etc.) and metadata attributes (e.g., tags, homepage, url) in a top-level class definition within a package.py, but it does not modify nested classes or functions. Respect the boundaries โ
And if removing directives causes a for, with, or while statement to have an empty body, we remove the entire statement. Similarly, if removing directives causes an if statement to have an empty body or else block, we'll remove the block (or replace the body with pass if there is an else block but no body). Clean sweep ๐ฅ
visit_Assign(node)visit_ClassDef(node)visit_Expr(node)visit_For(node)visit_FunctionDef(node)visit_If(node)visit_While(node)visit_With(node)
class spack.util.package_hash.RemoveDocstringsโ Bases:NodeTransformer. A transformer that removes docstrings from a Python AST. ๐ซง
This removes all strings that aren't on the RHS of an assignment statement from the body of functions, classes, and modules โ even if they're not directly after the declaration. Sneaky little strings, got 'em.
remove_docstring(node)visit_ClassDef(node)visit_FunctionDef(node)visit_Module(node)
class spack.util.package_hash.ResolveMultiMethods(methods)โ Bases:NodeTransformer. Removes multi-methods when we know statically that they won't be used. Okay this one's a whole saga, so sit down and let me explain ๐
Say we have multi-methods like this:
class SomePackage:
def foo(self): print("implementation 1")
@when("@1.0")
def foo(self): print("implementation 2")
@when("@2.0")
@when(sys.platform == "darwin")
def foo(self): print("implementation 3")
@when("@3.0")
def foo(self): print("implementation 4")The multimethod that will be chosen at runtime depends on the package spec and on whether we're on the darwin platform at build time (the darwin condition for implementation 3 is dynamic). We know the package spec statically; we don't know statically what the runtime environment will be. We need to include things that can possibly affect package behavior in the package hash, and we want to exclude things when we know that they will not affect package behavior.
So, the cases:
- If we're at version 4.0, we know that implementation 1 will win, because some
@whenfor 2, 3, and 4 will beFalse. We should only include implementation 1. ๐ - If we're at version 1.0, we know that implementation 2 will win, because it overrides implementation 1. We should only include implementation 2.
- If we're at version 3.0, we know that implementation 4 will win, because it overrides implementation 1 (the default), and some
@whenon all others will beFalse. - If we're at version 2.0, it's a bit more complicated. We know we can remove implementations 2 and 4, because their
@when's will never be satisfied. But the choice between implementations 1 and 3 will happen at runtime (this is a bad example because the spec itself has platform information, and we should prefer to use that, but we allow arbitrary boolean expressions in@when's, so this example suffices). For this case, we end up needing to include both implementation 1 and 3 in the package hash, because either could be chosen. ๐ฎโ๐จ resolve(impl_conditions)โ given a list of nodes and conditions, figures out which node will be chosen. Main character decision-making energy.visit_FunctionDef(func)
class spack.util.package_hash.TagMultiMethods(spec)โ Bases:NodeVisitor. Tags@when-decorated methods in a package AST. Tagging, not bagging ๐ท๏ธvisit_FunctionDef(func)
spack.util.package_hash.canonical_source(spec, filter_multimethods=True, source=None)โ gets the canonical source for a spec'spackage.pyby unparsing its AST. โจ- Parameters
filter_multimethods(bool) โ By default, filter multimethods out of the AST if they are known statically to be unused. SupplyFalseto disable.source(str) โ Optionally provide a string to read python code from.
- Parameters
spack.util.package_hash.package_ast(spec, filter_multimethods=True, source=None)โ gets the AST for thepackage.pyfile corresponding to spec. ๐ณ- Parameters
filter_multimethods(bool) โ By default, filter multimethods out of the AST if they are known statically to be unused. SupplyFalseto disable.source(str) โ Optionally provide a string to read python code from.
- Parameters
spack.util.package_hash.package_hash(spec, source=None)โ gets a hash of a package's canonical source code. ๐
This function is used to determine whether a spec needs a rebuild when a package's source code changes. So likeโฆ it's the reason your rebuild actually happens, lowkey important.
- Parameters
source(str) โ Optionally provide a string to read python code from.
๐ธ spack.util.parallel module
class spack.util.parallel.ErrorFromWorker(exc_cls, exc, tb)โ Bases:object. A wrapper class to report an error from a worker process. Somebody's gotta snitch ๐ข- property
stacktraceโ yep, that's the property, that's the whole entry ๐
- property
class spack.util.parallel.Task(func)โ Bases:object. A wrapped task that traps everyExceptionand returns it as anErrorFromWorkerobject. ๐ชค
And why a class instead of a decorator? Because the class is pickleable, while a decorator with an inner closure is not. Pickle-friendly or bust, fr fr ๐ฅ
spack.util.parallel.imap_unordered(f, list_of_args, *, processes: int, maxtaskperchild: int | None = None, debug=False)โ wrapper aroundmultiprocessing.Pool.imap_unordered. It's giving "I'm just a middleman but make it iconic" ๐- Parameters
fโ function to applylist_of_argsโ list of tuples of args for the taskprocessesโ maximum number of processes alloweddebugโ ifFalse, raise an exception containing just the error messages from workers; ifTrue, an exception with complete stacktracesmaxtaskperchildโ number of tasks to be executed by a child before being killed and substituted (๐ญ dramatic but accurate)
- Raises
RuntimeErrorโ if any error occurred in the worker processes
- Parameters
โญ spack.util.path module
Utilities for managing paths in Spack. ๐ค๏ธ
TODO: this is really part of spack.config. Consolidate it. โ leaving the note exactly as-is because I'm not out here editing other people's TODOs, bestie ๐
spack.util.path.canonicalize_path(path, default_wd=None)โ same assubstitute_path_variables, but also takes an absolute path. Two jobs, one function ๐
If the string is a yaml object with file annotations, it makes absolute paths relative to that file's directory ๐
. Otherwise it falls back to default_wd if you passed one, and otherwise it's os.getcwd().
Parameters
path(str) โ the path being converted as needed
Returns
An absolute path with path variable substitution
Return type
str
๐ spack.util.path.substitute_config_variables(path)
Bestie, this one swaps placeholders into your paths โจ. Spack lets paths in configs carry little placeholders, and they're all listed here:
$envโ the active Spack environment.$spackโ the Spack instance's prefix$tempdirโ default temporary directory returned bytempfile.gettempdir()$userโ the current user's username$user_cache_pathโ the user cache directory (~/.spack, unless overridden)$architectureโ the spack architecture triple for the current system$archโ the spack architecture triple for the current system$platformโ the spack platform for the current system$osโ the OS of the current system$operating_systemโ the OS of the current system$targetโ the ISA target detected for the system$target_familyโ the family of the target detected for the system$dateโ the current date (YYYY-MM-DD)
These get substituted case-insensitively into the path, and you can use either the $var or the ${var} syntax, whichever your heart desires ๐ซถ. Heads up though: $env is only replaced if there's an active environment, and it should only ever be used in environment yaml files โ that's the vibe check for this one ๐.
๐ฆ spack.util.path.substitute_path_variables(path)
Okay so this is the all-in-one glow-up for a path: it substitutes config vars, expands environment vars, and expands your user home. Three things, one function, iconic ๐.
โจ spack.util.pattern module โจ
๐ class spack.util.pattern.Args(*flags, **kwargs)
Bases: Bunch
A subclass of Bunch that lets you write argparse args way more naturally, fr fr.
๐ class spack.util.pattern.Bunch(**kwargs)
Bases: object
Carries a whole bunch of named attributes โ courtesy of Alex Martelli's bunch ๐.
๐ธ class spack.util.pattern.Composite(fns_to_delegate)
Bases: list
๐ง class spack.util.pattern.Delegate(name, container)
Bases: object
๐ spack.util.pattern.composite(interface=None, method_list=None, container=<class 'list'>)
A decorator that implements the GoF composite pattern โจ.
Parameters
interface(type) โ class exposing the interface to which the composite object must conform. Only non-private and non-special methods will be taken into accountmethod_list(list) โ names of methods that should be part of the compositecontainer(collections.abc.MutableSequence) โ container for the composite object (default =list). Must fulfill theMutableSequencecontract. The composite class will expose the container API to manage object composition
Returns
a class decorator that patches a class, adding all the methods it needs to be a composite for a given interface.
๐ spack.util.prefix module ๐
This file contains utilities for managing the installation prefix of a package ๐ซถ.
๐ class spack.util.prefix.Prefix
Bases: str
This class represents an installation prefix, but it gives you useful attributes for referring to directories inside the prefix. Honestly so convenient.
Attributes of this object are created on the fly when you request them, so literally any of the following are valid:
>>> prefix = Prefix("/usr")
>>> prefix.bin
/usr/bin
>>> prefix.lib64
/usr/lib64
>>> prefix.share.man
/usr/share/man
>>> prefix.foo.bar.baz
/usr/foo/bar/baz
>>> prefix.join("dashed-directory").bin64
/usr/dashed-directory/bin64Prefix objects behave identically to strings โ in fact they subclass str, so operators like + are totally legal:
print("foobar " + prefix)That prints foobar /usr ๐. All of this is meant to make custom installs easy.
โญ join(string: str) -> Prefix
Concatenate a string to a prefix.
Useful for strings that are not valid variable names โ which includes strings containing characters like - and . ๐.
Parameters
stringโ the string to append to the prefix
Returns
the newly created installation prefix
๐ spack.util.s3 module ๐
๐บ class spack.util.s3.UrllibS3Handler
Bases: BaseHandler
s3_open(req)
๐ธ class spack.util.s3.WrapStream(raw)
Bases: BufferedReader
๐ฆ detach()
Disconnect this buffer from its underlying raw stream and return it.
After the raw stream has been detached, the buffer is in an unusable state โ ๏ธ.
๐ read(*args, **kwargs)
Read and return up to n bytes.
If the size argument is omitted, None, or negative, read and return all data until EOF.
If the size argument is positive and the underlying raw stream is not interactive, multiple raw reads may be issued to satisfy the byte count (unless EOF is reached first). However, for interactive raw streams (as well as sockets and pipes), at most one raw read will be issued, and a short result does not imply that EOF is imminent ๐.
Return an empty bytes object on EOF.
Return None if the underlying raw stream was open in non-blocking mode and no data is available at the moment.
๐ฅ spack.util.s3.get_mirror_s3_connection_info(mirror, method)
Create s3 config for session/client from a Mirror instance (or just set defaults when no mirror is given).
โญ spack.util.s3.get_s3_session(url, method='fetch')
๐ซ spack.util.s3.s3_client_cache: Dict[Tuple[str, str], Any] = {}
Maps (mirror name, method) tuples to s3 client instances ๐๏ธ.
๐ง spack.util.spack_json module ๐ง
A simple wrapper around JSON to guarantee consistent use of load/dump.
๐ exception spack.util.spack_json.SpackJSONError(msg: str, json_error: BaseException)
Bases: SpackError
Raised when there are issues with JSON parsing ๐ญ.
๐ spack.util.spack_json.dump(data: Dict, stream: Any | None = None) -> str | None
Dump JSON with a reasonable amount of indentation and separation.
๐ฆ spack.util.spack_json.load(stream: Any) -> Dict
Spack JSON needs to be ordered to support specs ๐.
๐ spack.util.spack_yaml module ๐
Enhanced YAML parsing for Spack.
load()preserves YAML Marks on returned objects โ this lets us access file and line information later.- Our load methods use the
OrderedDictclass instead of YAML's default unordered dict.
๐ exception spack.util.spack_yaml.SpackYAMLError(msg, yaml_error)
Bases: SpackError
Raised when there are issues with YAML parsing ๐ญ.
โจ spack.util.spack_yaml.dump(data, stream=None, default_flow_style=False)
๐ธ spack.util.spack_yaml.load(*args, **kwargs)
๐ฅ spack.util.timer module ๐ฅ
Debug signal handler: prints a stack trace and enters the interpreter.
register_interrupt_handler() enables a Ctrl+C handler that prints a stack trace and drops the user into an interpreter ๐.
๐ class spack.util.timer.BaseTimer
Bases: object
duration(name=None)
measure(name)
property phases
start(name=None)
stop(name=None)
write_json(out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)
write_tty(out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)
๐ spack.util.timer.NULL_TIMER = <spack.util.timer.NullTimer object>
An instance of a do-nothing timer ๐ซถ.
โญ class spack.util.timer.NullTimer
Bases: BaseTimer
A timer interface that does nothing โ useful for "tell don't ask" style code when timers are optional.
๐ class spack.util.timer.TimeTracker(total, start, count, path)
Bases: tuple
count โ Alias for field number 2
path โ Alias for field number 3
start โ Alias for field number 1
total โ Alias for field number 0
๐ class spack.util.timer.Timer(now: ~typing.Callable[[], float] = <built-in function time>)
Bases: BaseTimer
A simple interval timer โฑ๏ธ.
๐ duration(name='_global')
Get the time in seconds of a named timer, or the total time if no name is passed. The duration is always 0 for timers that have not been started โ no error is raised, so chill ๐.
Parameters
name(str) โ (Optional) name of the timer
Returns
duration of timer.
Return type
float
๐ฆ measure(name)
Context manager that lets you time a block of code โจ.
Parameters
name(str) โ name of the timer
๐ธ property phases
Get all named timers (excluding the global/total timer).
๐ start(name='_global')
Start or restart a named timer, or the global timer when no name is given.
Parameters
name(str) โ optional name of the timer. When no name is passed, the global timer is started.
๐ง stop(name='_global')
Stop a named timer, or all timers when no name is given. Stopping a timer that hasn't started has no effect, so no drama ๐ .
Parameters
name(str) โ optional name of the timer. When no name is passed, all timers are stopped.
โญ write_json(out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, extra_attributes={})
Write a json object with times to file.
๐ฅ write_tty(out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)
Write a human-readable summary of timings (depth is 1).
๐ซ class spack.util.timer.TimerEvent(time, running, label)
Bases: tuple
label โ Alias for field number 2
running โ Alias for field number 1
time โ Alias for field number 0
spack.util.timer.global_timer_name = '_global'โ the name of the global timer, bestie. It's the one thatstart(),stop(), andduration()grab when you call them with no arguments at all. No name needed, it already knows. ๐
๐ธ spack.util.url module ๐ธ
Utility functions for parsing, formatting, and manipulating URLs. Basically the whole "links are hard" department, and honestly? They ate.
spack.util.url.default_download_filename(url: str) -> strโ computes a default file name for a given URL. Heads up though: it makes no request, so this is not the same ascurl -O, which pulls the remote file name out of the response header. Different vibe entirely, iykyk.spack.util.url.file_url_string_to_path(url)โ turns a file URL string into a path. That's it, that's the whole thing. ๐spack.util.url.format(parsed_url)โ formats a URL string. Returns a canonicalized version of the URL you handed it, as a string. Glow-up for your URL. โจspack.util.url.is_path_instead_of_url(path_or_url)โ so historically some config files and spack commands used paths where urls should've gone (yes, messy, we know ๐ญ). This utility validates and promotes paths to urls so you're not out here living a lie.spack.util.url.join(base_url, path, *extra, **kwargs)โ joins a base URL with one or more local URL path components. This one's the main character, so listen up ๐
If resolve_href is True, treat the base URL like it's the locator of a web page, and the remaining URL path components like they formed a relative URL to be resolved against it (i.e. as in posixpath.join(...)). The result is an absolute URL to the resource a user's browser would navigate to if they clicked a link with an "href" attribute equal to the relative URL.
If resolve_href is False (the default), then the URL path components are joined as in posixpath.join().
Note:file://URL path components are not canonicalized as part of this operation. If you want that, pass the joined url toformat().
Examples โ literally just look at them, they explain it better than I ever could:
base_url = 's3://bucket/index.html' body = fetch_body(prefix)
link = get_href(body) # link == '../other-bucket/document.txt'
# wrong - link is a local URL that needs to be resolved against
base_url spack.util.url.join(base_url, link)
's3://bucket/other_bucket/document.txt'
# correct - resolve local URL against base_url
spack.util.url.join(base_url, link, resolve_href=True)
's3://other_bucket/document.txt'
prefix = 'https://mirror.spack.io/build_cache'
# wrong - prefix is just a URL prefix spack.util.url.join(prefix,
'my-package', resolve_href=True) 'โ
https://mirror.spack.io/my-package'
# correct - simply append additional URL path components
spack.util.url.join(prefix, 'my-package', resolve_href=False) # default
'https://mirror.spack.io/build_cache/my-package'
# For canonicalizing file:// URLs, take care to explicitly differentiate
# between absolute and relative join components.See how the "wrong" ones are wrong? Yeah. Don't be wrong. ๐ซถ
spack.util.url.local_file_path(url)โ gets a local file path from a url. Ifurlis afile://URL, it returns the absolute path to the local file or directory it's referencing. Otherwise, it returnsNone. Simple as. ๐spack.util.url.parse_link_rel_next(link_value: str) -> str | Noneโ returns the next link from aLinkheader value, if there is one. If not,None. That's the whole function, no notes.spack.util.url.path_to_file_url(path)โ converts a path into a file URL. The reverse of the other one. Iconic symmetry honestly. ๐spack.util.url.validate_scheme(scheme)โ returns true if the URL scheme is generally known to Spack. It's mostly here to help tell paths and urls apart, because Windows paths likeC:/x/y/z(with a backward not forward slash) may totally parse as a URL with schemeCand path/x/y/z. Sneaky little guy. ๐
๐ฆ spack.util.web module ๐ฆ
The web-spidering department. All the fetching, all the parsing, all the vibes.
exception spack.util.web.DetailedHTTPError(req: Request, code: int, msg: str, hdrs: Message, fp: IO | None)โ Bases:HTTPError. It's an HTTP error but, like, with details. Main character error energy.class spack.util.web.ExtractMetadataParserโ Bases:HTMLParser. This parser takes an HTML page and selects the include-fragments (used on GitHub, โhttps://github.github.io/include-fragment-element), as well as a possible base url. Very niche, very slay.handle_starttag(tag, attrs)โ handles a start tag. Yes, that's the description. It's a handler. It handles. ๐
exception spack.util.web.HTMLParseErrorโ Bases:Exception. Raised when HTML parsing goes sideways, as HTML tends to do.class spack.util.web.LinkParserโ Bases:HTMLParser. This parser just takes an HTML page and strips out thehrefs on the links. Good enough for a really simple spider โ no cap, "good enough" is literally the spec here. ๐ท๏ธhandle_starttag(tag, attrs)โ again, it handles the start tag. Consistent queen.
exception spack.util.web.NoNetworkConnectionError(message, url)โ Bases:SpackWebError. Raised when an operation can't get an internet connection. Us when the wifi drops mid-spack install. ๐ญspack.util.web.SPACK_USER_AGENT = 'Spackbot/0.22.1'โ the User-Agent used inRequestobjects. That's the name Spack goes by on the internet. ๐class spack.util.web.SpackHTTPDefaultErrorHandlerโ Bases:HTTPDefaultErrorHandler.http_error_default(req, fp, code, msg, hdrs)โ the default http error handler. Handles the default http errors. You get it. ๐
exception spack.util.web.SpackWebError(message, long_message=None)โ Bases:SpackError. The superclass for Spack web spidering errors. All the other web errors look up to it.spack.util.web.base_curl_fetch_args(url, timeout=0)โ returns the basic fetch arguments typically used in calls tocurl. These args include the ones that make sure stuff behaves: failing on errors for codes over400, printing HTML headers, resolving3xxredirects, status or failure handling, and connection timeouts.
It also uses these configuration options to set an additional argument as needed:
config:connect_timeout(int): connection timeoutconfig:verify_ssl(str): Perform SSL verification
Parameters
url(str) โ URL whose contents will be fetchedtimeout(int) โ Connection timeout, which is only used if higher thanconfig:connect_timeout
Returns (list): list of argument strings. A whole list of them. โจ
spack.util.web.check_curl_code(returncode)โ checks standard return code failures for the provided arguments.
Parameters
returncode(int) โ curl return code
Raises FetchError if the curl returncode indicates failure. So like, if it's bad. ๐ฅ
spack.util.web.custom_ssl_certs() -> Tuple[bool, str] | Noneโ returns a tuple(is_file, path)if custom SSL certificates are configured and valid. Otherwise you get nothing, bestie.spack.util.web.fetch_url_text(url, curl=None, dest_dir='.')โ retrieves text-only URL content using the configured fetch method. It figures out the fetch method from:config:url_fetch_method(str): fetch method to use (e.g.,'curl')
If the method is curl, it also uses these configuration options:
config:connect_timeout(int): connection time outconfig:verify_ssl(str): Perform SSL verification
Parameters
url(str) โ URL whose contents are to be fetchedcurl(spack.util.executable.ExecutableorNone) โ (optional) curl executable if curl is the configured fetch methoddest_dir(str) โ (optional) destination directory for fetched text file, defaults to'.'
Returns (str or None): path to the fetched file
Raises FetchError if the curl returncode indicates failure
spack.util.web.get_header(headers, header_name)โ looks up a dict of headers for the given header value. Specifically, it looks up a dict of headers,[headers], for a header value given by[header_name]. It returnsheaders[header_name]ifheader_nameis inheaders. Otherwise, the first fuzzy match is returned, if any.
This fuzzy matching works by discarding word separators and capitalization, so for example "Content-length", "content_length", "conTENtLength", etc., all match. Slay, case-insensitivity. ๐
In the case of multiple fuzzy-matches, the returned value is the "first" such match given the underlying mapping's ordering โ or unspecified if no such ordering is defined. So like, don't count on it being predictable, bestie.
If header_name is not in headers, and no such fuzzy match exists, then a KeyError is raised. Rude but fair. ๐
spack.util.web.list_url(url, recursive=False)โ lists what's at a url.recursivedefaults toFalse, so it's not gonna go digging unless you tell it to.spack.util.web.parse_etag(header_value)โ parses a strong etag from anETag: <value>header value. Ngl, we don't allow for weakness indicators because it's unclear what that even means for cache invalidation. Weak etags, we don't know her. ๐spack.util.web.push_to_url(local_file_path, remote_path, keep_original=True, extra_args=None)โ pushes a local file up to a remote path.keep_originaldefaults toTrue, so your local copy stays put unless you say otherwise. ๐ซถspack.util.web.read_from_url(url, accept_content_type=None)โ reads from a url. You can pin anaccept_content_typeif you're picky about what comes back.spack.util.web.remove_url(url, recursive=False)โ removes a url. AgainrecursiveisFalseby default, because we're not out here deleting the whole world by accident. ๐ญspack.util.web.set_curl_env_for_ssl_certs(curl: Executable) -> Noneโ configures curl to use custom certs in a file at runtime. See: โhttps://curl.se/docs/sslcerts.htmlitem4. Go read item 4, it's important. ๐spack.util.web.spider(root_urls: str | Iterable[str], depth: int = 0, concurrency: int | None = None)โ gets web pages from root URLs. The actual crawling. The main event. ๐ธ๏ธ
If depth is specified (e.g., depth=2), then this will also follow up to <depth> levels of links from each root. Deeper = more links = more chaos.
Parameters
root_urlsโ root urls used as a starting point for spideringdepthโ level of recursion into linksconcurrencyโ number of simultaneous requests that can be sent
Returns โ a dict of pages visited (URL) mapped to their full text, and the set of visited links. Two things. It gives you two things. โจ
spack.util.web.ssl_create_default_context()โ creates the default SSL context forurllibwith custom certificates if configured. Safe and secure and iconic. ๐spack.util.web.url_exists(url, curl=None)โ determines whether a url exists. A scheme-specific process is used for Google Storage (gs) and Amazon Simple Storage Service (s3) URLs; otherwise, the configured fetch method defined byconfig:url_fetch_methodis used.
Parameters
url(str) โ URL whose existence is being checkedcurl(spack.util.executable.ExecutableorNone) โ (optional) curl executable if curl is the configured fetch method
Returns (bool): True if it exists; False otherwise. Binary. Decisive. ๐
spack.util.web.urlopen = <function _urlopen.<locals>.dispatch_open>โ dispatches to the correctOpenerDirector.open, based on Spack configuration. It's a dispatcher, it dispatches, that's the assignment. ๐
๐ง spack.util.windows_registry module ๐ง
Utility module for dealing with the Windows Registry. Yes, Spack goes there too.
class spack.util.windows_registry.HKEYโ Bases:object. It's the registry key class, holding it down for the Windows girlies. ๐ช
okay so these next ones are the predefined, open registry HKEYs ๐ and honestly the Microsoft docs said it best, so here's the quote:
An application must open a key before it can read data from the registry. To open a key, an application must supply a handle to another key in the registry that is already open. The system defines predefined keys that are always open. Predefined keys help an application navigate in the registry.
and here's the whole iconic lineup, bestie:
HKEY_CLASSES_ROOT = <spack.util.windows_registry._HKEY_CONSTANT object>HKEY_CURRENT_CONFIG = <spack.util.windows_registry._HKEY_CONSTANT object>HKEY_CURRENT_USER = <spack.util.windows_registry._HKEY_CONSTANT object>HKEY_LOCAL_MACHINE = <spack.util.windows_registry._HKEY_CONSTANT object>HKEY_PERFORMANCE_DATA = <spack.util.windows_registry._HKEY_CONSTANT object>HKEY_USERS = <spack.util.windows_registry._HKEY_CONSTANT object>
๐ exception spack.util.windows_registry.InvalidKeyError(key)
Bases: RegistryError
It's the Runtime Error that pops off when someone tries to access an invalid key in the Windows registry. invalid key access = not the move ๐ญ
๐ซ exception spack.util.windows_registry.InvalidRegistryOperation(name, e, *args, **kwargs)
Bases: RegistryError
A Runtime Error you bump into when a registry operation is invalid for, like, an indeterminate reason. No further explanation, she just says no ๐
๐จ exception spack.util.windows_registry.RegistryError
Bases: Exception
A RunTime Error concerning the Windows Registry. That's the whole description, and honestly? Slay, concise queen.
๐ class spack.util.windows_registry.RegistryKey(name, handle)
Bases: object
It's the class that wraps a Windows registry key ๐ so comfy, so cozy.
EnumKey(index)โ convenience wrapper aroundwinreg.EnumKeyEnumValue(index)โ convenience wrapper aroundwinreg.EnumValueOpenKeyEx(subname, **kwargs)โ convenience wrapper aroundwinreg.OpenKeyExQueryInfoKey()โ convenience wrapper aroundwinreg.QueryInfoKeyQueryValueEx(name, **kwargs)โ convenience wrapper aroundwinreg.QueryValueExget_subkey(sub_key)โ returns the subkey of namesub_keyin aRegistryKeyobjectget_value(val_name)โ returns the value associated with this key in aRegistryValueobjectproperty hkeyโ thehkeyproperty, no notes, she's just there ๐ซถproperty subkeysโ returns a list of all subkeys of this key asRegistryKeyobjectsproperty valuesโ returns all subvalues of this key asRegistryValueobjects, in a dictionary of value name :RegistryValueobjectwinreg_error_handler(name, *args, **kwargs)โ the error handler, takesname,*args,**kwargs. No description given, lowkey mysterious ๐ต๏ธโโ๏ธ
๐ธ class spack.util.windows_registry.RegistryValue(name, value, parent_key)
Bases: object
The class defining a Windows registry entry ๐ literally the definition of the thing, no cap.
๐ฆ class spack.util.windows_registry.WindowsRegistryView(key, root_key=<spack.util.windows_registry._HKEY_CONSTANT object>)
Bases: object
Okay so this is the whole interface for access, querying and searching of Windows registry entries ๐. This class represents a single key entrypoint into the Windows registry and gives you an interface to this key's values, its subkeys, and those subkeys' values. Important vibe check: this class cannot be used to move freely about the registry โ only subkeys/values of the root key you used to instantiate it. She's loyal to her root key, periodt ๐
๐ class KeyMatchConditions
Bases: object
static name_matcher(subkey_name)โ name matcher, static, does what it says ๐static regex_matcher(subkey_name)โ regex matcher, static, also does what it says ๐
๐ find_matching_subkey(subkey_name, recursive=True)
Performs a BFS of subkeys until a key matching the subkey name regex is found. Returns None or the first RegistryKey object corresponding to the requested key name.
Parameters:
subkey_name (str)โ subkey to be searched forrecursive (bool)โ perform a recursive search
Returns: the desired subkey as a RegistryKey object, or none
๐ find_subkey(subkey_name, recursive=True)
Performs a BFS of subkeys until the desired key is found. Returns None or a RegistryKey object corresponding to the requested key name.
Parameters:
subkey_name (str)โ subkey to be searched forrecursive (bool)โ perform a recursive search
Returns: the desired subkey as a RegistryKey object, or none
๐ find_subkeys(subkey_name, recursive=True)
Exactly the same as find_subkey, except this function tries to match a regex to multiple keys ๐ฏโโ๏ธ she's a multitasker.
Parameters:
subkey_name (str)
Returns: the desired subkeys as a list of RegistryKey objects, or none
๐ find_value(val_name, recursive=True)
If non-recursive, returns the RegistryValue object corresponding to the name.
Parameters:
val_name (str)โ name of value desired from registryrecursive (bool)โ optional argument; ifTrue, the registry is searched recursively for the value of nameval_name, else only the current key is searched
Returns: the desired registry value as a RegistryValue object if it exists, otherwise, None
๐ง get_matching_subkeys(subkey_name)
Returns all subkeys regex-matching the subkey name.
Note: this method obtains only direct subkeys of the given key and does not descend to transitive subkeys. For this behavior, see find_matching_subkeys.
โจ get_subkey(subkey_name)
Gets the subkey. That's it. That's the whole method ๐ซถ
โจ get_subkeys()
Gets the subkeys. Plural. Iconic.
โจ get_value(value_name)
Returns the registry value corresponding to the provided argument (if it exists) ๐
โจ get_values()
Gets the values. All of them, presumably. She's not elaborating ๐
๐ฉน invalid_reg_ref_error_handler()
The error handler for invalid registry refs. No parameters, no description, just vibes ๐
๐ท๏ธ property reg
The reg property. She's a property. Moving on ๐
๐ spack.version package ๐
This module implements Version and version-ish objects. These are:
StandardVersionโ a single version of a packageClosedOpenRangeโ a range of versions of a packageVersionListโ an ordered list ofVersionandVersionRangeelements
The set of Version and ClosedOpenRange is totally ordered, with < defined as Version(x) < VersionRange(Version(y), Version(x)) if Version(x) <= Version(y). math queen behavior, honestly ๐งฎ๐
๐ฏ class spack.version.ClosedOpenRange(lo: StandardVersion, hi: StandardVersion)
Bases: object
classmethod from_version_range(lo: StandardVersion, hi: StandardVersion)โ constructs aClosedOpenRangefrom thelo:hirangeintersection(other: ClosedOpenRange | ConcreteVersion)โ intersection time ๐ซถintersects(other: ConcreteVersion | ClosedOpenRange | VersionList)โ do they intersect? Find out โจoverlaps(other: ClosedOpenRange | ConcreteVersion | VersionList) -> boolโ returns abool, returns the truth, returns the overlap ๐satisfies(other: ClosedOpenRange | ConcreteVersion | VersionList)โ does it satisfy the other thing? Yes or no bestieunion(other: ClosedOpenRange | ConcreteVersion | VersionList)โ union, obviously, we love a merge ๐
๐ exception spack.version.EmptyRangeError(message, long_message=None)
Bases: VersionError
Raised when constructing an empty version range. Empty range = error, no cap. ๐ญ
๐ง class spack.version.GitVersion(string: str)
Bases: ConcreteVersion
A class to represent versions interpreted from git refs ๐ฟโจ
There are two distinct categories of git versions:
GitVersions instantiated with an associated reference version (e.g.'git.foo=1.2')GitVersions requiring commit lookups
Git ref versions that are not paired with a known version are handled separately from all other version comparisons. When Spack identifies a git ref version, it associates a CommitLookup object with the version. This object handles caching of information from the git repo ๐พ. When executing comparisons with a git ref version, Spack queries the CommitLookup for the most recent version previous to this git ref, as well as the distance between them expressed as a number of commits. If the previous version is X.Y.Z and the distance is D, the git commit version is represented by the tuple (X, Y, Z, '', D). The component '' cannot be parsed as part of any valid version, but it is a valid component ๐ฅ. This allows a git ref version to be less than (older than) every Version newer than its previous version, but still newer than its previous version. it's giving "impostor but make it ordered" ๐
To find the previous version from a git ref version, Spack queries the git repo for its tags ๐ท๏ธ. Any tag that matches a version known to Spack is associated with that version, as is any tag that is a known version prepended with the character v (i.e., a tag v1.0 is associated with the known version 1.0). Additionally, any tag that represents a semver version (X.Y.Z with X, Y, Z all integers) is associated with the version it represents, even if that version is not known to Spack. Each tag is then queried in git to see whether it is an ancestor of the git ref in question, and if so, the distance between the two. The previous version is the version that is an ancestor with the least distance from the git ref in question ๐งโโ๏ธ
This procedure can be circumvented if the user supplies a known version to associate with the GitVersion (e.g. [hash]=develop). If the user prescribes the version, then there's no need to do a lookup, and the standard version comparison operations are sufficient โจ
๐ซถ attach_lookup(lookup: AbstractRefLookup)
Use the git fetcher to look up a version for a commit.
Since we want to optimize the clone and lookup, we do the clone once and store it in the user-specified git repository cache ๐พ. We also need context of the package to get known versions, which could be tags if they are linked to Git Releases. If we are unable to determine the context of the version, we cannot continue โ sorry bestie ๐ญ. This implementation is alongside the GitFetcher because eventually the git repos cache will be one and the same with the source cache.
๐ท๏ธ property dashed: StandardVersion
The dashed property, typed StandardVersion. She's dashed. She's a property. โจ
๐ท๏ธ property dotted: StandardVersion
The dotted property, typed StandardVersion. Dotted, as opposed to dashed. We love a contrast ๐
โจ has_git_prefix
Checks for a git prefix. That's the whole vibe ๐
โจ intersection(other)
Intersection, but for git versions. Takes other ๐ธ
โจ intersects(other)
Do they intersect? Takes other, returns the truth ๐ซถ
โจ is_commit: bool
A bool attribute โ is it a commit or not? Iconic question, honestly ๐
โจ is_prerelease() -> bool
Returns a bool telling you whether it's a prerelease ๐ฅ
โจ isdevelop()
Checks if it's develop. No parameters, just the develop check ๐
still mid-class, bestie โ same list, same energy, no line left behind ๐
- property
joined:StandardVersion overlaps(other) -> boolref- property
ref_lookup - property
ref_version:StandardVersion satisfies(other: GitVersion | StandardVersion | ClosedOpenRange | VersionList)- property
underscored:StandardVersion up_to(index) -> StandardVersion
- class
spack.version.StandardVersion(string: str | None, version: Tuple[tuple, tuple], separators: tuple)โ Bases:ConcreteVersion. The class to represent versions, periodt ๐property dashedโ The dashed representation of the version.
Example:
>>> version = Version('1.2.3b')
>>> version.dashed
Version('1-2-3b')Returns: The version with separator characters replaced by dashes.
Return type: Version.
property dottedโ The dotted representation of the version.
Example:
>>> version = Version('1-2-3b')
>>> version.dotted
Version('1.2.3b')Returns: The version with separator characters replaced by dots.
Return type: Version.
property dotted_numeric_string: strโ Replaces all non-numeric components of the version with0. This can be used to pass Spack versions to libraries that have stricter version schema. ๐ซถ- static
from_string(string: str) intersection(other: ClosedOpenRange | StandardVersion)intersects(other: StandardVersion | GitVersion | ClosedOpenRange) -> boolis_prerelease() -> boolisdevelop()โ Triggers on the special case of the@develop-like version. ๐property joinedโ The joined representation of the version.
Example:
>>> version = Version('1.2.3b')
>>> version.joined
Version('123b')Returns: The version with separator characters removed.
Return type: Version.
overlaps(other) -> boolsatisfies(other: ClosedOpenRange | StandardVersion | GitVersion | VersionList) -> boolseparatorsstring- static
typemax() - static
typemin() property underscoredโ The underscored representation of the version.
Example:
>>> version = Version('1.2.3b')
>>> version.underscored
Version('1_2_3b')Returns: The version with separator characters replaced by underscores.
Return type: Version.
union(other: ClosedOpenRange | StandardVersion)up_to(index)โ The version up to the specified component.
Examples:
>>> version = Version('1.23-4b')
>>> version.up_to(1)
Version('1')
>>> version.up_to(2)
Version('1.23')
>>> version.up_to(3)
Version('1.23-4')
>>> version.up_to(4)
Version('1.23-4b')
>>> version.up_to(-1)
Version('1.23-4')
>>> version.up_to(-2)
Version('1.23')
>>> version.up_to(-3)
Version('1')Returns: The first index components of the version.
Return type: Version.
version
spack.version.Version(string: str | int) -> GitVersion | StandardVersion- exception
spack.version.VersionChecksumError(message, long_message=None)โ Bases:VersionError. Raised for version checksum errors. ๐ญ - exception
spack.version.VersionError(message, long_message=None)โ Bases:SpackError. This is raised when something is wrong with a version. fr fr - class
spack.version.VersionList(vlist=None)โ Bases:object. Sorted, non-redundant list ofVersionandClosedOpenRangeelements. โจadd(item: StandardVersion | GitVersion | ClosedOpenRange | VersionList)property concrete: ConcreteVersion | Noneproperty concrete_range_as_version: ConcreteVersion | Noneโ Likeconcrete, but collapsesVersionRange(x, x)toVersion(x). This is just for compatibility with old Spack. ๐ซถcopy()- static
from_dict(dictionary)โ Parse dict fromto_dict. highest() -> StandardVersion | Noneโ Get the highest version in the list.highest_numeric() -> StandardVersion | Noneโ Get the highest numeric version in the list.intersect(other) -> boolโ Intersect this spec's list with other. ReturnTrueif the spec changed as a result;Falseotherwise.intersection(other: VersionList) -> VersionListintersects(other)lowest() -> StandardVersion | Noneโ Get the lowest version in the list.overlaps(other) -> boolpreferred() -> StandardVersion | Noneโ Get the preferred (latest) version in the list.satisfies(other) -> boolto_dict()โ Generate human-readable dict for YAML.union(other: VersionList)update(other: VersionList)
- exception
spack.version.VersionLookupError(message, long_message=None)โ Bases:VersionError. Raised for errors looking up git commits as versions. ๐ฅ spack.version.VersionRange(lo: str | StandardVersion, hi: str | StandardVersion)spack.version.any_version: VersionList = [:]โ This version contains all possible versions. ๐spack.version.from_string(string) -> VersionList | ClosedOpenRange | StandardVersion | GitVersionโ Converts a string to a version object. This is private. Client code should usever(). iykykspack.version.is_git_version(string: str) -> boolspack.version.ver(obj) -> VersionList | ClosedOpenRange | StandardVersion | GitVersionโ Parses aVersion,VersionRange, orVersionListfrom a string or list of strings.
โจ Submodules โจ
๐ง spack.version.common module
- exception
spack.version.common.EmptyRangeError(message, long_message=None)โ Bases:VersionError. Raised when constructing an empty version range. ๐ - exception
spack.version.common.VersionChecksumError(message, long_message=None)โ Bases:VersionError. Raised for version checksum errors. - exception
spack.version.common.VersionError(message, long_message=None)โ Bases:SpackError. This is raised when something is wrong with a version. - exception
spack.version.common.VersionLookupError(message, long_message=None)โ Bases:VersionError. Raised for errors looking up git commits as versions. spack.version.common.is_git_version(string: str) -> bool
๐ฆ spack.version.git_ref_lookup module
- class
spack.version.git_ref_lookup.GitRefLookup(pkg_name)โ Bases:AbstractRefLookup. An object for cached lookups of git refs. ๐
GitRefLookup objects delegate to the MISC_CACHE for locking. GitRefLookup objects may be attached to a GitVersion to allow for comparisons between git refs and versions as represented by tags in the git repository.
property cache_keyproperty cache_pathproperty fetcherget(ref) -> Tuple[str | None, int]โ Get the version string and distance for a given git ref.
Parameters: ref (str) -- git ref to lookup
Returns: optional version string and distance
load_data()โ Load data if the path already exists.lookup_ref(ref) -> Tuple[str | None, int]โ Lookup the previous version and distance for a given commit.
We use git to compare the known versions from package to the git tags, as well as any git tags that are SEMVER versions, and find the latest known version prior to the commit, as well as the distance from that version to the commit in the git repo. Those values are used to compare Version objects. ๐ธ
property pkgproperty repository_uriโ Identifier for git repos used within the repo and metadata caches.save()โ Save the data to file
๐ spack.version.lookup module
- class
spack.version.lookup.AbstractRefLookupโ Bases:object.get(ref) -> Tuple[str | None, int]โ Get the version string and distance for a given git ref.
Parameters: ref (str) -- git ref to lookup
Returns: optional version string and distance
โญ spack.version.version_types module
- class
spack.version.version_types.ClosedOpenRange(lo: StandardVersion, hi: StandardVersion)โ Bases:object.- classmethod
from_version_range(lo: StandardVersion, hi: StandardVersion)โ ConstructClosedOpenRangefromlo:hirange. intersection(other: ClosedOpenRange | ConcreteVersion)intersects(other: ConcreteVersion | ClosedOpenRange | VersionList)overlaps(other: ClosedOpenRange | ConcreteVersion | VersionList) -> boolsatisfies(other: ClosedOpenRange | ConcreteVersion | VersionList)union(other: ClosedOpenRange | ConcreteVersion | VersionList)
- classmethod
- class
spack.version.version_types.ConcreteVersionโ Bases:object. - class
spack.version.version_types.GitVersion(string: str)โ Bases:ConcreteVersion. Class to represent versions interpreted from git refs. ๐ฆ
There are two distinct categories of git versions:
GitVersionsinstantiated with an associated reference version (e.g.'git.foo=1.2')GitVersionsrequiring commit lookups
Okay so git ref versions that aren't paired with a known version? They get handled totally separately from every other version comparison. No cap. When Spack spots a git ref version, it associates a CommitLookup object with it โ think of it as that version's own little assistant that handles caching of info from the git repo. ๐ซถ When Spack runs a comparison against a git ref version, it asks the CommitLookup for the most recent version previous to this git ref, plus the distance between them expressed as a number of commits. If the previous version is X.Y.Z and the distance is D, then the git commit version is represented by the tuple (X, Y, Z, '', D). That '' component absolutely cannot be parsed as part of any valid version โ but it is a valid component. Iconic loophole energy honestly. โจ This is what lets a git ref version be less than (older than) every Version newer than its previous version, while still being newer than its previous version.
So how does Spack find that previous version? It queries the git repo for its tags. ๐
Any tag that matches a version known to Spack gets associated with that version, and so does any tag that's a known version prepended with the character v (i.e., a tag v1.0 is associated with the known version 1.0). On top of that, any tag that represents a semver version (X.Y.Z with X, Y, Z all integers) is associated with the version it represents, even if Spack doesn't know that version. Then each tag gets queried in git to see whether it's an ancestor of the git ref in question, and if it is, the distance between the two. The previous version is whichever one is an ancestor with the least distance from the git ref in question. Closest ancestor wins, periodt. ๐
This whole procedure can be circumvented if the user supplies a known version to associate with the GitVersion (e.g. [hash]=develop). If the user prescribes the version, there's no need to do a lookup at all, and the standard version comparison operations are sufficient. Slay, less work for everyone. ๐
attach_lookup(lookup: AbstractRefLookup) โ use the git fetcher to look up a version for a commit. Since we want to optimize the clone and lookup, the clone happens once and gets stored in the user-specified git repository cache. We also need the context of the package to get the known versions โ which could be tags if they're linked to Git Releases. If we can't determine the context of the version, we cannot continue. ๐ญ This implementation lives alongside the GitFetcher because eventually the git repos cache will be one and the same with the source cache.
Also chillin' on GitVersion, no further notes needed:
property dashed: StandardVersionproperty dotted: StandardVersionhas_git_prefixintersection(other)intersects(other)is_commit: boolis_prerelease() -> boolisdevelop()property joined: StandardVersionoverlaps(other) -> boolrefproperty ref_lookupproperty ref_version: StandardVersionsatisfies(other: GitVersion | StandardVersion | ClosedOpenRange | VersionList)property underscored: StandardVersionup_to(index) -> StandardVersion
class spack.version.version_types.StandardVersion(string: str | None, version: Tuple[tuple, tuple], separators: tuple)
Bases: ConcreteVersion โ the class to represent versions. ๐ฆ
property dashedโ the dashed representation of the version. Like, check it:
>>> version = Version('1.2.3b')
>>> version.dashed
Version('1-2-3b')Returns the version with separator characters replaced by dashes. Return type: Version.
property dottedโ the dotted representation of the version. Behold:
>>> version = Version('1-2-3b')
>>> version.dotted
Version('1.2.3b')Returns the version with separator characters replaced by dots. Return type: Version.
property dotted_numeric_string: strโ replaces all non-numeric components of the version with0. This is the move when you need to pass Spack versions to libraries that have stricter version schemas. ๐ซถstatic from_string(string: str)intersection(other: ClosedOpenRange | StandardVersion)intersects(other: StandardVersion | GitVersion | ClosedOpenRange) -> boolis_prerelease() -> boolisdevelop()โ triggers on the special case of the@develop-like version.property joinedโ the joined representation of the version. Watch:
>>> version = Version('1.2.3b')
>>> version.joined
Version('123b')Returns the version with separator characters removed. Return type: Version.
overlaps(other) -> boolsatisfies(other: ClosedOpenRange | StandardVersion | GitVersion | VersionList) -> boolseparatorsstringstatic typemax()static typemin()property underscoredโ the underscored representation of the version:
>>> version = Version('1.2.3b')
>>> version.underscored
Version('1_2_3b')Returns the version with separator characters replaced by underscores. Return type: Version.
union(other: ClosedOpenRange | StandardVersion)up_to(index)โ the version up to the specified component. Lowkey the most useful thing ever:
>>> version = Version('1.23-4b')
>>> version.up_to(1)
Version('1')
>>> version.up_to(2)
Version('1.23')
>>> version.up_to(3)
Version('1.23-4')
>>> version.up_to(4)
Version('1.23-4b')
>>> version.up_to(-1)
Version('1.23-4')
>>> version.up_to(-2)
Version('1.23')
>>> version.up_to(-3)
Version('1')Returns the first index components of the version. Return type: Version.
version
spack.version.version_types.Version(string: str | int) -> GitVersion | StandardVersion
class spack.version.version_types.VersionList(vlist=None)
Bases: object โ a sorted, non-redundant list of Version and ClosedOpenRange elements. Tidy queen behavior. ๐
add(item: StandardVersion | GitVersion | ClosedOpenRange | VersionList)property concrete: ConcreteVersion | Noneproperty concrete_range_as_version: ConcreteVersion | Noneโ likeconcrete, but collapsesVersionRange(x, x)toVersion(x). This is just for compatibility with old Spack. ๐copy()static from_dict(dictionary)โ parse a dict fromto_dict.highest() -> StandardVersion | Noneโ get the highest version in the list.highest_numeric() -> StandardVersion | Noneโ get the highest numeric version in the list.intersect(other) -> boolโ intersect this spec's list withother. ReturnsTrueif the spec changed as a result;Falseotherwise.intersection(other: VersionList) -> VersionListintersects(other)lowest() -> StandardVersion | Noneโ get the lowest version in the list.overlaps(other) -> boolpreferred() -> StandardVersion | Noneโ get the preferred (latest) version in the list. Main character version, honestly. ๐satisfies(other) -> boolto_dict()โ generate a human-readable dict for YAML.union(other: VersionList)update(other: VersionList)
spack.version.version_types.VersionRange(lo: str | StandardVersion, hi: str | StandardVersion)
class spack.version.version_types.VersionStrComponent(data)
Bases: object
data: int | strstatic from_string(string)
spack.version.version_types.from_string(string) -> VersionList | ClosedOpenRange | StandardVersion | GitVersion
Converts a string to a version object. This one is private โ client code should use ver(). Don't @ me. ๐
spack.version.version_types.parse_string_components(string: str) -> Tuple[tuple, tuple]
spack.version.version_types.ver(obj) -> VersionList | ClosedOpenRange | StandardVersion | GitVersion
Parses a Version, VersionRange, or VersionList from a string or list of strings. This is the one you actually call, bestie. โจ
๐งฉ Submodules
spack.abi module
class spack.abi.ABI
Bases: object โ this class provides methods to test ABI compatibility between specs. The current implementation is rather rough and could be improved. (Their words, not mine! ๐ญ)
architecture_compatible(target: Spec, constraint: Spec) -> boolโ returnTrueif the architecture of the target spec is ABI compatible with the architecture of the constraint spec. If either the target or constraint specs have no architecture, the target is also defined as architecture ABI compatible to the constraint. Chill defaults fr fr.compatible(target: Spec, constraint: Spec, loose: bool = False) -> boolโ returnsTrueif the target spec is ABI compatible to the constraint spec.compiler_compatible(parent: Spec, child: Spec, loose: bool = False) -> boolโ returnTrueif the compilers forparentandchildare ABI compatible.
spack.audit module
Classes and functions to register audit checks for various parts of Spack and run them on-demand. ๐ต๏ธโโ๏ธ
To register a new class of sanity checks (e.g. sanity checks for compilers.yaml), the first action required is to create a new AuditClass object:
audit_cfgcmp = AuditClass(
tag='CFG-COMPILER',
description='Sanity checks on compilers.yaml',
kwargs=()
)This object is then used as a decorator to register functions that will each perform a single check:
@audit_cfgcmp
def _search_duplicate_compilers(error_cls):
passThese functions need to take as argument the keywords declared when creating the decorator object plus an error_cls argument at the end, acting as a factory to create Error objects. It should return a (possibly empty) list of errors.
Calls to each of these functions are triggered by the run method of the decorator object, that will forward the keyword arguments passed as input.
class spack.audit.AuditClass(group, tag, description, kwargs) Bases: Sequence
run(**kwargs)
spack.audit.CALLBACKS = {'CFG-COMPILER': <spack.audit.AuditClass object>, 'CFG-PACKAGES': <spack.audit.AuditClass object>, 'GENERIC': <spack.audit.AuditClass object>, 'PKG-ATTRIBUTES': <spack.audit.AuditClass object>, 'PKG-DIRECTIVES': <spack.audit.AuditClass object>, 'PKG-EXTERNALS': <spack.audit.AuditClass object>, 'PKG-HTTPS-DIRECTIVES': <spack.audit.AuditClass object>, 'PKG-PROPERTIES': <spack.audit.AuditClass object>} : Map an audit tag to a list of callables implementing checks
class spack.audit.Error(summary, details) Bases: object
Information on an error reported in a test.
spack.audit.GROUPS = {'configs': ['CFG-COMPILER', 'CFG-PACKAGES'], 'externals': ['PKG-EXTERNALS'], 'generic': ['GENERIC'], 'packages': ['PKG-DIRECTIVES', 'PKG-ATTRIBUTES', 'PKG-PROPERTIES'], 'packages-https': ['PKG-HTTPS-DIRECTIVES']} : Map a group of checks to the list of related audit tags
spack.audit.config_compiler = <spack.audit.AuditClass object> : Sanity checks on compilers.yaml
spack.audit.config_packages = <spack.audit.AuditClass object> : Sanity checks on packages.yaml
spack.audit.external_detection = <spack.audit.AuditClass object> : Sanity checks on package directives
spack.audit.generic = <spack.audit.AuditClass object> : Generic checks relying on global state
spack.audit.package_directives = <spack.audit.AuditClass object> : Sanity checks on package directives
spack.audit.packages_with_detection_tests() : Return the list of packages with a corresponding detection_test.yaml file.
spack.audit.run_check(tag, **kwargs) : Run the checks associated with a single tag.
Parameters
tag(str) -- tag of the check**kwargs-- keyword arguments forwarded to the checks
Returns Errors occurred during the checks
spack.audit.run_group(group, **kwargs) : Run the checks that are part of the group passed as argument.
Parameters
group(str) -- group of checks to be run**kwargs-- keyword arguments forwarded to the checks
Returns List of (tag, errors) that failed.
spack.binary_distribution module ๐๐ฆ
spack.binary_distribution.BINARY_INDEX: BinaryCacheIndex = <spack.binary_distribution.BinaryCacheIndex object> : Default binary cache index instance
class spack.binary_distribution.BinaryCacheIndex(cache_root: str | None = None) Bases: object
The BinaryCacheIndex tracks what specs are available on (usually remote) binary caches.
This index is "best effort", in the sense that whenever we don't find what we're looking for here, we will attempt to fetch it directly from configured mirrors anyway. Thus, it has the potential to speed things up, but cache misses shouldn't break any spack functionality.
At the moment, everything in this class is initialized as lazily as possible, so that it avoids slowing anything in spack down until absolutely necessary.
TODO: What's the cost if, e.g., we realize in the middle of a spack install that the cache is out of date, and we fetch directly? Does it mean we should have paid the price to update the cache earlier?
clear() : For testing purposes we need to be able to empty the cache and clear associated data structures.
find_built_spec(spec, mirrors_to_check=None) : Look in our cache for the built spec corresponding to spec.
If the spec can be found among the configured binary mirrors, a list is returned that contains the concrete spec and the mirror url of each mirror where it can be found. Otherwise, None is returned.
This method does not trigger reading anything from remote mirrors, but rather just checks if the concrete spec is found within the cache.
The cache can be updated by calling update() on the cache.
Parameters
spec(spack.spec.Spec) -- Concrete spec to findmirrors_to_check-- Optional mapping containing mirrors to check. IfNone, just assumes all configured mirrors.
Returns
An list of objects containing the found specs and mirror url where each can be found, e.g.:
[
{
"spec": <concrete-spec>,
"mirror_url": <mirror-root-url>
}
]find_by_hash(find_hash, mirrors_to_check=None) : Same as find_built_spec but uses the hash of a spec.
Parameters
find_hash(str) -- hash of the spec to searchmirrors_to_check-- Optional mapping containing mirrors to check. IfNone, just assumes all configured mirrors.
get_all_built_specs()
regenerate_spec_cache(clear_existing=False) : Populate the local cache of concrete specs (_mirrors_for_spec) from the locally cached buildcache index files. This is essentially a no-op if it has already been done, as we keep track of the index hashes for which we have already associated the built specs.
update(with_cooldown=False) : Make sure local cache of buildcache index files is up to date. If the same mirrors are configured as the last time this was called and none of the remote buildcache indices have changed, calling this method will only result in fetching the index hash from each mirror to confirm it is the same as what is stored locally. Otherwise, the buildcache index.json and index.json.hash files are retrieved from each configured mirror and stored locally (both in memory and on disk under _index_cache_root).
update_spec(spec, found_list) : Take list of {'mirror_url': m, 'spec': s} objects and update the local built_spec_cache
class spack.binary_distribution.BinaryCacheQuery(all_architectures) Bases: object
Callable object to query if a spec is in a binary cache
class spack.binary_distribution.BuildCacheDatabase(root) Bases: Database
A database for binary buildcaches.
A database supports writing buildcache index files, in which case certain fields are not needed in each install record, and no locking is required. To use this feature, it provides lock_cfg=NO_LOCK, and override the list of record_fields.
record_fields: Tuple[str, ...] = ('spec', 'ref_count', 'in_buildcache') : Fields written for each install record
class spack.binary_distribution.BuildManifestVisitor Bases: BaseDirectoryVisitor
Visitor that collects a list of files and symlinks that can be checked for need of relocation. It knows how to dedupe hardlinks and deal with symlinks to files and directories.
before_visit_dir(root, rel_path, depth) : Return True from this function to recurse into the directory at os.path.join(root, rel_path). Return False in order not to recurse further.
Parameters
root-- root directoryrel_path-- relative path to current directory fromrootdepth-- depth of current directory from the root directory
Returns True when the directory should be recursed into. False when not
Return type bool
before_visit_symlinked_dir(root, rel_path, depth) : Return True to recurse into the symlinked directory and False in order not to. Note: rel_path is the path to the symlink itself. Following symlinked directories blindly can cause infinite recursion due to cycles.
Parameters
root-- root directoryrel_path-- relative path to current symlink fromrootdepth-- depth of current symlink from the root directory
Returns True when the directory should be recursed into. False when not
Return type bool
seen_before(root, rel_path)
visit_file(root, rel_path, depth) : Handle the non-symlink file at os.path.join(root, rel_path)
Parameters
root-- root directoryrel_path-- relative path to current file fromrootdepth(int) -- depth of current file from the root directory
visit_symlinked_file(root, rel_path, depth) : Handle the symlink to a file at os.path.join(root, rel_path). Note: rel_path is the location of the symlink, not to what it is pointing to. The symlink may be dangling.
Parameters
root-- root directoryrel_path-- relative path to current symlink fromrootdepth-- depth of current symlink from the root directory
exception spack.binary_distribution.BuildcacheIndexError(message, long_message=None) Bases: SpackError
...raised when a buildcache just cannot be read, for any reason at all ๐ญ (yes, even the silly ones)
spack.binary_distribution.CURRENT_BUILD_CACHE_LAYOUT_VERSION = 2โ bestie this is the build cache layout version that this version of Spack creates. And what does Version 2 give you? It includes the parent directories of the package prefix in the tarball. That's it, that's the whole vibe โจexception spack.binary_distribution.CannotListKeys(message, long_message=None)โ Bases:GenerateIndexError. Raised when it can't list keys while generating the key index. rude but okay ๐class spack.binary_distribution.DefaultIndexFetcher(url, local_hash, urlopen=<function _urlopen.<locals>.dispatch_open>)โ Bases:object. A fetcher forindex.jsonthat uses a separateindex.json.hashfile as its cache invalidation strategy. lowkey elegant ngl ๐conditional_fetch() -> FetchIndexResultget_remote_hash()
class spack.binary_distribution.EtagIndexFetcher(url, etag, urlopen=<function _urlopen.<locals>.dispatch_open>)โ Bases:object. Same idea, different slay: this fetcher forindex.jsonuses ETags headers as the cache invalidation strategy ๐conditional_fetch() -> FetchIndexResult
exception spack.binary_distribution.FetchCacheError(errors)โ Bases:Exception. Thrown when fetching the cache straight up failed โ and it's usually a composite error list, so like, multiple problems holding hands ๐exception spack.binary_distribution.FetchIndexErrorโ Bases:Exception. No fancy description, it's just giving "index fetch said no" ๐class spack.binary_distribution.FetchIndexResult(etag, hash, data, fresh)โ Bases:tuple. A lil named tuple moment, fields in order:dataโ alias for field number 2etagโ alias for field number 0freshโ alias for field number 3hashโ alias for field number 1
exception spack.binary_distribution.GenerateIndexError(message, long_message=None)โ Bases:SpackError. Raised when it can't generate the key or package index for a mirror. iconic of it to simply refuse ๐ซถexception spack.binary_distribution.InvalidMetadataFile(message, long_message=None)โ Bases:SpackError. The metadata file is Not It.exception spack.binary_distribution.ListMirrorSpecsError(message, long_message=None)โ Bases:SpackError. Raised when it can't retrieve the list of specs from the mirror ๐exception spack.binary_distribution.NewLayoutException(msg)โ Bases:SpackError. Raised if the directory layout is different from the buildcache's. mismatch era ๐exception spack.binary_distribution.NoChecksumException(path, size, contents, algorithm, expected, computed)โ Bases:SpackError. Raised when a file fails checksum verification. trust issues, honestly.exception spack.binary_distribution.NoGpgException(msg)โ Bases:SpackError. Raised whengpg2is not inPATH. bestie go install gpg2 fr fr ๐ฅexception spack.binary_distribution.NoKeyException(msg)โ Bases:SpackError. Raised when gpg has no default key added. no key, no signature, no entry ๐exception spack.binary_distribution.NoOverwriteException(file_path)โ Bases:SpackError. Raised when a file would be overwritten. hands off the existing file โexception spack.binary_distribution.NoVerifyException(message, long_message=None)โ Bases:SpackError. Raised if a file fails signature verification. the signature said "not today" ๐ญexception spack.binary_distribution.NotInstalledError(specs: List[Spec])โ Bases:SpackError. Raised when a spec is not installed but got picked to be packaged anyway. main character spec with zero credentials ๐class spack.binary_distribution.OCIIndexFetcher(url: str, local_hash, urlopen=None)โ Bases:object. The OCI registry flavor of index fetching โจconditional_fetch() -> FetchIndexResultโ Download an index from an OCI registry type mirror. doing the absolute most, respectfully ๐ซถ
exception spack.binary_distribution.PickKeyException(keys)โ Bases:SpackError. Raised when multiple keys could be used to sign, and it can't pick. too many options, so real of it ๐class spack.binary_distribution.PushOptions(force, regenerate_index, unsigned, key)โ Bases:NamedTuple. A whole little bundle of push settings, one field at a time:force: boolโ overwrite existing tarball/metadata files in the buildcachekey: str | Noneโ what key to use for signingregenerate_index: boolโ regenerated indices after pushingunsigned: boolโ whether to sign or not.
exception spack.binary_distribution.PushToBuildCacheError(message, long_message=None)โ Bases:SpackError. Raised when it can't push objects to the binary mirror. the push was pushed back ๐ญexception spack.binary_distribution.UnsignedPackageException(message, long_message=None)โ Bases:SpackError. Raised if you try to install an unsigned package without using--no-check-signature. so like, the flag is not optional bestie, it's the whole point ๐ฅ
๐ฆ the actual functions, bestie ๐ฆ
spack.binary_distribution.binary_index_location()โ Sets up aBinaryCacheIndexfor remote buildcache dbs in the user's homedir. home sweet home ๐spack.binary_distribution.build_cache_keys_relative_path()โ no description in the source, but the name says it all. iykyk ๐spack.binary_distribution.build_cache_prefix(prefix)โ takes aprefix, does prefix things. keeping it mysterious โจspack.binary_distribution.build_cache_relative_path()โ same energy, no notes. ๐spack.binary_distribution.buildinfo_file_name(prefix)โ the filename of the binary package meta-data file. that's the whole job ๐ซถspack.binary_distribution.check_specs_against_mirrors(mirrors, specs, output_file=None)โ Checks every given spec against the buildcaches on the given mirrors and figures out if any of them need to be rebuilt. and when do they need rebuilding? when their hash doesn't exist in the mirror. hash missing = rebuild era ๐- Parameters
mirrors(dict) โ mirrors to check againstspecs(Iterable) โ specs to check against mirrorsoutput_file(str) โ path to the output file to be written. If provided, mirrors with missing or out-of-date specs get formatted as a JSON object and written to this file.
- Returns:
1if any spec was out-of-date on any mirror,0otherwise. yes, it's a 1 or 0 situation, very binary of it โญ
- Parameters
spack.binary_distribution.clear_spec_cache()โ clears the spec cache. bye bye cache ๐ธ (no extra description, don't come for me)spack.binary_distribution.compute_hash(data)โ computes a hash fromdata. the name carried the whole description and honestly it ate ๐spack.binary_distribution.dedupe_hardlinks_if_necessary(root, buildinfo)โ Updates abuildinfodict for old archives that did not dedupe hardlinks. And why do we care? because de-duping hardlinks is necessary when relocating files in parallel and in-place โ which means we must preserve inodes when relocating. no inodes, no peace ๐คspack.binary_distribution.download_buildcache_entry(file_descriptions, mirror_url=None)โ downloads a buildcache entry, and you can optionally point it at amirror_urlโจspack.binary_distribution.download_single_spec(concrete_spec, destination, mirror_url=None)โ downloads the buildcache files for exactly one concrete spec. one spec, one destiny ๐ฆ- Parameters
concrete_specโ the concrete spec to be downloadeddestination(str) โ path where to put the downloaded buildcachemirror_url(str) โ url of the mirror from which to download
- Parameters
spack.binary_distribution.download_tarball(spec, unsigned: bool | None = False, mirrors_for_spec=None)โ Downloads the binary tarball for a given package into the stage area, and returns the path to the downloaded tarball if it worked,Noneotherwise. iconic behavior tbh ๐- Parameters
spec(spack.spec.Spec) โ concrete specunsignedโ ifTrueorFalse, override the mirror signature verification defaultsmirrors_for_spec(list) โ optional list of concrete specs and mirrors obtained by callingbinary_distribution.get_mirrors_for_spec(). These get checked in order first before looking in other configured mirrors.
- Returns
Noneif the tarball could not be downloaded (maybe also verified, depending on whether new-style signed binary packages were found). Otherwise, return an object indicating the path to the downloaded tarball, the path to the downloaded specfile (in the case of new-style buildcache), and whether or not the tarball is already verified. and here's the shape of that object, verbatim:
- Parameters
{
"tarball_path": "path-to-locally-saved-tarfile",
"specfile_path": "none-or-path-to-locally-saved-specfile",
"signature_verified": "true-if-binary-pkg-was-already-verified"
}spack.binary_distribution.extract_tarball(spec, download_result, force=False, timer=<spack.util.timer.NullTimer object>)โ extracts the binary tarball for a given package into the install area. unzip and slay ๐งspack.binary_distribution.file_matches(path, regex)โ doespathmatchregex? that's the entire plot ๐spack.binary_distribution.generate_key_index(key_prefix, tmpdir=None)โ Creates the key index page, bestie. Specifically: it creates (or replaces) the"index.json"page at the location given inkey_prefix, and that page contains an entry for each key (.pub) underkey_prefix. every single one ๐ซถspack.binary_distribution.generate_package_index(cache_prefix, concurrency=32)โ Creates or replaces the build cache index on the given mirror. The buildcache index contains an entry for each binary package under thecache_prefix. ๐- Parameters
cache_prefix(str) โ base url of the binary mirror.concurrencyโ (int): the desired threading concurrency to use when fetching the spec files from the mirror.
- Returns:
None. that's it. that's the return. ๐
- Parameters
spack.binary_distribution.get_buildfile_manifest(spec)โ Returns a data structure with info about a build, and the things in it aretext_to_relocate,binary_to_relocate,binary_to_relocate_fullpath,link_to_relocate, andotherโ and "other" means it doesn't fit any of the previous checks (and should not be relocated). We exclude docs (man) and metadata (.spack). This can be used to find a particular kind of file in spack, or to generate the build metadata. so many jobs, one manifest โจspack.binary_distribution.get_buildinfo_dict(spec)โ creates metadata for a tarball. short and sweet ๐spack.binary_distribution.get_keys(install=False, trust=False, force=False, mirrors=None)โ Gets the pgp public keys available on a mirror, with the suffix.pub. keys era ๐ (well,.pubera)spack.binary_distribution.get_mirrors_for_spec(spec=None, mirrors_to_check=None, index_only=False)โ Checks whether a concrete spec exists on the mirrors and returns a list telling you which mirrors it can be found on. the "where is my package" function fr fr ๐ฆ- Parameters
spec(spack.spec.Spec) โ the spec to look for in the binary mirrorsmirrors_to_check(dict) โ optionally override the configured mirrors with the mirrors in this dictionary.index_only(bool) โ whenindex_onlyis set toTrue, only the local cache is checked and no requests are made. offline mode bestie, no cap ๐
- Returns A list of objects, each containing a
mirror_urland a spec key, indicating all the mirrors where the spec can be found. ๐
- Parameters
spack.binary_distribution.hashes_to_prefixes(spec)โ hands you back a dictionary mapping hashes to prefixes for a spec plus all its deps, and hard pass on the externals bestie, they're not invited ๐spack.binary_distribution.install_root_node(spec, unsigned=False, force=False, sha256=None)โ installs just the root node of a concrete spec out of a buildcache, no deps attached, main character only ๐
Checking the sha256 sum of a node before install is usually needed only for software installed during Spack's bootstrapping, since at that point we might not have a proper signature verification mechanism available. (iykyk ๐ญ)
Parameters:
specโ the spec to be installed (heads up, only the root node gets installed)unsigned(bool) โ ifTrue, allows installing unsigned binariesforce(bool) โ force installation if the spec is already present in the local storesha256(str) โ optional sha256 of the binary package, checked before installation
spack.binary_distribution.install_single_spec(spec, unsigned=False, force=False)โ installs one single concrete spec from a buildcache, that's it, that's the vibe โจ
Parameters:
spec(spack.spec.Spec) โ spec to be installedunsigned(bool) โ ifTrue, allows installing unsigned binariesforce(bool) โ force installation if the spec is already present in the local store
spack.binary_distribution.needs_rebuild(spec, mirror_url)โ no description given, it's giving mysterious and lowkey iconic ๐ต๏ธโโ๏ธspack.binary_distribution.push_keys(*mirrors, **kwargs)โ uploads pgp public keys to the mirrors you point at ๐spack.binary_distribution.push_or_raise(spec: Spec, out_url: str, options: PushOptions)โ builds a tarball from the given spec and drops it into the directory structure used at the mirror (following<tarball_directory_name>).
This method raises NoOverwriteException when force=False and the tarball or spec.json file already exist in the buildcache. It raises PushToBuildCacheError when the tarball or spec.json file cannot be pushed to the buildcache. So like, it will raise, that's the whole name ๐ซ
spack.binary_distribution.read_buildinfo_file(prefix)โ reads the buildinfo file. short and sweet fr frspack.binary_distribution.relocate_package(spec)โ relocates the given package ๐spack.binary_distribution.select_signing_key(key=None)โ no description provided, just vibes ๐ซspack.binary_distribution.sign_specfile(key, force, specfile_path)โ also no description, we love an enigmatic queen ๐spack.binary_distribution.specs_to_be_packaged(specs: List[Spec], root: bool = True, dependencies: bool = True) -> List[Spec]โ returns the list of nodes to be packaged, given a list of specs. RaisesNotInstalledErrorif a spec is not installed but got picked to be packaged ๐ฌ
Parameters:
specsโ list of root specs to be processedrootโ include the root of each spec in the nodesdependenciesโ include the dependencies of each spec in the nodes
spack.binary_distribution.tarball_directory_name(spec)โ returns the name of the tarball directory per the convention<os>-<architecture>/<compiler>/<package>-<version>/๐spack.binary_distribution.tarball_name(spec, ext)โ returns the name of the tarfile per the convention<os>-<architecture>-<package>-<dag_hash><ext>๐๏ธspack.binary_distribution.tarball_path_name(spec, ext)โ returns the full path+name for a given spec per the convention<tarball_directory_name>/<tarball_name>๐ค๏ธspack.binary_distribution.tarfile_of_spec_prefix(tar: TarFile, prefix: str) -> Noneโ creates a tarfile of an install prefix of a spec. Skips the existing buildinfo file, no double-dipping ๐ โโ๏ธ
Parameters:
tarโ tarfile object to add files toprefixโ absolute install prefix of spec
spack.binary_distribution.try_direct_fetch(spec, mirrors=None)โ tries to find the spec directly on the configured mirrors ๐spack.binary_distribution.try_fetch(url_to_fetch)โ utility function that tries to fetch a file from a url, stages it locally, and returns the path to the staged file. manifesting a successful download โจ
Parameters:
url_to_fetch(str) โ url pointing to remote resource to fetch
Returns:
- path to the locally staged resource, or
Noneif it could not be fetched ๐ญ
spack.binary_distribution.try_verify(specfile_path)โ utility function that attempts to verify a local file. Assumes the file is a clearsigned signature file. ๐
Parameters:
specfile_path(str) โ path to file to be verified
Returns:
Trueif the signature could be verified,Falseotherwise
spack.binary_distribution.update_cache_and_get_specs()โ gets all concrete specs for build caches available on configured mirrors. Initialization of internal cache data structures is done as lazily as possible, so this method will also attempt to initialize and update the local index cache โ which is essentially a no-op if it's been done already and nothing has changed on the configured mirrors. lazy girlie behavior, honestly relatable ๐ด
Throws:
FetchCacheError
๐ง spack.build_environment module
This module contains all the routines related to setting up the package build environment. All of it gets set up by package.py right before install() is called. ๐ ๏ธ
There are two parts to the build environment, bestie, and yep we're covering both:
- Python build environment (i.e. the
install()method)
This is how things are set up when install() is called. Spack takes advantage of each package being in its own module by adding a bunch of command-like functions (like configure(), make(), etc.) in the package's module scope. This lets package writers call them all directly in Package.install() without writing self. everywhere. No, this isn't Pythonic. Yes, it makes the code more readable and more like the shell script from which someone is likely porting. iconic trade-off tbh ๐
- Build execution environment
This is the set of environment variables, like PATH, CC, CXX, etc., that control the build. There are also a number of environment variables used to pass information (like RPATHs and other information about dependencies) to Spack's compiler wrappers. All of these env vars are also set up here. ๐
Skimming this module is a nice way to get acquainted with the types of calls you can make from within the install() function. do it, it's self-care ๐ซถ
exception spack.build_environment.ChildError(msg, module, classname, traceback_string, log_name, log_type, context)
Bases: InstallError
Special exception class for wrapping exceptions from child processes in Spack's build environment. ๐ง๐ฅ
The main features of a ChildError are:
- They're serializable, so when a child build fails, we can send one of these to the parent and let the parent report what happened.
- They have a
tracebackfield containing a traceback generated on the child immediately after failure. Spack will print this on failure in lieu of trying to runsys.excepthookon the parent process, so users will see the correct stack trace from a child. ๐ - They also contain
context, which shows context in thePackageimplementation where the error happened. This helps people debug Python code in their packages. To get it, Spack searches the stack trace for the deepest frame whereselfis in scope and is an instance ofPackageBase. This will generally find a useful spot in thepackage.pyfile. detective work, slay ๐ต๏ธโโ๏ธ
The long_message of a ChildError displays one of two things:
- If the original error was a
ProcessError, indicating a command died during the build, we'll show context from the build log. - If the original error was any other type of error, we'll show context from the Python code.
SpackError handles displaying the special traceback if we're in debug mode with spack -d. ๐
build_errors = [('spack.util.executable', 'ProcessError')]
property long_message
class spack.build_environment.EnvironmentVisitor(*roots: Spec, context: Context)
Bases: object
neighbors(item)
class spack.build_environment.MakeExecutable(name, jobs, **kwargs)
Bases: Executable
Special callable executable object for make so the user can specify parallelism options on a per-invocation basis. Specifying 'parallel' to the call will override whatever the package's global setting is, so you can either default to true or false and override particular calls. Specifying 'jobs_env' to a particular call will name an environment variable which will be set to the parallelism level (without affecting the normal invocation with -j). parallel girlie make it make sense ๐
class spack.build_environment.ModuleChangePropagator(package)
Bases: object
Wrapper class to accept changes to a package.py Python module, and propagate them in the MRO of the package. ๐
It is mainly used as a substitute of the package.py module, when calling the "setup_dependent_package" function during build environment setup.
propagate_changes_to_mro()
class spack.build_environment.SetupContext(*specs: Spec, context: Context)
Bases: object
This class encapsulates the logic to determine environment modifications, and is used as well to set globals in modules of package.py. ๐๏ธ
get_env_modifications() -> EnvironmentModifications
Returns the environment variable modifications for the given input specs and context. Environment modifications include:
- Updating
PATHfor packages that are required at runtime - Updating
CMAKE_PREFIX_PATHandPKG_CONFIG_PATHso that their respective tools can find Spack-built dependencies (whencontext=build) - Running custom package environment modifications:
setup_run_environment,setup_dependent_run_environment,setup_build_environment,setup_dependent_build_environment.
The (partial) order imposed on the specs is externals first, then topological from leaf to root. That way externals cannot contribute search paths that would shadow Spack's prefixes, and dependents override variables set by dependencies. no cap, order matters here ๐ฅ
set_all_package_py_globals()
Sets the globals in modules of package.py files. That's it, that's the whole girl ๐
exception spack.build_environment.StopPhase(message, long_message=None)
Bases: SpackError
Okay so this one is a pickle-able exception that's literally there to control stopped builds ๐ โ like, "bestie, we're pausing this build on purpose, don't panic."
class spack.build_environment.UseMode(*values)
Bases: Flag
This is the vibe-check enum for how a spec gets used in an environment ๐ฆ Here are all the flags:
ADDED = 32โ flag is set when the(node, mode)is finalized โจBUILDTIME = 16โ a spec that should be visible in search paths in a build env ๐จBUILDTIME_DIRECT = 8โ a spec that's a direct build or test dep ๐ROOT = 1โ entrypoint spec (a spec to be built; an env root, etc) ๐RUNTIME = 2โ a spec used at runtime, but no executables inPATH๐RUNTIME_EXECUTABLE = 4โ a spec used at runtime, with executables inPATH๐โโ๏ธโจ
spack.build_environment.clean_environment()
Literally justโฆ cleans the environment. No further notes, she's mysterious like that ๐
spack.build_environment.effective_deptypes(*specs: Spec, context: Context = Context.BUILD) -> List[Tuple[Spec, UseMode]]
Given a list of input specs and a context, this returns a list of tuples of all the specs that contribute to (environment) modifications, together with a flag saying exactly in what way they do so ๐ซถ
And here's the important bit: the list is ordered topologically from root to leaf, meaning environment modifications should be applied in reverse so that dependents override dependencies โ NOT the other way around. Periodt ๐
spack.build_environment.get_cmake_prefix_path(pkg)
Gets the CMake prefix path for pkg. Short king ๐
spack.build_environment.get_effective_jobs(jobs, parallel=True, supports_jobserver=False)
Returns the number of jobs โ or None if supports_jobserver is set and a jobserver is detected ๐
spack.build_environment.get_package_context(traceback, context=3)
Returns some context for an error message when the build fails ๐ญ Because nothing's worse than a build failing with zero explanation.
Parameters:
tracebackโ a traceback from some exception raised during installcontext(int) โ lines of context to show before and after the line where the error happened
Here's the tea: this function inspects the stack to find where we failed in the package file, and then it adds detailed context to the long_message from there ๐
spack.build_environment.get_rpath_deps(pkg: PackageBase) -> List[Spec]
Returns the immediate or transitive dependencies (depending on the package, bestie) that need to be rpath'ed ๐ And if a package shows up multiple times, the newest version is kept โ the older ones can go, we don't need them.
spack.build_environment.get_rpaths(pkg)
Gets a list of all the rpaths for a package. Obsessed with her honestly ๐
spack.build_environment.jobserver_enabled()
Returns true if a posix jobserver (make) is detected. Slay ๐ฅ
spack.build_environment.load_external_modules(pkg)
Traverses a package's spec DAG and loads any external modules ๐ซ
More specifically: it traverses a package's dependencies and loads any external modules associated with them.
Parameters:
pkg(spack.package_base.PackageBase) โ package to load deps for
spack.build_environment.set_compiler_environment_variables(pkg, env)
Sets the compiler environment variables for pkg into env. Lowkey essential ๐ง
spack.build_environment.set_package_py_globals(pkg, context: Context = Context.BUILD)
Populates the Python module of a package with some useful global names ๐ธ This makes things way easier for package writers. You're welcome, bestie ๐ซถ
spack.build_environment.set_wrapper_variables(pkg, env)
Sets environment variables used by the Spack compiler wrapper (which all have the prefix SPACK_) and also adds the compiler wrappers to PATH ๐
This determines the injected -L/-I/-rpath options; each of these specifies a search order and this function computes these options in a manner that is intended to match the DAG traversal order in SetupContext.
TODO: this is not the case yet, we're using post order, SetupContext is using topo order. ๐ She said "not yet" and honestly respect the honesty.
spack.build_environment.setup_package(pkg, dirty, context: Context = Context.BUILD)
Executes all environment setup routines ๐ฅ All of them. Every single one.
spack.build_environment.start_build_process(pkg, function, kwargs)
Creates a child process to do part of a Spack build ๐ถโจ
Parameters:
pkg(spack.package_base.PackageBase) โ package whose environment we should set up the child process forfunction(Callable) โ argless function to run in the child process
Usage:
def child_fun():
# do stuff
build_env.start_build_process(pkg, child_fun)The child process is run with the build environment set up by spack.build_environment ๐ This allows package authors to have full control over the environment, etc. without affecting other builds that might be executed in the same spack call.
If something goes wrong, the child process catches the error and passes it to the parent wrapped in a ChildError ๐ญ The parent is expected to handle (or re-raise) the ChildError.
This uses multiprocessing.Process to create the child process. The mechanism used to create the process differs on different operating systems and for different versions of Python. In some cases "fork" is used (i.e. the "fork" system call) and some cases it starts an entirely new Python interpreter process (in the docs this is referred to as the "spawn" start method). Breaking it down by OS:
- Linux always uses
fork๐ง - Mac OS uses
forkbefore Python 3.8 and"spawn"for 3.8 and after ๐ - Windows always uses the
"spawn"start method ๐ช
For more information on multiprocessing child process creation mechanisms, see โ https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
spack.build_environment.write_log_summary(out, log_type, log, last=None)
Writes a log summary out to out. The details are hers to keep, honestly ๐
๐ธ spack.builder module ๐ธ
spack.builder.BUILDER_CLS
Builder classes, as registered by the "builder" decorator. Look at this whole little registry, I'm obsessed:
BUILDER_CLS = {'cmake': <class 'spack.build_systems.cmake.CMakeBuilder'>,
'meson': <class 'spack.build_systems.meson.MesonBuilder'>,
'python_pip': <class 'spack.build_systems.python.PythonPipBuilder'>}class spack.builder.Builder(pkg)
Bases: Sequence
Okay so a builder is a class that, given a package object (i.e. associated with a concrete spec), knows how to install it ๐จโจ
And the way she behaves? Like a sequence โ when you iterate over her, she returns the "phases" of the installation in the correct order. Main character sequencing honestly ๐
Parameters:
pkg(spack.package_base.PackageBase) โ package object to be built
Attributes:
archive_files: List[str] = []โ list of glob expressions. Each expression must either be absolute or relative to the package source path ๐ Matching artifacts found at the end of the build process will be copied in the same directory tree as_spack_build_logfileand_spack_build_envfile.build_system: str | None = Noneโ build system name. Must also be defined in derived classes ๐build_time_test_callbacks: List[str]โ the build-time test callbacks, as the name literally saysinstall_time_test_callbacks: List[str]โ and these are the install-time ones ๐legacy_attributes: Tuple[str, ...] = ()โ the legacy attribute names, empty by defaultlegacy_methods: Tuple[str, ...] = ()โ the legacy method names, also empty by defaultphases: Tuple[str, ...] = ()โ sequence of phases. Must be defined in derived classes ๐ซถproperty prefixโ the prefix property, she's a propertyrun_after_callbacks = []โ callbacks that run after โจrun_before_callbacks = []โ callbacks that run before โจ
Methods:
setup_build_environment(env)โ sets up the build environment for a package ๐ธ
This method will be called before the current package prefix exists in Spack's store ๐
Parameters:
env(spack.util.environment.EnvironmentModifications) โ environment modifications to be applied when the package is built. Package authors can call methods on it to alter the build environment.
setup_dependent_build_environment(env, dependent_spec)โ sets up the build environment of packages that depend on this one ๐ซถ
This is similar to setup_build_environment, but it's used to modify the build environments of packages that depend on this one.
This gives packages like Python and others that follow the extension model a way to implement common environment or compile-time settings for dependencies ๐
This method will be called before the dependent package prefix exists in Spack's store.
Examples:
- Installing python modules generally requires
PYTHONPATHto point to thelib/pythonX.Y/site-packagesdirectory in the module's install prefix. This method could be used to set that variable ๐
Parameters:
env(spack.util.environment.EnvironmentModifications) โ environment modifications to be applied when the dependent package is built. Package authors can call methods on it to alter the build environment.dependent_spec(spack.spec.Spec) โ the spec of the dependent package about to be built. This allows the extendee (self) to query the dependent's state. Note that this package's spec is available asself.spec๐
property specโ the spec propertyproperty stageโ the stage propertytest()โ runs the test. No notes, she just tests ๐
class spack.builder.BuilderMeta(name, bases, attr_dict)
Bases: PhaseCallbacksMeta, ABCMeta
The metaclass behind the builder class โ she's giving very much "I make the class happen" energy ๐ง
class spack.builder.CallbackTemporaryStage(attribute_name, callbacks)
Bases: tuple
An object of this kind is a shared global state used to collect callbacks during class definition time, and is flushed when the class object is created at the end of the class definition ๐ Very "temporary but make it iconic."
Parameters:
attribute_name(str) โ name of the attribute that will be attached to the buildercallbacks(list) โ container used to temporarily aggregate the callbacks
Attributes:
attribute_nameโ alias for field number 0callbacksโ alias for field number 1
class spack.builder.InstallationPhase(name, builder)
Bases: object
One installation phase, holding its name and its builder ๐
๐ฅ
...is literally managing one (1) single phase of the installation, bestie. It's giving "I have one job and I do it flawlessly." ๐
So here's the tea: this descriptor stashes the name of the method it should search for execution at the moment it gets created. Then the method gets retrieved at __get__ time โ which is iconic because it means subclasses of whatever class declared the phases can override it. Main character energy for the subclasses, honestly. ๐ฅ
It also serves up hooks so you can run arbitrary callbacks before and after the phase. Obsessed with the flexibility.
copy()execute()
โจ class spack.builder.PhaseCallbacksMeta(name, bases, attr_dict)
Bases: type
Okay so this one lets you register arbitrary functions right during class definition and then run them later โ before or after a given install phase. It's giving "I'll handle it, don't worry." ๐ซถ
Every method you decorate with run_before or run_after gets temporarily stored in a global shared state when the Python interpreter parses a class that's being defined. Then, at class definition time, that temporary storage gets flushed and a list of callbacks gets attached to the class being defined. Slay, honestly. โจ
static run_after(phase, when=None)โ decorator to register a function for running after a given phase.
Parameters:
phase(str) โ the phase after which the function must run.when(str) โ condition under which the function is run (ifNone, it is always run).
static run_before(phase, when=None)โ decorator to register a function for running before a given phase.
Parameters:
phase(str) โ the phase before which the function must run.when(str) โ condition under which the function is run (ifNone, it is always run).
โจ spack.builder.builder(build_system_name)
A class decorator used to register the default builder for a given build-system. No cap, this is the vibe check for build systems. ๐
Parameters:
build_system_name(str) โ name of the build-system.
โจ spack.builder.buildsystem_name(pkg)
Give it a package object that has an associated concrete spec, and it hands you back the name of its build system. Lowkey so useful.
Parameters:
pkg(spack.package_base.PackageBase) โ package for which we want the build system name.
โจ spack.builder.create(pkg)
Same deal โ you pass in a package object with an associated concrete spec, and it returns the builder object that can actually install it. It's giving "I got you, bestie." ๐
Parameters:
pkg(spack.package_base.PackageBase) โ package for which we want the builder.
โจ spack.builder.run_after(phase, when=None)
Decorator to register a function for running after a given phase.
Parameters:
phase(str) โ phase after which the function must run.when(str) โ condition under which the function is run (ifNone, it is always run).
โจ spack.builder.run_before(phase, when=None)
Decorator to register a function for running before a given phase.
Parameters:
phase(str) โ phase before which the function must run.when(str) โ condition under which the function is run (ifNone, it is always run).
๐ธ spack.caches module
Caches that Spack uses to store data. Think of it as Spack's little storage girlies keeping everything organized. ๐๏ธ
โจ spack.caches.FETCH_CACHE: FsCache | Singleton = <spack.fetch_strategy.FsCache object>
Spack's local cache for downloaded source archives. Saving you from re-downloading the same tarball like a hundred times. ๐ญ
โจ spack.caches.MISC_CACHE: FileCache | Singleton = <spack.util.file_cache.FileCache object>
Spack's cache for small data. It's giving "tiny but mighty." ๐ฆ
โจ class spack.caches.MirrorCache(root, skip_unstable_versions)
Bases: object
store(fetcher, relative_dest)โ fetch and relocate the fetcher's target right into our mirror cache. Relocation services but make it archive storage. ๐symlink(mirror_ref)โ symlink a human-readable path in our mirror to the actual storage location. Bestie, it's just a shortcut, iconic.
โจ spack.caches.fetch_cache_location()
The filesystem cache of downloaded archives. This is the whole reason Spack doesn't repeatedly fetch the same files when building the same package different ways or multiple times. Thank u for your service. ๐
โจ spack.caches.misc_cache_location()
The MISC_CACHE is Spack's cache for small data. Currently the MISC_CACHE stores indexes for virtual dependency providers and for which packages provide which tags. Fr fr, that's what's in there. โญ
๐ spack.ci module
โจ class spack.ci.CDashHandler(ci_cdash)
Bases: object
Class for managing CDash data and processing. It's the whole CDash handler situation.
args()property build_nameโ returns the CDash build name.
A name gets generated if the current_spec property is set; otherwise, the value gets retrieved from the environment through the SPACK_CDASH_BUILD_NAME variable.
Returns: (str) the current spec's CDash build name.
property build_stampโ returns the CDash build stamp.
The one defined by the SPACK_CDASH_BUILD_STAMP environment variable is preferred โ because of how timestamps are represented, iykyk โ otherwise, one will be built.
Returns: (str) current CDash build stamp.
copy_test_results(source, dest)โ copy test results to the artifacts directory.create_buildgroup(opener, headers, url, group_name, group_type)populate_buildgroup(job_names)property project_encreport_skipped(spec: Spec, report_dir: str, reason: str | None)โ explicitly report skipping testing of a spec (e.g., its CI configuration identifies it as known to have broken tests, or the CI installation failed).
Parameters:
specโ spec being testedreport_dirโ directory where the report will be writtenreasonโ reason the test is being skipped
property upload_url
โจ class spack.ci.PushResult(success, url)
Bases: tuple
successโ alias for field number 0urlโ alias for field number 1
โจ class spack.ci.RebuildDecision
Bases: object
โจ class spack.ci.SpackCI(ci_config, spec_labels, stages)
Bases: object
The Spack CI object, used to generate the intermediate representation used by the CI generator(s). Middle-management but make it glamorous. ๐ผโจ
generate_ir()โ generate the IR from the Spack CI configurations.
โจ class spack.ci.TemporaryDirectory
Bases: object
โจ spack.ci.can_sign_binaries()
Utility method to figure out if this spack instance is capable of signing binary packages. Currently this is only possible if the spack gpg keystore contains exactly one secret key. Just one. Not two. Periodt. ๐
โจ spack.ci.can_verify_binaries()
Utility method to determine if this spack instance is capable (at least in theory) of verifying signed binaries. "In theory" is doing a lot of heavy lifting there ngl. ๐
โจ spack.ci.compute_affected_packages(rev1='HEAD^', rev2='HEAD')
Figure out which packages were added, removed, or changed between rev1 and rev2, and return the names as a set. Detective work, honestly. ๐ต๏ธโโ๏ธ
โจ spack.ci.copy_files_to_artifacts(src, artifacts_dir)
Copy file(s) into the given artifacts directory.
Parameters:
src(str) โ the glob-friendly path expression for the file(s) to copyartifacts_dir(str) โ the destination directory
โจ spack.ci.copy_stage_logs_to_artifacts(job_spec: Spec, job_log_dir: str) -> None
Copy the selected build stage file(s) into the given artifacts directory.
It looks for build logs in the stage directory of the given job_spec, and attempts to copy the files into the directory given by job_log_dir.
Parameters:
job_specโ spec associated with spack install logjob_log_dirโ path into which the build log should be copied
โจ spack.ci.copy_test_logs_to_artifacts(test_stage, job_test_dir)
Copy test log file(s) into the given artifacts directory.
Parameters:
test_stage(str) โ test stage pathjob_test_dir(str) โ the destination artifacts test directory
โจ spack.ci.create_buildcache(input_spec: Spec, *, destination_mirror_urls: List[str], sign_binaries: bool = False) -> List[PushResult]
Create the buildcache at the provided mirror(s). Okay but the keyword-only star is so real. ๐
Parameters:
input_specโ installed spec to package and pushdestination_mirror_urlsโ list of urls to push tosign_binariesโ whether or not to sign the buildcache entry
Returns: a list of PushResults, indicating success or failure.
โจ spack.ci.display_broken_spec_messages(base_url, hashes)
Fetch the broken spec file for each of the hashes under the base_url and print a message with some details about each one. Like a little bad-news bulletin, one per hash. ๐
โจ spack.ci.download_and_extract_artifacts(url, work_dir)
Look for gitlab artifacts.zip at the given url, and attempt to download and extract the contents into the given work_dir.
Parameters:
url(str) โ complete url toartifacts.zipfilework_dir(str) โ path to destination where artifacts should be extracted
โจ spack.ci.generate_gitlab_ci_yaml(env, print_summary, output_file, prune_dag=False, check_index_only=False, run_optimizer=False, use_dependencies=False, artifacts_root=None, remote_mirror_override=None)
Generate a gitlab yaml file so you can run a dynamic child pipeline from the spec matrix in the active environment. Whole lotta arguments, but we respect her range. ๐
Parameters:
env(spack.environment.Environment) โ activated environment object, which must contain agitlab-cisection describing how to map specs to runnersprint_summary(bool) โ should we print a summary of all the jobs in the stages in which they were placedoutput_file(str) โ file path where the generated file should be writtenprune_dag(bool) โ ifTrue, do not generate jobs for specs already exist built on the mirrorcheck_index_only(bool) โ set this toTrueand Spack goes straight for the mirror index, using literally only that to figure out whether the built specs already live on the mirror (this mode = faster yaml generation time, fr). Leave it falsy and it also checks every single spec directly by url โ lowkey useful when there's no index at all, or the one you've got might be out of date ๐run_optimizer(bool) โTruemeans Spack post-processes the yaml it just generated to try to shrink it down (it attempts to collect repeated configuration and swap it out for definitions). Less yaml, same vibes ๐use_dependencies(bool) โ if true, use"dependencies"instead of"needs"("needs"is the one that allows DAG scheduling). Highkey useful when your gitlab instance simply cannot be configured to handle more than a few"needs"per job ๐ซถartifacts_root(str) โ the path where artifacts like logs, environment files (spack.yaml,spack.lock), etc. should be written. GitLab requires this to be inside the project directory, no exceptions โจremote_mirror_override(str) โ typically you only need this when onespack.yamlis used to populate several mirrors with binaries, based on some criteria. Spack protected pipelines populate different mirrors based on branch name, and this option is what facilitates that. DEPRECATED ๐
spack.ci.get_change_revisions()
If this is a git repo, get the revisions to use when checking for changed packages and spack core modules ๐ฆ
spack.ci.get_job_name(spec: Spec, build_group: str = '')
Given a spec and possibly a build group, return the job name. And if the resulting name is longer than 255 characters? It gets truncated. That's it, that's the rule โ๏ธ
Parameters
spec(spack.spec.Spec) โ spec job will buildbuild_group(str) โ name of build group this job belongs to (a CDash notion)
Returns: the job name ๐ธ
spack.ci.get_spack_info()
If spack is running from a git repo, return the most recent git log entry โ otherwise, return a string containing the spack version. Either way you get something, bestie ๐
spack.ci.get_spec_filter_list(env, affected_pkgs, dependent_traverse_depth=None)
Given a list of package names and an active/concretized environment, return the set of all concrete specs from the environment that could have been affected by changing the list of packages ๐
If a dependent_traverse_depth is given, it's used to limit upward (in the parent direction) traversal of specs of touched packages. E.g. if 1 is provided, then only direct dependents of touched package specs are traversed to produce specs that could have been affected by changing the package โ while if 0 is provided, only the changed specs themselves are traversed. If None is given, upward traversal of touched package specs is done all the way to the environment roots. And a negative number? That results in no traversals at all, yielding an empty set. Zero traversal, zero notes ๐
Parameters
env(spack.environment.Environment) โ active concrete environmentaffected_pkgs(List[str]) โ affected package namesdependent_traverse_depthโ optional integer to limit dependent traversal, orNoneto disable the limit
Returns A set of concrete specs from the active environment, including those associated with affected packages, their dependencies and dependents, as well as their dependents dependencies. The whole little family tree ๐ณ
spack.ci.get_stack_changed(env_path, rev1='HEAD^', rev2='HEAD')
Given an environment manifest path and two revisions to compare, return whether or not the stack was changed. Returns True if the environment manifest changed between the provided revisions (or additionally if the .gitlab-ci.yml file itself changed). Returns False otherwise. Simple as ๐ฆ
spack.ci.import_signing_key(base64_signing_key)
Given a Base64-encoded gpg key, decode and import it to use for signing packages ๐
Parameters
base64_signing_key(str) โ a gpg key including the secret key, armor-exported and base64 encoded, so it can be stored in a gitlab CI variable. For an example of how to generate such a key, see:https://github.com/spack/spack-infrastructure/blob/main/gitlab-docker/files/gen-key
spack.ci.process_command(name, commands, repro_dir, run=True, exit_on_failure=True)
Create a script for and run the command. Then copy the script to the reproducibility directory. Very "document everything" of it ๐
Parameters
name(str) โ name of the command being processedcommands(list) โ list of arguments for a single command, or list of lists of arguments for multiple commands. No shell escape is performed.repro_dir(str) โ job reproducibility directoryrun(bool) โ run the script and return the exit code ifTrue
Returns: the exit code from processing the command โญ
spack.ci.push_to_build_cache(spec: Spec, mirror_url: str, sign_binaries: bool) -> bool
Push one or more binary packages to the mirror ๐
Parameters
specโ installed spec to pushmirror_urlโ URL of target mirrorsign_binariesโ ifTrue, spack will attempt to sign the binary package before pushing
spack.ci.read_broken_spec(broken_spec_url)
Read data from the broken specs file located at the url, and return it as a yaml object ๐
spack.ci.remove_other_mirrors(mirrors_to_keep, scope=None)
Remove all mirrors from the given config scope โ the exceptions being any listed in mirrors_to_keep, which is a list of mirror urls. Everyone else? Gone ๐
spack.ci.reproduce_ci_job(url, work_dir, autostart, gpg_url, runtime)
Given a url to gitlab artifacts.zip from a failed spack ci rebuild job, attempt to set up an environment in which the failure can be reproduced locally. This entails the following:
First, download and extract artifacts. Then look through those artifacts to glean some information needed for the reproducer (e.g. one of the artifacts contains information about the version of spack tested by gitlab, another is the generated pipeline yaml containing details of the job, like the docker image used to run it). The output of this function is a set of printed instructions for running docker, and then commands to run to reproduce the build once you're inside the container ๐ณโจ
spack.ci.run_standalone_tests(**kwargs)
Run stand-alone tests on the current spec ๐งช
Parameters kwargs (dict) โ dictionary of arguments used to run the tests
List of recognized keys:
"cdash"(CDashHandler): (optional) cdash handler instance"fail_fast"(bool): (optional) terminate tests after the first failure"log_file"(str): (optional) test log file name if NOT CDash reporting"job_spec"(Spec): spec that was built"repro_dir"(str): reproduction directory
spack.ci.setup_spack_repro_version(repro_dir, checkout_commit, merge_commit=None)
Look in the local spack clone to find the checkout_commit, and if provided, the merge_commit given as arguments. If those commits can be found locally, then clone spack and attempt to recreate a merge commit with the same parent commits as tested in gitlab. It looks something like:
git clone repo && cd repogit checkout <checkout_commit>git merge <merge_commit>
If there's no merge_commit provided, then skip step (3). No merge, no drama ๐
Parameters
repro_dir(str) โ location where spack should be clonedcheckout_commit(str) โ SHA of PR branch commitmerge_commit(str) โ SHA of target branch parent
Returns: True if the git repo state was successfully recreated, or False otherwise.
spack.ci.stage_spec_jobs(specs: List[Spec]) -> Tuple[Dict[str, Spec], Dict[str, Set[str]], List[Set[str]]]
Turn a DAG into a list of stages (set of nodes); the list is ordered topologically, so that each node in a stage has dependencies only in previous stages. It's giving "everything in its right place" ๐
Parameters specs โ specs to build
Returns: a tuple (nodes, edges, stages) where nodes maps labels to specs, edges maps labels to a set of labels of dependencies, and stages is a topologically ordered list of sets of labels ๐
spack.ci.translate_deprecated_config(config)
(no description, it just vibes โ this is the config translator, bestie ๐ซถ)
spack.ci.win_quote(quote_str: str) -> str
(also no description โ it's the Windows quoting helper, and that's all the manpage gives us ๐ )
spack.ci.write_broken_spec(url, pkg_name, stack_name, job_url, pipeline_url, spec_dict)
Given a url to write to and the details of the failed job, write an entry in the broken specs list. Filing the receipts, but for builds ๐
๐ง spack.ci_needs_workaround module
spack.ci_needs_workaround.convert_job(job_entry)
(Takes a job entry and converts it โ the workaround module's whole personality ๐)
spack.ci_needs_workaround.get_job_name(needs_entry)
(Grabs the job name out of a needs entry โจ)
spack.ci_needs_workaround.needs_to_dependencies(yaml)
(Swaps needs over to dependencies on the given yaml โ the actual workaround, iykyk ๐ฅ)
๐ spack.ci_optimization module
spack.ci_optimization.add_extends(yaml, key)
Modifies the given object "yaml" so that it includes an "extends" key whose value features "key" ๐ฆ
Here's the whole decision tree, no cap:
- If
"extends"is not in yaml, then yaml is modified such thatyaml["extends"] == key. - If
yaml["extends"]is a str, then yaml is modified such thatyaml["extends"] == [yaml["extends"], key]. - If
yaml["extends"]is a list that does not includekey, thenkeyis appended to the list. - Otherwise, yaml is left unchanged.
Slay, a cleanly specified mutation ๐
spack.ci_optimization.build_histogram(iterator, key)
Builds a histogram of values given an iterable of mappings and a key ๐ข
For each mapping "m" with key "key" in iterator, the value m[key] is considered.
Returns a list of tuples (hash, count, proportion, value), where:
"hash"is a sha1sum hash of the value.countis literally the number of occurrences of values that hash tohash. bestie, that's it, that's the whole thing ๐proportionis the proportion of all values considered above that hash tohashโ like, what fraction of the squad is in this bucket ๐ซถvalueis one of the values considered above that hash tohash. and here's the tea: which value gets picked when multiple values hash to the samehashis undefined, so don't come for me if it's not the one you wanted ๐ญ
The list is sorted in descending order by count, which means the most frequently occurring hashes show up first. iconic behavior, honestly โจ
๐ฆ spack.ci_optimization.common_subobject(yaml, sub)
Okay so this one factors the prototype object sub out of the values of mapping yaml. Let's go ๐
Picture a modified copy of yaml called new, where for each key key in yaml:
- if
yaml[key]matchessub, thennew[key] = subkeys(yaml[key], sub). - otherwise,
new[key] = yaml[key].
If that match criteria isn't satisfied for any such key, then you get (yaml, None) back, and the yaml object comes home unchanged. no notes. ๐
But if it is satisfied โ then every matching value in new gets modified the same way as add_extends(new[key], common_key), and then new[common_key] is set to sub. The common_key value is chosen so that it doesn't match any preexisting key in new. In that case you get (new, common_key) returned. slay ๐
๐ธ spack.ci_optimization.matches(obj, proto)
Returns True if the test object obj matches the prototype object proto. three rules, no cap:
- if
objandprotoare mappings,objmatchesprotoif(key in obj)and(obj[key] matches proto[key])for every key inproto. - if
objandprotoare sequences,objmatchesprotoif they're the same length and(a matches b)for every(a,b)inzip(obj, proto). - otherwise,
objmatchesprotoifobj == proto. simple as that ๐
Precondition: proto must not have any reference cycles. fr fr, don't do that to it ๐
๐ง spack.ci_optimization.optimizer(yaml)
That's the whole entry, bestie โ the signature is the vibe ๐ฆ
๐ spack.ci_optimization.print_delta(name, old, new, applied=None)
Also just vibing with its signature, no description attached โจ
โญ spack.ci_optimization.sort_yaml_obj(obj)
Same deal โ that's the full listing, no extra lore ๐
๐ spack.ci_optimization.subkeys(obj, proto)
Returns the test mapping obj after factoring out the items it has in common with the prototype mapping proto. okay so here's the definition, pay attention ๐ซถ
Consider a recursive merge operation, merge(a, b), on mappings a and b, that returns a mapping m whose keys are the union of the keys of a and b, and for every such key k, its corresponding value is:
merge(a[key], b[key])ifa[key]andb[key]are mappings, orb[key]if(key in b)and notmatches(a[key], b[key]), ora[key]otherwise
If obj and proto are mappings, the returned object is the smallest object a such that merge(a, proto) matches obj.
Otherwise, obj is returned. periodt ๐
๐ spack.ci_optimization.try_optimization_pass(name, yaml, optimization_pass, *args, **kwargs)
Try applying an optimization pass and return info about how it went. lowkey the most useful one here:
name is a string describing the nature of the pass. If it's a non-empty string, summary statistics also get printed to stdout. bonus content โจ
yaml is the object you're applying the pass to.
optimization_pass is the function implementing the pass to be applied.
args and kwargs are the additional arguments passed to the optimization pass. The pass is applied like:
>>> (new_yaml, *other_results) = optimization_pass(yaml, *args, **kwargs)And here's the guardrail: the pass's results are greedily rejected if it doesn't modify the original yaml document, or if it produces a yaml document that serializes to a larger string. no bloat allowed ๐ค
Returns (new_yaml, yaml, applied, other_results) if applied, or (yaml, new_yaml, applied, other_results) otherwise. note the swap, bestie, the swap matters ๐
๐ฅ spack.compiler module
๐ class spack.compiler.Compiler(cspec, operating_system, target, paths, modules=None, alias=None, environment=None, extra_rpaths=None, enable_implicit_rpaths=None, **kwargs)
Bases: object
This class encapsulates a Spack "compiler", which includes C, C++, and Fortran compilers. Subclasses should implement support for specific compilers, their possible names, arguments, and how to identify the particular type of compiler. main character class energy fr ๐
PrgEnv: str | None = NonePrgEnv_compiler: str | None = None- property
c11_flag - property
c99_flag cc_names: List[str] = []- property
cc_pic_flagโ Returns the flag used by the C compiler to produce Position Independent Code (PIC). - property
cc_rpath_arg - classmethod
cc_version(cc) compiler_environment()- property
compiler_verbose_output: str | Noneโ Verbose output from compiling a dummy C source file. Output is cached. - property
cxx11_flag - property
cxx14_flag - property
cxx17_flag - property
cxx98_flag cxx_names: List[str] = []- property
cxx_pic_flagโ Returns the flag used by the C++ compiler to produce Position Independent Code (PIC). - property
cxx_rpath_arg - classmethod
cxx_version(cxx) - property
debug_flags - property
default_libc: Spec | Noneโ Determine libc targeted by the compiler from the link line. - classmethod
default_version(cc)โ Override just this to override all compiler version functions. one override to rule them all ๐ - property
disable_new_dtags - property
enable_new_dtags - classmethod
extract_version_from_output(output)โ Extracts the version from the compiler's output. f77_names: List[str] = []- property
f77_pic_flagโ Returns the flag used by the F77 compiler to produce Position Independent Code (PIC). - property
f77_rpath_arg - classmethod
f77_version(f77) fc_names: List[str] = []- property
fc_pic_flagโ Returns the flag used by the FC compiler to produce Position Independent Code (PIC). - property
fc_rpath_arg - classmethod
fc_version(fc) get_real_version()โ Query the compiler for its version.
This is the "real" compiler version, regardless of what's in the compilers.yaml file, which the user can change to name their compiler. (yes, users rename compilers, it's a whole thing ๐ซ )
It uses the runtime environment of the compiler (modules and environment modifications) to enable the compiler to run properly on any platform. considerate king behavior ๐
ignore_version_errors: Sequence[int] = ()โ Return values to ignore when invoking the compiler to get its version.implicit_rpaths() -> List[str]is_supported_on_platform()โ Platform matcher for Platform objects supported by the compiler.- property
linker_argโ Flag that needs to be used to pass an argument to the linker. - property
openmp_flag - property
opt_flags - property
prefixโ Query the compiler for its install prefix. This is the install path as reported by the compiler. Note that paths forcc,cxx, etc. are not enough to find the install prefix of the compiler, since they can be symlinks, wrappers, or filenames instead of absolute paths. sneaky, sneaky ๐ญ prefixes: List[str] = []- property
real_versionโ Executable-reported compiler version used for API-determinations. E.g. C++11 flag checks. - property
required_libsโ For executables created with this compiler, the compiler libraries that would be generally required to run it. - classmethod
search_regexps(language) setup_custom_environment(pkg, env)โ Set any environment variables necessary to use the compiler.suffixes = ['-.*']- property
verbose_flag: str | Noneโ This property should be overridden in the compiler subclass if a verbose flag is available.
If it's not overridden, it's assumed to not be supported. no override = no vibes, apparently ๐
verify_executables()โ Raise an error if any of the compiler executables is not valid.
This method confirms that for all of the compilers (cc, cxx, f77, fc) that have paths, those paths exist and are executable by the current user. It raises a CompilerAccessError if any of the non-null paths for the compiler are not accessible. ๐จ
- property
version version_argument = '-dumpversion'โ Compiler argument that produces version information.version_regex = '(.*)'โ Regex used to extract the version from the compiler's output.
๐ spack.concretize module
Functions here are used to take abstract specs and make them concrete. For example, if a spec asks for a version between 1.8 and 1.9, these functions might take the most recent 1.9 version of the package available. Or, if the user didn't specify a compiler for a spec, then this will assign a compiler to the spec based on defaults or user preferences. it's giving "make it make sense" energy โจ
TODO: make this customizable and allow users to configure concretization policies. (future us problem, bestie ๐)
๐ class spack.concretize.Concretizer(abstract_spec=None)
Bases: object
You can subclass this class to override some of the default concretization strategies, or you can override all of them. your choice, your rules ๐
adjust_target(spec)โ Adjusts the target microarchitecture if the compiler is too old to support the default one.
Parameters: spec โ spec to be concretized.
Returns: True if spec was modified, False otherwise.
check_for_compiler_existence = Noneโ Controls whether we check that compiler versions actually exist during concretization. Used for testing and for mirror creation. ๐ชchoose_virtual_or_external(spec: Spec)โ Given a list of candidate virtual and external packages, try to find one that is most ABI compatible. the matchmaker of the concrete world ๐ซถconcretize_architecture(spec)โ if the spec is empty, just provide the defaults of the platform. If the architecture is not a string type, then check whether the platform, target, or operating system are concretized. If any of the fields got changed, returnTrue. If everything is concretized (i.e. the architecture attribute is a namedtuple of classes) then returnFalse. If the target is a string type, convert that string into a concretized architecture. And if it has no architecture but the root of the DAG does have one, use the root โ otherwise fall back to the defaults on the platform. ๐ซถconcretize_compiler(spec)โ if the spec already has a compiler we're done, periodt. If not, take the compiler used for the nearest ancestor that has a compiler spec and use that one. If the ancestor's compiler is not concrete, then use the preferred compiler as specified inspackconfig. โจ- The intuition, iykyk: use the
spackconfigdefault if no package that depends on this one has a strict compiler requirement. Otherwise, try to build with the compiler that will be used by libraries that link to this one, to maximize compatibility.
- The intuition, iykyk: use the
concretize_compiler_flags(spec)โ the compiler flags get updated to match those of the spec whose compiler is used, defaulting to no compiler flags in the spec. Default specs set at the compiler level will still be added later. ๐concretize_develop(spec)โ adds thedev_path=*variant to packages built from local source.concretize_variants(spec)โ if the spec already has variants filled in, return. Otherwise, add the user preferences frompackages.yamlor the default variants from the package specification. ๐concretize_version(spec)โ if the spec is already concrete, return. Otherwise take the preferred version fromspackconfig, and default to the package's version if there are no available versions. ๐ฆ- TODO: In many cases we probably want to look for installed versions of each package and use an installed version if we can link to it. The policy implemented here will tend to rebuild a lot of stuff becasue it will prefer a compiler in the spec to any compiler already- installed things were built with. There is likely some better policy that finds some middle ground between these two extremes. ๐ญ (yes bestie, that's a real TODO and it's giving "chaos but make it documented")
non_buildable_packages = {'glibc', 'musl'}โ packages that the old concretizer cannot deal with correctly, and cannot build anyway. Those will not be considered as providers for virtuals. ๐target_from_package_preferences(spec)โ returns the preferred target from the package preferences if there's any.- Parameters
- spec -- abstract spec to be concretized
- Parameters
exception spack.concretize.InsufficientArchitectureInfoError(spec, archs)โ Bases:SpackError. Raised when details on architecture cannot be collected from the system. ๐exception spack.concretize.NoBuildError(spec)โ Bases:SpecError. Raised when a package is configured with thebuildableoptionFalse, but no satisfactory external versions can be found.exception spack.concretize.NoCompilersForArchError(arch, available_os_targets)โ Bases:SpackError. (that's literally all it says, no cap)exception spack.concretize.NoValidVersionError(spec)โ Bases:SpackError. Raised when there is no way to have a concrete version for a particular spec.exception spack.concretize.UnavailableCompilerVersionError(compiler_spec, arch=None)โ Bases:SpackError. Raised when there is no available compiler that satisfies a compiler spec.spack.concretize.concretize_specs_together(*abstract_specs, **kwargs)โ given a number of specs as input, tries to concretize them together. โจ- Parameters
tests(boolorlistorset) --Falseto run no tests,Trueto test all packages, or a list of package names to run tests for some*abstract_specs-- abstract specs to be concretized, given either asSpecsor strings
- Returns: List of concretized specs
- Parameters
spack.concretize.disable_compiler_existence_check()spack.concretize.enable_compiler_existence_check()spack.concretize.find_spec(spec, condition, default=None)โ searches the dag from spec in an intelligent order and looks for a spec that matches a condition. ๐class spack.concretize.reverse_order(value)โ Bases:object. Helper for creating key functions. This is a wrapper that inverts the sense of the natural comparisons on the object. ๐
๐ spack.config module ๐
This module implements Spack's configuration file handling. No because literally, this is the whole config brain of Spack, bestie. ๐ง โจ
It implements Spack's configuration system, which handles merging multiple scopes with different levels of precedence. See the documentation on Configuration Scopes for details on how Spack's configuration system behaves. The scopes are:
defaultsystemsiteuser
And corresponding per-platform scopes. ๐
The important functions in this module are:
get_config()update_config()
get_config reads in YAML data for a particular scope and returns it. Callers can then modify the data and write it back with update_config. ๐
When read in, Spack validates configurations with jsonschemas. The schemas are in submodules of spack.schema. ๐
spack.config.COMMAND_LINE_SCOPES: List[str] = []โ configuration scopes added on the command line, set byspack.main.main()spack.config.CONFIG: Configuration | Singleton = <spack.config.Configuration object>โ this is the singleton configuration instance for Spack. โจspack.config.CONFIGURATION_DEFAULTS_PATHโ
('defaults', '/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/etc/spack/defaults')Path to the default configuration.
spack.config.CONFIG_DEFAULTSโ
{'config': {'build_jobs': 4, 'build_stage': '$tempdir/spack-stage', 'checksum': True, 'concretizer': 'clingo', 'connect_timeout': 10, 'debug': False, 'dirty': False, 'license_dir': '/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/etc/spack/licenses', 'verify_ssl': True}}Hard-coded default values for some key configuration options. This ensures that Spack will still work even if config.yaml in the defaults scope is removed. ๐ซถ
exception spack.config.ConfigError(message, long_message=None)โ Bases:ConfigErrorโฆ jk, Bases:SpackError. Superclass for all Spack config related errors. (the mama of all config errors, honestly)exception spack.config.ConfigFileError(message, long_message=None)โ Bases:ConfigError. Issue reading or accessing a configuration file.exception spack.config.ConfigFormatError(validation_error, data: Dict[str, Any], filename: str | None = None, line: int | None = None)โ Bases:ConfigError. Raised when a configuration format does not match its schema. ๐class spack.config.ConfigPathโ Bases:object.elementโ
'(?:(?:(?:"[^"]+")|(?:\'[^\']+\'))|(?:[^:\'"]+))'next_key_patternโ
'((?:(?:(?:"[^"]+")|(?:\'[^\']+\'))|(?:[^:\'"]+))[+-]?)(?:\\:|$)'static process(path)quoted_stringโ
'(?:"[^"]+")|(?:\'[^\']+\')'unquoted_stringโ
'[^:\'"]+'class spack.config.ConfigScope(name, path)โ Bases:object. This class represents a configuration scope. A scope is one directory containing named configuration files. Each file is a config "section" (e.g.,mirrors,compilers, etc.). ๐๏ธclear() -> Noneโ empty cached config information.get_section(section: str) -> Dict[str, Any] | Noneโ returns the data associated with a given sectionget_section_filename(section: str) -> strโ returns the filename associated with a given sectionproperty is_platform_dependent: boolโ returns true if the scope name is platform specific
exception spack.config.ConfigSectionError(message, long_message=None)โ Bases:ConfigError. Error for referring to a bad config section name in a configuration. ๐class spack.config.Configuration(*scopes: ConfigScope)โ Bases:object. A full Spack configuration, from a hierarchy of config files. This class makes it easy to add a new scope on top of an existing one. ๐ธclear_caches() -> Noneโ clears the caches for configuration files. This will cause files to be re-read upon the next request.property file_scopes: List[ConfigScope]โ list of writable scopes with an associated file.get(path: str, default: Any | None = None, scope: str | None = None) -> Anyโ get a config section or a single value from one. Accepts a path syntax that allows us to grab nested config map entries. Getting the'config'section would look like:
spack.config.get('config')and the dirty section in the config scope would be:
spack.config.get('config:dirty')We use : as the separator, like YAML objects. iykyk ๐
get_config(section: str, scope: str | None = None) -> Dict[str, Any]โ get configuration settings for a section. If scope isNoneor not provided, return the merged contents of all of Spack's configuration scopes. If scope is provided, return only the configuration as specified in that scope. This off the top-level name from the YAML section. That is, for a YAML config file that looks like this:
config:
install_tree:
root: $spack/opt/spack
build_stage:
- $tmpdir/$user/spack-stageget_config('config') will return:
{ 'install_tree': {
'root': '$spack/opt/spack',
}
'build_stage': ['$tmpdir/$user/spack-stage']
}get_config_filename(scope: str, section: str) -> str
Okay so you hand it a scope and a section and it hands you back the name of the configuration file for that combo. That's it. That's the whole job. Slay, honestly ๐
highest_precedence_non_platform_scope() -> ConfigScope
Gives you the non-internal, non-platform scope with the highest precedence. And just so we're clear on the terminology, bestie: platform-specific scopes are shaped like scope/platform. ๐ซถ
highest_precedence_scope() -> ConfigScope
Same vibe but a lil broader โ this one gives you the non-internal scope with the highest precedence, platform or not. ๐
matching_scopes(reg_expr) -> List[ConfigScope]
Returns a List[ConfigScope] of every scope whose name matches the regex you passed in. โจ
Example so it actually clicks: matching_scopes(r'^command') will return all scopes whose names begin with command. See? Iconic.
pop_scope() -> ConfigScope
Removes the highest precedence scope and returns it. Like popping off a stack, because that's literally what it is ๐
print_section(section: str, blame: bool = False, *, scope=None) -> None
Prints a configuration to stdout. You've got a section, an optional blame (defaults to False), and a keyword-only scope (defaults to None). Easy.
push_scope(scope: ConfigScope) -> None
Adds a higher precedence scope to the Configuration. Popping's counterpart, if you will. Balance. ๐ธ
remove_scope(scope_name: str) -> ConfigScope | None
Removes a scope by name. And here's the important deet: it has no effect when scope_name does not exist. So it's not gonna yell at you, it's just gonna do nothing. Very chill of it. ๐
scopes: Dict[str, ConfigScope]
That's the attribute, bestie. A Dict[str, ConfigScope]. It's just... sitting there, holding all your scopes. Not a method, don't @ me.
set(path: str, value: Any, scope: str | None = None) -> None
A convenience function for setting single values in config files. ๐
It accepts the path syntax described in get(). So if you know how get() paths work, you already know how this works. iykyk.
update_config(section: str, update_data: Dict, scope: str | None = None, force: bool = False) -> None
Updates the configuration file for a particular scope. Let's break it all the way down:
It overwrites the contents of a section in a scope with update_data, then writes out the config file. ๐ฅ
update_data should have the top-level section name stripped off โ don't worry, it'll be re-added for you. And the data itself can be a list, a dict, or literally any other yaml-ish structure. Flexible queen. ๐
โ ๏ธ Heads up though: configuration scopes that are still written in an old schema format will fail to update unlessforceisTrue. Not a drill, fr fr.
Parameters
sectionโ section of the configuration to be updatedupdate_dataโ data to be used for the updatescopeโ scope to be updatedforceโ force the update
class spack.config.ImmutableConfigScope(name, path)
Bases: ConfigScope
A configuration scope that cannot be written to. Read-only energy. ๐ซ
It's used for ConfigScopes passed on the command line. So when you type stuff at the prompt, this is the vibe it gets wrapped in. ๐ฆ
class spack.config.InternalConfigScope(name: str, data: Dict[str, Any] | None = None)
Bases: ConfigScope
An internal configuration scope that is not persisted to a file. It lives in memory and that's that. ๐ญ
Why does this exist? Because it's for spack internal use โ so that command-line options and config file settings are accessed the same way, and Spack can easily override settings from files. Clever, honestly. ๐
clear() -> None
Empties out the cached config information. Poof. ๐ช
get_section(section: str) -> Dict[str, Any] | None
Just reads from an internal dictionary. That's the whole description, and honestly? Respect for the brevity. ๐
get_section_filename(section: str) -> str
Returns the filename associated with a given section. ๐
spack.config.SCOPES_METAVAR = '{defaults,system,site,user,command_line}[/PLATFORM] or env:ENVIRONMENT'
This is the metavar to use for commands that accept scopes โ and the reason it exists is lowkey smart: it's shorter and more readable than listing out all the choices. Your terminal thanks you. ๐
okay so we are still deep in spack.config.SECTION_SCHEMAS bestie, and this is the exact moment the page decides to just dump the entire annotated value on us ๐ญ it's typed Dict[str, Any], and the value is one giant Python dict whose top-level keys are 'bootstrap', 'cdash', and 'ci' โ each one a whole JSON Schema wearing '$schema': 'http://json-schema.org/draft-07/schema#' and 'additionalProperties': False like it's a little outfit. no cap, this is it, character for character ๐
spack.config.SECTION_SCHEMAS: Dict[str, Any] = {'bootstrap': {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'bootstrap': {'properties': {'enable': {'type':
'boolean'}, 'root': {'type': 'string'}, 'sources': {'items':
{'additionalProperties': False, 'properties': {'metadata': {'type':
'string'}, 'name': {'type': 'string'}}, 'required': ['name',
'metadata'], 'type': 'object'}, 'type': 'array'}, 'trusted':
{'patternProperties': {'\\w[\\w-]*': {'type': 'boolean'}}, 'type':
'object'}}, 'type': 'object'}}, 'title': 'Spack bootstrap configuration
file schema', 'type': 'object'}, 'cdash': {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'cdash': {'additionalProperties': False,
'patternProperties': {'build-group': {'type': 'string'}, 'project':
{'type': 'string'}, 'site': {'type': 'string'}, 'url': {'type':
'string'}}, 'required': ['build-group'], 'type': 'object'}}, 'title':
'Spack cdash configuration file schema', 'type': 'object'}, 'ci':
{'$schema': 'http://json-schema.org/draft-07/schema#',
'additionalProperties': False, 'properties': {'ci': {'oneOf': [{'anyOf':
[{'additionalProperties': False, 'properties': {'broken-specs-url':
{'type': 'string'}, 'broken-tests-packages': {'items': {'type':
'string'}, 'type': 'array'}, 'enable-artifacts-buildcache': {'type':
'boolean'}, 'pipeline-gen': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'submapping': {'items': {'additionalProperties': False, 'properties':
{'build-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'build-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'match':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['match'],
'type': 'object'}, 'type': 'array'}}, 'required': ['submapping'],
'type': 'object'}, {'oneOf': [{'additionalProperties': False,
'properties': {'noop-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'noop-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type':
'object'}, {'additionalProperties': False, 'properties': {'build-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'build-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'copy-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'copy-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'reindex-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'reindex-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'signing-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'signing-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':okay breathe ๐ซถ that was a lot, and bestie it doesn't even close โ the dump literally cuts off mid 'signing-job-remove', right after 'image': ๐ that's just where this part of the page ends, not me skipping bits.
๐ธ what you actually just looked at
'bootstrap'โ schema with a'bootstrap'object holding'enable'('type': 'boolean'),'root'('string'),'sources'(anarraywhose items require both'name'and'metadata', each a'string', via'required': ['name', 'metadata']), and'trusted'(anobjectwhose keys match'\\w[\\w-]*'and map to'type': 'boolean'). Title:'Spack bootstrap configuration file schema'โจ'cdash'โ'additionalProperties': False, a'patternProperties'block with'build-group','project','site'and'url'all typed'string', and'required': ['build-group']. Title:'Spack cdash configuration file schema'๐'ci'โ the main character of this chunk ๐ also'additionalProperties': False, and its'ci'property is a'oneOf'wrapping an'anyOf'. Nesting dolls, honestly.'broken-specs-url'โ'string''broken-tests-packages'โ'array'of'string''enable-artifacts-buildcache'โ'boolean''pipeline-gen'โ'array', and its'items'are a'oneOf'(this is where it gets spicy ๐ฅ)
๐ฆ the 'pipeline-gen' items, decoded
- branch one โ
'match_behavior'is a'string'with'default': 'first'and'enum': ['first', 'merge'], plus'submapping'(an'array') which is required ('required': ['submapping']). Each submapping item is an object with'build-job','build-job-remove', and'match', and'match'is the required one ('required': ['match']) โ an'array'of'string'๐ - branch two โ itself a
'oneOf'of job objects, each one a'type': 'object':'noop-job'+'noop-job-remove''build-job'+'build-job-remove''copy-job'+'copy-job-remove''reindex-job'+'reindex-job-remove''signing-job'+'signing-job-remove'
๐ง every single job block is the same girl, different font
no fr, they're copy-paste twins with 'additionalProperties': True and these keys:
'after_script'โ'type': 'array', items are'anyOf'a'string'or an'array'of'string''before_script'โ same deal,'anyOf''string'/'array'of'string''image'โ'oneOf'a plain'string'or an'object'with'entrypoint'(an'array'of'string') and'name'('string')'script'โ'anyOf''string'/'array'of'string''tags'โ'array'of'string''variables'โ an'object'with'patternProperties'keyed by'[\\w\\d\\-_\\.]+', each value a'string'
and the -remove versions are the exact same shape, lowkey iconic for how committed they are to the bit ๐ญ the only difference you'll spot is that some of them carry 'additionalProperties': True and the wrapper objects use 'additionalProperties': False โ read it carefully, bestie, because the schema is not gonna forgive a vibe-based guess ๐
heads up: 'additionalProperties': False is not decorative โ in the places it appears, extra keys are just not allowed. don't freeball it.
and that's the tea on this slice โ we're mid-signing-job-remove, mid-'image', right as the property list is about to spill over. hold that thought ๐
๐ง schema hours, continued (no thoughts, just JSON)
okay bestie we are DEEP in the schema mines now and honestly? this part is pure structure. it's the JSON-schema equivalent of a group chat where everyone keeps forwarding the same message ๐
what you're looking at is a whole pile of oneOf / anyOf branches, and every branch is an object with additionalProperties: False โ meaning the validator is like "no extra keys, i don't know her." and inside basically every branch you get the same little job-shaped creature repeated over and over: after_script, before_script, image, script, tags, variables. like genuinely the same six keys, copy-pasted with love ๐ธ
here's the chunk, verbatim, no cap:
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}},
{'additionalProperties': False, 'properties': {'cleanup-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'cleanup-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'any-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'any-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'}]}]}, 'type': 'array'},
'rebuild-index': {'type': 'boolean'}, 'target': {'default': 'gitlab',
'enum': ['gitlab'], 'type': 'string'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'broken-specs-url':
{'type': 'string'}, 'broken-tests-packages': {'items': {'type':
'string'}, 'type': 'array'}, 'pipeline-gen': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'submapping': {'items': {'additionalProperties': False, 'properties':
{'build-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'build-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}, 'match':
{'items': {'type': 'string'}, 'type': 'array'}}, 'required': ['match'],
'type': 'object'}, 'type': 'array'}}, 'required': ['submapping'],
'type': 'object'}, {'oneOf': [{'additionalProperties': False,
'properties': {'noop-job': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}, 'noop-job-remove':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'}}, 'type':
'object'}, {'additionalProperties': False, 'properties': {'build-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'build-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'copy-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'copy-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':okay so let me annotate the โจloreโจ because your girl did read all of this and she has notes ๐
๐ the job shape (the recurring main character)
every time you see a job block, it's the same six keys, and they all mean the same thing every time:
after_scriptโ an array; each item is either astringor anarrayofstrings ๐ธbefore_scriptโ same deal: array of (string| array ofstring)imageโoneOftwo things: a plainstring, or an object withentrypoint(array ofstrings) andname(string)scriptโ array of (string| array ofstring), iconic behaviour honestlytagsโ array ofstrings, no dramavariablesโ an object where the keys are matched by the pattern[\\w\\d\\-_\\.]+and every value is astring
bestie notice the pattern:[\\w\\d\\-_\\.]+โ word chars, digits,-,_,.. that's the whole vibe. if your variable name has a space in it, the schema is gonna look at you like ๐ซค
๐ the named job slots
then we get branches that are objects with additionalProperties: False (so, only these keys) each containing one or two named jobs, and each job has additionalProperties: True (so, extra keys are chillin'):
cleanup-jobandcleanup-job-removeany-jobandany-job-removebuild-jobandbuild-job-removecopy-jobandcopy-job-removenoop-jobandnoop-job-remove
the -remove variants carry the exact same properties block โ same after_script, before_script, image, script, tags, variables. the way that a schema dev said "just copy it" and honestly? valid, it works ๐ซถ
๐ฅ pipeline-gen and friends
top-level-wise in this chunk we've got:
broken-specs-urlโ astring, that's itbroken-tests-packagesโ anarrayofstringspipeline-genโ anarray; each item is aoneOf:- a first flavour:
additionalProperties: False, withmatch_behavior(astring,default: 'first',enum: ['first', 'merge']) andsubmappingโ an array of objects that arerequired: ['match'], wherematchis an array ofstrings, plus optional job slotsbuild-job/build-job-remove(eachadditionalProperties: Truewith the usual six keys). the whole first flavour isrequired: ['submapping']โ - a second flavour: another
oneOfof objects that just hold job slots โnoop-job/noop-job-removein one,build-job/build-job-removein another,copy-job/copy-job-removein the next ๐ฆ
- a first flavour:
rebuild-indexโ aboolean, simple queentargetโ astring,default: 'gitlab', and theenumis literally just['gitlab']โ one (1) option and she's owning it ๐
and that's the schema chunk, no notes. stay tuned because the copy-job-remove block is still mid-sentence and we are absolutely not done ๐
okay so we're literally still inside the schema, mid-anyOf, no biggie ๐
โ and if you've been following along, you already know the vibe: this whole thing is a giant JSON Schema, and anyOf means "the config has to match at least one of these alternatives." We're picking our way through the list. Let's go ๐ซถ
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'reindex-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'reindex-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'signing-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'signing-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'cleanup-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'cleanup-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'},
{'additionalProperties': False, 'properties': {'any-job':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'anyOf': [{'type': 'string'}, {'items': {'type': 'string'}, 'type':
'array'}]}, 'type': 'array'}, 'before_script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'anyOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'type': 'object'},
'any-job-remove': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'before_script': {'items': {'anyOf': [{'type': 'string'}, {'items':
{'type': 'string'}, 'type': 'array'}]}, 'type': 'array'}, 'image':
{'oneOf': [{'type': 'string'}, {'properties': {'entrypoint': {'items':
{'type': 'string'}, 'type': 'array'}, 'name': {'type': 'string'}},
'type': 'object'}]}, 'script': {'items': {'anyOf': [{'type': 'string'},
{'items': {'type': 'string'}, 'type': 'array'}]}, 'type': 'array'},
'tags': {'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'type': 'object'}}, 'type': 'object'}]}]}, 'type': 'array'},
'rebuild-index': {'type': 'boolean'}, 'target': {'default': 'gitlab',
'enum': ['gitlab'], 'type': 'string'}, 'temporary-storage-url-prefix':
{'type': 'string'}}, 'type': 'object'}]}, {'anyOf':okay so let's โจdissectโจ this absolute icon of a block, because the pattern is so satisfying once you clock it ๐
Notice the pairs, bestie โ they come in twos the whole way down: reindex-job and reindex-job-remove, signing-job and signing-job-remove, cleanup-job and cleanup-job-remove, any-job and any-job-remove. And I'm not gonna sit here and invent lore about what the -remove ones do, because the schema doesn't tell me and I refuse to make things up ๐ we read what's there, no cap.
Every single one of those job objects is built the same way, and it's giving template energy:
'additionalProperties': Falseon the outer wrapper, but the job object itself says'additionalProperties': Trueโ so inside the job, extra keys are allowed ๐ธafter_script,before_script, andscriptโ each'items'is ananyOfbetween a{'type': 'string'}and an{'items': {'type': 'string'}, 'type': 'array'}. So: a string, or an array of strings. Choose your fighter ๐imageโ aoneOfof{'type': 'string'}or an object withentrypoint(that's'items': {'type': 'string'}, 'type': 'array') andname({'type': 'string'})tagsโ array of stringsvariablesโ an object whosepatternPropertieskey is the regex[\\w\\d\\-_\\.]+, and every matching key has to be a'string'๐ง
Then the tail end of this alternative adds three friends: 'rebuild-index': {'type': 'boolean'}, 'target' which has 'default': 'gitlab', 'enum': ['gitlab'] and 'type': 'string', and 'temporary-storage-url-prefix': {'type': 'string'} โญ
And that last line โ ..., 'type': 'object'}]}, {'anyOf': โ is the outer anyOf shutting one alternative and opening the next one. Which is where we're headed:
[{'additionalProperties': False, 'properties': {'after_script':
{'items': {'type': 'string'}, 'type': 'array'}, 'before_script':
{'items': {'type': 'string'}, 'type': 'array'}, 'bootstrap': {'items':
{'anyOf': [{'type': 'string'}, {'additionalProperties': False,
'properties': {'compiler-agnostic': {'default': False, 'type':
'boolean'}, 'name': {'type': 'string'}}, 'required': ['name'], 'type':
'object'}]}, 'type': 'array'}, 'broken-specs-url': {'type': 'string'},
'broken-tests-packages': {'items': {'type': 'string'}, 'type': 'array'},
'enable-artifacts-buildcache': {'type': 'boolean'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items': {'type':
'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type': 'string'},
{'properties': {'entrypoint': {'items': {'type': 'string'}, 'type':
'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]}, 'script':
{'items': {'type': 'string'}, 'type': 'array'}, 'tags': {'items':
{'type': 'string'}, 'type': 'array'}, 'variables': {'patternProperties':
{'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type': 'object'}}, 'required':
['tags'], 'type': 'object'}}, 'required': ['match'], 'type': 'object'},
'type': 'array'}, 'match_behavior': {'default': 'first', 'enum': ['first',
'merge'], 'type': 'string'}, 'rebuild-index': {'type': 'boolean'},
'script': {'items': {'type': 'string'}, 'type': 'array'},
'service-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'script': {'items': {'type': 'string'},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'signing-job-attributes': {'additionalProperties': True,
'properties': {'after_script': {'items': {'type': 'string'}, 'type':
'array'}, 'before_script': {'items': {'type': 'string'}, 'type':
'array'}, 'image': {'oneOf': [{'type': 'string'}, {'properties':
{'entrypoint': {'items': {'type': 'string'}, 'type': 'array'}, 'name':
{'type': 'string'}}, 'type': 'object'}]}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'},
'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+': {'type':
'string'}}, 'type': 'object'}}, 'required': ['mappings'], 'type':
'object'}, {'additionalProperties': False, 'properties':okay this alternative ate and left no crumbs ๐ฅ watch the vibe shift: after_script and before_script here are plain {'items': {'type': 'string'}, 'type': 'array'} โ no anyOf string-or-array situation like we had upstairs. Different block, different rules, that's just how it is ๐
Quick tour of the new keys in this one:
bootstrapโ array whose items are ananyOfof{'type': 'string'}or an object withcompiler-agnostic('default': False,'type': 'boolean') andname({'type': 'string'}), and that object has'required': ['name']๐broken-specs-urlโ{'type': 'string'}broken-tests-packagesโ array of stringsenable-artifacts-buildcacheโ{'type': 'boolean'}imageโ same string-or-objectoneOfas before, withentrypointandnamemappingsโ an array of objects, each with'additionalProperties': Falseand'required': ['match']. Inside each mapping:matchโ array of stringsremove-attributesโ object,'additionalProperties': False,'required': ['tags'], andtagsis an array of stringsrunner-attributesโ object with'additionalProperties': True,'required': ['tags'], and the full job-ish spread:after_script,before_script(both arrays of strings),image(the sameoneOf),script(array of strings),tags(array of strings),variables(the[\\w\\d\\-_\\.]+patternProperties object of strings)
match_behaviorโ'default': 'first','enum': ['first', 'merge'],'type': 'string'๐ฆrebuild-indexโ{'type': 'boolean'}scriptโ array of stringsservice-job-attributesโ'additionalProperties': True,'required': ['tags'], withafter_script,before_script,image,script,tags,variablesall as abovesigning-job-attributesโ same deal,'additionalProperties': True,'required': ['tags'], same six keystagsโ array of stringsvariablesโ same patternProperties situation
and the whole object bottoms out with 'required': ['mappings'] โ like, if you pick this alternative, mappings is not optional, periodt ๐
Then it wraps up with 'type': 'object'}, {'additionalProperties': False, 'properties': and we're off into yet another alternative, because of course we are ๐ญ
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'bootstrap': {'items': {'anyOf': [{'type': 'string'},
{'additionalProperties': False, 'properties': {'compiler-agnostic':
{'default': False, 'type': 'boolean'}, 'name': {'type': 'string'}},
'required': ['name'], 'type': 'object'}]}, 'type': 'array'},
'broken-specs-url': {'type': 'string'}, 'broken-tests-packages':
{'items': {'type': 'string'}, 'type': 'array'}, 'image': {'oneOf':
[{'type': 'string'}, {'properties': {'entrypoint': {'items': {'type':
'string'}, 'type': 'array'}, 'name': {'type': 'string'}}, 'type':and that's exactly where this part runs out of page ๐ so real talk about what we just saw: this third alternative opens with after_script and before_script as plain arrays of strings again, then bootstrap (string-or-object anyOf, compiler-agnostic defaulting to False, name required), then broken-specs-url, broken-tests-packages, and image โ and the image object is literally cut off mid-{'type': because the page ends. No enable-artifacts-buildcache in this one either, and no mappings yet โ we don't know what comes next, so no spoilers, iykyk ๐
the schema stays undefeated, bestie โ catch you in the next chunk ๐
โฆokay so we were deep in the ci schema and honestly? it was serving. Let's finish that one and then meet the whole config-file family, because every single Spack YAML file has a JSON Schema riding shotgun and you're about to meet them all ๐
๐ง the rest of ci (Spack CI configuration file schema)
'object'}]}, 'mappings': {'items': {'additionalProperties': False,
'properties': {'match': {'items': {'type': 'string'}, 'type': 'array'},
'remove-attributes': {'additionalProperties': False, 'properties':
{'tags': {'items': {'type': 'string'}, 'type': 'array'}}, 'required':
['tags'], 'type': 'object'}, 'runner-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'}}, 'required':
['match'], 'type': 'object'}, 'type': 'array'}, 'match_behavior':
{'default': 'first', 'enum': ['first', 'merge'], 'type': 'string'},
'rebuild-index': {'type': 'boolean'}, 'script': {'items': {'type':
'string'}, 'type': 'array'}, 'service-job-attributes':
{'additionalProperties': True, 'properties': {'after_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'before_script': {'items':
{'type': 'string'}, 'type': 'array'}, 'image': {'oneOf': [{'type':
'string'}, {'properties': {'entrypoint': {'items': {'type': 'string'},
'type': 'array'}, 'name': {'type': 'string'}}, 'type': 'object'}]},
'script': {'items': {'type': 'string'}, 'type': 'array'}, 'tags':
{'items': {'type': 'string'}, 'type': 'array'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['tags'], 'type': 'object'},
'signing-job-attributes': {'additionalProperties': True, 'properties':
{'after_script': {'items': {'type': 'string'}, 'type': 'array'},
'before_script': {'items': {'type': 'string'}, 'type': 'array'},
'image': {'oneOf': [{'type': 'string'}, {'properties': {'entrypoint':
{'items': {'type': 'string'}, 'type': 'array'}, 'name': {'type':
'string'}}, 'type': 'object'}]}, 'script': {'items': {'type': 'string'},
'type': 'array'}, 'tags': {'items': {'type': 'string'}, 'type':
'array'}, 'variables': {'patternProperties': {'[\\w\\d\\-_\\.]+':
{'type': 'string'}}, 'type': 'object'}}, 'required': ['tags'], 'type':
'object'}, 'tags': {'items': {'type': 'string'}, 'type': 'array'},
'temporary-storage-url-prefix': {'type': 'string'}, 'variables':
{'patternProperties': {'[\\w\\d\\-_\\.]+': {'type': 'string'}}, 'type':
'object'}}, 'required': ['mappings'], 'type': 'object'}]}]}}, 'title':
'Spack CI configuration file schema', 'type': 'object'},So like, what's it actually saying? Bestie, it's a whole mood:
mappingsโ the one thing that's required at the top level, an array of mapping objects. Nomappings, no schema, periodt.- Each mapping requires
match(an array of strings โ your spec matchers, iykyk). remove-attributesโ an object that requirestags(array of strings), nothing else allowed.runner-attributesโadditionalPropertiesistrue(go off, add whatever), and inside:after_script,before_script,script,tagsare arrays of strings,imageis either a plain string or an object withentrypoint(array of strings) andname(string), andvariablesis an object whose keys match[\\w\\d\\-_\\.]+and whose values are strings. It requirestags.match_behaviorโ defaults to'first', enum is['first', 'merge'], string. The icon of "pick one or blend them".rebuild-indexโ boolean, no notes.scriptโ array of strings, top level.service-job-attributesandsigning-job-attributesโ same shape asrunner-attributes, sameadditionalProperties: true, same required['tags'].tagsโ array of strings, top level.temporary-storage-url-prefixโ string.variablesโ top level, keys matching[\\w\\d\\-_\\.]+, string values.- Top level also has
additionalProperties: False, so no sneaky extra keys ๐ธ
๐ ๏ธ compilers โ Spack compiler configuration file schema
'compilers': {'$schema': 'http://json-schema.org/draft-07/schema#',
'additionalProperties': False, 'properties': {'compilers': {'items':
{'additionalProperties': False, 'properties': {'compiler':
{'additionalProperties': False, 'properties': {'alias': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'environment':
{'additionalProperties': False, 'default': {}, 'properties':
{'append_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'extra_rpaths': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}, 'flags': {'additionalProperties': False, 'properties':
{'cflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]},
'cppflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]},
'cxxflags': {'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'fflags':
{'anyOf': [{'type': 'string'}, {'type': 'null'}]}, 'ldflags': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'ldlibs': {'anyOf': [{'type':
'string'}, {'type': 'null'}]}}, 'type': 'object'}, 'implicit_rpaths':
{'anyOf': [{'items': {'type': 'string'}, 'type': 'array'}, {'type':
'boolean'}]}, 'modules': {'anyOf': [{'type': 'string'}, {'type':
'null'}, {'type': 'array'}]}, 'operating_system': {'type': 'string'},
'paths': {'additionalProperties': False, 'properties': {'cc': {'anyOf':
[{'type': 'string'}, {'type': 'null'}]}, 'cxx': {'anyOf': [{'type':
'string'}, {'type': 'null'}]}, 'f77': {'anyOf': [{'type': 'string'},
{'type': 'null'}]}, 'fc': {'anyOf': [{'type': 'string'}, {'type':
'null'}]}}, 'required': ['cc', 'cxx', 'f77', 'fc'], 'type': 'object'},
'spec': {'type': 'string'}, 'target': {'type': 'string'}}, 'required':
['paths', 'spec', 'modules', 'operating_system'], 'type': 'object'}},
'type': 'object'}, 'type': 'array'}}, 'title': 'Spack compiler
configuration file schema', 'type': 'object'},The tea โ on this one:
- Top-level
compilersis an array of items, eachadditionalProperties: False, each with acompilerobject. - Inside
compiler:aliasis string-or-null.environmentdefaults to{}and holdsappend_path,prepend_path,remove_path,set(each an object whose keys match\\w[\\w-]*with string-or-number values) andunset, which defaults to[]and is an array of string-or-number. AlladditionalProperties: Falseโ no freelancing ๐ extra_rpathsโ defaults to[], array of strings.flagsโ no extras allowed;cflags,cppflags,cxxflags,fflags,ldflags,ldlibs, each string-or-null.implicit_rpathsโ array of strings or a boolean. Choose your fighter.modulesโ string, null, or array. All the options.operating_systemโ string.pathsโ no extras;cc,cxx,f77,fc, each string-or-null, and all four are required.specโ string.targetโ string.- The
compilerobject requires['paths', 'spec', 'modules', 'operating_system']โ those four, no skipping ๐
๐งฉ concretizer โ Spack concretizer configuration file schema
'concretizer': {'$schema': 'http://json-schema.org/draft-07/schema#',
'additionalProperties': False, 'properties': {'concretizer':
{'additionalProperties': False, 'properties': {'duplicates':
{'properties': {'strategy': {'enum': ['none', 'minimal', 'full'],
'type': 'string'}}, 'type': 'object'}, 'enable_node_namespace': {'type':
'boolean'}, 'os_compatible': {'additionalProperties': {'type': 'array'},
'type': 'object'}, 'reuse': {'oneOf': [{'type': 'boolean'}, {'enum':
['dependencies'], 'type': 'string'}, {'properties': {'exclude':
{'items': {'type': 'string'}, 'type': 'array'}, 'from': {'items':
{'properties': {'exclude': {'items': {'type': 'string'}, 'type':
'array'}, 'include': {'items': {'type': 'string'}, 'type': 'array'},
'type': {'enum': ['local', 'buildcache', 'external'], 'type':
'string'}}, 'type': 'object'}, 'type': 'array'}, 'include': {'items':
{'type': 'string'}, 'type': 'array'}, 'roots': {'type': 'boolean'}},
'type': 'object'}]}, 'targets': {'properties': {'granularity': {'enum':
['generic', 'microarchitectures'], 'type': 'string'}, 'host_compatible':
{'type': 'boolean'}}, 'type': 'object'}, 'unify': {'oneOf': [{'type':
'boolean'}, {'enum': ['when_possible'], 'type': 'string'}]}}, 'type':
'object'}}, 'title': 'Spack concretizer configuration file schema',
'type': 'object'},Lowkey my fave. Look at this:
duplicatesโ object withstrategy, enum['none', 'minimal', 'full'].enable_node_namespaceโ boolean.os_compatibleโ object withadditionalPropertiesof type array. Vibes.reuseโ the main character honestly. It can be: a boolean; the string'dependencies'; or an object withexclude(array of strings),include(array of strings),roots(boolean), andfromโ an array of objects each withexclude,include, and atypeenum of['local', 'buildcache', 'external'].targetsโgranularityenum['generic', 'microarchitectures'], plushost_compatibleboolean.unifyโ boolean, or the string'when_possible'. Fr fr.- Everything is
additionalProperties: Falseat both levels, so don't be out here inventing keys โจ
โ๏ธ config โ Spack core configuration file schema
'config': {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'config': {'default': {}, 'deprecatedProperties':
{'error': False, 'message': 'config:terminal_title has been replaced by
install_status and is ignored', 'properties': ['terminal_title']},
'properties': {'additional_external_search_paths': {'items': {'type':
'string'}, 'type': 'array'}, 'aliases': {'patternProperties':
{'\\w[\\w-]*': {'type': 'string'}}, 'type': 'object'}, 'allow_sgid':
{'type': 'boolean'}, 'binary_index_root': {'type': 'string'},
'binary_index_ttl': {'minimum': 0, 'type': 'integer'}, 'build_jobs':
{'minimum': 1, 'type': 'integer'}, 'build_language': {'type': 'string'},
'build_stage': {'oneOf': [{'type': 'string'}, {'items': {'type':
'string'}, 'type': 'array'}]}, 'ccache': {'type': 'boolean'},
'checksum': {'type': 'boolean'}, 'concretizer': {'enum': ['original',
'clingo'], 'type': 'string'}, 'connect_timeout': {'minimum': 0, 'type':
'integer'}, 'db_lock_timeout': {'minimum': 1, 'type': 'integer'},
'debug': {'type': 'boolean'}, 'deprecated': {'type': 'boolean'},
'develop_stage_link': {'type': 'string'}, 'dirty': {'type': 'boolean'},
'environments_root': {'type': 'string'}, 'extensions': {'items':
{'type': 'string'}, 'type': 'array'}, 'flags': {'properties':
{'keep_werror': {'enum': ['all', 'specific', 'none'], 'type':
'string'}}, 'type': 'object'}, 'install_hash_length': {'minimum': 1,
'type': 'integer'}, 'install_missing_compilers': {'type': 'boolean'},
'install_path_scheme': {'type': 'string'}, 'install_status': {'type':
'boolean'}, 'install_tree': {'anyOf': [{'properties': {'padded_length':
{'oneOf': [{'minimum': 0, 'type': 'integer'}, {'type': 'boolean'}]},
'projections': {'patternProperties': {'all|\\w[\\w-]*': {'type':
'string'}}, 'type': 'object'}, 'root': {'type': 'string'}}, 'type':
'object'}, {'type': 'string'}]}, 'license_dir': {'type': 'string'},
'locks': {'type': 'boolean'}, 'misc_cache': {'type': 'string'},
'package_lock_timeout': {'anyOf': [{'minimum': 1, 'type': 'integer'},
{'type': 'null'}]}, 'shared_linking': {'anyOf': [{'enum': ['rpath',
'runpath'], 'type': 'string'}, {'properties': {'bind': {'type':
'boolean'}, 'type': {'enum': ['rpath', 'runpath'], 'type': 'string'}},
'type': 'object'}]}, 'source_cache': {'type': 'string'}, 'ssl_certs':
{'type': 'string'}, 'stage_name': {'type': 'string'},
'suppress_gpg_warnings': {'type': 'boolean'}, 'template_dirs': {'items':
{'type': 'string'}, 'type': 'array'}, 'test_stage': {'type': 'string'},
'url_fetch_method': {'enum': ['urllib', 'curl'], 'type': 'string'},
'verify_ssl': {'type': 'boolean'}}, 'type': 'object'}}, 'title': 'Spack
core configuration file schema', 'type': 'object'},Grab a snack, this is the big one ๐
Real talk: there's adeprecatedPropertiesblock here witherror: Falseand the message'config:terminal_title has been replaced by install_status and is ignored'for the propertyterminal_title. Soterminal_titleis deprecated โ it won't explode, it just gets ignored. Noted, bestie ๐
Every property the config object knows about:
additional_external_search_pathsโ array of stringsaliasesโ object, keys matching\\w[\\w-]*, values stringsallow_sgidโ booleanbinary_index_rootโ stringbinary_index_ttlโ integer, minimum0build_jobsโ integer, minimum1build_languageโ stringbuild_stageโ string, or array of stringsccacheโ booleanchecksumโ booleanconcretizerโ enum['original', 'clingo']connect_timeoutโ integer, minimum0db_lock_timeoutโ integer, minimum1debugโ booleandeprecatedโ booleandevelop_stage_linkโ stringdirtyโ booleanenvironments_rootโ stringextensionsโ array of stringsflagsโ object withkeep_werrorenum['all', 'specific', 'none']install_hash_lengthโ integer, minimum1install_missing_compilersโ booleaninstall_path_schemeโ stringinstall_statusโ booleaninstall_treeโ either a string, or an object withpadded_length(integer minimum0, or boolean),projections(keys matchingall|\\w[\\w-]*, string values), androot(string)license_dirโ stringlocksโ booleanmisc_cacheโ stringpackage_lock_timeoutโ integer minimum1, or nullshared_linkingโ enum['rpath', 'runpath']as a plain string, or an object withbind(boolean) andtype(enum['rpath', 'runpath'])source_cacheโ stringssl_certsโ stringstage_nameโ stringsuppress_gpg_warningsโ booleantemplate_dirsโ array of stringstest_stageโ stringurl_fetch_methodโ enum['urllib', 'curl']verify_sslโ boolean
And yeah โ default: {}, additionalProperties: False at the outer level, so the config object is the only key allowed up there. Iconic behaviour honestly.
๐ definitions โ Spack definitions configuration file schema
'definitions': {'$schema': 'http://json-schema.org/draft-07/schema#',
'additionalProperties': False, 'properties': {'definitions': {'default':
[], 'items': {'patternProperties': {'^(?!when$)\\w*': {'default': [],
'items': {'anyOf': [{'additionalProperties': False, 'properties':
{'exclude': {'items': {'type': 'string'}, 'type': 'array'}, 'matrix':
{'items': {'items': {'type': 'string'}, 'type': 'array'}, 'type':
'array'}}, 'type': 'object'}, {'type': 'string'}, {'type': 'null'}]},
'type': 'array'}}, 'properties': {'when': {'type': 'string'}}, 'type':
'object'}, 'type': 'array'}}, 'title': 'Spack definitions configuration
file schema', 'type': 'object'},The little guy. definitions defaults to [], and each item allows keys matching ^(?!when$)\\w* โ i.e. literally anything except when โ each defaulting to []. Those items are arrays whose entries are any of:
- an object with
additionalProperties: False, holdingexclude(array of strings) andmatrix(an array of arrays of strings) - a string
- a null
And there's a when property, type string, sitting at the object level. Periodt โจ
๐ฑ develop โ (title says) Spack repository configuration file schema
'develop': {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'develop': {'additionalProperties': False,
'default': {}, 'patternProperties': {'\\w[\\w-]*':
{'additionalProperties': False, 'properties': {'path': {'type':
'string'}, 'spec': {'type': 'string'}}, 'type': 'object'}}, 'type':
'object'}}, 'title': 'Spack repository configuration file schema',
'type': 'object'},Not me noticing the title on the develop schema literally says 'Spack repository configuration file schema' ๐ญ I don't make the rules, I just transcribe them faithfully. What it contains: develop defaults to {}, keys matching \\w[\\w-]*, each object additionalProperties: False with path (string) and spec (string). Cute, small, does its job ๐
๐ช mirrors โ and we're mid-sentence, sorry not sorry
'mirrors': {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'mirrors': {'additionalProperties': False,
'default': {}, 'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'additionalProperties': False, 'anyOf': [{'required':
['url']}, {'required': ['fetch']}, {'required': ['pull']}],
'properties': {'access_pair': {'items': {'maxItems': 2, 'minItems': 2,
'type': ['string', 'null']}, 'type': 'array'}, 'access_token': {'type':
['string', 'null']}, 'autopush': {'type': 'boolean'}, 'binary': {'type':
'boolean'}, 'endpoint_url': {'type': ['string', 'null']}, 'fetch':So mirrors defaults to {}, keys match \\w[\\w-]*, and each value is either a plain string or an object that must have one of url / fetch / pull, with properties including access_pair (array, minItems 2, maxItems 2, items string-or-null), access_token (string or null), autopush (boolean), binary (boolean), endpoint_url (string or null), and fetch โ which is where this part cuts off ๐
No worries though, the schema keeps going and so do we ๐ ๐ฅ
okay so this next chunk is literally a raw schema dump and I need you to not scroll past it ๐ญ โ this is the receipts. It's the JSON Schema that Spack uses to validate your mirror config and your modules config, straight out of the source, all $schema, additionalProperties: False, patternProperties and enum glory. No cap, it's dense, but every single key down there is a real thing you can actually put in your config files. Read it like a vibe check on your YAML ๐
{'anyOf': [{'type': 'string'}, {'additionalProperties': False,
'properties': {'access_pair': {'items': {'maxItems': 2, 'minItems': 2,
'type': ['string', 'null']}, 'type': 'array'}, 'access_token': {'type':
['string', 'null']}, 'endpoint_url': {'type': ['string', 'null']},
'profile': {'type': ['string', 'null']}, 'url': {'type': 'string'}},
'type': 'object'}]}, 'profile': {'type': ['string', 'null']}, 'push':
{'anyOf': [{'type': 'string'}, {'additionalProperties': False,
'properties': {'access_pair': {'items': {'maxItems': 2, 'minItems': 2,
'type': ['string', 'null']}, 'type': 'array'}, 'access_token': {'type':
['string', 'null']}, 'endpoint_url': {'type': ['string', 'null']},
'profile': {'type': ['string', 'null']}, 'url': {'type': 'string'}},
'type': 'object'}]}, 'signed': {'type': 'boolean'}, 'source': {'type':
'boolean'}, 'url': {'type': 'string'}}, 'type': 'object'}]}}, 'type':
'object'}}, 'title': 'Spack mirror configuration file schema', 'type':
'object'}, 'modules': {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'modules': {'additionalProperties': False,
'patternProperties': {'^(?!prefix_inspections$)\\w[\\w-]*$':
{'additionalProperties': False, 'default': {}, 'properties':
{'arch_folder': {'type': 'boolean'}, 'enable': {'default': [], 'items':
{'enum': ['tcl', 'lmod'], 'type': 'string'}, 'type': 'array'}, 'lmod':
{'allOf': [{'allOf': [{'properties': {'all': {'additionalProperties':
False, 'default': {}, 'properties': {'autoload': {'enum': ['none',
'run', 'direct', 'all'], 'type': 'string'}, 'conflict': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}, 'environment':
{'additionalProperties': False, 'default': {}, 'properties':
{'append_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'filter': {'additionalProperties': False, 'default': {}, 'properties':
{'exclude_env_vars': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}, 'load': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'prerequisites': {'enum': ['none',
'run', 'direct', 'all'], 'type': 'string'}, 'suffixes':
{'patternProperties': {'\\w[\\w-]*': {'type': 'string'}}, 'type':
'object', 'validate_spec': True}, 'template': {'type': 'string'}},
'type': 'object'}, 'defaults': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'exclude': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'exclude_implicits': {'default':
False, 'type': 'boolean'}, 'hash_length': {'default': 7, 'minimum': 0,
'type': 'integer'}, 'hide_implicits': {'default': False, 'type':
'boolean'}, 'include': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'naming_scheme': {'type': 'string'}, 'projections':
{'patternProperties': {'all|\\w[\\w-]*': {'type': 'string'}}, 'type':
'object'}, 'verbose': {'default': False, 'type': 'boolean'}}},
{'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}, '^[\\^@%+~]':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}}, 'validate_spec': True}],
'default': {}, 'type': 'object'}, {'properties': {'core_compilers':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'core_specs': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}, 'filter_hierarchy_specs': {'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type':
'object'}, 'hierarchy': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}]}, 'prefix_inspections':
{'additionalProperties': False, 'patternProperties': {'^[\\w-]*':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'}}, 'type':
'object'}, 'roots': {'properties': {'lmod': {'type': 'string'}, 'tcl':
{'type': 'string'}}, 'type': 'object'}, 'tcl': {'allOf': [{'allOf':
[{'properties': {'all': {'additionalProperties': False, 'default': {},
'properties': {'autoload': {'enum': ['none', 'run', 'direct', 'all'],
'type': 'string'}, 'conflict': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'environment': {'additionalProperties':
False, 'default': {}, 'properties': {'append_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'},
'filter': {'additionalProperties': False, 'default': {}, 'properties':
{'exclude_env_vars': {'default': [], 'items': {'type': 'string'},
'type': 'array'}}, 'type': 'object'}, 'load': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'prerequisites': {'enum': ['none',
'run', 'direct', 'all'], 'type': 'string'}, 'suffixes':
{'patternProperties': {'\\w[\\w-]*': {'type': 'string'}}, 'type':
'object', 'validate_spec': True}, 'template': {'type': 'string'}},
'type': 'object'}, 'defaults': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'exclude': {'default': [], 'items':
{'type': 'string'}, 'type': 'array'}, 'exclude_implicits': {'default':
False, 'type': 'boolean'}, 'hash_length': {'default': 7, 'minimum': 0,
'type': 'integer'}, 'hide_implicits': {'default': False, 'type':
'boolean'}, 'include': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'naming_scheme': {'type': 'string'}, 'projections':
{'patternProperties': {'all|\\w[\\w-]*': {'type': 'string'}}, 'type':
'object'}, 'verbose': {'default': False, 'type': 'boolean'}}},
{'patternProperties':
{'(?!hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all)(^\\w[\\w-]*)':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':๐ช so what did we just read (mirror edition)
The top-level title of that whole thing is literally 'Spack mirror configuration file schema' โ that's the name, bestie, that's the assignment ๐
. The mirror entries each accept either a plain string or a full object, and that object's keys are:
access_pairโ an array,minItems2andmaxItems2, with items typed['string', 'null']. Exactly two slots, no more, no less ๐.access_tokenโ type['string', 'null'].endpoint_urlโ type['string', 'null'].profileโ type['string', 'null'].urlโ type'string'.
Then the outer mirror options, same energy:
pushโanyOfa'string'or that same mirror object shape (the one withaccess_pair,access_token,endpoint_url,profile,url).signedโ'boolean'.sourceโ'boolean'.urlโ'string'.
And every one of those objects has 'additionalProperties': False, so if you invent a key? Schema says no โ.
๐ง modules schema โ the lmod/tcl situation
This one declares '$schema': 'http://json-schema.org/draft-07/schema#, and top-level additionalProperties is False too. Inside, the modules key holds named module sets matched by the pattern ^(?!prefix_inspections$)\w[\w-]*$ โ so any word-ish name, as long as it's not literally prefix_inspections. Each set can have:
arch_folderโ'boolean'.enableโ array, default[], items are'string'with enum['tcl', 'lmod']. Pick your fighters โ๏ธ.lmodandtclโ each wrapped inallOf, with avalidate_spec: Truemarker on the shared block.
The shared block (this shows up again and again, it's giving copy-paste main character energy) has:
allโ an object, default{}, containing:autoloadโ enum['none', 'run', 'direct', 'all'].conflictโ array of strings, default[].environmentโ object, default{}, withappend_path,prepend_path,remove_path,set(each apatternPropertiesobject over\w[\w-]*accepting a string or a number), andunset(array of string-or-number, default[]).filterโ object, default{}, withexclude_env_vars(array of strings, default[]).loadโ array of strings, default[].prerequisitesโ enum['none', 'run', 'direct', 'all'].suffixesโpatternPropertiesover\w[\w-]*mapping to'string', withvalidate_spec: True.templateโ'string'.
defaultsโ array of strings, default[].excludeโ array of strings, default[].exclude_implicitsโ boolean, defaultFalse.hash_lengthโ integer, default7, minimum0. Not a suggestion, it's a floor ๐ซถ.hide_implicitsโ boolean, defaultFalse.includeโ array of strings, default[].naming_schemeโ'string'.projectionsโpatternPropertiesmatchingall|\w[\w-]*โ'string'.verboseโ boolean, defaultFalse.
There's also a second patternProperties branch matching everything that isn't one of the reserved words โ the negative lookahead list is hierarchy|core_specs|verbose|hash_length|defaults|filter_hierarchy_specs|hide|include|exclude|projections|naming_scheme|core_compilers|all โ and those get the same autoload/conflict/environment/filter/load/prerequisites/suffixes/template object shape, default {}.
Plus a ^[\^@%+~] branch for keys starting with spec sigils โ same object shape again, because of course ๐ฅ.
Then the hierarchy-ish properties:
core_compilersโ array of strings, default[].core_specsโ array of strings, default[].filter_hierarchy_specsโpatternPropertiesover that same negative-lookahead pattern, each mapping to an array of strings, default[].hierarchyโ array of strings, default[].
Outside modules, you've got:
prefix_inspectionsโ object,additionalPropertiesFalse,patternProperties^[\w-]*mapping to arrays of strings, default[].rootsโ object withlmod('string') andtcl('string').
And then tcl starts all over again with the exact same allOf โ all โ defaults โ exclude โ exclude_implicits โ hash_length (default 7, minimum 0) โ hide_implicits (default False) โ include โ naming_scheme โ projections โ verbose (default False) parade, followed by that reserved-words patternProperties remixโฆ
โฆand that's exactly where this chunk cuts off โ mid-template, bestie ๐ญ hold that thought ๐
{'type': 'string'}}, 'type': 'object'}, '^[\\^@%+~]':
{'additionalProperties': False, 'default': {}, 'properties':
{'autoload': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'conflict': {'default': [], 'items': {'type': 'string'},
'type': 'array'}, 'environment': {'additionalProperties': False,
'default': {}, 'properties': {'append_path': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'prepend_path': {'patternProperties': {'\\w[\\w-]*':
{'anyOf': [{'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'},
'remove_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'set':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'unset': {'default': [],
'items': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}, 'type':
'array'}}, 'type': 'object'}, 'filter': {'additionalProperties': False,
'default': {}, 'properties': {'exclude_env_vars': {'default': [],
'items': {'type': 'string'}, 'type': 'array'}}, 'type': 'object'},
'load': {'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'prerequisites': {'enum': ['none', 'run', 'direct', 'all'], 'type':
'string'}, 'suffixes': {'patternProperties': {'\\w[\\w-]*': {'type':
'string'}}, 'type': 'object', 'validate_spec': True}, 'template':
{'type': 'string'}}, 'type': 'object'}}, 'validate_spec': True}],
'default': {}, 'type': 'object'}, {}]}, 'use_view': {'anyOf': [{'type':
'string'}, {'type': 'boolean'}]}}, 'type': 'object'}}, 'properties':
{'prefix_inspections': {'additionalProperties': False,
'patternProperties': {'^[\\w-]*': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}}, 'type': 'object'}}, 'type': 'object'}},
'title': 'Spack module file configuration file schema', 'type':
'object'}, 'packages': {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'packages': {'additionalProperties': False,
'default': {}, 'patternProperties': {'(?!^all$)(^\\w[\\w-]*)':
{'additionalProperties': False, 'default': {}, 'deprecatedProperties':
{'error': False, 'message': "setting 'compiler:', 'target:' or
'provider:' preferences in a package-specific section of packages.yaml
is deprecated, and will be removed in v0.23.\n\n\tThese preferences will
be ignored by Spack, and can be set only in the 'all' section of the
same file. You can run:\n\n\t\t$ spack audit configs\n\n\tto get better
diagnostics, including files:lines where the deprecated attributes are
used.\n\n\tUse requirements to enforce conditions on specific packages:
https://spack.readthedocs.io/en/latest/packages_yaml.html#package-requirements\n",
'properties': ['target', 'compiler', 'providers']}, 'properties':
{'buildable': {'default': True, 'type': 'boolean'}, 'compiler': {},
'conflict': {'items': {'oneOf': [{'additionalProperties': False,
'properties': {'message': {'type': 'string'}, 'spec': {'type':
'string'}, 'when': {'type': 'string'}}, 'type': 'object'}, {'type':
'string'}]}, 'type': 'array'}, 'externals': {'items':
{'additionalProperties': True, 'properties': {'extra_attributes':
{'additionalProperties': True, 'properties': {'environment':
{'additionalProperties': False, 'default': {}, 'properties':
{'append_path': {'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}}, 'type': 'object'}, 'prepend_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'remove_path':
{'patternProperties': {'\\w[\\w-]*': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}}, 'type': 'object'}, 'set': {'patternProperties':
{'\\w[\\w-]*': {'anyOf': [{'type': 'string'}, {'type': 'number'}]}},
'type': 'object'}, 'unset': {'default': [], 'items': {'anyOf': [{'type':
'string'}, {'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}},
'type': 'object'}, 'modules': {'items': {'type': 'string'}, 'type':
'array'}, 'prefix': {'type': 'string'}, 'spec': {'type': 'string'}},
'required': ['spec'], 'type': 'object'}, 'type': 'array'},
'package_attributes': {'additionalProperties': False,
'patternProperties': {'\\w+': {}}, 'type': 'object'}, 'permissions':
{'additionalProperties': False, 'properties': {'group': {'type':
'string'}, 'read': {'enum': ['user', 'group', 'world'], 'type':
'string'}, 'write': {'enum': ['user', 'group', 'world'], 'type':
'string'}}, 'type': 'object'}, 'prefer': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'}, 'providers':
{}, 'require': {'oneOf': [{'items': {'oneOf': [{'additionalProperties':
False, 'properties': {'any_of': {'items': {'type': 'string'}, 'type':
'array'}, 'message': {'type': 'string'}, 'one_of': {'items': {'type':
'string'}, 'type': 'array'}, 'spec': {'type': 'string'}, 'when':
{'type': 'string'}}, 'type': 'object'}, {'type': 'string'}]}, 'type':
'array'}, {'type': 'string'}]}, 'target': {}, 'variants': {'oneOf':
[{'type': 'string'}, {'items': {'type': 'string'}, 'type': 'array'}]},
'version': {'default': [], 'items': {'anyOf': [{'type': 'string'},
{'type': 'number'}]}, 'type': 'array'}}, 'type': 'object'}},
'properties': {'all': {'additionalProperties': False, 'default': {},
'deprecatedProperties': {'error': False, 'message': "setting version
preferences in the 'all' section of packages.yaml is deprecated and will
be removed in v0.23\n\n\tThese preferences will be ignored by Spack. You
can set them only in package-specific sections of the same file.\n",
'properties': ['version']}, 'properties': {'buildable': {'default':
True, 'type': 'boolean'}, 'compiler': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}, 'conflict': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'},
'package_attributes': {'additionalProperties': False,
'patternProperties': {'\\w+': {}}, 'type': 'object'}, 'permissions':
{'additionalProperties': False, 'properties': {'group': {'type':
'string'}, 'read': {'enum': ['user', 'group', 'world'], 'type':
'string'}, 'write': {'enum': ['user', 'group', 'world'], 'type':
'string'}}, 'type': 'object'}, 'prefer': {'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'message': {'type':
'string'}, 'spec': {'type': 'string'}, 'when': {'type': 'string'}},
'type': 'object'}, {'type': 'string'}]}, 'type': 'array'}, 'providers':
{'additionalProperties': False, 'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'default': [], 'items': {'type': 'string'}, 'type':
'array'}}, 'type': 'object'}, 'require': {'oneOf': [{'items': {'oneOf':
[{'additionalProperties': False, 'properties': {'any_of': {'items':
{'type': 'string'}, 'type': 'array'}, 'message': {'type': 'string'},
'one_of': {'items': {'type': 'string'}, 'type': 'array'}, 'spec':
{'type': 'string'}, 'when': {'type': 'string'}}, 'type': 'object'},
{'type': 'string'}]}, 'type': 'array'}, {'type': 'string'}]}, 'target':
{'default': [], 'items': {'type': 'string'}, 'type': 'array'},
'variants': {'oneOf': [{'type': 'string'}, {'items': {'type': 'string'},
'type': 'array'}]}, 'version': {}}, 'type': 'object'}}, 'type':
'object'}}, 'title': 'Spack package configuration file schema', 'type':
'object'}, 'repos': {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'repos': {'default': [], 'items': {'type':
'string'}, 'type': 'array'}}, 'title': 'Spack repository configuration
file schema', 'type': 'object'}, 'upstreams': {'$schema':
'http://json-schema.org/draft-07/schema#', 'additionalProperties':
False, 'properties': {'upstreams': {'default': {}, 'patternProperties':
{'\\w[\\w-]*': {'additionalProperties': False, 'default': {},
'properties': {'install_tree': {'type': 'string'}, 'modules':
{'properties': {'lmod': {'type': 'string'}, 'tcl': {'type': 'string'}},
'type': 'object'}}, 'type': 'object'}}, 'type': 'object'}}, 'title':
'Spack core configuration file schema', 'type': 'object'}, 'view':
{'$schema': 'http://json-schema.org/draft-07/schema#', 'properties':
{'view': {'anyOf': [{'type': 'boolean'}, {'type': 'string'},
{'patternProperties': {'\\w+': {'additionalProperties': False,
'properties': {'exclude': {'items': {'type': 'string'}, 'type':
'array'}, 'link': {'pattern': '(roots|all|run)', 'type': 'string'},
'link_type': {'type': 'string'}, 'projections': {'patternProperties':
{'all|\\w[\\w-]*': {'type': 'string'}}, 'type': 'object'}, 'root':
{'type': 'string'}, 'select': {'items': {'type': 'string'}, 'type':
'array'}}, 'required': ['root']}}, 'type': 'object'}]}}, 'title': 'Spack
view configuration file schema'}}okay so that whole gorgeous wall of curly braces and single quotes? it's the Dict from section names -> schema for that section ๐ every config section in Spack gets its own lil schema, and this bad boy holds all of them. it's giving dictionary-of-destiny, fr fr.
๐ธ spack.config.SingleFileScope and friends ๐ธ
spack.config.SingleFileScope(name: str, path: str, schema: Dict[str, Any], yaml_path: List[str] | None = None)โ bases:ConfigScope. literally the class that represents a configuration scope living inside a single YAML file. one file, one scope, no cap. โจget_section(section: str) -> Dict[str, Any] | Noneโ returns the data that's chillin' in a given section ๐ซถget_section_filename(section) -> strโ returns the filename tied to a given section, iykykis_platform_dependent: bool(property) โ returnstrueif the scope name is platform specific ๐
spack.config.YamlConfigDictโ the type used for raw YAML configuration. it's literally just an alias ofDict[str, Any]. icon behavior ๐spack.config.add(fullpath: str, scope: str | None = None) -> Noneโ adds the config you gave it to whatever config scope you named. heads up bestie:addtakes a path. if you wanna add from a filename, useadd_from_fileinstead ๐spack.config.add_default_platform_scope(platform: str) -> Noneโ adds the default platform scope for the platform you hand it ๐spack.config.add_from_file(filename: str, scope: str | None = None) -> Noneโ adds updates to a config straight from a filename. the filename-to-config pipeline, slay ๐spack.config.change_or_add(section_name: str, find_fn: Callable[[str], bool], update_fn: Callable[[str], None]) -> Noneโ changes or adds a subsection of config, but with extra logic so it picks a reasonable scope to actually apply the change to. and the way it searches? starts from the highest priority config scopes: the first one that matches the criteria (that's whatfind_fndecides) gets updated. if no such config exists, it grabs the first config scope that defines any config for the named section. and if literally no scopes define any related config at all? then it just updates the highest-priority config scope. main character energy, honestly ๐spack.config.collect_urls(base_url: str) -> listโ returns a list of configuration URLs ๐ธ simple, clean, ate.
Parameters
base_urlโ the URL for a configuration (yaml) file, or for a directory that's holding yaml file(s) ๐
Returns
A list of configuration file(s), or an empty list if there are none. That's it, that's the whole vibe โจ
๐
spack.config.config_paths_from_entry_points() -> List[Tuple[str, str]]
Load configuration paths from entry points. Okay so this is genuinely such a slay feature โ a python package can register entry point metadata so that Spack can find its configuration, just by adding this to the project's pyproject.toml:
[project.entry-points."spack.config"]
baz = "baz:get_spack_config_path"The function get_spack_config_path returns the path to the package's spack configuration scope. Iconic. No cap. ๐
๐ฆ spack.config.create() -> Configuration
The singleton Configuration instance.
This constructs one instance associated with this module and returns it. It's bundled inside a function so that configuration can be initialized lazily โ main character energy, shows up exactly when needed, not a second before ๐ธ
โญ spack.config.default_modify_scope(section: str = 'config') -> str
Return the config scope that commands should modify by default.
Commands that modify configuration by default modify the highest priority scope. The way that works is honestly so satisfying.
Parameters
section(bool) โ Section for which to get the default scope. If this is not'compilers', a general (non-platform) scope is used.
Ngl, the original really does sayboolthere forsection. We're just reporting the facts bestie ๐
๐ฅ spack.config.ensure_latest_format_fn(section: str) -> Callable[[Dict[str, Any]], bool]
Return a function that takes as input a dictionary read from a configuration file and updates it to the latest format.
The function returns True if there was any update, False otherwise. Little boolean vibe check, love that for us ๐
Parameters
sectionโ section of the configuration, e.g."packages","config", etc.
๐ spack.config.fetch_remote_configs(url: str, dest_dir: str, skip_existing: bool = True) -> str
Retrieve configuration file(s) at the specified URL. Fetching, but make it fashion โจ
Parameters
urlโ URL for a configuration (yaml) file or a directory containing yaml file(s)dest_dirโ destination directoryskip_existingโ Skip files that already exist indest_dirifTrue; otherwise, replace those files
Returns
Path to the corresponding file if URL is or contains a single file and it is the only file in the destination directory, or the root (dest_dir) directory if multiple configuration files exist or are retrieved.
๐ง spack.config.get(path: str, default: Any | None = None, scope: str | None = None) -> Any
The module-level wrapper for Configuration.get(). Short, sweet, does exactly what it says on the tin ๐ซถ
๐ spack.config.get_mark_from_yaml_data(obj)
Try to get spack.util.spack_yaml mark from YAML data.
We try the object, and if that fails we try its first member (if it's a container). Persistence! We love a try-hard in the best way ๐
Returns
The mark if one is found, otherwise None.
๐ spack.config.get_valid_type(path)
Returns an instance of a type that will pass validation for path.
The instance is created by calling the constructor with no arguments. If multiple types will satisfy validation for data at the configuration path given, the priority order is list, dict, str, bool, int, float. Literally a whole ranking system, and list is sitting at the top of the leaderboard ๐
๐ธ spack.config.matched_config(cfg_path: str) -> List[Tuple[str, Any]]
That's the whole entry, bestie โ no extra description given, so I'm not gonna invent one ๐ญ
๐ spack.config.merge_yaml(dest, source, prepend=False, append=False)
Merges source into dest; entries in source take precedence over dest.
This routine may modify dest and should be assigned to dest, in case dest was None to begin with, e.g.:
dest = merge_yaml(dest, source)In the result, elements from lists from source will appear before elements of lists from dest. Likewise, when iterating over keys or items in merged OrderedDict objects, keys from source will appear before keys from dest. Order matters, always has ๐
Config file authors can optionally end any attribute in a dict with :: instead of :, and the key will override that of the parent instead of merging. Sneaky little double-colon, we stan.
+ will extend the default prepend merge strategy to include string concatenation. - will change the merge strategy to append, it also includes string concatentation.
Yes, "concatentation." We're keeping the typo because it's canon now ๐
๐ spack.config.override(path_or_scope: ConfigScope | str, value: Any | None = None) -> Generator[Singleton | Configuration, None, None]
The simple way to override config settings within a context. Chef's kiss ๐ค
Parameters
path_or_scope(ConfigScopeorstr) โ scope or single option to overridevalue(objectorNone) โ value for the single option
Temporarily push a scope on the current configuration, then remove it after the context completes. If a single option is provided, create an internal config scope for it and push/pop that scope. In and out, no mess left behind, immaculate โจ
๐ฆ spack.config.parse_spec_from_yaml_string(string: str) -> Spec
Parse a spec from YAML and add file/line info to errors, if it's available.
Parse a Spec from the supplied string, but also intercept any syntax errors and add file/line information for debugging using file/line annotations from the string. Basically it's giving helpful bestie energy when things go wrong ๐ญ๐ซถ
Parameters
stringโ a string representing aSpecfrom config YAML.
๐ spack.config.process_config_path(path: str) -> List[str]
Process a path argument to config.set() that may contain overrides ('::' or trailing ':').
Colons will be treated as static strings if inside of quotes, e.g. this: this:is:a:path:'value:with:colon' will yield:
[this, is, a, path, value:with:colon]The path may consist only of keys (e.g. for a get) or may end in a value. Keys are always strings: if a user encloses a key in quotes, the quotes should be removed. Values with quotes should be treated as strings, but without quotes, may be parsed as a different yaml object (e.g. '{}' is a dict, but '"{}"' is a string).
This function does not know whether the final element of the path is a key or value, so:
- It must strip the quotes, in case it is a key (so we look for
"key"and not'"key"') - It must indicate somehow that the quotes were stripped, in case it is a value (so that we don't process
'"{}"'as a YAML dict)
Therefore, all elements with quotes are stripped, and then also converted to syaml_str (if treating the final element as a value, the caller should not parse it in this case). It's giving detective work honestly ๐๐
โญ spack.config.raw_github_gitlab_url(url: str) -> str
Transform a github URL to the raw form to avoid undesirable html. Goodbye, ugly html soup ๐
Parameters
urlโ url to be converted to raw form
Returns
Raw github/gitlab url or the original url
๐ spack.config.read_config_file(filename: str, schema: Dict[str, Any] | None = None) -> Dict[str, Any] | None
Read a YAML configuration file.
User can provide a schema for validation. If no schema is provided, we will infer the schema from the top-level key. Adaptive queen behavior ๐
๐ฅ spack.config.remove_yaml(dest, source)
UnMerges source from dest; entries in source take precedence over dest. It's the anti-merge, the un-invite, the "we're removing you from the group chat" ๐ญ
This routine may modify dest and should be assigned to dest, in case dest was None to begin with, e.g.:
dest = remove_yaml(dest, source)In the result, elements from lists from source will not appear as elements of lists from dest. Likewise, when iterating over keys or items in merged OrderedDict objects, keys from source will not appear as keys in dest.
Config file authors can optionally end any attribute in a dict with :: instead of :, and the key will remove the entire section from dest. Double colon: remover of worlds ๐
๐ธ spack.config.scopes() -> Dict[str, ConfigScope]
Convenience function to get list of configuration scopes. Short and slay โจ
๐ spack.config.set(path: str, value: Any, scope: str | None = None) -> None
Convenience function for setting single values in config files.
Accepts the path syntax described in get(). Consistent, iconic, no surprises ๐
๐ง spack.config.update_all(section_name: str, change_fn: Callable[[str], bool]) -> None
Change a config section, which may have details duplicated across multiple scopes. Spread everywhere but still one cohesive edit โ the way that works is honestly so satisfying ๐
๐ฆ spack.config.use_configuration(*scopes_or_paths: ConfigScope | str) -> Generator[Configuration, None, None]
Use the configuration scopes passed as arguments within the context manager. We're committing to this setup for the duration, bestie ๐ซถ
This function invalidates caches, and is therefore very slow.
Fr fr, "very slow" is in the docs. Do not come for me, I'm just the messenger ๐ญ
Parameters
*scopes_or_pathsโ scope objects or paths to be used
Returns
Configuration object associated with the scopes passed as arguments
๐ spack.config.validate(data: Dict[str, Any], schema: Dict[str, Any], filename: str | None = None) -> Dict[str, Any]
Validate data read in from a Spack YAML file. The vibe check of the config world ๐
Parameters
dataโ data read from a Spack YAML fileschemaโjsonschemato validatedata
This leverages the line information (start_mark, end_mark) stored on Spack YAML structures. Little breadcrumbs so you can actually find where it went wrong ๐
๐ spack.config.writable_scope_names() -> List[str]
Yep, that's the whole thing. Very demure, very minimal ๐
โญ spack.config.writable_scopes() -> List[ConfigScope]
Return list of writable scopes. Higher-priority scopes come first in the list. It's called ~~priority~~ hierarchy, and it's giving ordered excellence ๐
๐ธ spack.context module ๐ธ
This module provides classes used in user and build environment.
๐ class spack.context.Context(*values)
Bases: Enum
Enum used to indicate the context in which an environment has to be setup: build, run or test.
BUILD = 1RUN = 2TEST = 3
classmethod from_string(s: str)
๐ spack.cray_manifest module ๐
๐ exception spack.cray_manifest.ManifestValidationError(msg, long_msg=None)
Bases: SpackError
That's the whole entry โ an exception class doing exception class things ๐ญ
๐ฆ spack.cray_manifest.compiler_from_entry(entry: dict, manifest_path: str)
Present and accounted for, that's the whole signature ๐ซถ
๐ฅ spack.cray_manifest.default_path = '/opt/cray/pe/cpe-descriptive-manifest/'
Cray systems can store a Spack-compatible description of system packages here. Locked in at that exact path, no cap ๐
๐ spack.cray_manifest.entries_to_specs(entries)
Yep. That's the entry. We love a lowkey undocumented one โจ
๐ง spack.cray_manifest.read(path, apply_updates)
Also exactly as listed, bestie โ no extra description in the source ๐ธ
๐ spack.cray_manifest.spec_from_entry(entry)
Same energy: present, correct, and ready to convert an entry into a spec ๐
โจ spack.cray_manifest.translated_compiler_name(manifest_compiler_name)
Okay so when Spack builds a Compiler object it expects a name that matches one of the classes in spack.compilers โ but the names sitting in a Cray manifest? Lowkey not always the same. ๐ So this function is the friend that smooths it over: if we know the manifest name refers to a compiler Spack has, it translates it for you automatically. ๐ซถ
Heads up bestie: this will raise an error if there's no recorded translation and the name doesn't match a known compiler name. So it's not out here guessing โ it either knows the name or it tells you. ๐
๐ธ spack.database module
Spack's installation tracking database. It's giving very organized bestie with a spreadsheet. ๐โจ The database serves two purposes:
- It implements a cache on top of a potentially very large Spack directory hierarchy, speeding up many operations that would otherwise require filesystem access.
- It will allow us to track external installations as well as lost packages and their dependencies.
Real talk though โ prior to the implementation of this store, a directory layout served as the authoritative database of packages in Spack. This module provides a cache and a sanity checking mechanism for what is in the filesystem. Glow-up era. ๐
๐พ exception spack.database.CorruptDatabaseError(message, long_message=None)
- Bases:
SpackError
Raised when errors are found while reading the database. She looked at the DB and went "girl what happened here" ๐ญ โ then threw this.
๐ฆ spack.database.DEFAULT_INSTALL_RECORD_FIELDS = ('spec', 'ref_count', 'path', 'installed', 'explicit', 'installation_time', 'deprecated_for')
Default list of fields written for each install record. These are the receipts Spack keeps for every install, periodt. ๐งพ
๐ spack.database.DEFAULT_LOCK_CFG: LockConfiguration = (True, 120, None)
Default configuration for database locks. Lock it up bestie. ๐
๐ class spack.database.Database(root: str, upstream_dbs: List[Database] | None = None, is_upstream: bool = False, lock_cfg: LockConfiguration = (True, 120, None))
- Bases:
object
The main character of the module, no cap. Here's everything she can do:
add(spec_like, *args, **kwargs)
Adds it to the database. That's the vibe. โจ
all_hashes()
Return dag hash of every spec in the database. Every single one, she's not missing anybody. ๐
db_for_spec_hash(hash_key)
Tells you which DB a given hash key belongs to. ๐๏ธ
deprecate(spec_like, *args, **kwargs)
Marks a spec as deprecated. It's not gone, it's just... retired. ๐ซถ
deprecator(spec)
Return the spec that the given spec is deprecated for, or None. Like the "who replaced you" lookup. ๐
get_by_hash(dag_hash, default=None, installed=<built-in function any>)
Look up a spec by DAG hash โ or even just a DAG hash prefix, she's flexible like that. ๐
- Parameters
dag_hash(str) โ hash (or hash prefix) to look updefault(object or None) โ default value to return ifdag_hashis not in the DB (default:None)installed(bool or InstallStatus or Iterable or None) โ ifTrue, includes only installed specs in the search; ifFalseonly missing specs, and if any, all specs in database. If anInstallStatusor iterable ofInstallStatus, returns specs whose install status (installed, deprecated, or missing) matches (one of) theInstallStatus. (default:any)
installeddefaults toanyso that we can refer to any known hash. Note thatquery()andquery_one()differ in that they only return installed specs by default.- Returns โ a list of specs matching the hash or hash prefix
- Return type โ (list)
get_by_hash_local(dag_hash, default=None, installed=<built-in function any>)
Same energy as above, but she's only digging through this DB. Look up a spec in this DB by DAG hash, or by a DAG hash prefix. ๐ฏ
- Parameters
dag_hash(str) โ hash (or hash prefix) to look updefault(object or None) โ default value to return ifdag_hashis not in the DB (default:None)installed(bool or InstallStatus or Iterable or None) โ ifTrue, includes only installed specs in the search; ifFalseonly missing specs, and if any, all specs in database. If anInstallStatusor iterable ofInstallStatus, returns specs whose install status (installed, deprecated, or missing) matches (one of) theInstallStatus. (default:any)
installeddefaults toanyso that we can refer to any known hash. Note thatquery()andquery_one()differ in that they only return installed specs by default.- Returns โ a list of specs matching the hash or hash prefix
- Return type โ (list)
get_record(spec_like, *args, **kwargs)
Hands you the install record for a spec. The full receipt. ๐งพโจ
installed_extensions_for(spec_like, *args, **kwargs)
Pulls up the installed extensions for the spec you give it. A little family tree moment. ๐ณ
installed_relatives(spec_like, *args, **kwargs)
Same but for relatives โ dependencies and dependents doing their thing. ๐ซถ
is_occupied_install_prefix(path)
Checks whether that install prefix is already taken. Is this seat saved? ๐
mark(spec_like, *args, **kwargs)
Marks a spec in the database. ๐
missing(spec)
Tells you if a spec is missing. She's looking for it and it's not here, bestie. ๐ญ
query(*args, **kwargs)
Query the Spack database including all upstream databases. Very inclusive, we love to see it. ๐
Additional Arguments:
install_tree(str) โ query'all'(default),'local','upstream', or upstream path
Args:
query_specโ queries iterate through specs in the database and return those that satisfy the suppliedquery_spec. Ifquery_specisany, This will match all specs in the database. If it is a spec, we'll evaluatespec.satisfies(query_spec)known(bool or None) โ Specs that are "known" are those for which Spack can locate apackage.pyfile -- i.e., Spack "knows" how to install them. Specs that are unknown may represent packages that existed in a previous version of Spack, but have since either changed their name or been removedinstalled(bool or InstallStatus or typing.Iterable or None) โ ifTrue, includes only installed specs in the search; ifFalseonly missing specs, and if any, all specs in database. If anInstallStatusor iterable ofInstallStatus, returns specs whose install status (installed, deprecated, or missing) matches (one of) theInstallStatus. (default:True)explicit(bool or None) โ A spec that was installed following a specific user request is marked as explicit. If instead it was pulled-in as a dependency of a user requested spec it's considered implicit.start_date(datetime.datetime or None) โ filters the query discarding specs that have been installed beforestart_date.end_date(datetime.datetime or None) โ filters the query discarding specs that have been installed afterend_date.hashes(Container) โ list or set of hashes that we can use to restrict the searchin_buildcache(bool or None) โ Specs that are marked in this database as part of an associated binary cache arein_buildcache. All other specs are not. This field is used for querying mirror indices. Default isany.
Returns: list of specs that match the query
query_by_spec_hash(hash_key: str, data: Dict[str, InstallRecord] | None = None) -> Tuple[bool, InstallRecord | None]
Get a spec for hash, and whether it's installed upstream. Two-for-one. ๐ฅ
- Returns
(bool, optional InstallRecord): bool tells us whether the spec is installed upstream. Its InstallRecord is also returned if it's installed at all; otherwise None.
- Return type โ (tuple)
query_local(*args, **kwargs)
Query only the local Spack database. Home turf only. ๐
This function doesn't guarantee any sorting of the returned data for performance reason, since comparing specs for __lt__ may be an expensive operation. So like, don't come for her about the order, she's being efficient. ๐
Args:
query_specโ queries iterate through specs in the database and return those that satisfy the suppliedquery_spec. Ifquery_specisany, This will match all specs in the database. If it is a spec, we'll evaluatespec.satisfies(query_spec)known(bool or None) โ bestie, a spec that's "known" is one where Spack can actually find apackage.pyfile, like, Spack knows how to install it. Unknown specs might be packages from an older Spack that got renamed or straight-up removed since then.installed(bool or InstallStatus or typing.Iterable or None) โ okay so:True= only installed specs in the search;False= only missing specs; and if it's any / None, you get all specs in the database. If you hand it anInstallStatusor an iterable ofInstallStatus, you get specs whose install status (installed,deprecated, ormissing) matches (one of) theInstallStatus. Default isTrue. ๐explicit(bool or None) โ a spec that got installed because a user specifically asked for it is marked as explicit. If it got pulled in as a dependency of a user-requested spec instead, it's considered implicit.start_date(datetime.datetime or None) โ filters the query, dropping any specs installed beforestart_date.end_date(datetime.datetime or None) โ filters the query, dropping any specs installed afterend_date.hashes(Container) โ a list or set of hashes we can use to restrict the search ๐ฅin_buildcache(bool or None) โ specs marked in this database as part of an associated binary cache arein_buildcache. All the other specs are not. This field is used for querying mirror indices. Default is any.
Returns: list of specs that match the query โจ
query_local_by_spec_hash(hash_key)
Get a spec by hash out of the local database, no cap.
Returns: InstallRecord when it's installed locally, otherwise None.
Return type: (InstallRecord or None)
query_one(query_spec, known=<built-in function any>, installed=True)
Query for exactly one spec that matches the query spec. One. Single. Spec. ๐ฏ
Raises an assertion error if more than one spec matches the query. And returns None if no installed package matches.
read_transaction()
Get a read lock context manager, made for use in a with block. ๐ซถ
record_fields
record_fields: Tuple[str, ...] = ('spec', 'ref_count', 'path', 'installed', 'explicit', 'installation_time', 'deprecated_for')Fields written for each install record. That's the whole list, iconic behavior.
reindex(directory_layout)
Build the database index from scratch based on a directory layout.
Locks the DB if it isn't locked already. ๐
remove(spec_like, *args, **kwargs)
Literally just that. No extra notes, she's mysterious. ๐
specs_deprecated_by(spec)
Return all specs deprecated in favor of the given spec.
unused_specs
unused_specs(root_hashes: Container[str] | None = None, deptype: int | str | List[str] | Tuple[str, ...] = 3) -> List[Spec]Return all the specs that are currently installed but not needed by root specs. Spring cleaning energy ๐ธ
By default, roots are all explicit specs in the database. But if you pass in a set of root hashes, those get used as the roots instead.
Parameters
root_hashesโ optional list of roots to consider when evaluating needed installations.deptypeโ if a spec is reachable from a root via these dependency types, it's considered needed. By default onlylinkandrundependency types are considered.
update_explicit(spec, explicit)
Update the spec's explicit state in the database.
Parameters
spec(spack.spec.Spec) โ the spec whose install record is being updated.explicit(bool) โTrueif the package was requested explicitly by the user,Falseif it was pulled in as a dependency of an explicit package.
write_transaction()
Get a write lock context manager, for use in a with block. The โจwritingโจ sibling of read_transaction(). ๐
๐ง class spack.database.FailureTracker
spack.database.FailureTracker(root_dir: str | Path, default_timeout: float | None)Bases: object
Tracks installation failures. Bestie, this one's doing a lot of heavy lifting ๐ฅ
Prefix failure marking takes the form of a byte range lock on the *n*th byte of a file for coordinating between concurrent parallel build processes, plus a persistent file โ named with the full hash and containing the spec โ in a subdirectory of the database, so the persistence survives across overlapping but separate related build processes.
The failure lock file lives right alongside the install DB.
n is the sys.maxsize-bit prefix of the associated DAG hash, so the likelihood of collision is very low with no cleanup required. She thought about it ๐
clear(spec: Spec, force: bool = False) -> None
Removes any persistent and cached failure tracking for the spec.
see mark().
Parameters
specโ the spec whose failure indicators are being removed.forceโTrueif the failure information should be cleared when a failure lock exists for the file, orFalseif the failure should not be cleared (e.g., it may be associated with a concurrent build).
clear_all() -> None
Force remove install failure tracking files. All of them. Gone. ๐งน
dir
Ensure a persistent location for dealing with parallel installation failures (e.g., across near-concurrent processes).
has_failed(spec: Spec) -> bool
Return True if the spec is marked as failed.
lock_taken(spec: Spec) -> bool
Return True if another process has a failure lock on the spec.
mark(spec: Spec) -> Lock
Marks a spec as failing to install.
Parameters
specโ spec that failed to install ๐ญ
persistent_mark(spec: Spec) -> bool
Determine if the spec has a persistent failure marking.
class spack.database.ForbiddenLock
Bases: object
That's it. That's the whole description. She's a vibe, not a talker. ๐
exception spack.database.ForbiddenLockError(message, long_message=None)
Bases: SpackError
Raised when an upstream DB attempts to acquire a lock. Not today, upstream. ๐
๐ class spack.database.InstallRecord
spack.database.InstallRecord(spec: Spec, path: str, installed: bool, ref_count: int = 0, explicit: bool = False, installation_time: float | None = None, deprecated_for: Spec | None = None, in_buildcache: bool = False, origin=None)Bases: object
A record represents one installation in the DB. One record, one install, main character energy โญ
The record keeps track of the spec for the installation, its install path, AND whether or not it's installed. We need that installed flag in case a user either:
a. blew away a directory, or
b. used spack uninstall -f to get rid of it
If, in either case, the package was removed but others still depend on it, we still need to track its spec โ so we don't actually remove it from the database until a spec has no installed dependents left. Loyalty ๐ซถ
Parameters
specโ spec tracked by the install record.pathโ path where the spec has been installed.installedโ whether or not the spec is currently installed.ref_count(int) โ number of specs that depend on this one.explicit(bool or None) โ whether or not this spec was explicitly installed, or pulled-in as a dependency of something else.installation_time(datetime.datetime or None) โ time of the installation.
classmethod from_dict(spec, dictionary)
Yep. Builds one from a dict.
install_type_matches(installed)
Checks the install type against whatever you pass in.
to_dict(include_fields=('spec', 'ref_count', 'path', 'installed', 'explicit', 'installation_time', 'deprecated_for'))
Serializes it out, with exactly those default include_fields, character for character:
to_dict(include_fields=('spec', 'ref_count', 'path', 'installed', 'explicit', 'installation_time', 'deprecated_for'))class spack.database.InstallStatus
Bases: str
It's a string. That's the lore. ๐
๐ class spack.database.InstallStatuses
Bases: object
DEPRECATED = 'deprecated'INSTALLED = 'installed'MISSING = 'missing'
classmethod canonicalize(query_arg)
Takes your query_arg and makes it canonical, bestie.
exception spack.database.InvalidDatabaseVersionError(database, expected, found)
Bases: SpackError
Exception raised when the database metadata is newer than current Spack. iykyk โ your Spack is behind ๐ญ
property database_version_message
Gives you the message about the database version.
class spack.database.LockConfiguration
spack.database.LockConfiguration(enable: bool, database_timeout: int | None, package_timeout: int | None)Bases: NamedTuple
Data class to configure locks in Database objects. ๐
Parameters
enableโ whether to enable locks or not.database_timeoutโ timeout for the database lock.package_timeoutโ timeout for the package lock.database_timeout: int | Noneโ Alias for field number 1enable: boolโ Alias for field number 0package_timeout: int | Noneโ Alias for field number 2
exception spack.database.MissingDependenciesError(message, long_message=None)
Bases: SpackError
Raised when the DB cannot find records for dependencies. She looked everywhere and they're just not there ๐
okay so like, the database has some lock settings you can just... set, and they're giving very much "choose your own adventure" ๐โจ
spack.database.NO_LOCK: LockConfiguration = (False, None, None)โ tells a database to not use locks at all, no cap ๐spack.database.NO_TIMEOUT: LockConfiguration = (True, None, None)โ tells the database to use locks but like, with no timeout, so it'll wait forever if it has to ๐ซถ
exception spack.database.NoSuchSpecError(spec) โ Bases: KeyError. raised when a spec is literally not found in the database, bestie ๐ญ
exception spack.database.NonConcreteSpecAddError(message, long_message=None) โ Bases: SpackError. raised when you try to add a non-concrete spec to the DB, which is a no-no ๐
๐ง SpecLocker โ the lock manager of the group chat
class spack.database.SpecLocker(lock_path: str | Path, default_timeout: float | None) โ Bases: object. this icon manages acquiring and releasing read or write locks on concrete specs, periodt ๐
clear(spec: Spec) -> Tuple[bool, Lock | None]โ clears the lock for a spec โจclear_all(clear_fn: Callable[[Lock], Any] | None = None) -> Noneโ clears everything, no survivors ๐has_lock(spec: Spec) -> boolโ returnsTrueif the spec is already managed by this spec locker, iykyklock(spec: Spec, timeout: float | None = None) -> Lockโ returns a lock on a concrete spec.
okay so the lock is a byte range lock on the *n*th byte of a file, and the lock file is self.lock_path. n is the sys.maxsize-bit prefix of the DAG hash. this makes the likelihood of collision very low AND it gives us readers-writer lock semantics with just a single lockfile, so no cleanup required โ it's giving efficient, honestly ๐ธ
raw_lock(spec: Spec, timeout: float | None = None) -> Lockโ returns a raw lock for aSpec, but doesn't keep track of it, lowkey mysterious ๐ฆwrite_lock(spec: Spec) -> Generator[SpecLocker, None, None]โ the write-lock generator vibe check ๐ฅ
exception spack.database.UpstreamDatabaseLockingError(message, long_message=None) โ Bases: SpackError. raised when an operation would need to lock an upstream database, which is not the move ๐
โโ๏ธ
๐ง little helper functions
spack.database.failures_lock_path(root_dir: str | Path) -> Pathโ returns the path of the failures lock file, given the root directory. parameters:root_dirโ root directory containing the database directory ๐spack.database.lock_configuration(configuration)โ returns aLockConfigurationfrom aspack.config.Configurationobject โจspack.database.prefix_lock_path(root_dir: str | Path) -> Pathโ returns the path of the prefix lock file, given the root directory. parameters:root_dirโ root directory containing the database directory ๐spack.database.reader(version: StandardVersion) -> Type[SpecfileReaderBase]โ reader function, slay ๐
๐งฉ spack.dependency module
data structures that represent Spack's dependency relationships, periodt ๐
class spack.dependency.Dependency(pkg: PackageBase, spec: Spec, depflag: int = 5) โ Bases: object. class representing metadata for a dependency on a package.
okay so this class differs from spack.spec.DependencySpec because it represents metadata at the Package level. spack.spec.DependencySpec is a descriptor for an actual package configuration, while Dependency is a descriptor for a package's dependency requirements โ it's giving "same vibe, different layer" ๐ซถ
a dependency is a requirement for a configuration of another package that satisfies a particular spec. the dependency can have types, which determine how that package configuration is required, e.g. whether it is required for building the package, whether it needs to be linked to, or whether it's needed at runtime so that Spack can call commands from it ๐ฅ
a package can also depend on another package with patches. this is for cases where the maintainers of one package also maintain special patches for their dependencies. if one package depends on another with patches, a special version of that dependency with patches applied will be built for use by the dependent package. the patches are included in the new version's spec hash to differentiate it from unpatched versions of the same package, so that unpatched versions of the dependency package can coexist with the patched version โ iconic behavior honestly ๐
merge(other: Dependency)โ merge constraints, deptypes, and patches ofotherintoself๐ซproperty name: strโ get the name of the dependency package โจ
๐ฆ spack.deptypes module
data structures that represent Spack's edge types, no cap ๐
spack.deptypes.ALL: int = 15โ a flag with all dependency types set ๐spack.deptypes.ALL_FLAGS: Tuple[int, int, int, int] = (4, 1, 2, 8)โ an iterator of all flag componentsspack.deptypes.ALL_TYPES: Tuple[str, ...] = ('build', 'link', 'run', 'test')โ the types of dependency relationships that Spack understands ๐spack.deptypes.DEFAULT: int = 5โ default dependency type if none is specified ๐ธspack.deptypes.DEFAULT_TYPES: Tuple[str, ...] = ('build', 'link')โ default dependency type if none is specified (yes bestie it says it twice, that's the vibe)spack.deptypes.DepFlagโ type hint for the low-level dependency input (enum.Flagis too slow ๐ค)spack.deptypes.DepTypeโ individual dependency typesspack.deptypes.DepTypesโ type hint for the high-level dependency input. alias ofstr | List[str] | Tuple[str, ...]โจspack.deptypes.NONE: int = 0โ a flag with no dependency types set, the void ๐
๐ functions
spack.deptypes.canonicalize(deptype: str | List[str] | Tuple[str, ...]) -> intโ convert deptype user input to aDepFlag, or raiseValueError.
parameters: deptype โ string representing dependency type, or a list/tuple of such strings. can also be the builtin function all or the string 'all', which result in a tuple of all dependency types known to Spack ๐ฅ
spack.deptypes.flag_from_string(s: str) -> intโ turns a string into a flag, slay โจspack.deptypes.flag_from_strings(deptype: Iterable[str]) -> intโ transform an iterable of deptype strings into a flag ๐spack.deptypes.flag_to_chars(depflag: int) -> strโ create a string representing deptypes for many dependencies.
the string will be some subset of 'blrt', like 'bl ', 'b t', or ' lr ' where each letter in 'blrt' stands for 'build', 'link', 'run', and 'test' (the dependency types) ๐
for a single dependency, this just indicates that the dependency has the indicated deptypes. for a list of dependnecies, this shows whether ANY dpeendency in the list has the deptypes (so the deptypes are merged) โ yes the typos are canon, we don't fix 'em ๐ญ
spack.deptypes.flag_to_string(x: int) -> strโ flag to string conversion, cute ๐ฆspack.deptypes.flag_to_tuple(x: int) -> Tuple[str, ...]โ flag to tuple conversion, also cute ๐ธ
๐ช spack.directives module
this package contains directives that can be used within a package โจ
directives are functions that can be called inside a package definition to modify the package, for example:
class OpenMpi(Package):
depends_on("hwloc") provides("mpi") ...provides and depends_on are spack directives ๐ซถ
the available directives are:
build_systemconflictsdepends_onextendspatchprovidesresourcevariantversionrequiresredistribute
exception spack.directives.DirectiveError(message, long_message=None) โ Bases: SpackError. this is raised when something is wrong with a package directive, and it's giving "uh oh" ๐ญ
class spack.directives.DirectiveMeta(name, bases, attr_dict) โ Bases: type. flushes the directives that were temporarily stored in the staging area into the package ๐
๐ง static methods on DirectiveMeta
static directive(dicts=None)โ decorator for Spack directives.
Spack directives allow you to modify a package while it is being defined, e.g. to add version or dependency information. directives are one of the key pieces of Spack's package "language", which is embedded in python ๐
here's an example directive:
@directive(dicts='versions')
version(pkg, ...):
...this directive allows you write:
class Foo(Package):
version(...)the @directive decorator handles a couple things for you:
- adds the class scope (
pkg) as an initial parameter when called, like a class method would. this allows you to modify a package from within a directive, while the package is still being defined. - it automatically adds a dictionary called
"versions"to the package so that you can refer topkg.versionsโจ
the (dicts='versions') part ensures that ALL packages in Spack will have a versions attribute after they're constructed, and that if no directive actually modified it, it will just be an empty dict.
this is just a modular way to add storage attributes to the Package class, and it's how Spack gets information from the packages to the core โ iconic architecture honestly ๐
static pop_default_args()โ pop default arguments ๐static pop_from_context()โ pop the last constraint from the context ๐ซstatic push_default_args(default_args)โ push default arguments ๐ฆstatic push_to_context(when_spec)โ add a spec to the context constraints ๐ธ
class spack.directives.DisableRedistribute(source, binary) โ Bases: object. the "don't redistribute this" energy ๐
โโ๏ธ
spack.directives.build_system(*values, **kwargs) โ build system directive, slay ๐ฅ
spack.directives.conflicts(conflict_spec: Spec | str, when: Spec | str | bool | None = None, msg: str | None = None) โ allows a package to define a conflict.
currently, a "conflict" is a concretized configuration that is known to be non-valid. for example, a package that is known not to be buildable with intel compilers can declare:
conflicts('%intel')to express the same constraint only when the 'foo' variant is activated:
conflicts('%intel', when='+foo')parameters
conflict_spec(spack.spec.Spec) โ constraint defining the known conflictwhen(spack.spec.Spec) โ optional constraint that triggers the conflictmsg(str) โ optional user defined message ๐
spack.directives.depends_on
spack.directives.depends_on(spec: Spec | str, when: Spec | str | bool |
None = None, type: Tuple[str, ...] | str = ('build', 'link'), patches: โ
Callable[[PackageBase | Dependency], None] | str | List[Callable[[โ
PackageBase | Dependency], None] | str] | None = None)This one creates a dict of deps with specs defining when they apply ๐
Parameters:
specโ the package and constraints depended onwhenโ when the dependent satisfies this, it has the dependency represented byspectypeโstror tuple of legal Spack deptypespatchesโ single result ofpatch()directive, astrto be passed to patch, or a list of these
This directive is to be used inside a Package definition to declare that the package requires other packages to be built first. Also peep the section "Dependency specs" in the Spack Packaging Guide ๐ซถ
spack.directives.extends
spack.directives.extends(spec, when=None, type=('build', 'run'),
patches=None)Same as depends_on, but also adds this package to the extendee list. In case of Python, also adds a dependency on python-venv ๐
Keyword arguments can be passed to extends() so that extension packages can pass parameters to the extendee's extension mechanism. Slay.
spack.directives.license
spack.directives.license(license_identifier: str, checked_by: str | โ
List[str] | None = None, when: str | bool | None = None)Add a new license directive, to specify the SPDX identifier the software is distributed under ๐
Parameters:
license_identifiersโ SPDX identifier specifying the license(s) the software is distributed under.checked_byโ string or list of strings indicating which github user checked the license (if any).whenโ A spec specifying when the license applies.
spack.directives.maintainers
spack.directives.maintainers(*names: str)Add a new maintainer directive, to specify maintainers in a declarative way ๐
Parameters:
namesโ GitHub username for the maintainer
spack.directives.patch
spack.directives.patch(url_or_filename: str, level: int = 1, when: Spec
| str | bool | None = None, working_dir: str = '.', reverse: bool =
False, sha256: str | None = None, archive_sha256: str | None = None) ->
Callable[[PackageBase | Dependency], None]Packages can declare patches to apply to source. You can optionally provide a when spec to indicate that a particular patch should only be applied when the package's spec meets certain conditions (e.g. a particular version) ๐ฉน
Parameters:
url_or_filenameโ url or relative filename of the patchlevelโ patch level (as in the patch shell command)whenโ optional anonymous spec that specifies when to apply the patchworking_dirโ dir to change to before applyingreverseโ reverse the patchsha256โ sha256 sum of the patch, used to verify the patch (only required for URL patches)archive_sha256โ sha256 sum of the archive, if the patch is compressed (only required for compressed URL patches)
spack.directives.provides
spack.directives.provides(*specs: Spec | str, when: Spec | str | bool |
None = None)Allows packages to provide a virtual dependency โจ
If a package provides "mpi", other packages can declare that they depend on "mpi", and spack can use the providing package to satisfy the dependency.
Parameters:
*specsโ virtual specs provided by this packagewhenโ condition when this provides clause needs to be considered
spack.directives.redistribute
spack.directives.redistribute(source=None, binary=None, when: Spec | str
| bool | None = None)Can be used inside a Package definition to declare that the package source and/or compiled binaries should not be redistributed ๐
By default, Packages allow source/binary distribution (i.e. in mirrors). Because of this, and because overlapping enable/disable specs are not allowed, this directive only allows users to explicitly disable redistribution for specs.
Heads up bestie: overlapping enable/disable specs are not allowed, so this only lets you explicitly disable redistribution for specs. No sneaky mixing.
spack.directives.requires
spack.directives.requires(*requirement_specs: str, policy='one_of',
when=None, msg=None)Allows a package to request a configuration to be present in all valid solutions ๐
For instance, a package that is known to compile only with GCC can declare:
requires("%gcc")A package that requires Apple-Clang on Darwin can declare instead:
requires("%apple-clang", when="platform=darwin", msg="Apple Clang is required on Darwin")Parameters:
requirement_specsโ spec expressing the requirementwhenโ optional constraint that triggers the requirement. IfNonethe requirement is applied unconditionally.msgโ optional user defined message
spack.directives.resource
spack.directives.resource(**kwargs)Define an external resource to be fetched and staged when building the package. Based on the keywords present in the dictionary the appropriate FetchStrategy will be used for the resource. Resources are fetched and staged in their own folder inside spack stage area, and then moved into the stage area of the package that needs them ๐ง
List of recognized keywords:
'when': (optional) represents the condition upon which the resource is needed'destination': (optional) path where to move the resource. This path must be relative to the main package stage area.'placement': (optional) gives the possibility to fine tune how the resource is moved into the main package stage area.
spack.directives.variant
spack.directives.variant(name: str, default: Any | None = None,
description: str = '', values: Sequence | Callable[[Any], bool] | None =
None, multi: bool | None = None, validator: Callable[[str, str, Tuple[โ
Any, ...]], None] | None = None, when: str | bool | None = None, sticky:
bool = False)Define a variant for the package ๐ฆ
Packager can specify a default value as well as a text description.
Parameters:
nameโ Name of the variantdefaultโ Default value for the variant, if not specified otherwise the default will beFalsefor a boolean variant and'nothing'for a multi-valued variantdescriptionโ Description of the purpose of the variantvaluesโ Either a tuple of strings containing the allowed values, or a callable accepting one value and returningTrueif it is validmultiโ IfFalseonly one value per spec is allowed for this variantvalidatorโ Optional group validator to enforce additional logic. It receives the package name, the variant name and a tuple of values and should raise an instance ofSpackErrorif the group doesn't meet the additional constraintswhenโ Optional condition on which the variant appliesstickyโ The variant should not be changed by the concretizer to find a valid concrete spec
Raises DirectiveError โ If arguments passed to the directive are invalid
spack.directives.version
spack.directives.version(ver: str | int, checksum: str | None = None, *,
preferred: bool | None = None, deprecated: bool | None = None, no_cache:
bool | None = None, url: str | None = None, extension: str | None =
None, expand: bool | None = None, fetch_options: dict | None = None,
md5: str | None = None, sha1: str | None = None, sha224: str | None =
None, sha256: str | None = None, sha384: str | None = None, sha512: str
| None = None, git: str | None = None, commit: str | None = None, tag: โ
str | None = None, branch: str | None = None, get_full_repo: bool | None
= None, submodules: Callable[[PackageBase], str | List[str] | bool] | โ
bool | None = None, submodules_delete: bool | None = None, svn: str | โ
None = None, hg: str | None = None, cvs: str | None = None, revision: โ
str | None = None, date: str | None = None)Adds a version and, if appropriate, metadata for fetching its code ๐
The version directives are aggregated into a versions dictionary attribute with Version keys and metadata values, where the metadata is stored as a dictionary of kwargs.
The (keyword) arguments are turned into a valid fetch strategy for code packages later. See spack.fetch_strategy.for_package_version().
๐๏ธ spack.directory_layout module
class spack.directory_layout.DirectoryLayout
class spack.directory_layout.DirectoryLayout(root, **kwargs)Bases: object
A directory layout is used to associate unique paths with specs. Different installations are going to want different layouts for their install, and they can use this to customize the nesting structure of spack installs. The default layout is:
<install root>/<platform-os-target>/<compiler>-<compiler version>/<name>-<version>-<hash>
The hash here is a SHA-1 hash for the full DAG plus the build spec.
The installation directory projections can be modified with the projections argument.
all_deprecated_specs()all_specs()build_packages_path(spec)create_install_directory(spec)deprecated_file_path(deprecated_spec, deprecator_spec=None)โ Gets full path to spec file for deprecated spec.
If the deprecator_spec is provided, use that. Otherwise, assume deprecated_spec is already deprecated and its prefix links to the prefix of its deprecator.
disable_upstream_check()ensure_installed(spec)โ ThrowsInconsistentInstallDirectoryErrorif:specprefix does not existspecprefix does not contain a spec file, or- We read a spec with the wrong DAG hash out of an existing install directory.
env_metadata_path(spec)property hidden_file_regexesmetadata_path(spec)path_for_spec(spec)โ Return absolute path from the root to a directory for the spec.
Okay so the rest of the layout helpers, no cap ๐
read_spec(path)โ reads what's inside a file and parses it as a spec, literally that's it ๐relative_path_for_spec(spec)โ (that's the whole listing bestie, no description given)remove_install_directory(spec, deprecated=False)โ nukes a prefix, plus any empty parent directories hanging off the root ๐๏ธ. RaisesRemoveFailedErrorif something goes sideways.spec_file_path(spec)โ gets you the full path to the spec file ๐specs_by_hash()โ (just chillin' there with no description)write_host_environment(spec)โ the host environment is a json file with os, kernel, and spack versioning โจ. We use it in the case that an analysis later needs to easily access this information.write_spec(spec, path)โ writes a spec out to a file, done ๐ซถexception spack.directory_layout.DirectoryLayoutError(message, long_msg=None)โ bases:SpackError. The superclass for directory layout errors ๐exception spack.directory_layout.ExtensionAlreadyInstalledError(spec, ext_spec)โ bases:DirectoryLayoutError. Raised when an extension is added to a package that already has it ๐ญexception spack.directory_layout.ExtensionConflictError(spec, ext_spec, conflict)โ bases:DirectoryLayoutError. Also raised when an extension is added to a package that already has it โ yeah, same description, that's what the page says, don't come for me ๐exception spack.directory_layout.InconsistentInstallDirectoryError(message, long_msg=None)โ bases:DirectoryLayoutError. Raised when a package seems to be installed to the wrong place ๐exception spack.directory_layout.InvalidDirectoryLayoutParametersError(message, long_msg=None)โ bases:DirectoryLayoutError. Raised when a invalid directory layout parameters are supplied ๐ฉexception spack.directory_layout.InvalidExtensionSpecError(message, long_msg=None)โ bases:DirectoryLayoutError. Raised when an extension file has a bad spec in it ๐exception spack.directory_layout.RemoveFailedError(installed_spec, prefix, error)โ bases:DirectoryLayoutError. Raised when aDirectoryLayoutcannot remove an install prefix ๐คexception spack.directory_layout.SpecReadError(message, long_msg=None)โ bases:DirectoryLayoutError. Raised when directory layout can't read a spec ๐
๐ spack.error module
exception spack.error.FetchError(message, long_message=None)โ bases:SpackError. The superclass for fetch-related errors ๐ซถexception spack.error.NoHeadersError(message, long_message=None)โ bases:SpackError. Raised when package headers are requested but cannot be found ๐ญexception spack.error.NoLibrariesError(message_or_name, prefix=None)โ bases:SpackError. Raised when package libraries are requested but cannot be found ๐exception spack.error.SpackError(message, long_message=None)โ bases:Exception. This is the superclass for all Spack errors, the main character of error handling ๐. Subclasses can be found in the modules they have to do with.die()โ (listed with no description, slay)property long_messageโ the long version of the message, as a property โจprint_context()โ prints extended debug information about this exception ๐. This is usually printed when the top-level Spack error handler callsdie(), but it can be called separately beforehand if a lower-level error handler needs to print error context and continue without raising the exception to the top level โ very "I'll handle it myself" energy.
exception spack.error.SpecError(message, long_message=None)โ bases:SpackError. The superclass for all errors that occur while constructing specs ๐งexception spack.error.UnsatisfiableSpecError(provided, required, constraint_type)โ bases:SpecError. Raised when a spec conflicts with package constraints ๐จ. For original concretizer, provide the requirement that was violated when raising.exception spack.error.UnsupportedPlatformError(message)โ bases:SpackError. Raised by packages when a platform is not supported ๐spack.error.debug = 0โ at what level we should write stack traces or short error messages ๐ . This is module-scoped because it needs to be set very early, like, before anything else gets a say.
๐ฆ spack.extensions module
Service functions and classes to implement the hooks for Spack's command extensions โจ.
exception spack.extensions.CommandNotFoundError(cmd_name)โ bases:SpackError. Exception class thrown when a requested command is not recognized as such ๐ญexception spack.extensions.ExtensionNamingError(path)โ bases:SpackError. Exception class thrown when a configured extension does not follow the expected naming convention ๐ฉspack.extensions.ensure_extension_loaded(extension, *, path)โ (no description on this one, it's giving mysterious)spack.extensions.extension_name(path)โ returns the name of the extension in the path passed as argument ๐- Parameters:
path(str) โ path where the extension resides - Returns: The extension name.
- Raises
ExtensionNamingErrorโ ifpathdoes not match the expected format for a Spack command extension.
- Parameters:
spack.extensions.extension_paths_from_entry_points() -> List[str]โ loads extensions from a Python package's entry points ๐ธ. A python package can register entry point metadata so that Spack can find its extensions by adding the following to the project'spyproject.toml:
[project.entry-points."spack.extensions"]
baz = "baz:get_spack_extensions"The function get_spack_extensions returns paths to the package's spack extensions ๐ซถ
spack.extensions.get_command_paths()โ returns the list of paths where to search for command files ๐spack.extensions.get_extension_paths()โ returns the list of canonicalized extension paths fromconfig:extensions๐spack.extensions.get_module(cmd_name)โ imports the extension module for a particular command name and returns it ๐ฆ- Parameters:
cmd_name(str) โ name of the command for which to get a module (contains-, not_). Noted, bestie, hyphen gang.
- Parameters:
spack.extensions.get_template_dirs()โ returns the list of directories where to search for templates in extensions ๐spack.extensions.load_command_extension(command, path)โ loads a command extension from the path passed as argument ๐- Parameters:
command(str) โ name of the command (contains-, not_).path(str) โ base path of the command extension
- Returns: A valid module if found and loadable;
Noneif not found. Module loading exceptions are passed through โ so like, we don't catch them for you ๐
- Parameters:
spack.extensions.load_extension(name: str) -> strโ loads a single extension into the'spack.extensions'package ๐ง- Parameters:
nameโ name of the extension
- Parameters:
spack.extensions.path_for_extension(target_name: str, *, paths: List[str]) -> strโ returns the test root dir for a given extension ๐- Parameters:
target_name(str) โ name of the extension to test*pathsโ paths where the extensions reside
- Returns: Root directory where tests should reside or
None
- Parameters:
๐ฅ spack.fetch_strategy module
Fetch strategies are used to download source code into a staging area in order to build it ๐. They need to define the following methods:
fetch()โ this should attempt to download/check out source from somewhere.check()โ apply a checksum to the downloaded source code, e.g. for an archive. May not do anything if the fetch method was safe to begin with.expand()โ expand (e.g., an archive) downloaded file to source, with the standard stage source path as the destination directory.reset()โ restore original state of downloaded code. Used by clean commands. This may just remove the expanded source and re-expand an archive, or it may run something likegit reset --hard๐archive()โ archive a source directory, e.g. for creating a mirror.
class spack.fetch_strategy.BundleFetchStrategy(**kwargs)โ bases:FetchStrategy. Fetch strategy associated with bundle, or no-code, packages ๐ฆ.
Having a basic fetch strategy is a requirement for executing post-install hooks. Consequently, this class provides the API but does little more than log messages ๐ด
TODO: Remove this class by refactoring resource handling and the link between composite stages and composite fetch strategies (see #11981).
property cachableโ reportsFalseas there is no code to cache ๐fetch()โ simply reports success โ there is no code to fetch โจmirror_id()โBundlePackagesdon't have a mirror id.source_id()โBundlePackagesdon't have a source id.url_attr: str | None = ''โ there is no associated URL keyword inversion()for no-code packages but this property is required for some strategy-related functions (e.g.,check_pkg_attributes) ๐ซ
class spack.fetch_strategy.CacheURLFetchStrategy(url=None, checksum=None, **kwargs)โ bases:URLFetchStrategy. The resource associated with a cache URL may be out of date โฐfetch()โ fetch source code archive or repo.- Returns:
Trueon success,Falseon failure. - Return type:
bool
- Returns:
exception spack.fetch_strategy.ChecksumError(message, long_message=None)โ bases:FetchError. Raised when archive fails to checksum ๐class spack.fetch_strategy.CvsFetchStrategy(**kwargs)โ bases:VCSFetchStrategy. Fetch strategy that gets source code from a CVS repository ๐๏ธ. Use like this in a package:
version('name',
cvs=':pserver:anonymous@www.example.com:/cvsroot%module=modulename')Optionally, you can provide a branch and/or a date for the URL:
version('name',
cvs=':pserver:anonymous@www.example.com:/cvsroot%module=modulename',
branch='branchname', date='date')Repositories are checked out into the standard stage source path directory ๐
archive(destination)
Ok so this one wraps up the stuff you've already downloaded into an archive โ like, for a mirror. ๐
- For downloaded files, it should preserve the checksum of the original file. gotta keep that integrity, no cap.
- For repositories, it should just create an expandable tarball out of the downloaded repository. so it's tarball time for repos.
property cachable
Tells you whether the fetcher is even capable of caching the resource it goes and gets. ๐
This is generally determined by whether the resource is identifiably associated with a specific package version. iykyk.
- Returns โ
Trueif it can cache,Falseotherwise. - Return type โ
bool
property cvs
literally just a property named cvs. no description given in the docs, bestie โ that's the whole vibe. ๐ฆ
fetch()
Goes out and fetches the source code archive or repo. the main character moment of every fetch strategy. โจ
- Returns โ
Trueon success,Falseon failure. - Return type โ
bool
mirror_id()
This is a unique ID for a source that's meant to help identify reuse of resources across packages. ๐ฅ
It's unique like source-id, but heads up: it does not include the package name, and it's not necessarily easy for a human to create themselves. so don't @ me if you can't hand-write it.
optional_attrs: List[str] = ['branch', 'date']
the optional attributes list, verbatim. ๐ธ
reset()
Reverts to a freshly downloaded state โ like a factory reset but for your source. ๐
For archive files, this may just re-expand the archive. easy peasy.
source_id()
A unique ID for the source. ๐
It's intended that a human could easily generate this themselves using the information available to them in the Spack package. so like, it's for you.
The returned value is added to the content which determines the full hash for a package using str().
url_attr: str | None = 'cvs'
The URL attribute must be specified either at the package class level, or as a keyword argument to version(). it's used to distinguish fetchers for different versions in the package DSL. no skipping this one. ๐
exception spack.fetch_strategy.ExtrapolationError(message, long_message=None)
- Bases:
FetchError
Raised when we can't extrapolate a version for a package. ๐ญ it's giving "we tried, we really did".
exception spack.fetch_strategy.FailedDownloadError(url, msg='')
- Bases:
FetchError
Raised when a download fails. yeah. that's the whole thing. ๐
class spack.fetch_strategy.FetchAndVerifyExpandedFile(url, archive_sha256: str, expanded_sha256: str)
- Bases:
URLFetchStrategy
A fetch strategy that verifies the content digest during fetching, as well as after expanding it. double-checked, double-slay. ๐
expand()
Verify the checksum after expanding the archive.
class spack.fetch_strategy.FetchStrategy(**kwargs)
- Bases:
object
Superclass of all fetch strategies โ the mother of the whole fetch family tree. ๐
archive(destination)
Create an archive of the downloaded data for a mirror.
- For downloaded files, this should preserve the checksum of the original file.
- For repositories, it should just create an expandable tarball out of the downloaded repository.
property cachable
Whether the fetcher is capable of caching the resource it retrieves.
This generally is determined by whether the resource is identifiably associated with a specific package version.
- Returns โ
Trueif can cache,Falseotherwise. - Return type โ
bool
check()
Checksums the archive fetched by this FetchStrategy. yes, we're checking. always checking. ๐
expand()
Expands the downloaded archive into the stage source path.
fetch()
Fetch source code archive or repo.
- Returns โ
Trueon success,Falseon failure. - Return type โ
bool
classmethod matches(args)
A predicate that matches fetch strategies to arguments of the version directive. like a little matchmaker. ๐
- Parameters โ
argsโ arguments of theversiondirective
mirror_id()
A unique ID for a source that's intended to help identify reuse of resources across packages.
It's unique like source-id, but it doesn't include the package name and is not necessarily easy for a human to create themselves.
optional_attrs: List[str] = []
empty list, base class behaviour. โญ
reset()
Revert to freshly downloaded state.
For archive files, this may just re-expand the archive.
set_package(package)
hands the fetcher its package. that's the whole gig.
source_id()
A unique ID for the source.
It is intended that a human could easily generate this themselves using the information available to them in the Spack package.
The returned value is added to the content which determines the full hash for a package using str().
url_attr: str | None = None
The URL attribute must be specified either at the package class level, or as a keyword argument to version(). It is used to distinguish fetchers for different versions in the package DSL.
exception spack.fetch_strategy.FetcherConflict(message, long_message=None)
- Bases:
FetchError
Raised for packages with invalid fetch attributes. ๐ฉ not the vibe.
class spack.fetch_strategy.FsCache(root)
- Bases:
object
the filesystem cache situation. ๐
destroy()
nukes it. that's it, that's the method.
fetcher(target_path, digest, **kwargs)
builds you a fetcher for a target path and digest. plus whatever **kwargs you feel like throwing in. ๐ซถ
store(fetcher, relative_dest)
stores a fetcher at a destination relative to the cache root.
class spack.fetch_strategy.GCSFetchStrategy(*args, **kwargs)
- Bases:
URLFetchStrategy
A FetchStrategy that pulls from a GCS bucket. โ๏ธ google cloud girlies rise.
fetch()
Fetch source code archive or repo.
- Returns โ
Trueon success,Falseon failure. - Return type โ
bool
url_attr: str | None = 'gs'
The URL attribute must be specified either at the package class level, or as a keyword argument to version(). It is used to distinguish fetchers for different versions in the package DSL.
class spack.fetch_strategy.GitFetchStrategy(**kwargs)
- Bases:
VCSFetchStrategy
A fetch strategy that gets source code from a git repository. use it like this in a package:
version('name', git='https://github.com/project/repo.git')Optionally, you can provide a branch, or commit to check out, e.g.:
version('1.1', git='https://github.com/project/repo.git', tag='v1.1')You can use these three optional attributes in addition to git:
branch: particular branch to build from (default is the repository's default branch)tag: particular tag to check outcommit: particular commit hash in the repo
Repositories are cloned into the standard stage source path directory. ๐
archive(destination)
Create an archive of the downloaded data for a mirror.
- For downloaded files, this should preserve the checksum of the original file.
- For repositories, it should just create an expandable tarball out of the downloaded repository.
property cachable
Whether the fetcher is capable of caching the resource it retrieves.
This generally is determined by whether the resource is identifiably associated with a specific package version.
- Returns โ
Trueif can cache,Falseotherwise. - Return type โ
bool
clone(dest=None, commit=None, branch=None, tag=None, bare=False)
Clone a repository to a path. ๐งฌ
This method handles cloning from git, but does not require a stage.
- Parameters
dest(strorNone) โ the path into which the code is cloned. IfNone, requires a stage and uses the stage's source path.commit(strorNone) โ a commit to fetch from the remote. Only one ofcommit,branch, andtagmay be non-None.branch(strorNone) โ a branch to fetch from the remote.tag(strorNone) โ a tag to fetch from the remote.bare(bool) โ execute a "bare" git clone (--bareoption to git)
fetch()
Fetch source code archive or repo.
- Returns โ
Trueon success,Falseon failure. - Return type โ
bool
property git
the git property. โจ
property git_version
the version of git you've got. ๐
git_version_re = 'git version (\\S+)'
the regex it uses to sniff out that version, verbatim.
mirror_id()
A unique ID for a source that's intended to help identify reuse of resources across packages.
It's unique like source-id, but it doesn't include the package name and is not necessarily easy for a human to create themselves.
optional_attrs: List[str] = ['tag', 'branch', 'commit', 'submodules', 'get_full_repo', 'submodules_delete']
the optional attrs list for git, exactly as written. ๐
protocol_supports_shallow_clone()
Shallow clone operations (--depth #) are not supported by the basic HTTP protocol or by no-protocol file specifications. use (e.g.) https:// or file:// instead. ๐จ fr fr, don't try the bare ones.
reset()
Revert to freshly downloaded state.
For archive files, this may just re-expand the archive.
source_id()
A unique ID for the source.
It is intended that a human could easily generate this themselves using the information available to them in the Spack package.
The returned value is added to the content which determines the full hash for a package using str(). ๐
url_attr: str | None = 'git'โ so this URL attribute has to be set either on the package class itself, or as a keyword argument toversion(). It's literally the thing that tells the fetchers apart for different versions in the package DSL. ๐static version_from_git(git_exe)โ hand it a git executable and boom, you get back theVersion. Heads up though: this will fail if the output can't be parsed into a validVersion. ๐ญ
๐ class spack.fetch_strategy.GoFetchStrategy(**kwargs)
Bases: VCSFetchStrategy
Okay so this is the fetch strategy that employs the go get infrastructure โ it's giving Go girlie energy fr fr. ๐น
Use it in a package like this:
version('name',
go='github.com/monochromegane/the_platinum_searcher/...')Ngl,go getdoes not natively support versions โ they can be faked withgit. That's just how it is, bestie. ๐
The fetched source gets moved to the standard stage sourcepath directory during the expand step. โจ
archive(destination)โ creates an archive of the downloaded data for a mirror. For downloaded files this should preserve the checksum of the original file. For repositories it should just create an expandable tarball out of the downloaded repository. ๐expand()โ expands the downloaded archive into the stage source path.fetch()โ fetches the source code archive or repo. ReturnsTrueon success,Falseon failure. Return type:bool.property goโ it's a property. That's the whole vibe. ๐ฆproperty go_versionโ also a property, also iconic.reset()โ reverts to freshly downloaded state. For archive files, this may just re-expand the archive.url_attr: str | None = 'go'โ same deal as before: the URL attribute must be specified either at the package class level, or as a keyword argument toversion(). It's used to distinguish fetchers for different versions in the package DSL.
๐ธ class spack.fetch_strategy.HgFetchStrategy(**kwargs)
Bases: VCSFetchStrategy
Fetch strategy that gets source code from a Mercurial repository. Use it like this in a package:
version('name', hg='https://jay.grs.rwth-aachen.de/hg/lwm2')Optionally, you can provide a branch, or revision to check out, e.g.:
version('torus',
hg='https://jay.grs.rwth-aachen.de/hg/lwm2',
branch='torus')You can use the optional 'revision' attribute to check out a branch, tag, or particular revision in hg. To prevent non-reproducible builds, using a moving target like a branch is discouraged. ๐
- revision: Particular revision, branch, or tag.
Repositories are cloned into the standard stage source path directory. ๐ซถ
archive(destination)โ creates an archive of the downloaded data for a mirror. For downloaded files this should preserve the checksum of the original file. For repositories it should just create an expandable tarball out of the downloaded repository.property cachableโ whether the fetcher is capable of caching the resource it retrieves. This generally is determined by whether the resource is identifiably associated with a specific package version. ReturnsTrueif it can cache,Falseotherwise. Return type:bool.fetch()โ fetches the source code archive or repo. ReturnsTrueon success,Falseon failure. Return type:bool.property hgโ Returns:Executable: the hg executable. ๐งmirror_id()โ this is a unique ID for a source that is intended to help identify reuse of resources across packages. It's unique like source-id, but it does not include the package name and is not necessarily easy for a human to create themselves.optional_attrs: List[str] = ['revision']โ the optional attributes, right there in a list, exactly like that. โญreset()โ reverts to freshly downloaded state. For archive files, this may just re-expand the archive.source_id()โ a unique ID for the source. It's intended that a human could easily generate this themselves using the information available to them in the Spack package. The returned value is added to the content which determines the full hash for a package usingstr().url_attr: str | None = 'hg'โ the URL attribute must be specified either at the package class level, or as a keyword argument toversion(). It is used to distinguish fetchers for different versions in the package DSL.
๐ exception spack.fetch_strategy.InvalidArgsError(pkg=None, version=None, **args)
Bases: FetchError
Raised when a version can't be deduced from a set of arguments. Yeah, the args said nothing useful, so it throws hands. ๐ญ
๐ exception spack.fetch_strategy.NoArchiveFileError(message, long_message=None)
Bases: FetchError
Raised when an archive file is expected but none exists. It's giving "where is she" energy.
๐ exception spack.fetch_strategy.NoCacheError(message, long_message=None)
Bases: FetchError
Raised when there is no cached archive for a package. The cache said "not today" bestie.
๐ exception spack.fetch_strategy.NoDigestError(message, long_message=None)
Bases: FetchError
Raised after an attempt to checksum when the URL has no digest. No digest, no deal. ๐
๐ exception spack.fetch_strategy.NoStageError(method)
Bases: FetchError
Raised when fetch operations are called before set_stage(). Order matters, always has. โ
๐ฅ class spack.fetch_strategy.OCIRegistryFetchStrategy(url=None, checksum=None, **kwargs)
Bases: URLFetchStrategy
fetch()โ fetches the source code archive or repo. ReturnsTrueon success,Falseon failure. Return type:bool.
๐ class spack.fetch_strategy.S3FetchStrategy(*args, **kwargs)
Bases: URLFetchStrategy
FetchStrategy that pulls from an S3 bucket. Bucket girlie behavior. ๐ชฃ
fetch()โ fetches the source code archive or repo. ReturnsTrueon success,Falseon failure. Return type:bool.url_attr: str | None = 's3'โ the URL attribute must be specified either at the package class level, or as a keyword argument toversion(). It is used to distinguish fetchers for different versions in the package DSL.
๐ class spack.fetch_strategy.SvnFetchStrategy(**kwargs)
Bases: VCSFetchStrategy
Fetch strategy that gets source code from a subversion repository. Use it like this in a package:
version('name', svn='http://www.example.com/svn/trunk')Optionally, you can provide a revision for the URL:
version('name', svn='http://www.example.com/svn/trunk',
revision='1641')Repositories are checked out into the standard stage source path directory. ๐
archive(destination)โ creates an archive of the downloaded data for a mirror. For downloaded files this should preserve the checksum of the original file. For repositories it should just create an expandable tarball out of the downloaded repository.property cachableโ whether the fetcher is capable of caching the resource it retrieves. This generally is determined by whether the resource is identifiably associated with a specific package version. ReturnsTrueif it can cache,Falseotherwise. Return type:bool.fetch()โ fetches the source code archive or repo. ReturnsTrueon success,Falseon failure. Return type:bool.mirror_id()โ a unique ID for a source that is intended to help identify reuse of resources across packages. It's unique like source-id, but it does not include the package name and is not necessarily easy for a human to create themselves.optional_attrs: List[str] = ['revision']โ again, the list is literally['revision']. Slay.reset()โ reverts to freshly downloaded state. For archive files, this may just re-expand the archive.source_id()โ a unique ID for the source. It's intended that a human could easily generate this themselves using the information available to them in the Spack package. The returned value is added to the content which determines the full hash for a package usingstr().property svnโ property. That's it, that's the vibe check. ๐ฆurl_attr: str | None = 'svn'โ the URL attribute must be specified either at the package class level, or as a keyword argument toversion(). It is used to distinguish fetchers for different versions in the package DSL.
๐ class spack.fetch_strategy.URLFetchStrategy(url=None, checksum=None, **kwargs)
Bases: FetchStrategy
Okay so URLFetchStrategy pulls source code from a URL for an archive, checks the archive against a checksum, and decompresses the archive. The destination for the resulting file(s) is the standard stage path. Main character energy, honestly. โจ
archive(destination)โ just moves this archive to the destination. Simple. No notes.property archive_fileโ path to the source archive within this stage directory.property cachableโ whether the fetcher is capable of caching the resource it retrieves. This generally is determined by whether the resource is identifiably associated with a specific package version. ReturnsTrueif it can cache,Falseotherwise. Return type:bool.property candidate_urlsโ
Okay so this class keeps going โ here's the rest of its methods, and no, we are not skipping any of them ๐
check()โ checks the downloaded archive against a checksum digest. Literally a no-op if this stage checks code out of a repository (nothing to checksum, it just vibes โจ).curl(property) โ bestie it's a property namedcurl. That is the entire documentation. Iconic minimalism, no notes ๐.expand()โ expands the downloaded archive into the stage source path. Unzip and slay ๐ฆ.fetch()โ fetches the source code archive or repo.- Returns โ
Trueon success,Falseon failure. - Return type โ
bool
- Returns โ
mirror_id()โ a unique ID for a source, and the whole point is helping identify reuse of resources across packages. It's unique like source-id, but it does not include the package name, and it's not necessarily easy for a human to create themselves. So: unique, yes; human-friendly, absolutely not ๐.optional_attrsโList[str] = ['sha256', 'md5', 'sha1', 'sha224', 'sha384', 'sha512', 'checksum']reset()โ removes the source path if it exists, then re-expands the archive. Clean slate energy ๐ซ.source_id()โ a unique ID for the source. The idea is that a human could easily generate this themselves using the information available to them in the Spack package (so, the opposite ofmirror_id, lmao). The returned value gets added to the content which determines the full hash for a package usingstr().url_attr: str | None = 'url'โ the URL attribute has to be specified either at the package class level, or as a keyword argument toversion(). It's what distinguishes fetchers for different versions in the package DSL. Main character attribute, honestly ๐.
๐ง spack.fetch_strategy.VCSFetchStrategy(**kwargs)
Bases: FetchStrategy
Superclass for version control system fetch strategies. Ngl, this is the parent of all the git/svn/hg girlies ๐.
Like all fetchers, VCS fetchers are identified by the attributes passed to the version directive. The optional_attrs for a VCS fetch strategy represent types of revisions, e.g. tags, branches, commits, etc. โ all the little revision flavors ๐.
The required attributes (git, svn, etc.) are used to specify the URL and to distinguish a VCS fetch strategy from a URL fetch strategy. That's the vibe check that tells them apart ๐ฅ.
archive(destination, *, exclude: str | None = None)โ create an archive of the downloaded data for a mirror.- For downloaded files, this should preserve the checksum of the original file.
- For repositories, it should just create an expandable tarball out of the downloaded repository.
check()โ checksums the archive fetched by thisFetchStrategy.expand()โ expands the downloaded archive into the stage source path.
โจ spack.fetch_strategy.all_strategies
spack.fetch_strategy.all_strategies = [<class 'spack.fetch_strategy.BundleFetchStrategy'>, <class 'spack.fetch_strategy.URLFetchStrategy'>, <class 'spack.fetch_strategy.CacheURLFetchStrategy'>, <class 'spack.fetch_strategy.GoFetchStrategy'>, <class 'spack.fetch_strategy.GitFetchStrategy'>, <class 'spack.fetch_strategy.CvsFetchStrategy'>, <class 'spack.fetch_strategy.SvnFetchStrategy'>, <class 'spack.fetch_strategy.HgFetchStrategy'>, <class 'spack.fetch_strategy.S3FetchStrategy'>, <class 'spack.fetch_strategy.GCSFetchStrategy'>, <class 'spack.fetch_strategy.FetchAndVerifyExpandedFile'>]That's the list of all fetch strategies, created by the FetchStrategy metaclass. The whole roster, the full squad, all eleven of them ๐ซถ.
๐ spack.fetch_strategy.check_pkg_attributes(pkg)
Finds ambiguous top-level fetch attributes in a package. Currently this only ensures that two or more VCS fetch strategies are not specified at once โ like, pick a lane bestie, one VCS at a time ๐ญ.
๐ธ spack.fetch_strategy.fetcher(cls)
Decorator used to register fetch strategies. Cute little registration moment โจ.
๐ spack.fetch_strategy.for_package_version(pkg, version=None)
Determines a fetch strategy based on the arguments supplied to version() in the package description.
๐ฆ spack.fetch_strategy.from_kwargs(**kwargs)
Constructs an appropriate FetchStrategy from the given keyword arguments.
- Parameters โ
**kwargsโ dictionary of keyword arguments, e.g. from aversion()directive in a package. - Returns โ the fetch strategy that matches the args, based on attribute names (e.g.,
git,hg, etc.) - Return type โ
Callable - Raises
spack.error.FetchErrorโ if nofetch_strategymatches the args. So if nothing matches, it will throw hands ๐.
๐ spack.fetch_strategy.from_list_url(pkg)
If a package provides a URL which lists URLs for resources by version, this can create a fetcher for a URL discovered for the specified package's version. Very "found it in the list" of it ๐ .
โญ spack.fetch_strategy.from_url(url)
Given a URL, find an appropriate fetch strategy for it. Currently just gives you a URLFetchStrategy that uses curl.
TODO: make this return appropriate fetch strategies for other types of URLs. Future self, you got this ๐ซก.
๐ spack.fetch_strategy.from_url_scheme(url, *args, **kwargs)
Finds a suitable FetchStrategy by matching its url_attr with the scheme in the given url.
๐ซ spack.fetch_strategy.stable_target(fetcher)
Returns whether the fetcher target is expected to have a stable checksum. This is only true if the target is a preexisting archive file โ so, file yes, repo no, iykyk.
๐ spack.fetch_strategy.verify_checksum(file, digest)
๐ spack.fetch_strategy.warn_content_type_mismatch(subject, content_type='HTML')
๐ spack.filesystem_view module
๐ class spack.filesystem_view.FilesystemView(root: str, layout: DirectoryLayout, *, projections: Dict | None = None, ignore_conflicts: bool = False, verbose: bool = False, link_type: str = 'symlink')
Bases: object
Governs a filesystem view that is located at a certain root-directory. Packages are linked from their install directories into a common file hierarchy โ one big happy shared folder situation ๐ซถ.
In distributed filesystems, loading each installed package separately can lead to slow-downs due to too many directories being traversed. This can be circumvented by loading all needed modules into a common directory structure. Basically: fewer directories, less sadness โจ.
add_specs(*specs, **kwargs)โ add given specs to view.- Should accept
with_dependenciesas a keyword argument (defaultTrue) to indicate whether or not dependencies should be activated as well. - Should accept an
excludekeyword argument containing a list of regexps that filter out matching spec names. - This method should make use of
activate_standalone.
- Should accept
add_standalone(spec)โ add (link) a standalone package into this view.check_added(spec)โ check if the given concrete spec is active in this view.get_all_specs()โ get all specs currently active in this view.get_projection_for_spec(spec)โ get the projection in this view for a spec.get_spec(spec)โ return the actual spec linked in this view (i.e. do not look it up in the database by name).speccan be a name or a spec from which the name is extracted.- As there can only be a single version active for any spec, the name is enough to identify the spec in the view.
- If no spec is present, returns
None.
print_status(*specs, **kwargs)โ print a short summary about the given specs, detailing whetherโฆ- โฆthey are active in the view.
- โฆthey are active but the activated version differs.
- โฆthey are not active in the view.
- Takes the
with_dependencieskeyword argument so that the status of dependencies is printed as well.
remove_specs(*specs, **kwargs)โ removes given specs from view.- Should accept
with_dependenciesas a keyword argument (defaultTrue) to indicate whether or not dependencies should be deactivated as well. - Should accept
with_dependentsas a keyword argument (defaultTrue) to indicate whether or not dependents on the deactivated specs should be removed as well. - Should accept an
excludekeyword argument containing a list of regexps that filter out matching spec names. - This method should make use of
deactivate_standalone.
- Should accept
remove_standalone(spec)โ remove (unlink) a standalone package from this view.
๐ฆ class spack.filesystem_view.YamlFilesystemView(root: str, layout: DirectoryLayout, *, projections: Dict | None = None, ignore_conflicts: bool = False, verbose: bool = False, link_type: str = 'symlink')
Bases: FilesystemView
A filesystem view to work with a yaml based directory layout. Same energy, but with YAML on top ๐ .
add_specs(*specs, **kwargs)โ add given specs to view.- Should accept
with_dependenciesas a keyword argument (defaultTrue) to indicate whether or not dependencies should be activated as well. - Should accept an
excludekeyword argument containing a list of regexps that filter out matching spec names. - This method should make use of
activate_standalone.
- Should accept
add_standalone(spec)โ add (link) a standalone package into this view.check_added(spec)โ check if the given concrete spec is active in this view.clean()โ no description given, it's just here, doing its little cleanup moment ๐งน.get_all_specs()โ get all specs currently active in this view.get_conflicts(*specs)โ return a list of tuples (<spec>,<spec in view>) where the spec active in the view differs from the one to be activated.get_path_meta_folder(spec)โ get path to meta folder for either spec or spec name.get_projection_for_spec(spec)โ return the projection for a spec in this view.- Relies on the ordering of projections to avoid ambiguity. Order matters, bestie, always ๐ธ.
get_spec(spec)โ return the actual spec linked in this view (i.e. do not look it up in the database by name).speccan be a name or a spec from which the name is extracted.- As there can only be a single version active for any spec, the name is enough to identify the spec in the view.
- If no spec is present, returns
None.
link_meta_folder(spec)โ no description given, but the name isn't exactly hiding what it does ๐.merge(spec, ignore=None)โ merges stuff in, no extra deets given ๐print_conflict(spec_active, spec_specified, level='error')โ singular print function for spec conflicts. one conflict, one print, very demure โจprint_status(*specs, **kwargs)โ prints a short lil summary about the specs you handed it, detailing whetherโฆ- they're active in the view,
- they're active but the activated version differs,
- they're not active in the view.
It also takes the with_dependencies keyword argument so the status of dependencies gets printed too ๐ซถ
read_projections()โ reads those projections, that's the whole viberemove_files(files)โ removes the files, iconic behaviourremove_specs(*specs, **kwargs)โ removes the given specs from the view ๐ญ
It should accept with_dependencies as a keyword argument (default True) to say whether or not dependencies should be deactivated too.
It should also accept with_dependents as a keyword argument (default True) to say whether or not dependents of the deactivated specs should be removed as well.
And it should take an exclude keyword argument holding a list of regexps that filter out matching spec names ๐
This method should make use of deactivate_standalone.
remove_standalone(spec)โ remove (unlink) a standalone package from this view ๐งunlink_meta_folder(spec)โ unlinks that meta folder, periodtunmerge(spec, ignore=None)โ the un-merge, obviouslywrite_projections()โ writes the projections, slay
๐ฆ spack.graph module ๐ฆ
Functions for graphing DAGs of dependencies ๐
This file has the code for graphing DAGs of software packages (i.e. Spack specs). There are two main functions you probably actually care about:
graph_ascii() will output a colored graph of a spec in ascii format, kinda like the graph git shows you with git log --graph, e.g.:
o mpileaks
|\
| |\
| o | callpath
|/| |
| |\|
| |\ \
| | |\ \
| | | | o adept-utils
| |_|_|/|
|/| | | |
o | | | | mpi
/ / / /
| | o | dyninst
| |/| |
|/|/| |
| | |/
| o | libdwarf
|/ /
o | libelf
/
o boostgraph_dot() will output a graph of a spec (or multiple specs) in dot format โญ
โจ class spack.graph.AsciiGraph
Bases: object
write(spec, color=None, out=None)โ writes out an ascii graph of the spec you provided ๐
Arguments: spec โ the spec to graph. This only handles one spec at a time, bestie, one at a time.
Optional arguments:
- out โ file object to write out to (default is
sys.stdout) - color โ whether to write in color. Default is to autodetect based on the output file ๐
๐ class spack.graph.DAGWithDependencyTypes
Bases: DotGraphBuilder
DOT graph with link,run nodes grouped together and edges colored according to the dependency types ๐ฅ
edge_entry(edge)โ returns a tuple of(parent_id, child_id, edge_options)node_entry(node)โ returns a tuple of(node_id, node_options)visit(edge)โ visits an edge and builds up entries to render the graph
๐ class spack.graph.DotGraphBuilder
Bases: object
Visit edges of a graph and build DOT options for nodes and edges, iykyk
context()โ returns the context to be used to render the DOT graph templateedge_entry(edge: DependencySpec) -> Tuple[str, str, str]โ returns a tuple of(parent_id, child_id, edge_options)node_entry(node: Spec) -> Tuple[str, str]โ returns a tuple of(node_id, node_options)render() -> strโ returns a string with the output in DOT format โญvisit(edge: DependencySpec)โ visits an edge and builds up entries to render the graph
๐ธ class spack.graph.SimpleDAG
Bases: DotGraphBuilder
Simple DOT graph, with nodes colored uniformly and edges without properties. lowkey minimalist ๐
edge_entry(edge)โ returns a tuple of(parent_id, child_id, edge_options)node_entry(node)โ returns a tuple of(node_id, node_options)
๐ฆ class spack.graph.StaticDag
Bases: DotGraphBuilder
DOT graph for possible dependencies ๐
edge_entry(edge)โ returns a tuple of(parent_id, child_id, edge_options)node_entry(node)โ returns a tuple of(node_id, node_options)
๐ spack.graph.find(seq, predicate)
Find the index in seq for which predicate is True.
It searches the sequence and returns the index of the element for which the predicate evaluates to True. Returns -1 if the predicate does not evaluate to True for any element in seq ๐
๐ง spack.graph.graph_ascii(spec, node='o', out=None, debug=False, indent=0, color=None, depflag: int = 15)
That's it, that's the whole signature, go off โจ
๐ spack.graph.graph_dot(specs: List[Spec], builder: DotGraphBuilder | None = None, depflag: int = 15, out: TextIO | None = None)
DOT graph of the concrete specs passed as input.
Parameters
- specs โ specs to be represented
- builder โ builder to use to render the graph
- depflag โ dependency types to consider
- out โ optional output stream. If
None,sys.stdoutis used ๐ซถ
๐ซ spack.graph.static_graph_dot(specs: List[Spec], depflag: int = 15, out: TextIO | None = None)
Static DOT graph with edges to all possible dependencies.
Parameters
- specs โ abstract specs to be represented
- depflag โ dependency types to consider
- out โ optional output stream. If
None,sys.stdoutis used ๐
๐ spack.hash_types module ๐
Definitions that control how Spack creates Spec hashes. the lore โจ
๐ class spack.hash_types.SpecHashDescriptor(depflag: int, package_hash, name, override=None)
Bases: object
This class defines how hashes are generated on Spec objects.
Spec hashes in Spack are generated from a serialized (e.g., with YAML) representation of the Spec graph. That representation may only include certain dependency types, and it may optionally include a canonicalized hash of the package.py for each node in the graph ๐ฆ
We currently use different hashes for different use cases. so many hashes, so little time ๐ญ
- property
attrโ private attribute stored on spec ๐
โญ spack.hash_types.dag_hash = <spack.hash_types.SpecHashDescriptor object>
Spack's deployment hash. Includes all inputs that can affect how a package is built ๐ฅ
๐ spack.hash_types.package_hash = <spack.hash_types.SpecHashDescriptor object>
Package hash used as part of dag hash ๐
๐ฆ spack.hash_types.process_hash = <spack.hash_types.SpecHashDescriptor object>
Hash descriptor used only to transfer a DAG, as is, across processes ๐
๐ง spack.install_test module ๐ง
๐ธ class spack.install_test.PackageTest(pkg: Pb)
Bases: object
The class that manages stand-alone (post-install) package tests. this is her moment ๐ซถ
add_failure(exception: Exception, msg: str)โ add the failure details to the current list ๐ญarchive_install_test_log(dest_dir: str)โ archives that install test log, iconic- property
archived_install_test_log: strโ the archived install test log - property
logger: nixlog | winlog | Noneโ the current logger or, if there is none, sets to one โจ parts() -> intโ the total number of (checked) test partsphase_tests(builder: Builder, phase_name: str, method_names: List[str])โ execute the builder's package phase-time tests ๐
Parameters
- builder โ builder for the package being tested
- phase_name โ the name of the build-time phase (e.g.,
build,install) - method_names โ phase-specific callback method names
print_log_path()โ print the test log file path ๐ran_tests() -> boolโTrueif it ran tests,Falseotherwisestand_alone_tests(kwargs)โ run the package's stand-alone tests ๐
Parameters
- kwargs (
dict) โ arguments to be used by the test process
status(name: str, status: TestStatus, msg: str | None = None)โ track and print the test status for the test part namesummarize()โ collect test results summary lines for this spec โญtest_logger(verbose: bool = False, externals: bool = False)โ context manager for setting up the test logger ๐
Parameters
- verbose โ display verbose output, including echoing to stdout, otherwise suppress it
- externals โ
Truefor performing tests if external package,Falseto skip them
write_tested_status()โ write the overall status to the tested file ๐
If there are any test part failures, then the tests failed. If all test parts are skipped, then the tests were skipped. If any tests passed then the tests passed; otherwise, there were no tests executed.
๐ exception spack.install_test.SkipTest
Bases: Exception
Raised when a test (part) is being skipped. not me dodging the test ๐
๐ญ exception spack.install_test.TestFailure(failures: List[Tuple[BaseException, str]])
Bases: SpackError
Raised when package tests have failed for an installation. it's giving red flags ๐ฉ
๐ spack.install_test.TestFailureType
Stand-alone test failure info type
Alias of Tuple[BaseException, str]
๐ class spack.install_test.TestStatus(*values)
Bases: Enum
Names of the different stand-alone test states ๐
FAILED = 1NO_TESTS = -1PASSED = 2SKIPPED = 0lower()โ lowercases it, obviously โจ
๐ฆ class spack.install_test.TestSuite(specs, alias=None)
Bases: object
The class that manages specs for spack test run execution ๐
- property
content_hashโ the hash used to uniquely identify the test suite - property
current_test_cache_dirโ path to the test stage directory where the current spec's cached build-time files were automatically copied ๐
so bestie, picking up right where we left off โ that property from the last part returns the path to the current spec's staged, cached build-time files ๐
Return type: str
Raises TestSuiteSpecError โ if there is no spec being tested (rude, but okay ๐ญ)
property current_test_data_dirโ path to the test stage directory where the current spec's custom package (data) files got automatically copied โจ- Returns โ path to the current spec's staged, custom package (data) files
- Return type โ
str - Raises
TestSuiteSpecErrorโ if there is no spec being tested
ensure_stage()โ literally just makes sure the test suite stage directory exists. That's the whole assignment ๐ซถstatic from_dict(d)โ instantiates aTestSuitebased on a dictionaryspecsand an optional alias:specs: list of the test suite's specs in dictionary formalias: the test suite alias- Returns โ instance created from the specs
- Return type โ
TestSuite
static from_file(filename)โ instantiate aTestSuiteusing the specs and optional alias provided in the given file ๐- Parameters โ
filename(str) โ the path to the JSON file containing the test suite specs and optional alias - Raises
BaseExceptionโsjson.SpackJSONErrorif there's a problem parsing the file
- Parameters โ
log_file_for_spec(spec)โ the test log file path for the provided spec ๐- Parameters โ
spec(spack.spec.Spec) โ instance of the spec under test - Returns โ the path to the spec's log file
- Return type โ
str
- Parameters โ
property nameโ the name (alias or, if none, hash) of the test suite. iconic behavior honestly ๐property results_fileโ the path to the results summary file.property stageโ the root test suite stage directory ๐ธ- Returns โ the spec's test stage directory path
- Return type โ
str
test_dir_for_spec(spec)โ the path to the test stage directory for the provided spec.- Parameters โ
spec(spack.spec.Spec) โ instance of the spec under test - Returns โ the spec's test stage directory path
- Return type โ
str
- Parameters โ
classmethod test_log_name(spec)โ the standard log filename for a spec ๐- Parameters โ
spec(spack.spec.Spec) โ instance of the spec under test - Returns โ the spec's log filename
- Return type โ
str
- Parameters โ
classmethod test_pkg_id(spec)โ the standard install test package identifier ๐- Parameters โ
specโ instance of the spec under test - Returns โ the install test package identifier
- Return type โ
str
- Parameters โ
test_status(spec: Spec, externals: bool) -> TestStatus | Noneโ determine the overall test results status for the spec ๐- Parameters
specโ instance of the spec under testexternalsโTrueif externals are to be tested, elseFalse
- Returns โ the spec's test status if available or
None
- Parameters
tested_file_for_spec(spec)โ the test status file path for the spec.- Parameters โ
spec(spack.spec.Spec) โ instance of the spec under test - Returns โ the spec's test status file path
- Return type โ
str
- Parameters โ
classmethod tested_file_name(spec)โ the standard test status filename for the spec.- Parameters โ
spec(spack.spec.Spec) โ instance of the spec under test - Returns โ the spec's test status filename
- Return type โ
str
- Parameters โ
to_dict()โ build a dictionary for the test suite ๐ง- Returns โ the dictionary contains entries for up to two keys:
specs: list of the test suite's specs in dictionary formalias: the alias, or name, given to the test suite if provided
- Return type โ
dict
- Returns โ the dictionary contains entries for up to two keys:
write_reproducibility_data()โ yes, it's a whole mood. no extra details given, just vibes ๐write_test_result(spec, result)โ write the spec's test result to the test suite results file.- Parameters
spec(spack.spec.Spec) โ instance of the spec under testresult(str) โ result from the spec's test execution (e.g,PASSED)
- Parameters
exception spack.install_test.TestSuiteError(message, long_message=None)โ bases:SpackError. Raised when there is an error with the test suite ๐คexception spack.install_test.TestSuiteFailure(num_failures)โ bases:SpackError. Raised when one or more tests in a suite have failed ๐exception spack.install_test.TestSuiteNameError(message, long_message=None)โ bases:SpackError. Raised when there is an issue with the naming of the test suite.exception spack.install_test.TestSuiteSpecError(message, long_message=None)โ bases:SpackError. Raised when there is an issue associated with the spec being tested.
spack.install_test.cache_extra_test_sources(pkg: Pb, srcs: str | List[str])โ copy relative source paths to the corresponding install test subdir ๐๏ธ
This routine is intended as an optional install test setup helper for grabbing source files/directories during the installation process and copying them to the installation test subdirectory for subsequent use during install testing. ngl that's a mouthful, but it's literally just "grab this stuff now, test it later."
- Parameters
pkgโ package being testedsrcsโ relative path for file(s) and or subdirectory(ies) located in the staged source path that are to be copied to the corresponding location(s) under the install testing directory
- Raises
spack.installer.InstallErrorโ if any of the source paths are absolute or do not exist under the build stage
spack.install_test.check_outputs(expected: list | set | str, actual: str)โ ensure the expected outputs are contained in the actual outputs โ- Parameters
expectedโ expected raw output string(s)actualโ actual output string
- Raises
RuntimeErrorโ the expected output is not found in the actual output
- Parameters
spack.install_test.copy_test_files(pkg: Pb, test_spec: Spec)โ copy the spec's cached and custom test files to the test stage directory ๐- Parameters
pkgโ package being testedtest_specโ spec being tested, where the spec may be virtual
- Raises
TestSuiteErrorโ package must be part of an active test suite
- Parameters
spack.install_test.find_required_file(root: str, filename: str, expected: int = 1, recursive: bool = True) -> str | List[str]โ find the required file(s) under the root directory ๐- Parameters
rootโ root directory for the searchfilenameโ name of the file being locatedexpectedโ expected number of files to be found under the directory (default is1)recursiveโTrueif subdirectories are to be recursively searched, elseFalse(default isTrue)
- Returns: the path(s), relative to
root, to the required file(s) - Raises
ExceptionโSkipTestwhen number of files detected does not matchexpected
- Parameters
spack.install_test.get_all_test_suites()โ retrieves all validly stagedTestSuites ๐- Returns โ a list of
TestSuiteobjects, which may be empty if there are none - Return type โ
list
- Returns โ a list of
spack.install_test.get_escaped_text_output(filename: str) -> List[str]โ retrieve and escape the expected text output from the file ๐- Parameters โ
filenameโ path to the file - Returns โ escaped text lines read from the file
- Parameters โ
spack.install_test.get_named_test_suites(name)โ retrieves test suites with the provided name.- Returns โ a list of matching
TestSuiteinstances, which may be empty if none - Return type โ
list - Raises
ExceptionโTestSuiteNameErrorif no name is provided
- Returns โ a list of matching
spack.install_test.get_test_stage_dir()โ retrieves theconfig:test_stagepath to the configured test stage root directory ๐บ๏ธ- Returns โ absolute path to the configured test stage root or, if none, the default test stage path
- Return type โ
str
spack.install_test.get_test_suite(name: str) -> TestSuite | Noneโ ensure there is only one matching test suite with the provided name. one (1) suite, no duplicates, we don't do chaos here ๐- Returns โ the name if one matching test suite, else
None - Raises
TestSuiteNameErrorโ if there are more than one matchingTestSuites
- Returns โ the name if one matching test suite, else
spack.install_test.install_test_root(pkg: Pb)โ the install test root directory.- Parameters โ
pkgโ package being tested
- Parameters โ
spack.install_test.overall_status(current_status: TestStatus, substatuses: List[TestStatus]) -> TestStatusโ determine the overall status based on the current and associated sub status values ๐ฆ- Parameters
current_statusโ the current overall status, assumed to default toPASSED๐substatusesโ the status of each test part, or the overall status of each test spec
Returns: the test status that wraps up the main test and all its subtests. like, the whole picture โจ
spack.install_test.print_message(logger: nixlog | winlog, msg: str, verbose: bool = False)
Print the message to the log, and optionally echo it out too. bestie, sometimes you want it in the file, sometimes you want it loud ๐ฃ๏ธ
Parameters
loggerโ an instance of the output logger (e.g.nixlogorwinlog)msgโ the message being outputverboseโTrueshows verbose output,Falsesuppresses it (Falseis the default)
spack.install_test.process_test_parts(pkg: Pb, test_specs: List[Spec], verbose: bool = False)
Process the test parts that are associated with the package. very "let's get into it" energy ๐ง
Parameters
pkgโ the package being testedtest_specsโ the list of test specsverboseโ display verbose output (suppressed by default)
Raises TestSuiteError โ the package must be part of an active test suite
spack.install_test.results_filename = 'results.txt'
The name of the test suite results (summary) file ๐
spack.install_test.spack_install_test_log = 'install-time-test-log.txt'
The name of the Spack install phase-time test log file ๐
spack.install_test.test_suite_filename = 'test_suite.lock'
The name of the test suite's (JSON) lock file ๐
spack.install_test.test_function_names(pkg: Pb | Type[Pb], add_virtuals: bool = False) -> List[str]
Grab the names of every non-empty test function. lowkey a name-collector โจ
Parameters
pkgโ the package (or package class) of interestadd_virtualsโTrueadds the test methods of the provided package's virtual,Falsereturns only the package's own test functions
Returns: the names of the non-empty test functions
Raises ValueError โ happens if pkg isn't a package class
spack.install_test.test_functions(pkg: Pb | Type[Pb], add_virtuals: bool = False) -> List[Tuple[str, Callable]]
Grab all the non-empty test functions, not just their names โ the full package deal ๐ซถ
Parameters
pkgโ the package (or package class) of interestadd_virtualsโTrueadds the test methods of the provided package's virtual,Falsereturns only the package's own test functions
Returns: a list of non-empty test functions as (name, function)
Raises ValueError โ happens if pkg isn't a package class
spack.install_test.test_part(pkg: Pb, test_name: str, purpose: str, work_dir: str = '.', verbose: bool = False)
spack.install_test.test_process(pkg: Pb, kwargs)
these two are just sitting here as signatures with no write-up in the docs โ no cap, that's literally all the page gives us ๐
spack.install_test.virtuals(pkg)
Return a list of the unique virtuals for the package. iconic dedupe behaviour ๐
Parameters
pkgโ the package of interest
Returns: the names of the unique virtual packages
spack.install_test.write_test_suite_file(suite)
Write the test suite out to its (JSON) lock file. ๐โจ
spack.install_test.write_test_summary(counts: Counter)
Write a summary of the totals for each relevant status category. stats girl summer ๐
Parameters
countsโ the counts of the occurrences of the relevant test status types
๐งจ spack.installer module
This module is the whole package-installation situation, encapsulated. ๐ซง
The PackageInstaller coordinates concurrent builds of packages for the same Spack instance by leveraging the dependency DAG and file system locks. It also goes ahead with installing the non-dependent packages of failed dependencies, so that you still install as many of a package's dependencies as possible. honestly? resilient queen ๐
Traversing the dependency DAG bottom-up while prioritizing packages with no uninstalled dependencies lets multiple processes do concurrent builds of separate packages that are associated with a spec.
File system locks keep everything coordinated so that no two processes ever try to build the same package โ or a failed dependency package. ๐
When dependency packages fail to install, their dependents' build tasks get removed from the current process. A failure file is also written (and locked) so that other processes can detect the failure and adjust their build tasks accordingly. team work fr fr ๐ซถ
This module supports coordinating local and distributed concurrent installations of packages in a Spack instance. (Yes, distributed. On a shared file system. she's versatile ๐)
exception spack.installer.BadInstallPhase(pkg_name, phase)
Bases: InstallError
Raised when an install phase option isn't allowed for a package. ๐ซ
class spack.installer.BuildProcessInstaller(pkg: PackageBase, install_args: dict)
Bases: object
This class implements the part of installation that happens in the child process. tiny baby process, big job ๐งธ
run() -> bool
Main entry point, called from build_process, to kick off the install in the child.
class spack.installer.BuildRequest(pkg: PackageBase, install_args: dict)
Bases: object
A class for representing an installation request. ๐จ
get_depflags(pkg: PackageBase) -> int
Figure out the required dependency types for the associated package.
Parameters
pkgโ the explicit or implicit package being installed
Returns: the required dependency type(s) for the package
Return type: tuple
has_dependency(dep_id) -> bool
Returns True if the package id represents a known dependency of the requested package, False otherwise.
run_tests(pkg: PackageBase) -> bool
Decide whether the tests should be run for the provided packages.
Parameters
pkgโ the explicit or implicit package being installed
Returns: True if they should be run; False otherwise
Return type: bool
property spec: Spec
The specification associated with the package.
traverse_dependencies(spec=None, visited=None) -> Iterator[Spec]
Yield any dependencies of the appropriate type(s). ๐ฑ
class spack.installer.BuildTask(pkg: PackageBase, request: BuildRequest | None, compiler: bool, start: float, attempts: int, status: str, installed: Set[str])
Bases: object
A class representing the build task for a package. one task, one vibe check โ
add_dependent(pkg_id: str) -> None
Make sure the dependent package id is in the task's list so it gets properly updated when this package is installed.
Parameters
pkg_idโ the package identifier of the dependent package
property cache_only: bool
property explicit: bool
The package was explicitly requested by the user. main character energy ๐
flag_installed(installed: List[str]) -> None
Make sure the dependency isn't still considered uninstalled.
Parameters
installedโ the identifiers of the packages that have been installed so far
property is_root: bool
The package was requested directly, but may or may not be explicit in an environment.
property key: Tuple[int, int]
The key is the tuple (# uninstalled dependencies, sequence). not me obsessing over a tuple ๐
next_attempt(installed) -> BuildTask
Create a new, updated task for the next installation attempt. round two, bestie ๐
property priority
The priority is based on the remaining uninstalled dependencies.
property use_cache: bool
exception spack.installer.ExternalPackageError(message, long_msg=None, pkg=None)
Bases: InstallError
Raised by install() when a package is only for external use. ๐ช
class spack.installer.InstallAction
Bases: object
INSTALL = 1
Do a standard install
NONE = 0
Don't perform an install
OVERWRITE = 2
Do an overwrite install
exception spack.installer.InstallError(message, long_msg=None, pkg=None)
Bases: SpackError
Raised when something goes wrong during install or uninstall. ๐ญ
The error can be annotated with a pkg attribute so the caller can get the package for which the exception was raised. very "here's the receipt" of it all ๐งพ
exception spack.installer.InstallLockError(message, long_msg=None, pkg=None)
Bases: InstallError
Raised during install when something goes wrong with package locking. ๐๐
class spack.installer.InstallStatus(pkg_count: int)
Bases: object
get_progress() -> str
next_pkg(pkg: PackageBase)
set_term_title(text: str)
again โ no descriptions for these three in the docs, so we're just showing you the signatures as-is ๐
class spack.installer.OverwriteInstall(installer: PackageInstaller, database: Database, task: BuildTask, install_status: InstallStatus)
Bases: object
install()
Try to run the install task, overwriting the package prefix. If that fails, try to recover the original install prefix. If that fails too, mark the spec as uninstalled. This function always the original install error if installation fails. extreme makeover: prefix edition ๐๏ธ๐
class spack.installer.PackageInstaller(installs: List[Tuple[PackageBase, dict]] = [])
Bases: object
The class for managing the whole install process for a Spack instance, based on a bottom-up DAG approach. ๐ธ
This installer can coordinate concurrent batch and interactive, local and distributed (on a shared file system) builds for the same Spack instance. she does it all, no cap ๐
install() -> None
Install the requested package(s) and/or their associated dependencies.
spack.installer.STATUS_ADDED = 'queued'
Build status indicating the task has been added. โจ
spack.installer.STATUS_DEQUEUED = 'dequeued'
Build status indicating the task has been popped from the queue.
spack.installer.STATUS_FAILED = 'failed'
Build status indicating the spec failed to install. ๐ญ
spack.installer.STATUS_INSTALLED = 'installed'
Build status indicating the spec was sucessfully installed. ๐
spack.installer.STATUS_INSTALLING = 'installing'
Build status indicating the spec is being installed (possibly by another process). ๐ฅ
spack.installer.STATUS_REMOVED = 'removed'โ a build status that literally just means "this task got removed" โ and the reason it exists is iconic: it keeps the priority queue invariants intact ๐ don't touch it, it's doing its job.class spack.installer.TermStatusLine(enabled: bool)โ Bases:object.
okay so this one is for distributed builds, and its whole vibe is telling you "bestie, another process is installing other packages rn" so you're not sitting there confused ๐ซถ
add(pkg_id: str)โ adds a package to the waiting list, and if it's brand new it updates the status line for you.clear()โ wipes the status line. clean slate energy โจ
exception spack.installer.UpstreamPackageError(message, long_msg=None, pkg=None)โ Bases:InstallError. raised during install when something goes sideways with an upstream package โ like, not our package's fault, the upstream one is being messy ๐ญspack.installer.archive_install_logs(pkg: PackageBase, phase_log_dir: str) -> Noneโ copies the install logs over to their destination directory(ies). params, keep up:pkgis the package that was built and installed, andphase_log_diris the path to the archive directory ๐spack.installer.build_process(pkg: PackageBase, install_args: dict) -> boolโ does the actual installation/build of the package. the important lore: this runs in a separate child process, and it gets its own process and python module space all set up bybuild_environment.start_build_process(). basically it wraps an instance ofBuildProcessInstallerso we can spin one up in a subprocess without it being a whole thing. and yes โ this function's return value gets handed back to the parent process ๐
Parameters:
pkgโ the package being installed.install_argsโ arguments todo_install()from the parent process.
spack.installer.combine_phase_logs(phase_log_files: List[str], log_path: str) -> Noneโ reads a set or list of logs and smooshes them into one single file. every phase makes its own log, so this function's whole purpose is basicallycat-ing all those separate phase log output files intopkg.log_path. it's written generally on purpose โ give it some list of files plus a log path, and it combines them ๐ธ
Parameters:
phase_log_filesโ a list or iterator of logs to combinelog_pathโ the path to combine them to
spack.installer.dump_packages(spec: Spec, path: str) -> Noneโ dumps all the package information for a spec and its dependencies. it creates a package repository insidepathfor every namespace in the spec DAG, and then fills those repos with package files and patch files for every single node in the DAG. thorough queen behavior ๐
Parameters:
specโ the Spack spec whose package information is to be dumpedpathโ the path to the build packages directory
spack.installer.get_dependent_ids(spec: Spec) -> List[str]โ returns a list of package ids for the spec's dependents.
Parameters: spec โ Concretized spec
Returns: list of package ids
spack.installer.install_msg(name: str, pid: int, install_status: InstallStatus) -> strโ colorizes the name/id of the package being installed, because plain text is boring and we're not doing that ๐จ
Parameters:
nameโ Name/id of the package being installedpidโ id of the installer process
Return: Colorized installing message
spack.installer.log(pkg: PackageBase) -> Noneโ copies provenance into the install directory on success. success only, no cap โ
Parameters: pkg โ the package that was built and installed
spack.installer.package_id(spec: Spec) -> strโ a "unique" package identifier for installation purposes. it's used to track build tasks, locks, install, and failure statuses, and it has to distinguish between combinations of compilers and packages for combinatorial environments โ so yeah, it's doing a lot ๐
Parameters: pkg โ the package from which the identifier is derived
spack.installer.print_install_test_log(pkg: PackageBase) -> Noneโ outputs the install test log file path, but only if there are test failures. no failures, no output, that's the rule ๐
Parameters: pkg โ instance of the package under test
โจ spack.main module โจ
okay bestie we're in the big one โ this is the implementation of the Spack command line executable ๐ in a normal Spack installation, this gets invoked from the bin/spack script after the system path is set up. main character module, fr fr ๐
spack.main.SHOW_BACKTRACE = Falseโ whether to print backtraces on error. default is a no ๐class spack.main.SpackArgumentParser(prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True, exit_on_error=True)โ Bases:ArgumentParser.add_command(cmd_name)โ adds one subcommand to this parser.add_subparsers(**kwargs)โ makes sure the sensible defaults get propagated to subparsers ๐ฆformat_help(level='short')โ formats the help, default level is'short'.format_help_sections(level)โ formats help on sections for a particular verbosity level.
Parameters: level (str) โ 'short' or 'long' (more commands shown for long)
class spack.main.SpackCommand(command_name, subprocess=False)โ Bases:object. a callable object that invokes a spack command (for testing). example usage, copy it exactly:
install = SpackCommand('install')
install('-v', 'mpich')use this to invoke Spack commands directly from Python and check their output. testing era ๐ฅ
exception spack.main.SpackCommandErrorโ Bases:Exception. raised whenSpackCommandexecution fails. it happens, don't spiral ๐ญclass spack.main.SpackHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)โ Bases:RawTextHelpFormatter.add_arguments(actions)โ adds the arguments, obviously ๐
spack.main.add_all_commands(parser)โ adds all spack subcommands to the parser. every single one, no skips.spack.main.allows_unknown_args(command)โ implements really simple argument injection for unknown arguments. here's the tea: commands can add an optional argument called "unknown args" to indicate they can handle unknonwn args, and then we'll pass the unknown args in. yep, that's the whole mechanism ๐spack.main.finish_parse_and_run(parser, cmd_name, main_args, env_format_error)โ finishes parsing after we already know which command to run.spack.main.get_spack_commit()โ gets the Spack git commit sha.
Returns: (str or None) the commit sha if available, otherwise None
spack.main.get_version()โ gets a descriptive version of this instance of Spack. it outputs'<PEP440 version> (<git commit sha>)'โ and the commit sha only gets added when it's actually available โญspack.main.index_commands()โ creates an index of commands by section for this help level.spack.main.intro_by_level = {'long': 'Complete list of spack commands:', 'short': 'These are common spack commands:'}โ intro text for help at the different levels ๐ซถspack.main.levels = ['short', 'long']โ the help levels, in order of detail (i.e., number of commands shown).spack.main.main(argv=None)โ this is the entry point for the Spack command. and the lore is:main()itself is just an error handler โ it handles errors for everything in Spack that makes it to the top level. all the actual logic is in_main()๐
Parameters: argv (list or None) โ command line arguments, NOT including the executable name. if None, it parses from sys.argv.
spack.main.make_argument_parser(**kwargs)โ creates a basic argument parser without any subcommands added.spack.main.options_by_level = {'long': 'all', 'short': ['h', 'k', 'V', 'color']}โ controls the top-level spack options shown in basic vs. advanced help ๐spack.main.print_setup_info(*info)โ prints the basic information needed bysetup-env.[c]sh.
Parameters: info (list) โ list of things to print: comma-separated list of 'csh', 'sh', or 'modules'
and why is this in main.py? to make it fast ๐โโ๏ธ the setup scripts need to invoke spack in login scripts, so it needs to be quick. no time to waste.
spack.main.required_command_properties = ['level', 'section', 'description']โ the properties that commands are required to set. non-negotiable.spack.main.resolve_alias(cmd_name: str, cmd: List[str]) -> Tuple[str, List[str]]โ resolves aliases in the given command.
Parameters:
cmd_nameโ command name.cmdโ command line arguments.
Returns: new command name and arguments.
spack.main.restore_macos_dyld_vars()โ okay this one has real plot: Spack mutatesDYLD_*variables inspack loadandspack env activate. unlike Linux, macOS SIP clears these variables in new processes, meaning thatos.environ["DYLD_*"]in our Python process is not the same as the user's shell. therefore, we store the user'sDYLD_*variables inSPACK_DYLD_*and restore them here ๐spack.main.section_descriptions = {'admin': 'administration', 'basic': 'query packages', 'build': 'build packages', 'config': 'configuration', 'developer': 'developer', 'environment': 'environment', 'extensions': 'extensions', 'help': 'more help', 'packaging': 'create packages', 'system': 'system'}โ longer text for each section, to show in help โจspack.main.section_order = {'basic': ['list', 'info', 'find'], 'build': ['fetch', 'stage', 'patch', 'configure', 'build', 'restage', 'install', 'uninstall', 'clean'], 'packaging': ['create', 'edit']}โ preferential command order for some sections (e.g., the build pipeline is in execution order, not alphabetical โ respect the pipeline ๐ ).spack.main.send_warning_to_tty(message, *args)โ redirects messages totty.warn.spack.main.set_working_dir()โ changes the working directory togetcwd, or the spack prefix if there's no cwd ๐ธspack.main.setup_main_options(args)โ configures spack globals based on the basic options.spack.main.spack_working_dir = Noneโ the recorded directory where the spack command was originally invoked ๐
spack.main.stat_names is literally just a flex โ it's the dict that holds the names of profile statistics ๐
spack.main.stat_names = {'calls': (((1, -1),), 'call count'), 'cumtime': (((3, -1),), 'cumulative time'), 'cumulative': (((3, -1),), 'cumulative time'), 'filename': (((4, 1),), 'file name'), 'line': (((5, 1),), 'line number'), 'module': (((4, 1),), 'file name'), 'name': (((6, 1),), 'function name'), 'ncalls': (((1, -1),), 'call count'), 'nfl': (((6, 1), (4, 1), (5, 1)), 'name/file/line'), 'pcalls': (((0, -1),), 'primitive call count'), 'stdname': (((7, 1),), 'standard name'), 'time': (((2, -1),), 'internal time'), 'tottime': (((2, -1),), 'internal time')}๐ช spack.mirror module
Okay so bestie, this file is where all the mirror magic lives ๐ Spack mirrors are organized hierarchies stuffed with specially named archive files, and the whole point is: spack knows where to find files in a mirror even if the main server for a certain package is down ๐ค Like, if the computer where spack is running isn't even connected to the internet, a mirror still lets spack download packages straight from it (think intranet vibes, iykyk) โจ
class spack.mirror.Mirror(data: str | dict, name: str | None = None)โ Bases:object. It represents a named location for storing source tarballs and binary packages ๐
Mirrors have a fetch_url that says where and how artifacts are fetched from them, and a push_url that says where and how artifacts get pushed to them. These two URLs are usually the same, no cap ๐ซถ
autopush: boolโ a property, periodtbinaryโ also a propertydisplay(max_len=0)fetch_urlโ a property that gets the valid, canonicalized fetch URL โจstatic from_json(stream, name=None)static from_local_path(path: str)static from_url(url: str)โ creates an anonymous mirror by URL, and this method validates the URL (so behave ๐ฅ)static from_yaml(stream, name=None)get_access_pair(direction: str) -> List | Noneget_access_token(direction: str) -> str | Noneget_endpoint_url(direction: str) -> str | Noneget_profile(direction: str) -> str | Noneget_url(direction: str) -> strnameโ property ๐ท๏ธpush_urlโ property that gets the valid, canonicalized fetch URL (yes, it's worded exactly like that, we're keeping it ๐)signed: boolโ propertysourceโ propertyto_dict()to_json(stream=None)to_yaml(stream=None)update(data: dict, direction: str | None = None) -> boolโ modifies the mirror with the given data, and it handles expanding those trivial by-URL mirror definitions into something richer with a dict if needed ๐
Parameters:
- data (
dict) โ the data to update the mirror with - direction (
str) โ the direction to update the mirror in (fetchorpushorNonefor a top-level update)
Returns: True if the mirror got updated, False otherwise ๐ซ
Return type: bool
class spack.mirror.MirrorCollection(mirrors=None, scope=None, binary: bool | None = None, source: bool | None = None, autopush: bool | None = None)โ Bases:Mapping. It's literally a mapping of mirror names to mirrors ๐display()static from_dict(d)static from_json(stream, name=None)static from_yaml(stream, name=None)lookup(name_or_url)โ looks up and returns aMirror. If thisMirrorCollectioncontains a namedMirrorunder the name [name_or_url], then that mirror gets returned. Otherwise, [name_or_url] is assumed to be a mirror URL, and an anonymous mirror with the given URL is returned ๐ฆto_dict(recursive=False)to_json(stream=None)to_yaml(stream=None)
exception spack.mirror.MirrorError(msg, long_msg=None)โ Bases:SpackError. The superclass of all mirror-creation related errors, so if mirror stuff explodes, it's this icon's fault ๐ญclass spack.mirror.MirrorReference(cosmetic_path, global_path=None)โ Bases:object.
A MirrorReference stores the relative paths where you can store a package/resource in a mirror directory ๐
The appropriate storage location is given by storage_path. The cosmetic_path property gives you a reference that a human could generate themselves just by reading the details of the package ๐
You can iterate over a MirrorReference object to get every possible name that might refer to the resource in a mirror โ and that includes names generated by previous naming schemes that are no longer reported by storage_path or cosmetic_path (legacy representation stays valid, periodt) ๐
storage_pathโ property
class spack.mirror.MirrorStatsโ Bases:objectadded(resource)already_existed(resource)error()next_spec(spec)stats()
class spack.mirror.OCIImageLayout(digest: Digest)โ Bases:object. Follows the OCI Image Layout Specification to archive blobs ๐ง
Paths are of the form blobs/<algorithm>/<digest> โจ
spack.mirror.add(mirror: Mirror, scope=None)โ adds a named mirror in the given scopespack.mirror.create(path, specs, skip_unstable_versions=False)โ creates a directory to be used as a spack mirror and fills it with package archives ๐ธ
Parameters:
- path โ path to create a mirror directory hierarchy in
- specs โ any package versions matching these specs will be added to the mirror
- skip_unstable_versions โ if true, this skips adding resources when they don't have a stable archive checksum (as determined by
fetch_strategy.stable_target)
Return Value: returns a tuple of lists: (present, mirrored, error) ๐ซ
- present: package specs that were already present
- mirrored: package specs that were successfully mirrored
- error: package specs that failed to mirror due to some error
spack.mirror.create_mirror_from_package_object(pkg_obj, mirror_cache, mirror_stats)โ adds a single package object to a mirror ๐ซถ
The package object only needs to have an associated spec with a concrete version.
Parameters:
- pkg_obj (
spack.package_base.PackageBase) โ package object with to be added (yep, that's the wording, leaving it ๐ซถ) - mirror_cache (
spack.caches.MirrorCache) โ mirror where to add the spec - mirror_stats (
spack.mirror.MirrorStats) โ statistics on the current mirror
Returns: True if the spec was added successfully, False otherwise
spack.mirror.get_all_versions(specs)โ given a set of initial specs, returns a new set of specs that includes each version of each package in the original set ๐
Heads up bestie ๐ if any spec in the original set specifies properties other than version, that info is omitted in the new set. So like, the new set of specs won't include variant settings. Not me losing your variants, I promise it's documented behaviour.
spack.mirror.get_matching_versions(specs, num_versions=1)โ gets a spec for EACH known version matching any spec in the list ๐ฅ For concrete specs, this retrieves the concrete version and, if more than one version per spec is requested, it retrieves the latest versions of the package.spack.mirror.mirror_archive_paths(fetcher, per_package_ref, spec=None)โ returns aMirrorReferenceobject which keeps track of the relative storage path of the resource associated with the specified fetcher ๐spack.mirror.mirror_cache_and_stats(path, skip_unstable_versions=False)โ returns both a mirror cache and a mirror stats, starting from the path where a mirror ought to be created โจ
Parameters:
- path (
str) โ path to create a mirror directory hierarchy in - skip_unstable_versions โ if true, this skips adding resources when they don't have a stable archive checksum (as determined by
fetch_strategy.stable_target)
spack.mirror.remove(name, scope)โ removes the named mirror in the given scopespack.mirror.require_mirror_name(mirror_name)โ finds a mirror by name and raises if it doesn't exist ๐คspack.mirror.supported_url_schemes = ('file', 'http', 'https', 'sftp', 'ftp', 's3', 'gs', 'oci')โ literally what schemes do we support, that's the whole vibe check ๐
๐ง spack.mixins module
This module is where the extra behavior that can be attached to any given package lives ๐
spack.mixins.filter_compiler_wrappers(*files, **kwargs)โ substitutes any path referring to a Spack compiler wrapper with the path of the underlying compiler that has been used ๐ ๏ธ
The way this matters: if this isn't done, the files will have CC, CXX, F77, and FC set to Spack's generic cc, c++, f77, and f90. We want them bound to whatever compiler they were built with, no cap ๐
Parameters:
*filesโ files to be filtered relative to the search root (which is, by default, the installation prefix)**kwargsโ allowed keyword arguments:afterโ specifies after which phase the files should be filtered (defaults to'install')relative_rootโ path relative to prefix where to start searching for the files to be filtered. If not set, the install prefix will be used as the search root. It is highly recommended to set this, because searching from the installation prefix may affect performance severely in some cases ๐ญ (it's giving "trust me bestie, set it")ignore_absent,backupโ these two keyword arguments, if present, will be forwarded tofilter_file(see its documentation for more information on their behavior)recursiveโ this keyword argument, if present, will be forwarded tofind(see its documentation for more information on the behavior)
๐งฉ spack.multimethod module ๐งฉ
Okay so this module is where Spack keeps its multi-method utilities. Think of multi-methods like overloaded methods: same name, but we pick which version to run based on some criteria. For overloaded methods, that criteria would be the types of the arguments โ here it's specs and package vibes.
In Spack, multi-methods make package authors' lives easier. They let methods like install() (or other methods called by install()) declare multiple versions that get called when the package is instantiated with different specs. Example: if the package is built with OpenMPI on x86_64, you might want a different install method than if it was built for mpich2 on BlueGene/Q. Same energy for different package versions โ you might want a different type of install.
Multi-methods give you a simple decorator-based syntax so you don't end up with a rat's nest of if statements. That said, sometimes regular old conditionals are clearer, so package authors should use their judgement. No cap.
exception spack.multimethod.MultiMethodError(message)โ Bases:SpackError. Superclass for multimethod dispatch errors.class spack.multimethod.MultiMethodMeta(name, bases, attr_dict)โ Bases:type. This lets us track the class's dict during instantiation.exception spack.multimethod.NoSuchMethodError(cls, method_name, spec, possible_specs)โ Bases:SpackError. Raised when we can't find a version of a multi-method. ๐ญclass spack.multimethod.SpecMultiMethod(default=None)โ Bases:object. This implements a multi-method for Spack specs. Packages are instantiated with a particular spec, and you may want to execute different versions of methods based on what the spec looks like. For example, you might want to call a different version ofinstall()for one platform than you call on another.
The SpecMultiMethod class implements a callable object that handles method dispatch. When it is called, it looks through registered methods and their associated specs, and it tries to find one that matches the package's spec. If it finds one (and only one), it will call that method.
This is intended for use with decorators (see below). The decorator (see docs below) creates SpecMultiMethods and registers method versions with them.
To register a method, you can do something like this:
mm = SpecMultiMethod() mm.register("^chaos_5_x86_64_ib",
some_method)The object registered needs to be a Spec or some string that will parse to be a valid spec.
When the mm is actually called, it selects a version of the method to call based on the sys_type of the object it is called on.
See the docs for decorators below for more details. โจ
register(spec, method)โ Register a version of a method for a particular spec.
spack.multimethod.default_args(**kwargs)โ that's the whole signature, bestie.class spack.multimethod.when(condition)โ Bases:object.
๐ฆ spack.package module ๐ฆ
spack.util.package is a set of useful build tools and directives for packages. Everything in this module is automatically imported into Spack package files. Handy.
๐ ๏ธ spack.package_base module ๐ ๏ธ
This is where most of the action happens in Spack. The Spack package class structure is based strongly on Homebrew (http://brew.sh/), mainly because Homebrew makes it very easy to create packages. Iconic behavior.
exception spack.package_base.ActivationError(msg, long_msg=None)โ Bases:ExtensionError. Raised when there are problems activating an extension.exception spack.package_base.DependencyConflictError(conflict)โ Bases:SpackError. Raised when the dependencies cannot be flattened as asked for.class spack.package_base.DetectablePackageMeta(name, bases, attr_dict)โ Bases:type. Check if a package is detectable and add default implementations for the detection function.TAG = 'detectable'.exception spack.package_base.ExtensionError(message, long_msg=None)โ Bases:PackageError. Superclass for all errors having to do with extension packages.spack.package_base.FLAG_HANDLER_TYPEโ described as โAllowed URL schemes for spack packages.โ Its alias isCallable[[str, Iterable[str]], Tuple[Iterable[str] | None, Iterable[str] | None, Iterable[str] | None]]. That's the docstring, periodt.exception spack.package_base.InvalidPackageOpError(message, long_msg=None)โ Bases:PackageError. Raised when someone tries perform an invalid operation on a package. (Yes, โtries performโ โ we keep the facts exactly.)exception spack.package_base.ManualDownloadRequiredError(message, long_msg=None)โ Bases:InvalidPackageOpError. Raised when attempting an invalid operation on a package that requires a manual download.exception spack.package_base.NoURLError(cls)โ Bases:PackageError. Raised when someone tries to build a URL for a package with no URLs.class spack.package_base.PackageBase(spec)โ Bases:WindowsRPath,PackageViewMixin,RedistributionMixin. This is the superclass for all spack packages.
The Package class
At its core, a package consists of a set of software to be installed. A package may focus on a piece of software and its associated software dependencies, or it may simply be a set, or bundle, of software. The former requires defining how to fetch, verify (via, e.g., sha256), build, and install that software and the packages it depends on, so that dependencies can be installed along with the package itself. The latter โ sometimes referred to as a no-source package โ requires only defining the packages to be built.
Packages are written in pure Python.
There are two main parts of a Spack package:
- The package class. Classes contain directives, which are special functions, that add metadata (versions, patches, dependencies, and other information) to packages (see
directives.py). Directives provide the constraints that are used as input to the concretizer. - Package instances. Once instantiated, a package is essentially a software installer. Spack calls methods like
do_install()on the Package object, and it uses those to drive user-implemented methods likepatch(),install(), and other build steps. To install software, an instantiated package needs a concrete spec, which guides the behavior of the various install methods.
Packages are imported from repos (see repo.py).
Package DSL
Look in lib/spack/docs or check https://spack.readthedocs.io for the full documentation of the package domain-specific language. That used to be partially documented here, but as it grew, the docs here became increasingly out of date. We love an honest manpage. ๐
Package Lifecycle
A package's lifecycle over a run of Spack looks something like this:
p = Package() # Done for you by spack
p.do_fetch() # downloads tarball from a URL (or VCS)
p.do_stage() # expands tarball in a temp directory
p.do_patch() # applies patches to expanded source
p.do_install() # calls package's install() function
p.do_uninstall() # removes install directoryalthough packages that do not have code have nothing to fetch so omit p.do_fetch().
There are also some other commands that clean the build area:
p.do_clean() # removes the stage directory entirely
p.do_restage() # removes the build directory and
# re-expands the archive.The convention used here is that a do_* function is intended to be called internally by Spack commands (in spack.cmd). These aren't for package writers to override, and doing so may break the functionality of the Package class. So maybe don't, bestie. ๐
Package creators have a lot of freedom, and they could technically override anything in this class. That is not usually required.
For most use cases. Package creators typically just add attributes like homepage and, for a code-based package, url, or functions such as install(). There are many custom Package subclasses in the spack.build_systems package that make things even easier for specific build systems.
classmethod all_patches()โ Retrieve all patches associated with the package. Retrieves patches on the package itself as well as patches on the dependencies of the package.property all_urls: List[str]โ A list of all URLs in a package. Check both class-level and version-specific URLs. Returns a list of URLs.all_urls_for_version(version: StandardVersion) -> List[str]โ Return all URLs derived fromversion_urls(),url,urls, andlist_url(if it contains a version) in a package in that order.- version โ the version for which a URL is sought.
archive_install_test_log()โ Archive the install-phase test log, if present.classmethod build_system_flags(name: str, flags: Iterable[str]) -> Tuple[Iterable[str] | None, Iterable[str] | None, Iterable[str] | None]โflag_handlerthat passes flags to the build system arguments. Any package usingbuild_system_flagsmust also implementflags_to_build_system_args, or derive from a class that implements it. Currently,AutotoolsPackageandCMakePackageimplement it.property builderโ that's the property, no extra deets in this chunk.cache_extra_test_sources(srcs)โ Copy relative source paths to the corresponding install test subdir. This method is intended as an optional install test setup helper for grabbing source files/directories during the installation process and copying them to the installation test subdirectory for subsequent use during install testing. ๐ง
srcs(str or list) โ relative path for files and or subdirectories located in the staged source path that are to be copied to the corresponding location(s) under the install testing directory. basically: these are the things that get scooped up from staging and dropped into the test dir, slay ๐
๐ธ attributes & methods, continued ๐ธ
cmake_prefix_paths(property) โ the cmake prefix paths, babes. that's the whole vibe. โจcompiler(property) โ get thespack.compiler.Compilerobject used to build this package. that's your builder bestie right there ๐ซถconfigure_args_path(property) โ return the configure args file path associated with staging.conflictsโ typed asDict[Spec, List[Tuple[Spec, str | None]]]. yes it's a whole nested type situation, no we're not scared of it ๐content_hash(content=None)โ create a hash based on the artifacts and patches used to build this package. it's giving fingerprint but make it build metadata ๐ฅ
This includes:
- source artifacts (tarballs, repositories) used to build;
- content hashes (sha256's) of all patches applied by Spack; and
- canonicalized contents the package.py recipe used to build.
This hash is only included in Spack's DAG hash for concrete specs, but if it happens to be called on a package with an abstract spec, only applicable (i.e., determinable) portions of the hash will be included. so like, abstract specs get the partial tea, periodt โ
dependenciesโ typed asDict[Spec, Dict[str, Dependency]]. the whole dependency web, lowkey gorgeous ๐ธ๏ธdependencies_by_name(when: bool = False)โ a classmethod, bestie. note thewhenarg defaults toFalseโจdependencies_of_type(deptypes: int)โ a classmethod. get names of dependencies that can possibly have these deptypes.
This analyzes the package and determines which dependencies can be a certain kind of dependency. Note that they may not always be this kind of dependency, since dependencies can be optional, so something may be a build dependency in one configuration and a run dependency in another. it's giving "possibility, not promise" โ don't come for me ๐ญ
dependency_names()โ a classmethod, returns the dependency names. short king, does what it says ๐do_clean()โ removes the package's build stage and source tarball. bye bye to the mess ๐งนdo_deprecate(deprecator, link_fn)โ deprecate this package in favor ofdeprecatorspec. it's the "i'm stepping down, she's the main character now" moment ๐do_fetch(mirror_only=False)โ creates a stage directory and downloads the tarball for this package. Working directory will be set to the stage directory. get that tarball, girl ๐do_install(**kwargs)โ called by commands to install a package and or its dependencies.
Package implementations should override install() to describe their build process. (yes, that's your moment, bestie ๐)
Parameters:
cache_only(bool) โ Fail if binary package unavailable.dirty(bool) โ Don't clean the build environment before installing.explicit(bool) โ True if package was explicitly installed, False if package was implicitly installed (as a dependency).fail_fast(bool) โ Fail if any dependency fails to install; otherwise, the default is to install as many dependencies as possible (i.e., best effort installation).fake(bool) โ Don't really build; install fake stub files instead.force(bool) โ Install again, even if already installed.install_deps(bool) โ Install dependencies before installing this packageinstall_source(bool) โ By default, source is not installed, but for debugging it might be useful to keep it around.keep_prefix(bool) โ Keep install prefix on failure. By default, destroys it. ๐keep_stage(bool) โ By default, stage is destroyed only if there are no exceptions during build. Set toTrueto keep the stage even with exceptions.restage(bool) โ Force spack to restage the package source.skip_patch(bool) โ Skip patch stage of build ifTrue.stop_before(str) โ stop execution before this installation phase (orNone)stop_at(str) โ last installation phase to be executed (orNone)tests(bool or list or set) โFalseto run no tests,Trueto test all packages, or a list of package names to run tests for someuse_cache(bool) โ Install from binary package, if available.verbose(bool) โ Display verbose build output (by default, suppresses it)
do_patch()โ applies patches if they haven't been applied already. no double-patching in this house ๐ซdo_restage()โ reverts expanded/checked out source to a pristine state. reset era, we love to see it โจdo_stage(mirror_only=False)โ unpacks and expands the fetched tarball. unboxing vid but make it build system ๐do_test(dirty=False, externals=False)โ runs the tests. note dem defaults, bestie:dirty=False,externals=False๐งชdo_uninstall(force=False)โ uninstall this package by spec. it's over bestie ๐download_instr(property) โ defines the default manual download instructions. Packages can override the property to provide more information.
Returns: default manual download instructions
Return type: (str)
env_flags(name: str, flags: Iterable[str]) -> Tuple[Iterable[str] | None, Iterable[str] | None, Iterable[str] | None]โ a classmethod. flag_handler that adds all flags to canonical environment variables. yes that's a three-tuple of optional iterables, we stay unbothered ๐env_mods_path(property) โ return the build environment modifications file path associated with staging.env_path(property) โ return the build environment file path associated with staging.extendable = Falseโ most packages are NOT extendable. Set toTrueif you want extensions. default is a hard no, babe ๐ โโ๏ธextendee_spec(property) โ Spec of the extendee of this package, orNoneif it is not an extensionextends(spec)โ returnsTrueif this package extends the given spec.
If self.spec is concrete, this returns whether this package extends the given spec.
If self.spec is not concrete, this returns whether this package may extend the given spec. it's giving "maybe, we'll see" ๐ฆ
fetch_options: Dict[str, Any] = {}โ set of additional options used when fetching package versions. starts empty, fill her up ๐fetch_remote_versions(concurrency: int | None = None) -> Dict[StandardVersion, str]โ find remote versions of this package.
Uses list_url and any other URLs listed in the package file.
Returns: a dictionary mapping versions to URLs
Return type: dict
fetcher(property) โ the fetcher, obviously ๐find_valid_url_for_version(version)โ returns a URL from which the specified version of this package may be downloaded after testing whether the url is valid. Will tryurl,urls, andlist_urlbefore failing. she's persistent, she's thorough ๐version: class Versionโ The version for which a URL is sought.
See Class Version (version.py)
flag_handler(property) โ typed asCallable[[str, Iterable[str]], Tuple[Iterable[str] | None, Iterable[str] | None, Iterable[str] | None]]. look at that signature, it's literally iconic ๐ฅflags_to_build_system_args(flags)โ converts flags into build system args. ๐format_doc(**kwargs)โ a classmethod. Wrap doc string at 72 characters and format nicely. 72 chars, no more, no notes โ๏ธfullname = 'spack.package_base'โ that's the full name, bestie ๐fullnames = ['spack.package_base']โ and there's the list version. plural queen ๐global_license_dir = '/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/etc/spack/licenses'โ the global license dir, exactly that path, no cap ๐global_license_file(property) โ returns the path where a global license file for this particular package should be stored.has_code = Trueโ most Spack packages are used to install source or binary code while those that do not can be used to install a set of other Spack packages. default isTrue, i.e. yes there's real code in here ๐งฌhome(property) โ the home. that's it, that's the vibe ๐homepage: str | None = Noneโ package homepage where users can find more information about the package. defaults toNone๐inject_flags(name: str, flags: Iterable[str]) -> Tuple[Iterable[str] | None, Iterable[str] | None, Iterable[str] | None]โ a classmethod. flag_handler that injects all flags through the compiler wrapper. ๐install_configure_args_path(property) โ return the configure args file path on successful installation.install_env_path(property) โ return the build environment file path on successful installation.install_log_path(property) โ return the (compressed) build log file path on successful installation. compressed, so it doesn't eat all your disk ๐๏ธinstall_test_root(property) โ return the install test root directory.installed(property) โ whether it's installed, periodt โinstalled_upstream(property) โ whether it's installed upstream. different vibe, same energy โ๏ธis_extension(property) โ is it an extension or not, bestie ๐keep_werror: str | None = Noneโ defaults toNone, keep that in mind ๐งlanguages: Dict[Spec, Set[str]]โ typed asDict[Spec, Set[str]], the whole language map situation ๐license_comment = '#'โ string. Contains the symbol used by the license manager to denote a comment. Defaults to#. ๐ฌlicense_files: List[str] = []โ list of strings. These are files that the software searches for when looking for a license. All file paths must be relative to the installation directory. More complex packages like Intel may require multiple licenses for individual components. Defaults to the empty list. so like, empty by default but she can hold many ๐
license_required = Falseโ boolean, bestie.Truemeans this software needs a license,Falsemeans every singlelicense_*attribute gets ignored. Defaults toFalse.license_url = ''โ string, a URL that points at the license setup instructions for the software. Defaults to the empty string.license_vars: List[str] = []โ list of strings: the environment variables you can set to tell the software where to look for a license when it's not in the usual spot. Defaults to the empty list.list_depth = 0โ the link depth to whichlist_urlshould be searched for new versions. ๐ฆlist_url: str | None = Noneโ the default list URL, aka the place to find available versions, no cap.property log_pathโ returns the build log file path associated with staging. Yes it's a property, yes it just gives you the path. Slay.maintainers: List[str] = []โ list of strings holding the GitHub usernames of the package maintainers. Important: do not include@here, specifically so we don't unnecessarily ping the users. Respect the no-ping rule, fr fr. ๐manual_download = Falseโ boolean. Set it toTruefor packages that need a manual download. Right now it's used by package sanity tests and by the generation of a more meaningful fetch failure error. It's giving "we can't fetch this for you, bestie."metadata_attrs = ['homepage', 'url', 'urls', 'list_url', 'extendable', 'parallel', 'make_jobs', 'maintainers', 'tags']โ the list of attributes that get excluded from a package's hash. Iconic little exclusion list.property metadata_dirโ returns the install metadata directory.module = <module 'spack.package_base' from '/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/lib/spack/spack/package_base.py'>โ literally the module object itself, path and all. That's the whole value, verbatim. ๐name = 'package_base'โ the name, and it is exactly'package_base'.namespace = 'spack'โ the namespace, and it is exactly'spack'.nearest_url(version)โ finds the URL with the "closest" version to version. The precedence order is literally this, in order:- Find the next lowest or equal version with a URL.
- If no lower URL, return the next higher URL.
- If no higher URL, return
None.
non_bindable_shared_objects: List[str] = []โ list of shared objects that should get replaced with a different library at runtime. Typically this includes stub libraries likelibcuda.so. When you link against a library listed here, the dependent only records its soname or filename, not its absolute path, so the dynamic linker will go search for it. Heads up: it accepts both file names and directory names โ for example["libcuda.so", "stubs"]ensureslibcuda.soand all libraries in thestubsdirectory are not bound by path. ๐ซถpackage_dir = '/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/lib/spack/spack'โ yep, that's the whole string, keep it exactly.parallel = Trueโ by default we build in parallel. Subclasses can override this, obviously.patches: Dict[Spec, List[Patch]]โ the patches mapping, typed exactly like that.property phase_log_filesโ finds the sorted phase log files written to the staging directory. Sorted, because chaos is not the vibe.classmethod possible_dependencies(transitive: bool = True, expand_virtuals: bool = True, depflag: int = 15, visited: dict | None = None, missing: dict | None = None, virtuals: set | None = None) -> Dict[str, Set[str]]โ returns a dict of the possible dependencies of this package. โจ
Parameters
- transitive (
boolorNone) โ return all transitive dependencies ifTrue, only direct dependencies ifFalse(defaultTrue). Note the double period in the original, we're keeping it. - expand_virtuals (
boolorNone) โ expand virtual dependencies into all possible implementations (defaultTrue). - depflag โ dependency types to consider.
- visited (
dictorNone) โ dict of names of dependencies visited so far, mapped to their immediate dependencies' names. - missing (
dictorNone) โ dict to populate with packages and their missing dependencies. - virtuals (
set) โ if provided, populate with virtuals seen so far.
Returns
dictionary mapping dependency names to their immediate dependencies
Return type
(dict)
Each item in the returned dictionary maps a (potentially transitive) dependency of this package to its possible immediate dependencies. If expand_virtuals is False, virtual package names will be inserted as keys mapped to empty sets of dependencies โ so virtuals, if not expanded, are treated as though they have no immediate dependencies.
Missing dependencies are ignored by default, but if a missing dict is provided it gets populated with package names mapped to any dependencies they have that are in no repositories. This is only populated if transitive is True.
And a note, bestie: the returned dict includes the package itself. Don't forget that part. ๐ฅ
property prefixโ gets the prefix into which this package should be installed.provided: Dict[Spec, Set[Spec]]โ the provided mapping, exactly that type.provided_together: Dict[Spec, List[Set[str]]]โ the provided-together mapping, exactly that type.classmethod provided_virtual_names()โ returns a sorted list of names of virtuals that can be provided by this package.provides(vpkg_name)โTrueif this package provides a virtual package with the specified name. Simple as that. ๐remove_prefix()โ removes the prefix for a package along with any empty parent directories. Cleanup era.requirements: Dict[Spec, List[Tuple[Tuple[Spec, ...], str, str | None]]]โ the requirements mapping, typed exactly like that, nested tuples and all.property rpathโ gets the rpath this package links with, as a list of paths.property rpath_argsโ gets the rpath args as a string, with-Wl,-rpath,for each element.run_after_callbacks = []โ empty list, that's the whole story.run_before_callbacks = []โ also an empty list.run_test(exe, options=[], expected=[], status=0, installed=False, purpose=None, skip_missing=False, work_dir=None)โ runs the test and confirms the expected results are obtained.
It logs any failures and continues โ they'll be re-raised later. ๐ธ
Parameters
- exe (
str) โ the name of the executable. - options (
strorlist) โ list of options to pass to the runner. - expected (
strorlist) โ list of expected output strings. Each string is a regex expected to match part of the output. - status (
intorlist) โ possible passing status values, with0meaning the test is expected to succeed. - installed (
bool) โ ifTrue, the executable must be in the install prefix. - purpose (
str) โ message to display before running the test. - skip_missing (
bool) โ skip the test if the executable is not in the install prefixbindirectory or the provided work_dir. - work_dir (
strorNone) โ path to the smoke test directory.
run_tests = Falseโ by default we do not run tests within a package'sinstall(). Defaults, okay?sanity_check_is_dir: List[str] = []โ list of prefix-relative directory paths (or a single path). If these don't exist after install, or if they exist but are not directories, sanity checks will fail. ๐ญsanity_check_is_file: List[str] = []โ list of prefix-relative file paths (or a single path). If these don't exist after install, or if they exist but are not files, sanity checks fail. Same energy, different check.setup_dependent_package(module, dependent_spec)โ sets up Python module-scope variables for dependent packages. Called before theinstall()method of dependents.
The default implementation does nothing, but an extendable package can override it to set up the module of its extensions. This is useful when there are some common steps to installing all extensions for a certain package.
Examples, because examples ate:
- Extensions often need to invoke the Python interpreter from the Python installation being extended. This routine can put a
python()Executableobject in the module scope for the extension package to simplify extension installs. - MPI compilers could set some variables in the dependent's scope that point to
mpicc,mpicxx, etc., letting them be called by common name regardless of which MPI is used. - BLAS/LAPACK implementations can set some variables indicating the path to their libraries, since these paths differ by BLAS/LAPACK implementation.
Parameters
- module (
spack.package_base.PackageBase.module) โ the Python module object of the dependent package. Packages can use this to set module-scope variables for the dependent to use. - dependent_spec (
spack.spec.Spec) โ the spec of the dependent package about to be built. This lets the extendee (self) query the dependent's state. Note that this package's spec is available asself.spec.
setup_dependent_run_environment(env, dependent_spec)โ sets up the run environment of packages that depend on this one. ๐ง
Okay so picking right back up where we left off, no awkward re-intro needed ๐
...and this one is the run-environment sibling. It's similar to setup_run_environment, but it's the one you reach for to modify the run environments of packages that depend on this one. It's giving "I'm the extender, you're the extendee" energy fr fr.
This hands packages like Python โ and honestly anyone else riding the extension model โ a clean way to implement shared environment or run-time settings for their dependencies. Iconic behavior, no cap ๐โจ
Parameters
env(spack.util.environment.EnvironmentModifications) โ the environment modifications to be applied when the dependent package is run. Package authors can call methods on it to alter the build environment.dependent_spec(spack.spec.Spec) โ the spec of the dependent package that's about to be run. This lets the extendee (that'sself, bestie) query the dependent's state. Heads up tho: this package's own spec is available asself.spec๐ซถ
๐ธ setup_run_environment(env)
Sets up the run environment for a package. Simple, clean, does what it says on the tin.
Parameters
env(spack.util.environment.EnvironmentModifications) โ the environment modifications to be applied when the package is run. Package authors can call methods on it to alter the run environment. Go off, customize ๐ซ
๐ property stage
Gets the build staging area for this package.
Real talk: this automatically instantiates a Stage object if the package doesn't have one yet โ but it does not create the Stage directory on the filesystem. So like, it builds the idea of the stage, not the actual folder. Don't be surprised when there's nothing on disk yet ๐
๐ฆ test()
Bestie this one shows up with literally zero description attached โ the manpage gives it no notes, so I'm not gonna invent any. It exists, it's there, that's the whole vibe ๐
โญ test_requires_compiler: bool = False
Set this to True to say "hey, this stand-alone test needs a compiler." It's used to make sure a compiler โ plus build dependencies like 'cmake' โ are actually available before anyone tries to build a custom test code. Very "let me check my bag before we leave the house" of it ๐
๐ test_suite: TestSuite | None = None
A TestSuite instance, used to manage stand-alone tests for one or more specs. Defaults to None, because we're not assuming anything about your testing lifestyle.
๐ property tester
Again โ no description in the source, so no lore from me. It's a property called tester, and that's genuinely all we've got ๐ฅ
๐ property times_log_path
Returns the times log json file. That's it, that's the whole thing โจ
๐ transitive_rpaths = True
When True, add RPATHs for the entire DAG. When False, add RPATHs only for the immediate dependencies. So like โ whole friend group vs. just your close circle. Default's the whole group chat ๐
๐ง static uninstall_by_spec(spec, force=False, deprecator=None)
No description given in the manpage for this one either โ I'm just here to make sure you see the signature exactly as it is:spec,force=False,deprecator=None. Don't let me catch you guessing at what the defaults are ๐
๐ซถ unit_test_check()
Hook for unit tests to assert things about package internals. Obsessed with a designated check-in point tbh.
Unit tests can override this function to run checks after Package.install and all the post-install hooks run, but before the database is updated.
The overridden function may signal that the install procedure should bail out early (before the database gets touched) by returning False โ or really any value where bool(result) is False.
Returns True to continue, False to skip install()
Return type (bool)
๐ update_external_dependencies(extendee_spec=None)
A method to override in package classes when you wanna handle external dependencies. Your code, your rules โจ
๐ url_for_version(version)
Returns a URL from which the specified version of this package may be downloaded.
version: class Version The version for which a URL is sought.
See Class Version (version.py) โ go look it up bestie, it's literally right there ๐
๐ฆ url_version(version)
Given a version, this returns a string that should be substituted into the package's URL so you can download that version.
By default this just returns the version string. Subclasses may need to override this tho โ the manpage's own example: boost versions, where you need to make sure there are _'s in the download URL. Little detail, huge consequences ๐
๐ธ variants: Dict[str, Tuple[Variant, Spec]]
That's the whole entry โ a type annotation and nothing else. No prose, no parameters, no commentary. Simply existing, simply iconic โญ
๐
property version
Yep โ no description here either. It's a property, it's called version, and the manpage said what it said (nothing) ๐ซถ
๐ classmethod version_urls() -> Dict[StandardVersion, str]
A dict of explicitly defined URLs for versions of this package.
Returns A dict mapping version to url, ordered by version.
Important caveat, keep this one close: a version's URL only shows up in the result if it has an explicitly defined url argument. So this list may legitimately be empty if a package only defines url at the top level. Not a bug, just how it works ๐ซ
๐ versions: dict
Another one with no accompanying description. It's adict, it's calledversions, we move ๐ง
โจ view()
Create a view with the prefix of this package as the root. Extensions added to this view will modify the installation prefix of this package. Very main-character-of-your-own-prefix energy ๐
๐ฅ virtual = False
By default, packages are not virtual. Virtual packages override this attribute. So if you're building a virtual, you're the exception, and you know it ๐
๐ property virtuals_provided
The virtual packages provided by this package, alongside its spec.
๐ exception spack.package_base.PackageError(message, long_msg=None)
Bases: SpackError
Raised when something is wrong with a package definition. That's the whole mood โ your package definition has a problem and Spack is telling you about it ๐ญ
๐ง class spack.package_base.PackageMeta(name, bases, attr_dict)
Bases: PhaseCallbacksMeta, DetectablePackageMeta, DirectiveMeta, MultiMethodMeta
The package metaclass, here to support directives (like depends_on) and phases. It's the backstage crew making the whole show work, ngl ๐ญ
๐ exception spack.package_base.PackageStillNeededError(spec, dependents)
Bases: InstallError
Raised when a package is still needed by another one on uninstall. Translation: you tried to yeet something that other packages are still emotionally (and functionally) attached to. Not happening โ
๐ซถ class spack.package_base.PackageViewMixin
Bases: object
This one collects all the functionality related to adding installed Spack packages to views. Packages can customize how they get added to views by overriding these functions. Fully DIY, fully your call ๐
๐ add_files_to_view(view, merge_map, skip_if_exists=True)
Given a map of package files to destination paths in the view, add the files to the view. By default this adds all files. Alternative implementations may skip some files โ for example, if other packages linked into the view already include that file. Sharing is caring, etc ๐
Parameters
view(spack.filesystem_view.FilesystemView) โ the view that's being updated.merge_map(dict) โ maps absolute source paths to absolute dest paths for all files from this package.skip_if_exists(bool) โ whenTrue, don't link files in the view when they already exist. WhenFalse, always link files, without checking whether they already exist. Bold. Reckless. Respect โจ
๐ธ remove_files_from_view(view, merge_map)
Given a map of package files to files currently linked in the view, remove the files from the view. The default implementation removes all files. Alternative implementations might not remove everything โ for example, if two packages include the same file, it should only be removed once both packages are removed. Fair's fair ๐ซถ
๐ฆ view_destination(view)
The target root directory โ every file gets added relative to this directory.
โญ view_file_conflicts(view, merge_map)
Report any files that would prevent adding this package to the view. The default implementation looks for any files that already exist. Alternative implementations may allow some of those files to exist in the view โ and in that case, they'd be omitted from the results. Very "we can coexist, mostly" ๐
๐ view_source()
The source root directory that'll be added to the view โ files are added such that their path relative to the view destination matches their path relative to the view source. Symmetry, we love to see it ๐
๐ class spack.package_base.RedistributionMixin
Bases: object
All the logic for determining whether a Package is source/binary redistributable. The bouncer at the club door of your mirror ๐ชโจ
๐ disable_redistribute: Dict[Spec, DisableRedistribute]
Stores whether a given Spec source/binary should not be redistributed.
๐ property redistribute_binary
Whether it should be possible to create a binary out of an installed instance of this package.
๐ซ classmethod redistribute_source(spec)
Whether it should be possible to add the source of this package to a Spack mirror.
๐ง class spack.package_base.WindowsRPath
Bases: object
The collection of functionality surrounding Windows RPATH-specific features.
Here's the deal, and I need you to hear me on this: this is essentially meaningless for all other platforms, because they use RPATH. All methods within this class are no-ops on non-Windows. Packages can customize and manipulate this class the way they would a genuine RPATH โ i.e. adding directories that contain runtime library dependencies. Windows-only party, everyone else's invite got lost ๐
๐ win_add_library_dependent()
Return an extra set of directories that require linking for the package.
This method should be overridden by packages that produce binaries/libraries/python extension modules/etc. that get installed into directories outside a package's bin, lib, and lib64 directories, but still require linking against one of the package's dependencies, or against other components of the package itself. No-op otherwise.
Returns List of additional directories that require linking.
๐ win_add_rpath()
Return an extra set of rpaths for the package.
This method should be overridden by packages that need to include additional paths to be searched by rpath. No-op otherwise.
Returns List of additional rpaths.
๐ฅ windows_establish_runtime_linkage()
Establish RPATH on Windows.
It performs symlinking to incorporate rpath dependencies into Windows runtime search paths. Chain it together, make it work, periodt ๐
spack.package_base.build_system_flags(name: str, flags: Iterable[str]) -> Tuple[Iterable[str] | None, Iterable[str] | None, Iterable[str] | None]โ okay bestie, this is a flag_handler that passes flags to the build system arguments โจ. Huge caveat though: any package that uses build_system_flags also has to implementflags_to_build_system_args, or derive from a class that does. RN,AutotoolsPackageandCMakePackageare the ones that implement it ๐ .spack.package_base.deprecated_version(pkg: PackageBase, version: str | StandardVersion) -> boolโ returnsTrueiff the version is deprecated, no cap ๐
Parameters:
- pkg โ the package whose version is being checked.
- version โ the version being checked.
spack.package_base.detectable_packages = {}โ registers which packages are the detectable ones, by repo and package name. Heads up: it needs a pass of package repositories to be filled ๐ซถ.spack.package_base.env_flags(name: str, flags: Iterable[str]) -> Tuple[Iterable[str] | None, Iterable[str] | None, Iterable[str] | None]โ a flag_handler that adds all flags to the canonical environment variables ๐ธ.spack.package_base.flatten_dependencies(spec, flat_dir)โ makes each dependency of spec present in flat_dir via symlink ๐.spack.package_base.inject_flags(name: str, flags: Iterable[str]) -> Tuple[Iterable[str] | None, Iterable[str] | None, Iterable[str] | None]โ a flag_handler that injects all the flags through the compiler wrapper ๐.spack.package_base.install_dependency_symlinks(pkg, spec, prefix)โ executes a dummy install and flattens dependencies โจ. You can use this routine in apackage.pydefinition by settinginstall = install_dependency_symlinks. It literally comes in handy for creating a common location for the installation of third-party libraries ๐.spack.package_base.on_package_attributes(**attr_dict)โ a decorator that executes an instance function only if the object has attr values ๐. It runs the decorated method only if, at the moment of calling, the instance has attributes equal to certain values.
Parameters:
- attr_dict (dict) โ dictionary mapping attribute names to their required values.
spack.package_base.possible_dependencies(*pkg_or_spec: str | Spec | Type[PackageBase], transitive: bool = True, expand_virtuals: bool = True, depflag: int = 15, missing: dict | None = None, virtuals: set | None = None) -> Dict[str, Set[str]]โ gets the possible dependencies of a bunch of packages. For the full tea, seePackageBase.possible_dependenciesโ.spack.package_base.preferred_version(pkg: PackageBase)โ returns a sorted list of the preferred versions of the package, periodt ๐.
Parameters:
- pkg โ the package whose versions are to be assessed.
spack.package_base.spack_times_log = 'install_times.json'โ the filename of the json with total build and phase times (in seconds) โฑ๏ธ.spack.package_base.use_cray_compiler_names()โ compiler names for builds that rely on cray compiler names, iykyk ๐ฅ๏ธ.
๐ spack.package_prefs module
๐ class spack.package_prefs.PackagePrefs(pkgname, component, vpkg=None, all=True)
Bases: object.
This class defines the sort order for a set of specs โจ. Spack's package preference implementation uses PackagePrefss to define sort order โ and yes, the class looks at Spack's packages.yaml configuration and, when called on a spec, returns a key you can use to sort that spec in order of the user's preferences ๐
.
You can use it like this:
# key function sorts CompilerSpecs for mpich in order of preference kf = PackagePrefs('mpich', 'compiler')
compiler_list.sort(key=kf)Or like this:
# key function to sort VersionLists for OpenMPI in order of preference. kf = PackagePrefs('openmpi', 'version')
version_list.sort(key=kf)Optionally, you can sort in order of preferred virtual dependency providers ๐ฆ. To do that, provide 'providers' and a third argument denoting the virtual package (e.g., mpi):
kf = PackagePrefs('trilinos', 'providers', 'mpi')
provider_spec_list.sort(key=kf)- classmethod
has_preferred_providers(pkgname, vpkg)โ whether a specific package has preferred vpkg providers ๐. - classmethod
has_preferred_targets(pkg_name)โ whether a specific package has preferred vpkg providers. (Yes, that's literally what the docstring says โ same wording, no cap ๐ญ.) - classmethod
order_for_package(pkgname, component, vpkg=None, all=True)โ given a package name, a sort component (e.g. version, compiler, โฆ), and an optional vpkg, returns the list from the packages config ๐. - classmethod
preferred_variants(pkg_name)โ returns aVariantMapof preferred variants/values for a spec ๐.
๐ exception spack.package_prefs.VirtualInPackagesYAMLError(message, long_message=None)
Bases: SpackError.
Raised when a disallowed virtual is found in packages.yaml ๐จ.
๐ง spack.package_prefs.get_package_dir_permissions(spec)
Returns the permissions configured for the spec. It includes the GID bit if group permissions are on โ which makes the group attribute sticky for the directory ๐. Package-specific settings take precedent over settings for all. (That's "precedent" as written in the original, bestie โ not me ๐.)
๐ธ spack.package_prefs.get_package_group(spec)
Returns the unix group associated with the spec. Package-specific settings take precedence over settings for all ๐ซถ.
โญ spack.package_prefs.get_package_permissions(spec)
Returns the permissions configured for the spec. Package-specific settings take precedence over settings for all ๐.
๐ฅ spack.package_prefs.is_spec_buildable(spec)
Returns true if the spec is configured as buildable โ .
๐ฆ spack.package_prefs.spec_externals(spec)
Returns a list of external specs (with the external directory path filled in), one for each known external installation ๐.
๐ spack.package_test module
spack.package_test.compare_output(current_output, blessed_output)โ compares blessed and current output of executables ๐.spack.package_test.compare_output_file(current_output, blessed_output_file)โ same as above, but when the blessed output is given as a file ๐.spack.package_test.compile_c_and_execute(source_file, include_flags, link_flags)โ compiles C@p source_filewith@p include_flagsand@p link_flags, then runs it and returns the output โ๏ธ.
๐ง spack.parser module
Parser for spec literals โจ.
Here is the EBNF grammar for a spec:
spec = [name] [node_options] { ^[edge_properties] node } |
[name] [node_options] hash |
filename
node = name [node_options] |
[name] [node_options] hash |
filename
node_options = [@(version_list|version_pair)] [%compiler] { variant }
edge_properties = [ { bool_variant | key_value } ]
hash = / id
filename = (.|/|[a-zA-Z0-9-_]*/)([a-zA-Z0-9-_./]*)(.json|.yaml)
name = id | namespace id
namespace = { id . }
variant = bool_variant | key_value | propagated_bv | propagated_kv
bool_variant = +id | ~id | -id
propagated_bv = ++id | ~~id | --id
key_value = id=id | id=quoted_id
propagated_kv = id==id | id==quoted_id
compiler = id [@version_list]
version_pair = git_version=vid
version_list = (version|version_range) [ { , (version|version_range)} ]
version_range = vid:vid | vid: | :vid | :
version = vid
git_version = git.(vid) | git_hash
git_hash = [A-Fa-f0-9]{40}
quoted_id = " id_with_ws " | ' id_with_ws '
id_with_ws = [a-zA-Z0-9_][a-zA-Z_0-9-.\s]*
vid = [a-zA-Z0-9_][a-zA-Z_0-9-.]*
id = [a-zA-Z0-9_][a-zA-Z_0-9-]*Identifiers using the <name>=<value> command โ like architectures and compiler flags โ require a space before the name ๐ซถ.
There is one context-sensitive part: ids in versions may contain ., while other ids may not ๐.
There is one ambiguity: since - is allowed in an id, you need to put whitespace space before -variant for it to be tokenized properly. You can either use whitespace, or you can just use ~variant since it means the same thing ๐
. Spack uses ~variant in directory names and in the canonical form of specs to avoid ambiguity. Both are provided because ~ can cause shell expansion when it is the first character in an id typed on the command line ๐ธ.
spack.parser.ALL_TOKENS = re.compile('(?P<START_EDGE_PROPERTIES>(?:\\^\\[))|(?P<END_EDGE_PROPERTIES>(?:\\]))|(?P<DEPENDENCY>(?:\\^))|(?P<VERSION_HASH_PAIR>(?:@(?:(?:(?:git\\.(?:(?:[a-zA-Z_0-9][a-zA-Z_0-9./\\-]*)))|(?:(?:[A-Fa-f0-9]{40})))โ regex to scan a valid text ๐.spack.parser.ANALYSIS_REGEX = re.compile('(?P<START_EDGE_PROPERTIES>(?:\\^\\[))|(?P<END_EDGE_PROPERTIES>(?:\\]))|(?P<DEPENDENCY>(?:\\^))|(?P<VERSION_HASH_PAIR>(?:@(?:(?:(?:git\\.(?:(?:[a-zA-Z_0-9][a-zA-Z_0-9./\\-]*)))|(?:(?:[A-Fa-f0-9]{40})))โ regex to analyze an invalid text ๐.
spack.parser.ERROR_HANDLING_REGEXES
bestie this is the entire guest list ๐ it's literally described as the "List of all valid regexes followed by error analysis regexes" โ every valid pattern first, then the error-analysis ones tossed on at the end. Copy it exactly, no editing, no "cleaning it up", okay? ๐
spack.parser.ERROR_HANDLING_REGEXES =
['(?P<START_EDGE_PROPERTIES>(?:\\^\\[))',
'(?P<END_EDGE_PROPERTIES>(?:\\]))', '(?P<DEPENDENCY>(?:\\^))',
'(?P<VERSION_HASH_PAIR>(?:@(?:(?:(?:git\\.(?:(?:[a-zA-Z_0-9][a-zA-Z_0-9./\\-]*)))|(?:(?:[A-Fa-f0-9]{40}))))=(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))))',
'(?P<GIT_VERSION>@(?:(?:(?:git\\.(?:(?:[a-zA-Z_0-9][a-zA-Z_0-9./\\-]*)))|(?:(?:[A-Fa-f0-9]{40})))))',
'(?P<VERSION>(?:@\\s*(?:(?:(?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)(?!\\s*=))?)|=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))(?:\\s*,\\s*(?:(?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)(?!\\s*=))?)|=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)))*)))',
'(?P<PROPAGATED_BOOL_VARIANT>(?:(?:\\+\\+|~~|--)\\s*[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*))',
'(?P<BOOL_VARIANT>(?:[~+-]\\s*[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*))',
'(?P<PROPAGATED_KEY_VALUE_PAIR>(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*==(?:(?:[a-zA-Z_0-9\\-+\\*.,:=\\~\\/\\\\]+)|(?:\'(?:[^\']|(?<=\\\\)\')*\'|\\"(?:[^\\"]|(?<=\\\\)\\")*\\"))))',
'(?P<KEY_VALUE_PAIR>(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*=(?:(?:[a-zA-Z_0-9\\-+\\*.,:=\\~\\/\\\\]+)|(?:\'(?:[^\']|(?<=\\\\)\')*\'|\\"(?:[^\\"]|(?<=\\\\)\\")*\\"))))',
'(?P<COMPILER_AND_VERSION>(?:%\\s*(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*)(?:[\\s]*)@\\s*(?:(?:(?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)(?!\\s*=))?)|=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))(?:\\s*,\\s*(?:(?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)(?!\\s*=))?)|=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)))*)))',
'(?P<COMPILER>(?:%\\s*(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*)))',
'(?P<FILENAME>(?:(?:\\.|\\/|[a-zA-Z0-9-_]*\\/)(?:[a-zA-Z0-9-_\\.\\/]*)(?:\\.json|\\.yaml)))',
'(?P<FULLY_QUALIFIED_PACKAGE_NAME>(?:(?:(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-]*)(?:\\.(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-]*))+)))',
'(?P<UNQUALIFIED_PACKAGE_NAME>(?:(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-]*)))',
'(?P<DAG_HASH>(?:/(?:[a-zA-Z_0-9]+)))', '(?P<WS>(?:\\s+))',
'(?P<UNEXPECTED>(?:.[\\s]*))']class spack.parser.EdgeAttributeParser(ctx, literal_str) โ Bases: object
Yeah it inherits from plain old object, nothing dramatic ๐ธ its little attribute squad is ctx and literal_str, and it's got one method:
parse()โ that's it, that's the whole method. No parameters listed, no drama ๐
class spack.parser.ErrorTokenType(*values) โ Bases: TokenBase
It's the "Enum with regexes for error analysis" โ literally its whole job is being the error-analysis enum ๐
UNEXPECTED = 1โ the one and only value, iconic behavior fr fr
class spack.parser.FileParser(ctx) โ Bases: object
This one's whole mission? "Parse a single spec from a JSON or YAML file" ๐๏ธ hold ctx in your heart, and then:
parse(initial_spec: Spec) -> Specโ "Parse a spec tree from a specfile."- Parameters:
initial_specโ object where to parse the spec - Return: The
initial_specpassed as argument, once constructed
- Parameters:
spack.parser.GIT_REF = '(?:[a-zA-Z_0-9][a-zA-Z_0-9./\\-]*)'
"Git refs include branch names, and can contain "." and "/"" โ so yeah, dots and slashes are lowkey invited to this party ๐
spack.parser.IDENTIFIER = '(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-]*)'
The valid name for specs and variants, periodt. And bestie, read the receipts: here we are not using the previous "w[w.-]*" since that would match most characters that can be part of a word in any language ๐ญ so we keep it strict.
spack.parser.NO_QUOTES_NEEDED = re.compile('^[a-zA-Z0-9,/_.-]+$')
The vibe check for quoting: "Variant/flag values that match this can be left unquoted in Spack output" โจ
spack.parser.QUOTED_VALUE = '(?:\'(?:[^\']|(?<=\\\\)\')*\'|\\"(?:[^\\"]|(?<=\\\\)\\")*\\")'
"Quoted values can be anything in between quotes, including escaped quotes." โ so escaped quotes are absolutely allowed to crash the party, no cap ๐ฆ
spack.parser.SPLIT_KVP = re.compile('^([a-zA-Z_0-9][a-zA-Z_0-9\\-.]*)(==?)(.*)$')
"Regex with groups to use for splitting (optionally propagated) key-value pairs" โ the groups are doing the heavy lifting here, and they know it ๐ช
spack.parser.STRIP_QUOTES = re.compile('^([\'\\"])(.*)\\1$')
"Regex to strip quotes. Group 2 will be the unquoted string." โ Group 2 is the main character, everyone else is supporting cast ๐
class spack.parser.SpecNodeParser(ctx) โ Bases: object
"Parse a single spec node from a stream of tokens" โ a whole little specialist for one node at a time ๐ง its attributes are ctx, has_compiler, and has_version, and it has:
parse(initial_spec: Spec | None = None) -> Spec | Noneโ "Parse a single spec node from a stream of tokens"- Parameters:
initial_specโ object to be constructed - Return: The object passed as argument
- Parameters:
class spack.parser.SpecParser(literal_str: str) โ Bases: object
The big one, "Parse text into specs" ๐ฅ it carries ctx and literal_str, and its methods are:
all_specs() -> List[Spec]โ "Return all the specs that remain to be parsed"next_spec(initial_spec: Spec | None = None) -> Spec | Noneโ "Return the next spec parsed from text."- Parameters:
initial_specโ object where to parse the spec. IfNonea new one will be created. - Return: The spec that was parsed
- Parameters:
tokens() -> List[Token]โ "Return the entire list of token from the initial text. White spaces are filtered out." ๐ซง bye whitespace, you're not on the list
exception spack.parser.SpecParsingError(message, token, text) โ Bases: SpecSyntaxError
"Error when parsing tokens" ๐ that's the whole lore.
exception spack.parser.SpecSyntaxError โ Bases: Exception
"Base class for Spec syntax errors" โ the mother of all spec syntax drama ๐
exception spack.parser.SpecTokenizationError(matches, text) โ Bases: SpecSyntaxError
"Syntax error in a spec string" โ short, sweet, devastating ๐ญ
spack.parser.TOKEN_REGEXES
This one is "List of all the regexes used to match spec parts, in order of precedence" โ so the order is NOT a suggestion, it's the whole point ๐
spack.parser.TOKEN_REGEXES = ['(?P<START_EDGE_PROPERTIES>(?:\\^\\[))',
'(?P<END_EDGE_PROPERTIES>(?:\\]))', '(?P<DEPENDENCY>(?:\\^))',
'(?P<VERSION_HASH_PAIR>(?:@(?:(?:(?:git\\.(?:(?:[a-zA-Z_0-9][a-zA-Z_0-9./\\-]*)))|(?:(?:[A-Fa-f0-9]{40}))))=(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))))',
'(?P<GIT_VERSION>@(?:(?:(?:git\\.(?:(?:[a-zA-Z_0-9][a-zA-Z_0-9./\\-]*)))|(?:(?:[A-Fa-f0-9]{40})))))',
'(?P<VERSION>(?:@\\s*(?:(?:(?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)(?!\\s*=))?)|=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))(?:\\s*,\\s*(?:(?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)(?!\\s*=))?)|=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)))*)))',
'(?P<PROPAGATED_BOOL_VARIANT>(?:(?:\\+\\+|~~|--)\\s*[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*))',
'(?P<BOOL_VARIANT>(?:[~+-]\\s*[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*))',
'(?P<PROPAGATED_KEY_VALUE_PAIR>(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*==(?:(?:[a-zA-Z_0-9\\-+\\*.,:=\\~\\/\\\\]+)|(?:\'(?:[^\']|(?<=\\\\)\')*\'|\\"(?:[^\\"]|(?<=\\\\)\\")*\\"))))',
'(?P<KEY_VALUE_PAIR>(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*=(?:(?:[a-zA-Z_0-9\\-+\\*.,:=\\~\\/\\\\]+)|(?:\'(?:[^\']|(?<=\\\\)\')*\'|\\"(?:[^\\"]|(?<=\\\\)\\")*\\"))))',
'(?P<COMPILER_AND_VERSION>(?:%\\s*(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*)(?:[\\s]*)@\\s*(?:(?:(?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)(?!\\s*=))?)|=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))(?:\\s*,\\s*(?:(?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b))?:(?:=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)(?!\\s*=))?)|=?(?:[a-zA-Z0-9_][a-zA-Z_0-9\\-\\.]*\\b)))*)))',
'(?P<COMPILER>(?:%\\s*(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-.]*)))',
'(?P<FILENAME>(?:(?:\\.|\\/|[a-zA-Z0-9-_]*\\/)(?:[a-zA-Z0-9-_\\.\\/]*)(?:\\.json|\\.yaml)))',
'(?P<FULLY_QUALIFIED_PACKAGE_NAME>(?:(?:(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-]*)(?:\\.(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-]*))+)))',
'(?P<UNQUALIFIED_PACKAGE_NAME>(?:(?:[a-zA-Z_0-9][a-zA-Z_0-9\\-]*)))',
'(?P<DAG_HASH>(?:/(?:[a-zA-Z_0-9]+)))', '(?P<WS>(?:\\s+))']notice how there's no UNEXPECTED here? yeah, that one's an error-handling-only guest ๐ซ
class spack.parser.Token(kind: TokenBase, value: str, start: int | None = None, end: int | None = None) โ Bases: object
"Represents tokens; generated from input by lexer and fed to parse()." โ it's giving middle-man but make it essential ๐ its attributes are end, kind, start, and value. All four. Love that for them.
class spack.parser.TokenBase(new_class_name, /, names, *, module=None, qualname=None, type=None, start=1, boundary=None) โ Bases: Enum
"Base class for an enum type with a regex value" โ the blueprint, the moment, the mother ๐
class spack.parser.TokenContext(token_stream: Iterator[Token]) โ Bases: object
"Token context passed around by parsers" โ it literally gets passed around, it's the social butterfly of the parser ๐ฆ its attributes are current_token, next_token, and token_stream, and its methods:
accept(kind: TokenType)โ "If the next token is of the specified kind, advance the stream and returnTrue. Otherwise returnFalse." โ yes bestie, advance +True, or no advance +False, that's the whole deal โจadvance()โ "Advance one token" โ one. singular. no skipping ๐ฅexpect(*kinds: TokenType)โ it exists, it's listed, it takes*kinds: TokenType๐
class spack.parser.TokenType(*values) โ Bases: TokenBase
"Enumeration of the different token kinds in the spec grammar."
Okay so the order of declaration here is extremely important, because text containing specs gets parsed with a single regex made by "|".join(...) of all the regexes in the order of declaration. Do not rearrange. Do not shuffle. Do not "optimize". ๐
The values, in all their numbered glory:
BOOL_VARIANT = 8COMPILER = 12COMPILER_AND_VERSION = 11DAG_HASH = 16DEPENDENCY = 3END_EDGE_PROPERTIES = 2FILENAME = 13FULLY_QUALIFIED_PACKAGE_NAME = 14GIT_VERSION = 5KEY_VALUE_PAIR = 10PROPAGATED_BOOL_VARIANT = 7PROPAGATED_KEY_VALUE_PAIR = 9START_EDGE_PROPERTIES = 1UNQUALIFIED_PACKAGE_NAME = 15VERSION = 6VERSION_HASH_PAIR = 4WS = 17
spack.parser.VALUE = '(?:[a-zA-Z_0-9\\-+\\*.,:=\\~\\/\\\\]+)'
"These are legal values that can be parsed bare, without quotes on the command line." โ bare-faced and unbothered, we love a low-maintenance value ๐ธ
spack.parser.parse(text: str) -> List[Spec]
"Parse text into a list of strings" (yes, that's what it says โ a list of strings, even though the return type is specs, we don't question the docs, we just report them, iykyk ๐).
- Parameters:
text(str) โ text to be parsed - Returns: List of specs
spack.parser.parse_one_or_raise(text: str, initial_spec: Spec | None = None) -> Spec
"Parse exactly one spec from text and return it, or raise" โ exactly one, bestie, not two, not zero ๐
- Parameters:
text(str) โ text to be parsedinitial_specโ buffer where to parse the spec. IfNonea new one will be created.
spack.parser.quote_if_needed(value: str) -> str
"Add quotes around the value if it requires quotes." It'll add quotes around the value unless it matches NO_QUOTES_NEEDED. And specifically:
- single quotes by default ๐
- double quotes around any value that contains single quotes
If double quotes are used, we json-escape the string โ that is, we escape \, ", and control codes. Very "we did the escaping so you don't have to" of it ๐
spack.parser.strip_quotes_and_unescape(string: str) -> str
"Remove surrounding single or double quotes from string, if present." โ if present!! conditional queen ๐
spack.parser.tokenize(text: str) -> Iterator[Token]
"Return a token generator from the text passed as input."
Raises SpecTokenizationError โ if we can't tokenize anymore, but didn't reach the end of the input text. ๐ญ so like, if the input just... stops making sense? yeah that's a whole error.
๐ฉน spack.patch.FilePatch
class spack.patch.FilePatch(pkg: PackageBase, relative_path: str, level: int,
working_dir: str, reverse: bool = False, ordering_key: Tuple[str, int] | None = None)Bases: Patch
Okay so this one describes a patch that gets pulled from a file that lives right there in the repo. Like, it's local local. ๐
sha256(property,str) โ gets you the patch's checksum, bestie. Returns the sha256 of the patch file. ๐to_dict()โDict[str, Any]โ hands you a dictionary-flavored representation of the patch. Returns a dictionary representation. cute. โจ
๐ spack.patch.NoSuchPatchError
exception spack.patch.NoSuchPatchError(message, long_message=None)Bases: SpackError
This one gets raised when the patch file you're looking for literally doesn't exist. gone. not found. deceased. ๐ญ
๐งฌ spack.patch.Patch
class spack.patch.Patch(pkg: PackageBase, path_or_url: str, level: int,
working_dir: str, reverse: bool = False)Bases: object
The base class for all the patches. the mother of patches, fr.
Heads up, this is important: the owning package is not necessarily the package the patch actually gets applied to โ in the case where a dependent package patches its own dependency, it's the dependent's fullname. keep that in mind, bestie ๐ง
apply(stage: Stage)โNoneโ applies the patch to the source code chillin' inside a stage. stage = the stage where the source code lives. ๐ฌsha256: strโ the hash field, just vibing there.to_dict()โDict[str, Any]โ dictionary representation of the patch. Returns a dictionary representation. ๐
๐๏ธ spack.patch.PatchCache
class spack.patch.PatchCache(repository: RepoPath, data: Dict[str, Any] | None = None)Bases: object
An index of the patches used in a repository, keyed by sha256 hash. ๐
The whole point: this lets us look up patches without loading every single package (slay, performance). It's also needed to properly implement dependency patching, because we need a way to look up patches that come from packages that are not in the Spec sub-DAG.
The patch index looks like this in a file (it's YAML, but we write JSON):
patches:
sha256:
namespace1.package1:
<patch json>
namespace2.package2:
<patch json>
... etc. ...classmethod from_json(stream, repository)โPatchCacheโ spins up a fresh newPatchCachefrom JSON. Params: stream = stream of data; repository = repository containing package. Returns a newPatchCacheinstance. ๐ฃpatch_for_package(sha256: str, pkg: PackageBase)โPatchโ looks up a patch in the index and builds a patch object for it. We build these objects lazily because building them requires info about the package's location in its repo. Params: sha256 = the hash to look up; pkg = the Package object to get the patch for. Returns the patch object. ๐งto_json(stream: Any)โNoneโ dumps a JSON representation to a stream. Param: stream = stream of data. ๐คupdate(other: PatchCache)โNoneโ updates this cache with the contents of another one. Param: other = another patch cache to merge. ๐ซupdate_package(pkg_fullname: str)โNoneโ updates the patch cache. Param: pkg_fullname = the package to update. ๐
๐ค spack.patch.PatchDirectiveError
exception spack.patch.PatchDirectiveError(message, long_message=None)Bases: SpackError
Raised when the wrong arguments get handed to the patch directive. like, read the docs first babes. ๐
๐ spack.patch.PatchLookupError
exception spack.patch.PatchLookupError(message, long_message=None)Bases: NoSuchPatchError
Raised when a patch file can't be located from its sha256. the hash said she was there but she was not. ๐
๐ spack.patch.UrlPatch
class spack.patch.UrlPatch(pkg: PackageBase, url: str, level: int = 1, *,
working_dir: str = '.', reverse: bool = False, sha256: str,
ordering_key: Tuple[str, int] | None = None, archive_sha256: str | None = None)Bases: Patch
Describes a patch that gets retrieved from a URL โ so it's coming from the internet, main character energy. ๐โจ
apply(stage: Stage)โNoneโ applies the patch to the source in a stage. Param: stage = stage where source code lives. ๐ฌstage(property,Stage) โ the stage in which we download (and unpack) the URL patch. Returns the stage object. ๐ฆto_dict()โDict[str, Any]โ dictionary representation. Returns a dictionary representation. ๐
๐ ๏ธ spack.patch.apply_patch
spack.patch.apply_patch(stage: Stage, patch_path: str, level: int = 1,
working_dir: str = '.', reverse: bool = False) -> NoneApplies the patch sitting at patch_path to the code in the stage. the workhorse function, honestly iconic.
Params:
- stage โ stage with code that will be patched
- patch_path โ filesystem location for the patch to apply
- level โ patch level
- working_dir โ relative path within the stage to change to
- reverse โ reverse the patch
๐งพ spack.patch.from_dict
spack.patch.from_dict(dictionary: Dict[str, Any], repository: RepoPath | None = None) -> PatchCreates a patch from a json dictionary. ๐
Params:
- dictionary โ dictionary representation of a patch
- repository โ repository containing package
Returns: a patch object.
Raises ValueError โ if owner or url/relative_path are missing in the dictionary. so likeโฆ don't forget those. ๐ฌ
๐ spack.paths module
Defines paths that are part of Spack's directory structure. ๐บ๏ธ
Real talk: do not import other spack modules here. This module is used all throughout Spack and it should bring in a minimal number of external dependencies. keep it light. ๐ชถ
spack.paths.bin_path=/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/binโ the bin directory in the spack prefix ๐๏ธspack.paths.default_misc_cache_path=/home/abuild/.spack/cacheโ transient caches for Spack data (virtual cache, patch sha256 lookup, etc.) ๐งบspack.paths.default_monitor_path=/home/abuild/.spack/reports/monitorโ spack monitor analysis directories ๐spack.paths.default_test_path=/home/abuild/.spack/testโ installation test (spack test) output ๐งชspack.paths.default_user_bootstrap_path=/home/abuild/.spack/bootstrapโ the bootstrap store for bootstrapping clingo and other tools ๐ฅพspack.paths.prefix=/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1โ this file lives in$prefix/lib/spack/spack/__file__๐spack.paths.reports_path=/home/abuild/.spack/reportsโ junit, cdash, etc. reports about builds ๐spack.paths.sbang_script=/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/bin/sbangโ the sbang script in the spack installation ๐spack.paths.spack_root=/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1โ a synonym forprefix๐ฏspack.paths.spack_script=/home/abuild/rpmbuild/BUILD/spack-0.22.1-build/spack-0.22.1/bin/spackโ the spack script itself โญspack.paths.system_config_path=/etc/spackโ system configuration location ๐ฅ๏ธspack.paths.user_config_path=/home/abuild/.spackโ user configuration location ๐งspack.paths.user_repos_cache_path=/home/abuild/.spack/git_reposโ git repositories fetched to compare commits to versions ๐
๐ฏ spack.projections module
spack.projections.get_projection(projections, spec)Gets the projection for a spec out of a projections dict. that's it, that's the whole function. short and sweet. ๐ฌ
๐งฉ spack.provider_index module
Classes and functions to manage providers of virtual dependencies. virtual deps are such a vibe honestly.
๐๏ธ spack.provider_index.ProviderIndex
class spack.provider_index.ProviderIndex(repository: Repo | RepoPath,
specs: List[Spec] | None = None, restrict: bool = False)Bases: _IndexBase
copy()โ returns a deep copy of this index. no sharing, no aliasing, we're independent girlies. ๐ซถstatic from_json(stream, repository)โ constructs a provider index from its JSON representation. Param: stream = stream where to read from the JSON data. ๐ฅmerge(other)โ merges another provider index into this one. Param: other (ProviderIndex) = provider index to be merged. ๐ซremove_provider(pkg_name)โ removes a provider from theProviderIndex. bye babe ๐to_json(stream=None)โ dumps a JSON representation of this object. Param: stream = stream where to dump. ๐คupdate(spec)โ updates the provider index with additional virtual specs. Param: spec = spec potentially providing additional virtual specs. โจ
๐ฅ spack.provider_index.ProviderIndexError
exception spack.provider_index.ProviderIndexError(message, long_message=None)Bases: SpackError
Raised when there's a problem with a ProviderIndex. something went wrong with the index and we're being dramatic about it. ๐ญ
๐ spack.relocate module
๐ฑ spack.relocate.InstallRootStringError
exception spack.relocate.InstallRootStringError(file_path, root_path)Bases: SpackError
(it's an error, bestie. it gets raised when things go wrong with install root strings.) ๐
๐ spack.relocate.fixup_macos_rpath
spack.relocate.fixup_macos_rpath(root, filename)Applies rpath fixups to the given file. Params: root = absolute path to the parent directory; filename = relative path to the library or binary. Returns True if fixups were applied, else False. โ
๐ spack.relocate.fixup_macos_rpaths
spack.relocate.fixup_macos_rpaths(spec)Removes duplicate and nonexistent rpaths. ๐งน
So here's the tea: some autotools packages write their own -rpath entries on top of the ones the Spack compiler wrappers implicitly add. On Linux these duplicate rpaths get eliminated, but on macOS they turn into multiple entries โ which makes it way harder to adjust with install_name_tool -delete_rpath. annoying but we handle it. ๐
๐ spack.relocate.is_binary
spack.relocate.is_binary(filename)Returns true if a file is binary, False otherwise. Param: filename = file to be tested. Returns True or False. is it a binary or is it not, that is the question ๐ฆ
spack.relocate.macho_find_paths(orig_rpaths, deps, idpath, old_layout_root, prefix_to_prefix)
okay so this one figures out where all your mach-o paths are supposed to go ๐ Here's what you feed it:
- original rpaths from the mach-o binaries
- dependency libraries for those mach-o binaries
- the id path of the mach-o libraries
- the old install directory layout root
- a
prefix_to_prefixdictionary that maps prefixes in the old directory layout โ directories in the new directory layout
And what you get back? paths_to_paths, a dictionary mapping all of the old paths to new paths. slay.
spack.relocate.macho_make_paths_normal(orig_path_name, rpaths, deps, idpath)
Returns a dictionary mapping the relativized rpaths โ the original rpaths ๐ฆ This dict is the thing that gets used to replace paths inside mach-o binaries.
Real talk on the mechanics: it replaces @loader_path''@loader_path' with the dirname of the origname path name in rpaths and deps, and idpath gets replaced with the original path name. (yes, that double-quoted @loader_path thing is exactly how it shows up โ no cap, don't @ me ๐)
spack.relocate.macho_make_paths_relative(path_name, old_layout_root, rpaths, deps, idpath)
Mirror image of the last one, bestie โ returns a dictionary mapping the original rpaths โ the relativized rpaths, which again is what gets used to replace paths in mach-o binaries ๐
The deets: old_dir gets replaced with a relative path from the dirname of path_name in rpaths and deps, and idpath gets replaced with @rpath/libname. cute.
spack.relocate.macholib_get_paths(cur_path)
Grab the rpaths, the dependent libraries, and the library id of mach-o objects. That's it, that's the whole vibe check โจ
spack.relocate.make_elf_binaries_relative(new_binaries, orig_binaries, orig_layout_root)
Swaps the original RPATHs in the new binaries so they're relative to the original layout root ๐ธ
Parameters โ gather round:
new_binaries(list) โ the new binaries whose RPATHs are getting made relativeorig_binaries(list) โ the original binariesorig_layout_root(str) โ the path used as a base for making RPATHs relative
spack.relocate.make_link_relative(new_links, orig_links)
Computes the relative target from the original link, then makes the new link relative. chain reaction, but make it filesystem ๐
Parameters:
new_links(list) โ new links to be made relativeorig_links(list) โ original links
spack.relocate.make_macho_binaries_relative(cur_path_names, orig_path_names, old_layout_root)
Replaces old RPATHs with paths relative to old_dir inside binary files. short king of a docstring, honestly ๐
spack.relocate.modify_macho_object(cur_path, rpaths, deps, idpath, paths_to_paths)
This function is how you make machO buildcaches on macOS โ it swaps old paths for new ones using install_name_tool ๐ ๏ธ
Inputs:
- the mach-o binary to be modified
- original rpaths
- original dependency paths
- original id path, if it's a mach-o library
- a dictionary mapping paths in the old install layout โ the new install layout
spack.relocate.modify_object_macholib(cur_path, paths_to_paths)
This is the linux-side bestie ๐ญ โ used when installing machO buildcaches on linux by rewriting mach-o loader commands for dependency library paths of mach-o binaries, plus the id path for mach-o libraries. Rewritting of rpaths is handled by replace_prefix_bin (and yes, "Rewritting" is spelled exactly like that in the original โ we respect the source in this house ๐).
Inputs:
- the mach-o binary to be modified
- a dictionary mapping paths in the old install layout โ the new install layout
spack.relocate.needs_binary_relocation(m_type, m_subtype)
Returns True if the file with the MIME type/subtype you passed in needs binary relocation, False otherwise โ
โ
Parameters:
m_type(str) โ MIME type of the filem_subtype(str) โ MIME subtype of the file
spack.relocate.needs_text_relocation(m_type, m_subtype)
Same energy, different file โ returns True if the file with the given MIME type/subtype needs text relocation, False otherwise ๐
Parameters:
m_type(str) โ MIME type of the filem_subtype(str) โ MIME subtype of the file
spack.relocate.new_relocate_elf_binaries(binaries, prefix_to_prefix)
Takes a list of binaries plus an ordered dictionary of prefix โ prefix mapping, and updates the rpaths accordingly. lowkey my favorite kind of function: input, mapping, done ๐ฅ
spack.relocate.relocate_elf_binaries(binaries, orig_root, new_root, new_prefixes, rel, orig_prefix, new_prefix)
Relocates the binaries you pass in by changing their RPATHs โจ
It uses patchelf to grab the original RPATHs, then replaces them with rpaths from the new directory layout.
New RPATHs come from a dictionary mapping the prefixes in the old directory layout โ the prefixes in the new directory layout, but only if the rpath was in the old layout root. i.e. system paths are not replaced. read that again, bestie, it matters ๐
Parameters:
binaries(list) โ list of binaries that might need relocation, located in the new prefixorig_root(str) โ original root to be substitutednew_root(str) โ new root to be used, only relevant for relative RPATHsnew_prefixes(dict) โ dictionary that maps the original prefixes to where they should be relocatedrel(bool) โTrueif the RPATHs are relative,Falseif they're absoluteorig_prefix(str) โ prefix where the executable was originally locatednew_prefix(str) โ prefix where we want to relocate the executable
spack.relocate.relocate_links(links, prefix_to_prefix)
Relocates links to a new install prefix. that's the whole tweet ๐ฆ
spack.relocate.relocate_macho_binaries(path_names, old_layout_root, new_layout_root, prefix_to_prefix, rel, old_prefix, new_prefix)
Uses the macholib python package to pull the rpaths, depedent libraries and library identity for libraries out of the MachO object (again: "depedent" is verbatim, we do not correct the source ๐). Then it modifies them using replacement paths queried from the dictionary mapping old layout prefixes โ hashes, and the dictionary mapping hashes โ the new layout prefixes. two dicts, one glow-up ๐ซ
spack.relocate.relocate_text(files, prefixes)
Relocates a text file from the original installation prefix โ the new prefix.
Oh and โ relocation also affects the path in Spack's sbang script. yes, that one. it's involved ๐
Parameters:
files(list) โ text files to be relocatedprefixes(OrderedDict) โ string prefixes which need to be changed
spack.relocate.relocate_text_bin(binaries, prefixes)
Replaces null-terminated path strings that are hard-coded into binaries.
โ ๏ธ heads up bestie: the new install prefix must be shorter than the original one. not equal, not longer โ shorter. this is not a drill ๐ญ
Parameters:
binaries(list) โ binaries to be relocatedprefixes(OrderedDict) โ string prefixes which need to be changed.
Raises spack.relocate_text.BinaryTextReplaceError โ when the new path is longer than the old path ๐
spack.relocate.warn_if_link_cant_be_relocated(link, target)
no docstring for this one, bestie โ it's just the signature, standing there, looking iconic ๐
๐ธ spack.relocate_text module ๐ธ
This module contains pure-Python classes and functions for replacing paths inside text files and binaries. pure Python, no compiled nonsense, we love to see it ๐
class spack.relocate_text.BinaryFilePrefixReplacer(prefix_to_prefix, suffix_safety_size=7)
Bases: PrefixReplacer
classmethod binary_text_regex(binary_prefixes, suffix_safety_size=7)
Creates a regex that looks for exact matches of prefixes, and also tries to match a C-string type null terminator in a small lookahead window ๐
Parameters:
binary_prefixes(list) โ list of byte strings of prefixes to matchsuffix_safety_size(int) โ sizeof the lookahed for null-terminated string. (yes, "lookahed". we keep it. it's canon now ๐ )
Returns: compiled regex
classmethod from_strings_or_bytes(prefix_to_prefix: Dict[str | โ bytes, str | bytes], suffix_safety_size: int = 7) -> BinaryFilePrefixReplacer
Creates a BinaryFilePrefixReplacer from an ordered prefix-to-prefix map ๐ฆ
Parameters:
prefix_to_prefix(OrderedDict) โ ordered mapping of prefix to prefix.suffix_safety_size(int) โ number of bytes to retain at the end of a C-string to avoid binary string-aliasing issues.
exception spack.relocate_text.BinaryStringReplacementError(file_path, old_len, new_len)
Bases: SpackError
exception spack.relocate_text.BinaryTextReplaceError(msg)
Bases: SpackError
exception spack.relocate_text.CannotGrowString(old, new)
Bases: BinaryTextReplaceError
exception spack.relocate_text.CannotShrinkCString(old, new, full_old_string)
Bases: BinaryTextReplaceError
class spack.relocate_text.PrefixReplacer(prefix_to_prefix: Dict[bytes, bytes])
Bases: object
The base class for applying a prefix-to-prefix map to a list of binaries or text files ๐ Child classes implement _apply_to_file to do the actual work โ which is different depending on whether you're dealing with binaries or text files, because of course it is ๐ญ
apply(filenames: list)โ returns a list of files that were modified โจapply_to_file(f)apply_to_filename(filename)- property
is_noop: boolโ returns true when the prefix-to-prefix map is mapping everything to the same location (identity) or when there are no prefixes to replace. i.e. nothing to do, we can all go home ๐
class spack.relocate_text.TextFilePrefixReplacer(prefix_to_prefix: Dict[bytes, bytes])
Bases: PrefixReplacer
This class applies prefix-to-prefix mappings for relocation on text files ๐
note that UTF-8 encoding is assumed. don't come at me with your weird encodings, bestie ๐
classmethod from_strings_or_bytes(prefix_to_prefix: Dict[str | โ bytes, str | bytes]) -> TextFilePrefixReplacer
Creates a TextFilePrefixReplacer from an ordered prefix-to-prefix map. same vibe as the binary one, just text-coded ๐ธ
spack.relocate_text.encode_path(p: str | bytes) -> bytes
encode the path, get bytes. tiny function, main character energy ๐
spack.relocate_text.filter_identity_mappings(prefix_to_prefix)
Drops mappings that aren't changed. identity mappings? not on my watch โ๏ธ
spack.relocate_text.utf8_path_to_binary_regex(prefix: str)
Creates a binary regex that matches the input path in utf8 ๐
spack.relocate_text.utf8_paths_to_single_binary_regex(prefixes)
Creates a (binary) regex that matches any input path in utf8. one regex to rule them all ๐
๐ซถ spack.repo module ๐ซถ
exception spack.repo.BadRepoError(message, long_message=None)
Bases: RepoError
Raised when the repo layout is invalid. like, structurally not okay. we've all been there ๐ญ
exception spack.repo.FailedConstructorError(name, exc_type, exc_obj, exc_tb)
Bases: RepoError ๐
Okay so this one gets yeeted when a package's class constructor just... fails. That's it, that's the whole vibe. Construction did not slay. ๐ญ
class spack.repo.FastPackageChecker(packages_path)
Bases: Mapping โจ
Bestie, this is the cache that maps package names to the stats it scooped up on the package.py files that go with them. For each repository there's a cache living at class level, shared among literally all the instances that refer to it โ one cache to rule them all, no cap. And the update of the global cache? Super lazy, happens during instance initialization. Iconic low-effort behavior honestly. ๐ฆ
invalidate()โ Regenerate the cache for this checker. Fresh start, we love a reset. ๐last_mtime()modified_since(since: float) -> List[str]
class spack.repo.GitExe
Bases: object
That's it. That's the whole entry. No description, just vibes. ๐ซถ
class spack.repo.Indexer(repository)
Bases: object
An adaptor for indexes that need to be generated whenever repos get updated. Think of it as the little helper that keeps everything in sync. ๐
create()needs_update(pkg)โ Whether an update is needed when the package file hasn't changed.- Returns:
Trueif this package needs its index updated,Falseotherwise. - Return type:
(bool)
- Returns:
So here's the tea: we already automatically update indexes when package files change, but other files (like patches, hello ๐) may change underneath the package file. This method is your girl for checking those extra package-specific files whenever they're loaded, so it can tell the RepoIndex to update the index just for that one package. ๐
abstractmethod read(stream)โ Read this index from a provided file object.abstractmethod update(pkg_fullname)โ Update the index in memory with information about a package.abstractmethod write(stream)โ Write the index to a file object.
exception spack.repo.InvalidNamespaceError(message, long_message=None)
Bases: RepoError
Raised when an invalid namespace shows up and we're like... girl, what. ๐
class spack.repo.MockRepositoryBuilder(root_directory, namespace=None)
Bases: object
Build a mock repository in a directory. Your own little sandbox, adorable. ๐ธ
add_package(name, dependencies=None)โ Create a mock package in the repository, using a Jinja2 template.- Parameters:
name(str) โ name of the new packagedependencies(list) โ list of("dep_spec", "dep_type", "condition")tuples. Both"dep_type"and"condition"can default toNone, in which casespack.dependency.default_deptypeandspack.spec.Spec()are used. Defaults, we love to see it. โจ
- Parameters:
recipe_filename(name)remove(name)
spack.repo.NOT_PROVIDED = <object object>
A guaranteed unused default value for some functions. It's giving "placeholder that minds its own business". ๐
exception spack.repo.NoRepoConfiguredError(message, long_message=None)
Bases: RepoError
Raised when there are, and I cannot stress this enough, no repositories configured. Empty. Void. Nothing. ๐ญ
spack.repo.PATH: RepoPath | Singleton = <spack.repo.RepoPath object>
The singleton repo path instance. One of her, only her. ๐
class spack.repo.PatchIndexer(repository)
Bases: Indexer
Lifecycle methods for the patch cache. ๐ง
needs_update()โ Whether an update is needed when the package file hasn't changed.- Returns:
Trueif this package needs its index updated,Falseotherwise. - Return type:
(bool)
- Returns:
Same deal as before, bestie: indexes update automatically when package files change, but other files (patches, we see you) may change underneath the package file. This method checks those extra package-specific files whenever they're loaded so the RepoIndex knows to update the index just for that package. ๐
read(stream)โ Read this index from a provided file object.update(pkg_fullname)โ Update the index in memory with information about a package.write(stream)โ Write the index to a file object.
class spack.repo.ProviderIndexer(repository)
Bases: Indexer
Lifecycle methods for virtual package providers. Look at her go. ๐ฅ
read(stream)โ Read this index from a provided file object.update(pkg_fullname)โ Update the index in memory with information about a package.write(stream)โ Write the index to a file object.
spack.repo.ROOT_PYTHON_NAMESPACE = 'spack.pkg'
Package modules are imported as spack.pkg.<repo-namespace>.<pkg-name>. That's the naming convention, memorize it, there will be a quiz. โจ
class spack.repo.Repo(root, cache=None)
Bases: object
Class representing a package repository in the filesystem. Main character energy, honestly. ๐
Each package repository must have a top-level configuration file called repo.yaml.
Currently, repo.yaml this must define:
namespace:โ A Python namespace where the repository's packages should live.
Let's get into the methods, because there are so many and they all ate:
all_package_classes()โ Iterator over all package classes in the repository.
Use this with care, because loading packages is slow. Like, genuinely slow. Don't say I didn't warn you. ๐
all_package_names(include_virtuals=False)โ Returns a sorted list of all package names in theRepo. Sorted, because we're organized. ๐all_package_paths()dirname_for_package_name(pkg_name)โ Get the directory name for a particular package. This is the directory that contains itspackage.pyfile.dump_provenance(spec, path)โ Dump provenance information for a spec to a particular path.
This dumps the package file and any associated patch files. Raises UnknownPackageError if not found. So like, if it's not there, expect that error, no surprises. ๐
exists(pkg_name)โ Whether a package with the supplied name exists.extensions_for(extendee_spec)filename_for_package_name(pkg_name)โ Get the filename for the module we should load for a particular package. Packages for aRepolive in$root/<package_name>/package.py
This will return a proper package.py path even if the package doesn't exist yet, so callers will need to ensure the package exists before importing. Yes, you read that right โ it's giving you a path to something that might not be real. Handle it. ๐
get(spec)โ Returns the package associated with the supplied spec.get_pkg_class(pkg_name)โ Get the class for the package out of its module.
First loads (or fetches from cache) a module for the package. Then extracts the package class from the module according to Spack's naming convention. Two steps, both essential. ๐ฆ
property indexโ Construct the index for this repo lazily. Lazy queen. ๐is_prefix(fullname)โTrueiffullnameis a prefix of thisRepo's namespace.is_virtual(pkg_name)โ ReturnTrueif the package with this name is virtual,Falseotherwise.
This function uses the provider index. If you're calling from a code block that's used to construct the provider index, use the is_virtual_safe function instead. Important!! โ ๏ธ
- Parameters:
pkg_name(str) โ name of the package we want to check
is_virtual_safe(pkg_name)โ ReturnTrueif the package with this name is virtual,Falseotherwise.
This function doesn't use the provider index. That's the whole point of the "safe". ๐ซถ
- Parameters:
pkg_name(str) โ name of the package we want to check
last_mtime()โ Time a package file in this repo was last updated.package_path(name)โ Get path topackage.pyfile for this repo.packages_with_tags(*tags: str) -> Set[str]partition_package_name(pkg_name: str) -> Tuple[str, str]property patch_indexโ Index of patches and packages they're defined on.property provider_indexโ A provider index with names specific to this repo.providers_for(vpkg_spec)purge()โ Clear entire package instance cache. Wipe it. Gone. Clean slate. ๐real_name(import_name)โ Allow users to import Spack packages using Python identifiers.
A python identifier might map to many different Spack package names due to hyphen/underscore ambiguity. Okay so like: Easy example:num3proxy -> 3proxyAmbiguous:foo_bar -> foo_bar, foo-barMore ambiguous:foo_bar_baz -> foo_bar_baz, foo-bar-baz, foo_bar-baz, foo-bar_bazThe ambiguity is real, bestie. ๐ญ
property tag_indexโ Index of tags and which packages they're defined on.
exception spack.repo.RepoError(message, long_message=None)
Bases: SpackError
Superclass for repository-related errors. The mother of all repo problems. ๐
class spack.repo.RepoIndex(package_checker: FastPackageChecker, namespace: str, cache: FileCache)
Bases: object
Container class that manages a set of Indexers for a Repo. โจ
This class is responsible for checking packages in a repository for updates (using FastPackageChecker) and for regenerating indexes when they're needed.
Indexers should be added to the RepoIndex using add_indexer(name, indexer), and they should support the interface defined by Indexer, so that the RepoIndex can read, generate, and update stored indices. Teamwork makes the dream work. ๐ซถ
Generated indexes are accessed by name via __getitem__().
add_indexer(name: str, indexer: Indexer)โ Add an indexer to the repo index.- Parameters:
nameโ name of this indexerindexerโ object implementing theIndexerinterface
- Parameters:
class spack.repo.RepoLoader(fullname, repo, package_name)
Bases: _PrependFileLoader
Loads a Python module associated with a package in a specific repository. Doing the import so you don't have to. ๐
class spack.repo.RepoPath(*repos, **kwargs)
Bases: object
A RepoPath is a list of repos that function as one. Many repos, one energy. ๐
โจ
It functions exactly like a Repo, except it operates on the combined results of all the Repos in its list instead of on a single package repository. group project but make it functional ๐
Parameters
repos(list) โ list ofRepoobjects, or paths, to put in thisRepoPath
๐ธ the methods, bestie ๐ธ
all_package_classes()โ no docstring in the source, but she's there and she's real โจall_package_names(include_virtuals=False)โ same deal, documented purely by existing ๐all_package_paths()โ also just vibing with no description ๐ซถdirname_for_package_name(pkg_name)โ no notes, only vibesdump_provenance(spec, path)โ dump provenance information for a spec to a particular path. This dumps the package file and any associated patch files. RaisesUnknownPackageErrorif it's not found. so it's the receipts-dropping method, okurrr ๐ฅexists(pkg_name)โ whether a package with the given name exists in the path's repos. โ ๏ธ note that virtual packages do not "exist" โ they're giving ghost energy, not real-package energy ๐extensions_for(extendee_spec)โ no docstring given, she simply isfilename_for_package_name(pkg_name)โ again, no notes ๐ฅfirst_repo()โ get the first repo in precedence order. main character energy, first in line ๐get(spec)โ returns the package associated with the supplied spec.get_pkg_class(pkg_name)โ find a class for the spec's package and return the class object.get_repo(namespace, default=<object object>)โ get a repository by namespace.
Parameters:
namespaceโ look up this namespace in theRepoPath, and return it if found.
Optional arguments:
defaultโ ifdefaultis provided, return it when the namespace isn't found. If not, raise anUnknownNamespaceError. no default, no mercy ๐ญ
is_virtual(pkg_name)โ returnTrueif the package with this name is virtual,Falseotherwise.
This function uses the provider index. If you're calling from a code block that is used to construct the provider index, use the is_virtual_safe function instead. โ ๏ธ like, genuinely โ don't cause that recursion, bestie.
Parameters:
pkg_name(str) โ name of the package we want to check
is_virtual_safe(pkg_name)โ returnTrueif the package with this name is virtual,Falseotherwise. This one doesn't use the provider index, so it's the safe-to-call-from-index-construction version ๐ง
Parameters:
pkg_name(str) โ name of the package we want to check
last_mtime()โ time a package file in this repo was last updated.package_path(name)โ get path topackage.pyfile for this repo.packages_with_tags(*tags: str, full: bool = False) -> Set[str]โ returns a set of packages matching any of the tags in the input.
Parameters:
fullโ ifTrue, the package names in the output are fully-qualified
property patch_indexโ mergedPatchIndexfrom allRepos in theRepoPath.property provider_indexโ mergedProviderIndexfrom allRepos in theRepoPath.providers_for(vpkg_spec)โ no docstring, only presence ๐put_first(repo)โ addrepofirst in the search path.put_last(repo)โ addrepolast in the search path.remove(repo)โ remove a repo from the search path.repo_for_pkg(spec)โ given a spec, get the repository for its package.property tag_indexโ mergedTagIndexfrom allRepos in theRepoPath.
class spack.repo.ReposFinder
Bases: object
A MetaPathFinder class that loads a Python module corresponding to a Spack package. ๐ฆ
Return a loader based on the inspection of the current global repository list.
compute_loader(fullname)โ no extra notesfind_spec(fullname, python_path, target=None)โ no extra notes
class spack.repo.SpackNamespace(namespace)
Bases: ModuleType
Allow lazy loading of modules. she'll get there when she gets there ๐
class spack.repo.SpackNamespaceLoader
Bases: object
create_module(spec)exec_module(module)
class spack.repo.TagIndexer(repository)
Bases: Indexer
Lifecycle methods for a TagIndex on a Repo. ๐
read(stream)โ read this index from a provided file object.update(pkg_fullname)โ update the index in memory with information about a package.write(stream)โ write the index to a file object.
exceptions ๐
exception spack.repo.UnknownEntityError(message, long_message=None)โ Bases:RepoError. Raised when we encounter a package Spack doesn't have.exception spack.repo.UnknownNamespaceError(namespace, name=None)โ Bases:UnknownEntityError. Raised when we encounter an unknown namespace.exception spack.repo.UnknownPackageError(name, repo=None)โ Bases:UnknownEntityError. Raised when we encounter a package Spack doesn't have.
module-level functions in spack.repo ๐
spack.repo.add_package_to_git_stage(packages)โ add a package to the git stage withgit add.spack.repo.all_package_names(include_virtuals=False)โ convenience wrapper aroundspack.repo.all_package_names(). yes, the doc really wraps around itself like that ๐ซspack.repo.autospec(function)โ decorator that automatically converts the first argument of a function to aSpec.spack.repo.create(configuration)โ create aRepoPathfrom a configuration object.
Parameters:
configuration(spack.config.Configuration) โ configuration object
spack.repo.create_or_construct(path, namespace=None)โ create a repository, or just return aRepoif it already exists. why build when you can reuse, periodt โจspack.repo.create_repo(root, namespace=None, subdir='packages')โ create a new repository inrootwith the specified namespace.
If the namespace isn't provided, use the basename of root. Returns the canonicalized path and namespace of the created repository. ๐
spack.repo.diff_packages(rev1, rev2)โ compute package lists for the two revisions and return a tuple containing all the packages inrev1but not inrev2, and all the packages inrev2but not inrev1.spack.repo.get_all_package_diffs(type, rev1='HEAD^1', rev2='HEAD')โ show packages changed, added, or removed (or any combination of those) since a commit.
Parameters:
type(str) โ string containing one or more of'A','R','C'rev1(str) โ revision to compare against, default is'HEAD^'rev2(str) โ revision to compare torev1, default is'HEAD'
Returns: a set containing names of affected packages.
๐ ok so heads up: the signature literally saysrev1='HEAD^1'and the docs say'HEAD^'. I'm not fixing it, I'm just reporting it. copy it exactly as written bestie, no cap.
spack.repo.is_package_file(filename)โ determine whether we are in a package file from a repo.spack.repo.list_packages(rev)โ list all packages associated with the given revision.spack.repo.namespace_from_fullname(fullname)โ return the repository namespace only for the full module name.
For instance:
namespace_from_fullname('spack.pkg.builtin.hdf5') == 'builtin'Parameters:
fullname(str) โ full name for the Python module
spack.repo.packages_path()โ get the test repo if it is active, otherwise the builtin repo.spack.repo.partition_package_name(pkg_name: str) -> Tuple[str, str]โ given a package name that might be fully-qualified, returns the namespace part (if present) and the unqualified package name.
If the package name is unqualified, the namespace is an empty string.
Parameters:
pkg_nameโ a package name, either unqualified like"llvl", or fully-qualified like"builtin.llvm"
โญ yes, "llvl" is spelled that way in the original. leaving it exactly as-is, iykyk.
spack.repo.python_package_for_repo(namespace)โ returns the full namespace of a repository, given its relative one.
For instance:
python_package_for_repo('builtin') == 'spack.pkg.builtin'Parameters:
namespace(str) โ repo namespace
spack.repo.use_repositories(*paths_and_repos, **kwargs)โ use the repositories passed as arguments within the context manager. ๐ซถ
Parameters:
*paths_and_reposโ paths to the repositories to be used, or already constructedRepoobjectsoverride(bool) โ ifTrueuse only the repositories passed as input; ifFalseadd them to the top of the list of current repositories.
Returns: corresponding RepoPath object.
๐ธ spack.report module ๐ธ
Tools to produce reports of spec installations. receipts, but make it structured ๐
class spack.report.BuildInfoCollector(specs: List[Spec])
Bases: InfoCollector
Collect information for the PackageInstaller._install_task method. ๐ง
Parameters:
specsโ specs whose install information will be recorded
methods
extract_package_from_signature(instance, *args, **kwargs)โ return the package instance, given the signature of the wrapped function.fetch_log(pkg)โ return the stdout log associated with the function being monitored.
Parameters:
pkgโ package under consideration
init_spec_record(input_spec, record)โ add additional entries to a spec record when entering the collection context.on_success(pkg, kwargs, package_record)โ add additional properties on function call success. ๐
class spack.report.InfoCollector(wrap_class: Type, do_fn: str, specs: List[Spec])
Bases: object
Base class for context manager objects that collect information during the execution of certain package functions. ๐
The data collected is available through the specs attribute once exited, and it's organized as a list where each item represents the installation of one spec. ๐ฆ
do_fn: strโ action to be reported onextract_package_from_signature(instance, *args, **kwargs)โ return the package instance, given the signature of the wrapped function.fetch_log(pkg: PackageBase) -> strโ return the stdout log associated with the function being monitored.
Parameters:
pkgโ package under consideration
init_spec_record(input_spec: Spec, record)โ adds extra lil' entries to a spec record when you're entering the collection context ๐input_specs: List[Spec]โ the specs that are gonna get acted onon_success(pkg: PackageBase, kwargs, package_record)โ adds more properties once the function call succeeds, slayspecs: List[Dict[str, Any]]โ this is literally where we stash the data that ends up in our reportwrap_class: Typeโ the class you're wrapping a function for
๐
class spack.report.TestInfoCollector(specs: List[Spec], record_directory: str)
Bases: InfoCollector
Collects the tea for the PackageBase.do_test method ๐ซถ
Parameters
- specs โ specs whose install information will be recorded
- record_directory โ record directory for test log paths
Members
dir: strextract_package_from_signature(instance, *args, **kwargs)โ returns the package instance, given the signature of the wrapped functionfetch_log(pkg: PackageBase)โ returns the stdout log tied to the function being monitored- Parameters: pkg โ package under consideration
on_success(pkg, kwargs, package_record)โ adds extra properties on function call success โจ
๐ธ spack.report.build_context_manager(reporter: Reporter, filename: str, specs: List[Spec])
Decorates a package so a report gets generated right after the installation function runs, no cap.
Parameters
- reporter โ object that generates the report
- filename โ filename for the report
- specs โ specs that need reporting
๐ spack.report.test_context_manager(reporter: Reporter, filename: str, specs: List[Spec], raw_logs_dir: str)
Same vibe, but it decorates a package to generate a report after the test function is executed ๐
Parameters
- reporter โ object that generates the report
- filename โ filename for the report
- specs โ specs that need reporting
- raw_logs_dir โ record directory for test log paths
๐ฆ spack.resource module
Describes an optional resource you need for a build. Typically it's a bunch of sources that can be built in-tree inside another package to unlock optional features ๐
class spack.resource.Resource(name, fetcher, destination, placement)
Bases: object
Represents an optional resource that a package fetches. It's literally just an aggregation of a name, a fetcher, a destination and a placement โ iconic little bundle ๐
๐ง spack.rewiring module
exception spack.rewiring.PackageNotInstalledError(spliced_spec, build_spec, dep)โ Bases:RewireError. Raised when thebuild_specfor a splice was not installed ๐ญexception spack.rewiring.RewireError(message, long_msg=None)โ Bases:SpackError. Raised when something goes sideways with rewiring ๐spack.rewiring.rewire(spliced_spec)โ given a spliced spec, this function conducts all the rewiring on every node in that spec's DAG, fr frspack.rewiring.rewire_node(spec, explicit)โ rewires a single node, caring only about references to its subgraph. Binaries, text, and links all get changed in accordance with the splice, and then the resulting package is 'installed.' ๐ซถ
๐ spack.spec module
Spack gives you very fine-grained control over how packages are installed, built, and configured. To make that easy, it's got its own syntax for declaring a dependence โ and a descriptor of a particular package configuration is called a "spec" โจ
The syntax looks like this:
$ spack install mpileaks ^openmpi @1.2:1.4 +debug %intel @12.1 target=zen
0 1 2 3 4 5 6The first part is the command, spack install. Everything after that is a spec for a particular installation of the mpileaks package. Here's the breakdown, bestie ๐
- The package to install
- A dependency of the package, prefixed by
^ - A version descriptor for the package. This can be a specific version, like
"1.2", or a range of versions, e.g."1.2:1.4". And if multiple specific versions or multiple ranges are acceptable, you can separate them with commas โ like, if a package only builds with versions 1.0, 1.2-1.4, and 1.6-1.8 of mavpich, you could say:
depends_on("mvapich@1.0,1.2:1.4,1.6:1.8")- A compile-time variant of the package. If you need
openmpibuilt in debug mode for your package to work, you can require it by adding+debugto theopenmpispec when you depend on it. If you do NOT want the debug option enabled, swap that for-debug. And if you want the variant to be propagated through all your package's dependencies, use"++"for enabling and"--"or"~~"for disabling ๐ฅ - The name of the compiler to build with
- The versions of the compiler to build with. Heads up: the identifier for a compiler version is the same
@used for a package version. A version list denoted by@is associated with the compiler only if it comes immediately after the compiler name โ otherwise it gets associated with the current package spec. iykyk ๐ - The architecture to build with. You need this on machines where cross-compilation is required
๐ Exceptions, in all their dramatic glory
exception spack.spec.AmbiguousHashError(msg, *specs)โ Bases:SpecErrorexception spack.spec.ArchitecturePropagationError(message, long_message=None)โ Bases:SpecError. Raised when the double equal symbols are used to assign the spec's architecture ๐exception spack.spec.DuplicateArchitectureError(message, long_message=None)โ Bases:SpecError. Raised when the same architecture shows up in a spec twiceexception spack.spec.DuplicateCompilerSpecError(message, long_message=None)โ Bases:SpecError. Raised when the same compiler shows up in a spec twiceexception spack.spec.DuplicateDependencyError(message, long_message=None)โ Bases:SpecError. Raised when the same dependency shows up in a spec twiceexception spack.spec.InconsistentSpecError(message, long_message=None)โ Bases:SpecError. Raised when two nodes in the same spec DAG have inconsistent constraints ๐exception spack.spec.InvalidDependencyError(pkg, deps)โ Bases:SpecError. Raised when a dependency in a spec isn't actually a dependency of the packageexception spack.spec.InvalidHashError(spec, hash)โ Bases:SpecErrorexception spack.spec.MultipleProviderError(vpkg, providers)โ Bases:SpecError. Raised when there is no package that provides a particular virtual dependencyexception spack.spec.NoProviderError(vpkg)โ Bases:SpecError. Raised when there is no package that provides a particular virtual dependency
๐ class spack.spec.CompilerSpec(*args)
Bases: object
The CompilerSpec field represents the compiler (or range of compiler versions) that a package should be built with. CompilerSpecs have a name and a version list ๐ซถ
property concreteโ aCompilerSpecis concrete if its versions are concrete and there's an available compiler with the right versionconstrain(other: CompilerSpec) -> boolโ intersects self's versions with other. Returns whether theCompilerSpecchangedcopy()property display_strโ equivalent to{compiler.name}{@compiler.version}for Specs, without the extra@=for readabilitystatic from_dict(d)intersects(other: CompilerSpec) -> boolโ returnsTrueif all concrete specs matching self also match other, otherwiseFalse. For compiler specs this means the compiler name has to be the same for self and other, and the version ranges should intersect- Parameters: other โ spec to be satisfied
namesatisfies(other: CompilerSpec) -> boolโ returnsTrueif all concrete specs matching self also match other, otherwiseFalse. For compiler specs this means the compiler name has to be the same for self and other, and the version range of self is a subset of that of other- Parameters: other โ spec to be satisfied
to_dict()property versionversions
๐ class spack.spec.Spec(spec_like=None, normal=False, concrete=False, external_path=None, external_modules=None)
Bases: object
abstract_hash = Noneadd_dependency_edge(dependency_spec: Spec, *, depflag: int, virtuals: Tuple[str, ...])โ adds a dependency edge to this spec- Parameters:
- dependency_spec โ spec of the dependency
- deptypes โ dependency types for this edge
- virtuals โ virtuals provided by this edge
- Parameters:
property anonymousattach_git_version_lookup()property build_speccformat(*args, **kwargs)โ same asformat, but color defaults toautoinstead ofFalseclear_cached_hashes(ignore=())โ clears all cached hashes in a Spec while preserving other propertiesclear_dependencies()โ trims the dependencies of this specclear_edges()โ trims the dependencies and dependents of this specproperty colored_strcommon_dependencies(other)โ returns the names of dependencies that self and other have in commonproperty concreteโ a spec is concrete if it describes a single build of a package. More formally, it's concrete ifconcretize()has been called on it and it's been marked_concrete. Concrete specs either can be or have been built. All constraints are resolved, optional dependencies have been added or removed, a compiler has been chosen, and all variants have values ๐concretize(tests=False)โ turns the current spec into a concrete one, like giving it a full-on makeover with actual versions and everything locked in ๐
Params:
tests(bool or list) โ ifFalse, just ignore 'test' dependencies, no thoughts head empty; if it's a list of names, turn 'test' dependencies on for the packages in that list; ifTrue, turn 'test' dependencies on for all the packages. periodt.
concretized(tests=False)โ the non-destructive version ofconcretize(). ๐ซถ Like, it won't mess with your original spec at all.
First it clones, then hands you back a concrete version of this package without modifying this package. Bestie behavior fr fr.
Params:
tests(bool or list) โ ifFalse, disregard 'test' dependencies; if a list of names, activate them for the packages in the list; ifTrue, activate 'test' dependencies for all packages.
constrain(other, deps=True)โ intersects self with other in-place (yes, in-place, like it's editing the doc directly ๐ญ). ReturnsTrueif self changed,Falseif not.
Params:
otherโ the constraint being added to selfdepsโ ifFalse, constrain only the root node; otherwise constrain dependencies too.
Raises spack.error.UnsatisfiableSpecError โ when self literally cannot be constrained. it's giving impossible ๐
constrained(other, deps=True)โ same vibe but polite: returns a constrained copy, leaving this spec untouched. โจcopy(deps: bool | str | List[str] | Tuple[str, ...] | int = True, **kwargs)โ makes a copy of this spec. iconic.
Params:
depsโ defaults toTrue. If it's a boolean, it controls whether dependencies get copied (copied ifTrue). If you pass aDepTypesorDepFlag, only the matching dependencies are copied.kwargsโ extra arguments for internal use (passed straight to_dup), iykyk.
Returns: a copy of this spec.
Examples โ deep copy with dependencies:
spec.copy()
spec.copy(deps=True)Shallow copy (no dependencies):
spec.copy(deps=False)Only build and run dependencies: deps=('build', 'run')
- property
cshort_specโ returns an auto-colorized version ofself.short_spec. cute AND functional ๐ dag_hash(length=None)โ this is Spack's default hash, the one used to identify installations. main character energy ๐
Same as the full hash (includes the package hash plus build/link/run deps). Tells us when package files and any dependencies have changed.
NOTE: versions of Spack prior to 0.18 only included link and run deps. so like, ancient history bestie ๐ฆ
dag_hash_bit_prefix(bits)โ grab the first <bits> bits of the DAG hash as an integer type. numbers slay.static default_arch()โ returns an anonymous spec for the default architecture. no name, no problem ๐ถ๏ธdependencies(name=None, deptype: str | List[str] | Tuple[str, ...] | int = 15) -> List[Spec]โ returns a list of direct dependencies (the nodes in the DAG).
Params:
name(str) โ filter dependencies by package namedeptypeโ allowed dependency types
dependents(name=None, deptype: str | List[str] | Tuple[str, ...] | int = 15) -> List[Spec]โ returns a list of direct dependents (also nodes in the DAG, just the other direction ๐).
Params:
name(str) โ filter dependents by package namedeptypeโ allowed dependency types
detach(deptype='all')โ removes any reference that dependencies have of this node. like unfollowing but for a DAG ๐
Params:
deptype(str or tuple) โ dependency types tracked by the current spec
direct_dep_difference(other)โ returns the dependencies in self that aren't in other. it's giving "what's mine that isn't yours" ๐- property
edge_attributes: strโ helper method to print edge attributes in spec literals. lowkey just a lil helper ๐ edges_from_dependents(name=None, depflag: int = 15)โ returns a list of edges connecting this node in the DAG to its parents.
Params:
name(str) โ filter dependents by package namedepflagโ allowed dependency types
edges_to_dependencies(name=None, depflag: int = 15)โ returns a list of edges connecting this node in the DAG to its children.
Params:
name(str) โ filter dependencies by package namedepflagโ allowed dependency types
static ensure_external_path_if_external(external_spec)โ no extra description in the source, bestie โ it's just vibes and a signature โจstatic ensure_no_deprecated(root)โ raises if a deprecated spec is anywhere in the dag. ๐จ
Params:
root(Spec) โ root spec to be analyzed
Raises SpecDeprecatedError โ if any deprecated spec is found. not me catching the deprecated ones ๐
static ensure_valid_variants(spec)โ makes sure the variants attached to a spec are actually valid. quality control, we love to see it.
Params:
spec(Spec) โ spec to be analyzed
Raises spack.variant.UnknownVariantError โ on the first unknown variant found. it stops at the first one, no cap.
eq_dag(other, deptypes=True, vs=None, vo=None)โTrueif the full dependency DAGs of the specs are equal. full comparison, not just the surface-level vibe check.eq_node(other)โ equality with another spec, not including dependencies. just the node itself, nothing attached ๐- property
externalโ (property, no description in the source โ it's mysterious like that ๐ถ๏ธ) - property
external_pathโ (also a property, also mysterious ๐) static extract_json_from_clearsig(data)โ signature only in the source, no description. iconic enigma behavior.flat_dependencies(disconnect: bool = False)โ builds aDependencyMapof all of this spec's dependencies, with their constraints merged. ๐งฉ
Params:
disconnectโ ifTrue, disconnect all dependents and dependencies among nodes in this spec's DAG.
format(format_string: str = '{name}{@versions}{%compiler.name}{@compiler.versions}{compiler_flags}{variants}{arch=architecture}{/abstract_hash}', color: bool | None = False) -> strโ prints out particular pieces of a spec, depending on what's in the format string. this one is a whole personality ๐ธ
Using the {attribute} syntax, any field of the spec can be selected. And those attributes can be recursive โ like, s.format({compiler.version}) will print the version of the compiler. nesting! we love recursion!
Commonly used attributes of the Spec for format strings include:
name
version
compiler
compiler.name
compiler.version
compiler_flags
variants
architecture
architecture.platform
architecture.os
architecture.target
prefixSome additional special-case properties can be added:
hash[:len] The DAG hash with optional length argument
spack_root The spack root directory
spack_install The spack install directoryThe ^ sigil can be used to access dependencies by name. So s.format({^mpi.name}) prints the name of the MPI implementation in the spec. ๐ซถ
The @, %, arch=, and / sigils can be used to include the sigil with the printed string. These sigils may only be used with the appropriate attributes, listed below:
@ ``{@version}``, ``{@compiler.version}``
% ``{%compiler}``, ``{%compiler.name}``
arch= ``{arch=architecture}``
/ ``{/hash}``, ``{/hash:7}``, etcThe @ sigil may also be used for any other property named version. Sigils printed with the attribute string are only printed if the attribute string is non-empty, and are colored according to the color of the attribute.
Sigils are not used for printing variants. Variants listed by name naturally print with their sigil. For example, spec.format('{variants.debug}') would print either +debug or ~debug depending on the name of the variant. Non-boolean variants print as name=value. To print variant names or values independently, use spec.format('{variants.<name>.name}') or spec.format('{variants.<name>.value}'). ๐
Spec format strings use \ as the escape character. Use \{ and \} for literal braces, and \\ for the literal \ character. escape responsibly bestie ๐ญ
Params:
format_stringโ string containing the format to be expandedcolorโTruefor a colorized result;Falsefor no color;Nonefor auto color.
format_path(format_string: str, _path_ctor: Callable[[Any], PurePath] | None = None) -> strโ given aformat_stringthat's meant to be a path, this generates a string likeSpec.formatwould, but it eliminates the extra path separators introduced by formatting Spec properties. ๐ค๏ธ
Path separators you explicitly add to the string are preserved, so for example "{name}/{version}" would generate a directory based on the Spec's name, and a subdirectory based on its version. This function guarantees though that the resulting string would only have two directories โ i.e. that if under normal circumstances str(Spec.version) would contain a path separator, it would not in this case. so tidy. so clean. ๐งผ
static from_detection(spec_str, extra_attributes=None)โ constructs a spec from a spec string determined during external detection, then attaches extra attributes to it. ๐
Params:
spec_str(str) โ spec stringextra_attributes(dict) โ dictionary containing extra attributes
Returns: external spec โจ
Return type: spack.spec.Spec โ yeah bestie, that's the tail of the last one, moving on ๐
โจ static from_dict(data) โจ
Constructs a spec from JSON or YAML, because sometimes your spec just wants to come back home from a little serialization vacation ๐ด
Parameters:
dataโ a nesteddict/listdata structure read from YAML or JSON, fr fr.
๐ธ static from_json(stream) ๐ธ
Constructs a spec from JSON. Just JSON things โจ
Parameters:
streamโ a string or a file object to read from.
๐ฆ static from_literal(spec_dict, normal=True) ๐ฆ
Builds a Spec from a dictionary containing the spec literal. Okay so this one is lowkey the main character of the constructors ๐
The dictionary must have a single top level key representing the root, plus as many secondary level keys as you need in the spec.
Those keys can be a string, or a Spec, or a tuple containing the Spec and the dependency types. Versatile icon ๐
Parameters:
spec_dict(dict) โ the dictionary containing the spec literalnormal(bool) โ ifTrue, the same key appearing at different levels of thespec_dictwill map to the same object in memory. No cap, that matters.
Examples:
A simple spec foo with no dependencies:
{'foo': None}A spec foo with a (build, link) dependency bar:
{'foo':
{'bar:build,link': None}}A spec with a diamond dependency and various build types:
{'dt-diamond': {
'dt-diamond-left:build,link': {
'dt-diamond-bottom:build': None
},
'dt-diamond-right:build,link': {
'dt-diamond-bottom:build,link,run': None
}
}}The same spec but with a double copy of dt-diamond-bottom and no diamond structure at all:
{'dt-diamond': {
'dt-diamond-left:build,link': {
'dt-diamond-bottom:build': None
},
'dt-diamond-right:build,link': {
'dt-diamond-bottom:build,link,run': None
}
}, normal=False}And constructing a spec using a Spec object as the key, which is genuinely so iconic:
mpich = Spec('mpich')
libelf = Spec('libelf@1.8.11')
expected_normalized = Spec.from_literal({
'mpileaks': {
'callpath': {
'dyninst': {
'libdwarf': {libelf: None},
libelf: None
},
mpich: None
},
mpich: None
},
})๐ static from_signed_json(stream) ๐
Constructs a spec from a clearsigned json spec file. Signature verified, trust issues resolved ๐
Parameters:
streamโ a string or file object to read from.
โญ static from_specfile(path) โญ
Constructs a spec from a JSON or YAML spec file path. Point it at the file, it does the rest ๐ซถ
๐ static from_yaml(stream) ๐
Constructs a spec from YAML, for all the indentation girlies โจ
Parameters:
streamโ a string or file object to read from.
๐ property fullname ๐
It's a property, it's called fullname, that's the whole vibe ๐
๐ง index(deptype='all') ๐ง
Returns a dictionary that points to all the dependencies in this spec. A little directory of everyone this spec is linked with, iykyk ๐
๐ฅ static inject_patches_variant(root) ๐ฅ
This one's here, it's static, it takes a root. The vibes are immaculate, the description is minimal ๐ซถ
๐ install_status() ๐
Helper for tree to print DB install status. Just a lil helper doing its best ๐ญ
๐ property installed ๐
Installation status of a package.
Returns: True if the package has been installed, False otherwise. Periodt.
๐ธ property installed_upstream ๐ธ
Whether the spec is installed in an upstream repository.
Returns: True if the package is installed in an upstream, False otherwise.
๐ intersects(other: str | Spec, deps: bool = True) -> bool ๐
Returns True if there exists at least one concrete spec that matches both self and other, otherwise False.
This operation is commutative, bestie โ and if two specs intersect, it means one can constrain the other. They're compatible, they're giving mutual constraint energy ๐
Parameters:
otherโ spec to be checked for compatibilitydepsโ ifTrue, check compatibility of dependency nodes too; ifFalse, only check the root
๐ฆ property is_develop ๐ฆ
Returns whether the Spec represents a user-developed package in a Spack Environment (i.e. using spack develop). Dev mode: activated ๐ปโจ
๐ lookup_hash() ๐
Given a spec with an abstract hash, returns a copy of the spec with all properties and dependencies by looking up the hash in the environment, store, or finally, binary caches. This is non-destructive โ your original spec stays unbothered, as it should ๐ซถ
โญ node_dict_with_hashes(hash=<spack.hash_types.SpecHashDescriptor object>) โญ
Returns a node_dict of this spec with the dag hash added. If this spec is concrete, the full hash is added as well. And if 'build' is in the hash_type, the build hash is also added. Hashes on hashes on hashes ๐ฅ
๐ normalize(force=False, tests=False, user_spec_deps=None, disconnect=True) ๐
Okay so when specs are parsed, any dependencies specified are hanging off the root, and ONLY the ones that were explicitly provided are there. Normalization turns a partial flat spec into a DAG, where:
- Known dependencies of the root package are in the DAG.
- Each node's dependencies dict only contains its known direct deps.
- There is only ONE unique spec for each package in the DAG.
- This includes virtual packages. If there's a non-virtual package that provides a virtual package that is in the spec, then we replace the virtual package with the non-virtual one. Swapped in, slay ๐
TODO: normalize should probably implement some form of cycle detection, to ensure that the spec is actually a DAG. (Not me side-eyeing the missing cycle check ๐ญ)
๐ normalized() ๐
Returns a normalized copy of this spec without modifying this spec. Non-destructive queen ๐
๐ง property os ๐ง
A property. It's called os. Genuinely that's it, bestie ๐
๐ฅ static override(init_spec, change_spec) ๐ฅ
Static. Takes init_spec and change_spec. Overriding things, main character energy only ๐
๐ property package ๐
Another property, this one's package. Short and sweet ๐
๐ property package_class ๐
Internal package call gets only the class object for a package. Use this to just get package metadata โ no full package instantiation, we're being efficient โจ
๐ธ package_hash() ๐ธ
Compute the hash of the contents of the package for this node. Snapshot the package, hash it, done ๐ธ
๐ฆ property patches ๐ฆ
Returns patch objects for any patch sha256 sums on this Spec.
This is for use after concretization, to iterate over any patches associated with this spec.
TODO: this only checks in the package; it doesn't resurrect old patches from install directories, but it probably should. (Ghost patches, we see you ๐ป)
โญ property platform โญ
Property. Name's platform. That's the tweet ๐ฆ
๐ property prefix ๐
Property. Name's prefix. Love that for us ๐ซถ
๐ process_hash(length=None) ๐
Hash used to transfer specs among processes.
This hash includes build and test dependencies, and it's only used to serialize a spec and pass it around among processes. Inter-process communication, but make it hashed ๐
๐ง process_hash_bit_prefix(bits) ๐ง
Get the first <bits> bits of the DAG hash as an integer type. Chop chop โ๏ธ
๐ฅ replace_hash() ๐ฅ
Given a spec with an abstract hash, attempt to populate all properties and dependencies by looking up the hash in the environment, store, or finally, binary caches. This is destructive โ unlike lookup_hash(), it edits the spec in place. Handle with care, bestie ๐ญโ ๏ธ
๐ property root ๐
Follow dependent links and find the root of this spec's DAG.
Spack specs have a single root (the package being installed). Follow the chain, find your origin story ๐ฑ
๐ satisfies(other: str | Spec, deps: bool = True) -> bool ๐
Returns True if all concrete specs matching self also match other, otherwise False.
Parameters:
otherโ spec to be satisfieddepsโ ifTrue, descend to dependencies; otherwise only check the root node
๐ property short_spec ๐
Returns a version of the spec with the dependencies hashed instead of completely enumerated. Compact, cute, to the point ๐
๐ฆ property spack_install ๐ฆ
Special field for using {spack_install} in Spec.format(). A little placeholder with big purpose โจ
โญ property spack_root โญ
Special field for using {spack_root} in Spec.format(). Same energy, different name ๐
๐ spec_hash(hash) ๐
Utility method for computing different types of Spec hashes.
Parameters:
hash(spack.hash_types.SpecHashDescriptor) โ type of hash to generate
๐ฅ splice(other, transitive) ๐ฅ
Splices dependency "other" into this ("target") Spec, and returns the result as a concrete Spec. If transitive, then other and its dependencies will be extrapolated to a list of Specs and spliced in accordingly.
For example, let there exist a dependency graph as follows:
T | Z<-HIn this example, Spec T depends on H and Z, and H also depends on Z. Suppose, however, that we wish to use a different H, known as H'. This function will splice in the new H' in one of two ways:
- Transitively, where
H'depends on theZ'it was built with, and the newT*also directly depends on this newZ', or - Intransitively, where the new
T*andH'both depend on the originalZ.
Since the Spec returned by this splicing function is no longer deployed the same way it was built, any such changes are tracked by setting the build_spec to point to the corresponding dependency from the original Spec.
TODO: Extend this for non-concrete Specs. (Concrete-only for now, we're working on it ๐ซถ)
property spliced ๐งฌ
Returns whether or not this Spec is being deployed as built โ i.e. whether or not this Spec has ever been spliced. That's the whole vibe check, short and sweet โจ
property target ๐ฏ
Okay so this one is justโฆ a property that exists. The manpage gives it zero description, and I'm not about to invent lore for it, bestie ๐ so: it's here, it's real, that's all I got.
to_dict(hash=<spack.hash_types.SpecHashDescriptor object>) ๐ฆ
Create a dictionary suitable for writing this spec to YAML or JSON. Iconic serialization moment.
This dictionaries like the one that is ultimately written to a spec.json file in each Spack installation directory. For example, for sqlite:
{
"spec": {
"_meta": {
"version": 2
},
"nodes": [
{
"name": "sqlite",
"version": "3.34.0",
"arch": {
"platform": "darwin",
"platform_os": "catalina",
"target": "x86_64"
},
"compiler": {
"name": "apple-clang",
"version": "11.0.0"
},
"namespace": "builtin",
"parameters": {
"column_metadata": true,
"fts": true,
"functions": false,
"rtree": false,
"cflags": [],
"cppflags": [],
"cxxflags": [],
"fflags": [],
"ldflags": [],
"ldlibs": []
},
"dependencies": [
{
"name": "readline",
"hash": "4f47cggum7p4qmp3xna4hi547o66unva",
"type": [
"build",
"link"
]
},
{
"name": "zlib",
"hash": "uvgh6p7rhll4kexqnr47bvqxb3t33jtq",
"type": [
"build",
"link"
]
}
],
"hash": "tve45xfqkfgmzwcyfetze2z6syrg7eaf",
},
# ... more node dicts for readline and its dependencies ...
]
}
}Note that this dictionary starts with the 'spec' key, and what follows is a list starting with the root spec, followed by its dependencies in preorder. Each node in the list also has a 'hash' key that contains the hash of the node without the hash field included. iykyk ๐ซถ
In the example, the package content hash is not included in the spec, but if package_hash were true there would be an additional field on each node called package_hash.
from_dict() can be used to read back in a spec that has been converted to a dictionary, serialized, and read back in. Round trip slay ๐
Parameters
deptype(tuple or str) โ dependency types to include when traversing the spec.package_hash(bool) โ whether to include package content hashes in the dictionary.
to_json(stream=None, hash=<spack.hash_types.SpecHashDescriptor object>) ๐ธ
This one's got the signature and not much else in the docs โ it's to_json with an optional stream and a hash descriptor. No extra description given, so I'm not gonna fabricate one ๐
to_node_dict(hash=<spack.hash_types.SpecHashDescriptor object>) ๐ณ
Create a dictionary representing the state of this Spec.
to_node_dict creates the content that is eventually hashed by Spack to create identifiers like the DAG hash (see dag_hash()). Example result of to_node_dict for the sqlite package:
{
'sqlite': {
'version': '3.28.0',
'arch': {
'platform': 'darwin',
'platform_os': 'mojave',
'target': 'x86_64',
},
'compiler': {
'name': 'apple-clang',
'version': '10.0.0',
},
'namespace': 'builtin',
'parameters': {
'fts': 'true',
'functions': 'false',
'cflags': [],
'cppflags': [],
'cxxflags': [],
'fflags': [],
'ldflags': [],
'ldlibs': [],
},
'dependencies': {
'readline': {
'hash': 'zvaa4lhlhilypw5quj3akyd3apbq5gap',
'type': ['build', 'link'],
}
},
}
}Heads up bestie: the dictionary returned does not include the hash of the root of the spec, though it does include hashes for each dependency, and (optionally) the package file corresponding to each node.
See to_dict() for a "complete" spec hash, with hashes for each node and nodes for each dependency (instead of just their hashes).
Parameters
hash(spack.hash_types.SpecHashDescriptor)
to_yaml(stream=None, hash=<spack.hash_types.SpecHashDescriptor object>) ๐
Same situation as to_json โ signature only, optional stream, hash descriptor. That's the tea โ
traverse(**kwargs) ๐ถโโ๏ธ
Shorthand for traverse_nodes(). Literally just a little nickname, no cap.
traverse_edges(**kwargs) ๐ถโโ๏ธ๐จ
Shorthand for traverse_edges(). (Yes, the description says the same name as the method. That's what the manpage says, and I am a humble messenger ๐ซก)
tree(...) -> str ๐ฒ
Full signature, exactly as written:
tree(*, color: bool | None = None, depth: bool = False, hashes: bool = False, hashlen: int | None = None, cover: str = 'nodes', indent: int = 0, format: str = '{name}{@versions}{%compiler.name}{@compiler.versions}{compiler_flags}{variants}{arch=architecture}{/abstract_hash}', deptypes: Tuple[str, ...] | str = 'all', show_types: bool = False, depth_first: bool = False, recurse_dependencies: bool = True, status_fn: Callable[[Spec], InstallStatus] | None = None, prefix: Callable[[Spec], str] | None = None) -> strPrints out this spec and its dependencies, tree-formatted with indentation. It's giving directory tree but make it package manager. ๐ฆ
Status function may either output a boolean or an InstallStatus.
Parameters
colorโ ifTrue, always colorize the tree. IfFalse, don't colorize the tree. IfNone, use the default fromllnl.tty.color.depthโ print the depth from the root.hashesโ ifTrue, print the hash of each node.hashlenโ length of the hash to be printed.coverโ either"nodes"or"edges".indentโ extra indentation for the tree being printed.formatโ format to be used to print each node.deptypesโ dependency types to be represented in the tree.show_typesโ ifTrue, show the (merged) dependency type of a node.depth_firstโ ifTrue, traverse the DAG depth first when representing it as a tree.recurse_dependenciesโ ifTrue, recurse on dependencies.status_fnโ optional callable that takes a node as an argument and return its installation status.prefixโ optional callable that takes a node as an argument and return its installation prefix.
trim(dep_name) โ๏ธ
Remove any package that is or provides dep_name transitively from this tree. This can also remove other dependencies if they are only present because of dep_name. Cleanup era. ๐งน
update_variant_validate(variant_name, values) ๐ง
If it is not already there, adds the variant named variant_name to the spec spec based on the definition contained in the package metadata. Validates the variant and values before returning.
Used to add values to a variant without being sensitive to the variant being single or multi-valued. If the variant already exists on the spec it is assumed to be multi-valued and the values are appended. Multitasking queen behavior. ๐
Parameters
variant_nameโ the name of the variant to add or append to.valuesโ the value or values (as a tuple) to add/append to the variant.
validate_detection() ๐
Validate the detection of an external spec.
This method is used as part of Spack's detection protocol, and is not meant for client code use. > Fr fr, stay out of this one โ it's internal plumbing, not your personal playground ๐
validate_or_raise() ๐จ
Checks that names and values in this spec are real. If they're not, it will raise an appropriate exception. Manifesting no exceptions for you bestie. โจ
property version ๐ท๏ธ
A property, listed plainly. No description in the source, so nothing made up from me ๐ธ
property virtual ๐
Same deal โ it's a property, that's the whole entry. Short and iconic.
virtual_dependencies() ๐ง
Return list of any virtual deps in this spec. That's it, that's the function ๐
exception spack.spec.SpecDeprecatedError(message, long_message=None) โ ๏ธ
Bases: SpecError
Raised when a spec concretizes to a deprecated spec or dependency. Deprecated things are deprecated โ heed the warning, bestie. ๐ญ
exception spack.spec.SpecParseError(parse_error) ๐งฉ
Bases: SpecError
Wrapper for ParseError for when we're parsing specs.
property long_messageโ a property on this exception; the manpage doesn't say anything more about it, so I'm not gonna speculate ๐
exception spack.spec.UnsatisfiableArchitectureSpecError(provided, required) ๐
Bases: UnsatisfiableSpecError
Raised when a spec architecture conflicts with package constraints. Architecture beef = error. ๐ฆ
Okay so these next ones are the โจexceptionโจ girlies that live under spack.spec, and honestly they're all just variations of "bestie, these two specs are NOT compatible" ๐ญ every single one of them is a UnsatisfiableSpecError at heart.
exception spack.spec.UnsatisfiableCompilerFlagSpecError(provided, required)
Bases: UnsatisfiableSpecError
Raised when a spec variant conflicts with package constraints.
exception spack.spec.UnsatisfiableCompilerSpecError(provided, required)
Bases: UnsatisfiableSpecError
Raised when a spec compiler conflicts with package constraints.
exception spack.spec.UnsatisfiableDependencySpecError(provided, required)
Bases: UnsatisfiableSpecError
Raised when some dependency of constrained specs are incompatible. ๐ซ
exception spack.spec.UnsatisfiableProviderSpecError(provided, required)
Bases: UnsatisfiableSpecError
Raised when a provider is supplied but constraints don't match a vpkg requirement.
exception spack.spec.UnsatisfiableSpecNameError(provided, required)
Bases: UnsatisfiableSpecError
Raised when two specs aren't even for the same package. like, they're not even the same genre?? ๐
exception spack.spec.UnsatisfiableVersionSpecError(provided, required)
Bases: UnsatisfiableSpecError
Raised when a spec version conflicts with package constraints.
exception spack.spec.UnsupportedCompilerError(compiler_name)
Bases: SpecError
Raised when the user asks for a compiler spack doesn't know about. ๐
๐ฆ spack.spec_list module ๐ฆ
exception spack.spec_list.InvalidSpecConstraintError(message, long_message=None)
Bases: SpecListError
Error class for invalid spec constraints at concretize time.
class spack.spec_list.SpecList(name='specs', yaml_list=None, reference=None)
Bases: object
Here's everything this icon can do:
add(spec)extend(other, copy_reference=True)property is_matrixremove(spec)replace(idx: int, spec: str)โ Replace the existing spec at the index with the new one. โจ
Parameters:
- idx โ index of the spec to replace in the speclist
- spec โ new spec
property specs: List[Spec]property specs_as_constraintsproperty specs_as_yaml_listupdate_reference(reference)
exception spack.spec_list.SpecListError(message, long_message=None)
Bases: SpackError
Error class for all errors related to SpecList objects.
exception spack.spec_list.UndefinedReferenceError(message, long_message=None)
Bases: SpecListError
Error class for undefined references in Spack stacks. ๐
๐ spack.stage module ๐
class spack.stage.DevelopStage(name, dev_path, reference_link)
Bases: LockableStagingDir
property archive_filecache_local()check()create()โ Ensures the top-level (config:build_stage) directory exists.destroy()expand_archive()property expandedโ ReturnsTruesince thesource_pathmust exist. ๐fetch(*args, **kwargs)requires_patch_success = Falserestage()
class spack.stage.LockableStagingDir(name, path, keep, lock)
Bases: object
A directory whose lifetime can be managed with a context manager (but persists if the user requests it). Instances can have a specified name, and if they do, then for all instances that have the same name, only one can enter the context manager at a time. ๐
create()โ Ensures the top-level (config:build_stage) directory exists.destroy()
class spack.stage.ResourceStage(fetch_strategy: FetchStrategy, root: Stage, resource: Resource, **kwargs)
Bases: Stage
expand_archive()โ Changes to the stage directory and attempt to expand the downloaded archive. Fail if the stage is not set up or if the archive is not yet downloaded.restage()โ Removes the expanded archive path if it exists, then re-expands the archive.
exception spack.stage.RestageError(message, long_message=None)
Bases: StageError
"Error encountered during restaging.
class spack.stage.Stage(url_or_fetch_strategy, name=None, mirror_paths=None, keep=False, path=None, lock=True, search_fn=None)
Bases: LockableStagingDir
Manages a temporary stage directory for building. ๐ ๏ธ
A Stage object is a context manager that handles a directory where some source code is downloaded and built before being installed. It handles fetching the source code, either as an archive to be expanded or by checking it out of a repository. A stage's lifecycle looks like this:
with Stage() as stage: # Context manager creates and destroys the
# stage directory
stage.fetch() # Fetch a source archive into the stage.
stage.expand_archive() # Expand the archive into source_path.
<install> # Build and install the archive.
# (handled by user of Stage)When used as a context manager, the stage is automatically destroyed if no exception is raised by the context. If an exception is raised, the stage is left in the filesystem and NOT destroyed, for potential reuse later. ๐ญ (yes, it's giving "I'll leave my stuff here just in case")
You can also use the stage's create/destroy functions manually, like this:
stage = Stage()
try:
stage.create() # Explicitly create the stage directory.
stage.fetch() # Fetch a source archive into the stage.
stage.expand_archive() # Expand the archive into source_path.
<install> # Build and install the archive.
# (handled by user of Stage)
finally:
stage.destroy() # Explicitly destroy the stage directory.There are two kinds of stages: named and unnamed. Named stages can persist between runs of spack, e.g. if you fetched a tarball but didn't finish building it, you won't have to fetch it again. ๐
Unnamed stages are created using standard mkdtemp mechanisms or similar, and are intended to persist for only one run of spack.
property archive_fileโ Path to the source archive within this stage directory.cache_local()cache_mirror(mirror, stats)โ Perform a fetch if the resource is not already cached. โจ
Parameters:
- mirror (
spack.caches.MirrorCache) โ the mirror to cache this Stage's resource in - stats (
spack.mirror.MirrorStats) โ this is updated depending on whether the caching operation succeeded or failed
check()โ Check the downloaded archive against a checksum digest. No-op if this stage checks code out of a repository.destroy()โ Removes this stage directory.disable_mirrors()โ The Stage will not attempt to look for the associated fetcher target in any of Spack's mirrors (including the local download cache).expand_archive()โ Changes to the stage directory and attempt to expand the downloaded archive. Fail if the stage is not set up or if the archive is not yet downloaded.property expandedโ ReturnsTrueif source path expanded; elseFalse.property expected_archive_filesโ Possible archive file paths.fetch(mirror_only=False, err_msg=None)โ Retrieves the code or archive.
Parameters:
- mirror_only (
bool) โ only fetch from a mirror - err_msg (
strorNone) โ the error message to display if all fetchers fail, orNonefor the default fetch failure message
requires_patch_success = Truerestage()โ Removes the expanded archive path if it exists, then re-expands the archive.property save_filenameproperty source_pathโ Returns the well-known source directory path.steal_source(dest)โ Copy thesource_pathdirectory in its entirety to directorydest. ๐
This operation creates/fetches/expands the stage if it is not already, and destroys the stage when it is done.
class spack.stage.StageComposite
Bases: Composite
Composite for Stage type objects. The first item in this composite is considered to be the root package, and operations that return a value are forwarded to it.
property archive_fileproperty expandedclassmethod from_iterable(iterable: Iterable[Stage]) -> StageCompositeโ Create a new composite from an iterable of stages.property keepproperty pathproperty requires_patch_successproperty source_path
exception spack.stage.StageError(message, long_message=None)
Bases: SpackError
"Superclass for all errors encountered during staging.
exception spack.stage.StagePathError(message, long_message=None)
Bases: StageError
"Error encountered with stage path.
exception spack.stage.VersionFetchError(message, long_message=None)
Bases: StageError
Raised when we can't determine a URL to fetch a package. ๐ญ
spack.stage.compute_stage_name(spec)
Determine stage name given a spec.
spack.stage.create_stage_root(path: str) -> None
Create the stage root directory and ensure appropriate access perms. ๐
spack.stage.ensure_access(file)
Ensure we can access a directory and die with an error if we can't. (dramatic but fair ๐ )
spack.stage.get_checksums_for_versions(url_by_version: Dict[StandardVersion, str], package_name: str, *, first_stage_function: Callable[[Stage, str], None] | None = None, keep_stage: bool = False, concurrency: int | None = None, fetch_options: Dict[str, str] | None = None) -> Dict[StandardVersion, str]
Computes the checksums for each version passed in input, and returns the results. โญ
Archives are fetched according to the usl dictionary passed as input.
The first_stage_function argument is basically your all-access backstage pass ๐
โ it lets whoever's calling go peek inside the first downloaded archive, like, for example, to work out which build system it's dealing with. Iconic behaviour, honestly.
โจ Parameters โจ
url_by_versionโ the URL, keyed by version ๐package_nameโ name of the packagefirst_stage_functionโ a function that takes aStageand a URL; this is run on the stage of the first URL downloadedkeep_stageโ whether to keep the staging area when the command completesbatchโ whether to ask the user how many versions to fetch (false) or fetch all the versions (true) ๐ฅfetch_optionsโ options used for the fetcher (such astimeoutorcookies)concurrencyโ maximum number of workers to use for retrieving archives
๐ธ Returns ๐ธ
A dictionary mapping each version to the corresponding checksum. Everything matched up, no cap ๐
๐ฆ spack.stage.get_stage_root()
Okay so this one gets literally no blurb in the manpage โ just the name and that's it bestie. No description, no params, nothing. Not me inventing lore ๐ โ this is the whole entry.
๐ spack.stage.interactive_version_filter(...)
spack.stage.interactive_version_filter(
url_dict: ~typing.Dict[~spack.version.version_types.StandardVersion, str],
known_versions: ~typing.Iterable[~spack.version.version_types.StandardVersion] = (),
*,
initial_verion_filter: ~spack.version.version_types.VersionList | None = None,
url_changes: ~typing.Set[~spack.version.version_types.StandardVersion] = {},
input: ~typing.Callable[[...], str] = <built-in function input>
) -> Dict[StandardVersion, str] | NoneInteractively filter the list of spidered versions. You, personally, in the driver's seat ๐
โจ Parameters โจ
url_dictโ dictionary of versions to URLsknown_versionsโ versions that can be skipped because they are already known
๐ Returns ๐
A filtered dictionary of versions to URLs โ or None if the user decides they want to quit. Respect the choice ๐ซถ
โญ spack.stage.purge()
Remove all build directories in the top-level stage path. Poof. Gone. Clean slate energy โจ
๐ฆ spack.store module ๐ฆ
Components that manage Spack's installation tree. Okay this one's a whole thing, buckle up ๐
An install tree, or "build store", consists of two parts:
- A package database that tracks what is installed.
- A directory layout that determines how the installations are laid out.
The store holds all the install prefixes for packages installed by Spack. The simplest store could just be prefixes named by DAG hash, but that's not what we do โ we use a fancier directory layout so browsing the store and debugging are both way easier. Main character choices only ๐
spack.store.DEFAULT_INSTALL_TREE_ROOT = '/opt/spack'
Default installation root, relative to the Spack install path. The default-est default that ever defaulted ๐
๐ exception spack.store.MatchError(message, long_message=None)
Bases: SpackError
Error occurring when trying to match specs in store against a constraint. It's giving "that didn't match and we're upset about it" ๐ญ
spack.store.STORE: Store | Singleton = <spack.store.Store object>
Singleton store instance. One store, one vibe, periodt ๐
๐ช class spack.store.Store(...)
spack.store.Store(
root: str,
unpadded_root: str | None = None,
projections: Dict[str, str] | None = None,
hash_length: int | None = None,
upstreams: List[Database] | None = None,
lock_cfg: LockConfiguration = (False, None, None)
)Bases: object
A store is a path full of installed Spack packages. Literally a whole directory of them.
Stores consist of packages installed according to a DirectoryLayout, plus a database of their contents.
The directory layout controls what paths look like and how Spack makes sure each unique spec gets its own unique directory (or doesn't โ though bestie, we do not recommend that ๐).
The database is a single file that caches metadata for the entire Spack installation. It saves us from having to spider the install tree just to figure out what's actually in there โ efficiency, we love her โจ
The store can also lock installation prefixes, and mark installation failures. Multi-talented fr fr.
โจ Parameters โจ
rootโ path to the root of the install treeunpadded_rootโ path to the root of the install tree without padding. Thesbangscript has to be installed here to work with padded rootsprojectionsโ expression according to guidelines that describes how to construct a path to a package prefix in this storehash_lengthโ length of the hashes used in the directory layout. Spec hash suffixes will be truncated to this lengthupstreamsโ optional list of upstream databaseslock_cfgโ lock configuration for the database
Methods
reindex()โNoneโ convenience function to reindex the store DB with its own layout. Like hitting refresh on your whole life ๐
๐ง spack.store.create(configuration: Configuration | Singleton) -> Store
Create a store from the configuration passed as input.
configurationโ configuration to create a store
โญ spack.store.ensure_singleton_created() -> None
Ensures the lazily evaluated singleton is created. Lazy in the implementation sense, not the personality sense ๐
๐ spack.store.find(...)
spack.store.find(
constraints: str | List[str] | List[Spec],
multiple: bool = False,
query_fn: Callable[[Any], List[Spec]] | None = None,
**kwargs
) -> List[Spec]Returns a list of specs matching the constraints passed as inputs.
At least one spec per constraint must match โ otherwise the function errors out with an appropriate message. No silent failures here, we're dramatic about it ๐ค
By default this function queries the current store, but you can pass a custom query function to hit any other source of concretized specs (e.g. a binary cache). The query function must accept a spec as its first argument.
โจ Parameters โจ
constraintsโ spec(s) to be matched against installed packagesmultipleโ ifTrue, multiple matches per constraint are admittedquery_fn(Callable) โ query function to get matching specs. By defaultspack.store.STORE.db.query**kwargsโ keyword arguments forwarded to the query function
๐ธ spack.store.parse_install_tree(config_dict)
Parse config settings and return values relevant to the store object.
Parameters
config_dict(dict) โ dictionary of config values, as returned fromspack.config.get('config')
Returns
Triple of the install tree root, the unpadded install tree root (before padding was applied), and the projections for the install tree.
Return type
(tuple)
Encapsulate backwards compatibility capabilities for install_tree and deprecated values that are now parsed as part of install_tree. Old stuff, still handled, we respect the elders ๐ซถ
๐ spack.store.reinitialize()
Restore globals to the same state they'd have at start-up. Returns a token containing the state of the store before reinitialization. Keep that token, you'll want it later ๐
๐ spack.store.restore(token)
Restore the environment from a token returned by reinitialize. Undo button, but make it official.
๐ฆ spack.store.specfile_matches(filename: str, **kwargs) -> List[Spec]
Same as find but it reads the query from a spec file. Same energy, different source.
Parameters
filenameโ YAML or JSON file from which to read the query**kwargsโ keyword arguments forwarded to "find"
๐ spack.store.use_store(...)
spack.store.use_store(
path: str | Path,
extra_data: Dict[str, Any] | None = None
) -> Generator[Store, None, None]Use the store passed as argument within the context manager. Very with-statement of it โจ
Parameters
pathโ path to the storeextra_dataโ extra configuration under"config:install_tree"to be taken into account
Yields
The Store object associated with the context manager's store.
๐ง spack.subprocess_context module ๐ง
This module handles transmission of Spack state to child processes started using the 'spawn' start method. Notably, installations are performed in a subprocess and require transmitting the Package object (in such a way that the repository is available for importing when it is deserialized); installations performed in Spack unit tests may include additional modifications to global state in memory that must be replicated in the child process. So yeah โ packing up your whole vibe and shipping it to the baby process ๐ฆ
๐ class spack.subprocess_context.PackageInstallContext(pkg)
Bases: object
Captures the in-memory process state of a package installation that needs to be transmitted to a child process.
restore()
class spack.subprocess_context.SpackTestProcess(fn)
Bases: object
create()
class spack.subprocess_context.TestPatches(module_patches, class_patches)
Bases: object
restore()
๐ class spack.subprocess_context.TestState
Bases: object
Spack tests may modify state that is normally read from disk in memory; this object is responsible for properly serializing that state to be applied to a subprocess. This isn't needed outside of a testing environment, but this logic is designed to behave the same inside or outside of tests. Consistent queen ๐
restore()
Functions
spack.subprocess_context.append_patch(patch)spack.subprocess_context.clear_patches()spack.subprocess_context.serialize(obj)spack.subprocess_context.store_patches()
๐ spack.tag module ๐
Classes and functions to manage package tags. Little labels, big purpose ๐ท๏ธ
๐ class spack.tag.TagIndex(repository)
Bases: Mapping
Maps tags to list of packages.
Methods
copy()โ return a deep copy of this index. No shared references, we're not doing that ๐ธstatic from_json(stream, repository)get_packages(tag)โ returns all packages associated with the tagmerge(other)โ merge another tag index into this oneother(TagIndex) โ tag index to be merged
property tagsto_json(stream)update_package(pkg_name)โ updates a package in the tag indexpkg_name(str) โ name of the package to be removed from the index
๐ญ exception spack.tag.TagIndexError(message, long_message=None)
Bases: SpackError
Raised when there's a problem with a TagIndex. Something went wrong in tag-land ๐
๐ฅ spack.tag.packages_with_tags(tags, installed, skip_empty)
Returns a dict, indexed by tag, containing lists of names of packages containing the tag or, if no tags, for all available tags.
Parameters
tags(list or None) โ list of tags of interest orNonefor allinstalled(bool) โTrueif you want the names of packages that are installed; otherwiseFalseif you want, like, all the packages with the tag ๐ท๏ธskip_empty(bool) โTrueif you wanna exclude tags that have no packages attached; otherwiseFalseif you want an entry for literally every tag, even the lonely ones with zero tagged packages ๐ซถ
๐ฏ spack.target module
class spack.target.Target(name, module_name=None)
Bases: object
static from_dict_or_value(dict_or_value)โจproperty name๐ท๏ธoptimization_flags(compiler)โ gives you back the flags you need to optimize for this target, using the compiler you hand it.
Parameters:
compiler(spack.spec.CompilerSpecorspack.compiler.Compiler) โ an object holding both the name and the version of the compiler we're trying to use ๐
to_dict_or_value()โ returns a dict or a value that represents the current target.
String values are kept around for backward compatibility with the generic targets, like e.g. x86_64 or ppc64. More specific micro-architectures will instead return a dictionary containing info on the name, features, vendor, generation and parents of the current target ๐
๐งต spack.tengine module
class spack.tengine.Context
Bases: object
The base class for context classes that get used with the template engine โจ
context_properties = []
to_dict()โ returns a dictionary containing all the context properties ๐
class spack.tengine.ContextMeta(name, bases, attr_dict)
Bases: type
The meta class for Context. It exists to cut down on the boilerplate in client code ๐
classmethod context_property(func)โ a decorator that adds a function name to the list of new context properties, and then returns a property ๐ซ
spack.tengine.context_property(func)
A saner way to use the decorator ๐ธ
spack.tengine.curly_quote(text)
Wraps each line of text in curly braces ๐
spack.tengine.make_environment(dirs: Tuple[str, ...] | None = None)
Returns a configured environment for template rendering โจ
spack.tengine.prepend_to_line(text, token)
Sticks a token at the front of every line in text ๐
spack.tengine.quote(text)
Quotes each line in text ๐
๐ณ spack.traverse module
spack.traverse.traverse_edges
spack.traverse.traverse_edges(specs, root=True, order='pre',
cover='nodes', direction='children', deptype: int | str |
~typing.List[str] | ~typing.Tuple[str, ...] = 'all', depth=False,
key=<built-in function id>, visited=None)A generator that yields edges from the DAG, starting from a list of root specs ๐
Parameters:
specs(list) โ list of root specs (these count as depth 0)root(bool) โ yield the root nodes themselvesorder(str) โ what order of traversal to use in the DAG. For depth-first search this can bepreorpost. For BFS this should bebreadth. For topological order usetopo๐งญcover(str) โ decides how thoroughly to cover the dag. Possible values:nodesโ visit each unique node in the dag only once.edgesโ if a node has been visited once but is reached along a new path, it's accepted, but not recursively followed. This traverses each 'edge' in the DAG once.pathsโ explore every unique path reachable from the root. This descends into visited subtrees and will accept nodes multiple times if they're reachable by multiple paths ๐ค๏ธ
direction(str) โchildrenorparents. Ifchildren, it does a traversal of this spec's children. Ifparents, it traverses upwards in the DAG towards the root โฌ๏ธdeptypeโ allowed dependency typesdepth(bool) โ whenFalse, yield just edges. WhenTrue, yield the tuple(depth, edge), wheredepthcorresponds to the depth at whichedge.specwas discovered ๐ณ๏ธkeyโ function that takes a spec and outputs a key for the uniqueness testvisited(set orNone) โ a set of nodes not to follow ๐ซ
Returns: a generator that yields DependencySpec if depth is False, or a tuple of (depth, DependencySpec) if depth is True ๐
spack.traverse.traverse_nodes
spack.traverse.traverse_nodes(specs, root=True, order='pre',
cover='nodes', direction='children', deptype: int | str |
~typing.List[str] | ~typing.Tuple[str, ...] = 'all', depth=False,
key=<built-in function id>, visited=None)A generator that yields specs from the DAG, starting from a list of root specs ๐
Parameters:
specs(list) โ list of root specs (again, considered depth 0)root(bool) โ yield the root nodes themselvesorder(str) โ what order of traversal to use in the DAG. For depth-first search this can bepreorpost. For BFS this should bebreadth.cover(str) โ decides how extensively to cover the dag. Possible values:nodesโ visit each unique node in the dag only once.edgesโ if a node has been visited once but is reached along a new path, it's accepted, but not recursively followed. This traverses each 'edge' in the DAG once.pathsโ explore every unique path reachable from the root. This descends into visited subtrees and will accept nodes multiple times if they're reachable by multiple paths ๐ค๏ธ
direction(str) โchildrenorparents. Ifchildren, it does a traversal of this spec's children. Ifparents, it traverses upwards in the DAG towards the root.deptypeโ allowed dependency typesdepth(bool) โ whenFalse, yield just edges. WhenTrue, yield the tuple(depth, edge), wheredepthcorresponds to the depth at whichedge.specwas discovered.keyโ function that takes a spec and outputs a key for the uniqueness testvisited(set orNone) โ a set of nodes not to follow ๐ซ
Yields: by default Spec, or a tuple (depth, Spec) if depth is set to True โจ
spack.traverse.traverse_tree
spack.traverse.traverse_tree(specs, cover='nodes', deptype: int | str |
~typing.List[str] | ~typing.Tuple[str, ...] = 'all', key=<built-in
function id>, depth_first=True)A generator that yields (depth, DependencySpec) tuples in depth-first pre-order, so you can literally print a tree straight out of it ๐ฒ
Parameters:
specs(list) โ list of root specs (depth 0, you know the drill)cover(str) โ decides how extensively to cover the dag. Possible values:nodesโ visit each unique node in the dag only once.edgesโ if a node has been visited once but is reached along a new path, it's accepted, but not recursively followed. This traverses each 'edge' in the DAG once.pathsโ explore every unique path reachable from the root. This descends into visited subtrees and will accept nodes multiple times if they're reachable by multiple paths ๐ค๏ธ
deptypeโ allowed dependency typeskeyโ function that takes a spec and outputs a key for the uniqueness testdepth_first(bool) โ explore the tree in depth-first or breadth-first order. When you setdepth_first=Trueandcover=nodes, each spec only shows up once at the shallowest level, which is super useful when you're rendering the tree in a terminal ๐ฅ๏ธ
Returns: a generator that yields (depth, DependencySpec) tuples in an order such that a tree can be printed ๐
๐ spack.url module
Okay so this module has methods for parsing names and versions of packages straight out of URLs. The whole idea is to let package creators supply nothing more than the download location of the package, and have spack figure out the version and name info from there. Iconic behaviour, honestly ๐
Example: when spack is given the following URL:
https://www.hdfgroup.org/ftp/HDF/releases/HDF4.2.12/src/hdf-4.2.12.tar.gzIt can figure out that the package name is hdf, and that it's at version 4.2.12. This is useful for making the creation of packages simple: a user just supplies a URL and skeleton code is generated automatically โจ
Spack can also figure out that it can most likely download 4.2.6 at this URL:
https://www.hdfgroup.org/ftp/HDF/releases/HDF4.2.6/src/hdf-4.2.6.tar.gzThis is useful if a user asks for a package at a particular version number; spack doesn't need anyone to tell it where to get the tarball, even though it's never been told about that version before ๐คฏ
exception spack.url.UndetectableNameError(path)
Bases: UrlParseError
Raised when we can't parse a package name from a string ๐ญ
exception spack.url.UndetectableVersionError(path)
Bases: UrlParseError
Raised when we can't parse a version from a string ๐ญ
exception spack.url.UrlParseError(msg, path)
Bases: SpackError
Raised when the URL module can't parse something correctly ๐จ
spack.url.color_url(path, **kwargs)
Colors up the parts of the url according to Spack's parsing ๐จ
The colors are:
Cyan: the version found by parse_version_offset(). Red: the name found by parse_name_offset().
Green: instances of the version string from substitute_version(). Magenta: instances of the name (protected from substitution).
Parameters:
path(str) โ the filename or URL for the packageerrors(bool) โ append parse errors at the end of the stringsubs(bool) โ color substitutions as well as the parsed name/version ๐
spack.url.find_all(substring, string)
Returns a list containing the indices of every single occurrence of substring in string ๐
spack.url.find_versions_of_archive
spack.url.find_versions_of_archive(archive_urls: str | Sequence[str],
list_url: str | None = None, list_depth: int = 0, concurrency: int |
None = 32, reference_package: Any | None = None) -> Dict[
StandardVersion, str]Hiii bestie, this one's the lil web-scraper goblin ๐ธ๏ธ โ it goes out and hunts down new versions of a tarball for you. Basically it vibes on the internet so you don't have to.
Lowkey the important bit: this function has opinions about which URLs it likes best, and it ranks them in this exact order โ links found on the scraped page that match a url generated by the reference package, then found and in the archive_urls list, then found and derived from those in the archive_urls list, and if none are found for a version then the item in the archive_urls list is included for the version. Periodt. ๐
Parameters
archive_urlsโ URL or sequence of URLs for different versions of a package. Typically these are just the tarballs from the package file itself. By default, this searches the parent directories of archives.list_urlโ URL for a listing of archives. Spack will scrape these pages for download links that look like the archive URL. ๐list_depthโ max depth to follow links onlist_urlpages. Defaults to0.concurrencyโ maximum number of concurrent requests. โกreference_packageโ a spack package used as a reference for url detection. Uses theurl_for_versionmethod on the package to produce reference urls which, if found, are preferred. ๐ฅ
spack.url.parse_name
spack.url.parse_name(path, ver=None)Try to determine the name of a package from its filename or URL. It's giving detective energy ๐ โ just squinting at a string and being like "okay so who are you".
Parameters
path(str) โ The filename or URL for the packagever(str) โ The version of the package
Returns: The name of the package
Return type: str
Raises UndetectableNameError โ If the URL does not match any regexes ๐ญ
spack.url.parse_name_and_version
spack.url.parse_name_and_version(path)Okay this one's the two-for-one combo deal ๐ โ it tries to determine the name of a package and extract its version from its filename or URL. Name and version, bestie. Efficient queen.
Parameters
path(str) โ The filename or URL for the package
Returns: a tuple containing the package (name, version)
Return type: tuple
Raises
UndetectableVersionErrorโ If the URL does not match any regexesUndetectableNameErrorโ If the URL does not match any regexes
spack.url.parse_name_offset
spack.url.parse_name_offset(path, v=None)Try to determine the name of a package from its filename or URL โ but this time with receipts ๐. It tells you not just the name but exactly where it lives in the string.
Parameters
path(str) โ The filename or URL for the packagev(str) โ The version of the package
Returns
A tuple containing: name of the package, first index of name, length of name, the index of the matching regex, the matching regex
Return type: tuple
Raises UndetectableNameError โ If the URL does not match any regexes ๐ญ
spack.url.parse_version
spack.url.parse_version(path: str) -> StandardVersionTry to extract a version string from a filename or URL. Just the version this time, no name, we're being minimal ๐ฆ.
Parameters
pathโ The filename or URL for the package
Returns: The version of the package
Raises UndetectableVersionError โ If the URL does not match any regexes
spack.url.parse_version_offset
spack.url.parse_version_offset(path)Try to extract a version string from a filename or URL โ and again, we get the coordinates too ๐บ๏ธ.
Parameters
path(str) โ The filename or URL for the package
Returns
A tuple containing: version of the package, first index of version, length of version string, the index of the matching regex, the matching regex
Return type: tuple
Raises UndetectableVersionError โ If the URL does not match any regexes
spack.url.strip_name_suffixes
spack.url.strip_name_suffixes(path, version)Okay so most tarballs contain a package name followed by a version number. But some of them are messy and sneak extraneous information in-between the name and version, like:
rgb-1.0.6converge_install_2.3.16jpegsrc.v9b
These strings are not part of the package name and should be ignored. ๐ โโ๏ธ This function strips the version number and any extraneous suffixes off and returns the remaining string.
The goal? That the name is always the last thing in path, so you get:
rgbconvergejpeg
Clean. Gorgeous. Iconic. โจ
Parameters
path(str) โ The filename or URL for the packageversion(str) โ The version detected for this URL
Returns: The path with any extraneous suffixes removed
Return type: str
spack.url.substitute_version
spack.url.substitute_version(path, new_version)Given a URL or archive name, find the version in the path and substitute the new version for it. Replace all occurrences of the version if they don't overlap with the package name. Swapping out the glow-up version ๐
Simple example:
substitute_version('http://www.mr511.de/software/libelf-0.8.13.tar.gz', '2.9.3')
>>> 'http://www.mr511.de/software/libelf-2.9.3.tar.gz'Complex example:
substitute_version('https://www.hdfgroup.org/ftp/HDF/releases/HDF4.2.12/src/hdf-4.2.12.tar.gz', '2.3')
>>> 'https://www.hdfgroup.org/ftp/HDF/releases/HDF2.3/src/hdf-2.3.tar.gz'See how it hit both of them in the complex one? That's the "all occurrences" part doing its thing ๐ฅ
spack.url.substitution_offsets
spack.url.substitution_offsets(path)This returns offsets for substituting versions and names in the provided path. It is a helper for substitute_version(). Little sidekick energy ๐ซถ โ doesn't do the work itself, just hands over the coordinates.
spack.url.wildcard_version
spack.url.wildcard_version(path)Find the version in the supplied path, and return a regular expression that will match this path with any version in its place. Basically turns your very specific URL into a "any version goes" matching machine ๐โจ.
๐ฆ spack.user_environment module ๐ฆ
spack.user_environment.environment_modifications_for_specs
spack.user_environment.environment_modifications_for_specs(*specs: Spec,
view=None, set_package_py_globals: bool = True)List of environment (shell) modifications to be processed for spec. It's literally a to-do list for your shell, bestie ๐
This list is specific to the location of the spec or its projection in the view.
Parameters
specsโ spec(s) for which to list the environment modificationsviewโ view associated with the spec passed as first argumentset_package_py_globalsโ whether or not to set the global variables in thepackage.pyfiles (this may be problematic when using buildcaches that have been built on a different but compatible OS) โ ๏ธ
spack.user_environment.prefix_inspections
spack.user_environment.prefix_inspections(platform)Get list of prefix inspections for platform.
Parameters
platform(str) โ the name of the platform to consider. The platform determines what environment variables Spack will use for some inspections.
Returns
A dictionary mapping subdirectory names to lists of environment variables to modify with that directory if it exists.
spack.user_environment.project_env_mods
spack.user_environment.project_env_mods(*specs: Spec, view, env:
EnvironmentModifications) -> NoneGiven a list of environment modifications, project paths changes to the view. It's giving "same energy, different address" ๐ โก๏ธ๐ก
spack.user_environment.spack_loaded_hashes_var
spack.user_environment.spack_loaded_hashes_var = 'SPACK_LOADED_HASHES'Environment variable name Spack uses to track individually loaded packages ๐ Keep an eye on this one, it's how Spack remembers who's loaded.
spack.user_environment.unconditional_environment_modifications
spack.user_environment.unconditional_environment_modifications(view)List of environment (shell) modifications to be processed for view.
This list does not depend on the specs in this environment โ so it's the same no matter which specs you've got going on. No cap ๐งข
๐ spack.variant module ๐
The variant module contains data structures that are needed to manage variants both in packages and in specs. Okay so this is the whole personality-options department, and it's kind of the main character of this section ๐ฅ
class spack.variant.AbstractVariant(name, value, propagate=False)
Bases: object
A variant that has not yet decided who it wants to be. ๐ญ It behaves like a multi valued variant which could do things.
This kind of variant is generated during parsing of expressions like foo=bar and differs from multi valued variants because it will satisfy any other variant with the same name. Why? Because it could do it if it grows up to be a multi valued variant with the right set of values. It's giving potential energy, bestie ๐ฆ
compatible(other)
Returns True if self and other are compatible, False otherwise.
As there is no semantic check, two VariantSpec are compatible if either they contain the same value or they are both multi-valued.
Parameters
otherโ instance against which we test compatibility
Returns: True or False
Return type: bool
constrain(other)
Modify self to match all the constraints for other if both instances are multi-valued. Returns True if self changed, False otherwise.
Parameters
otherโ instance against which we constrainself
Returns: True or False
Return type: bool
copy()
Returns an instance of a variant equivalent to self.
Returns: a copy of self
Return type: AbstractVariant
>>> a = MultiValuedVariant('foo', True)
>>> b = a.copy()
>>> assert a == b
>>> assert a is not bLike, same vibes, different object โจ
static from_node_dict(name, value)
Reconstruct a variant from a node dict. Glow-up-from-parts situation ๐ง
intersects(other)
Returns True if there are variant matching both self and other, False otherwise.
satisfies(other)
Returns true if other.name == self.name, because any value that other holds and is not in self yet could be added.
Parameters
otherโ constraint to be met for the method to returnTrue
Returns: True or False
Return type: bool
sticking with the AbstractVariant crew for one more sec โ here's the rest of what that class exposes ๐
value(property) โ hands you back a tuple of strings containing all the values stashed in the variant.- Returns: values stored in the variant
- Return type:
tuple
yaml_entry()โ gives you a key, value tuple that's ready to drop straight into a yaml dict, no cap.- Returns:
(name, value_representation) - Return type:
tuple
- Returns:
class spack.variant.BoolValuedVariant(name, value, propagate=False)
Bases: SingleValuedVariant
A variant that's strictly a yes-or-no girlie โ it holds either True or False. ๐ซถ
And plot twist: BoolValuedVariant can also hold the value '*', which exists so that comparisons between foo=* and +foo or ~foo can be coerced. iykyk โจ
class spack.variant.DisjointSetsOfValues(*sets)
Bases: Sequence
This one lets you pull combos from one of many mutually exclusive sets โ like, pick a lane, bestie, but you've got options for which lane ๐
Heads up tho: the value ('none',) is reserved to denote the empty set, which means no other set is allowed to contain the item 'none'. That's just the rules ๐
Parameters:
*sets(list) โ mutually exclusive sets of values
allow_empty_set() โ adds the empty set to the current list of disjoint sets.
feature_values โ an attribute used to track values which correspond to features that can be enabled or disabled as understood by the package's build system. very demure, very build-system-aware ๐
prohibit_empty_set() โ removes the empty set from the current list of disjoint sets (the opposite of its sibling up there, iconic duo).
validator (property)
with_default(default) โ sets the default value and returns self, so you can keep chaining. fluent queen ๐
with_error(error_fmt) โ sets the error message format and returns self. also chainable, also serving.
with_non_feature_values(*values) โ marks a few values as not being tied to a feature.
exception spack.variant.DuplicateVariantError(message, long_message=None)
Bases: SpecError
Raised when the same variant shows up in a spec twice. bestie said it twice so it must be true โ except no, the parser says absolutely not ๐
exception spack.variant.InconsistentValidationError(vspec, variant)
Bases: SpecError
Raised if the wrong validator is used to validate a variant. mismatched energy, we don't love it ๐
exception spack.variant.InvalidVariantForSpecError(variant, when, spec)
Bases: SpecError
Raised when an invalid conditional variant is specified. the when clause said what it said and the spec did not listen.
exception spack.variant.InvalidVariantValueCombinationError(message, long_message=None)
Bases: SpecError
Raised when a variant has values '*' or 'none' mixed in with other values. pick a struggle, variant ๐ญ
exception spack.variant.InvalidVariantValueError(variant, invalid_values, pkg)
Bases: SpecError
Raised when a valid variant has at least one invalid value. so close, and yet so far ๐
class spack.variant.MultiValuedVariant(name, value, propagate=False)
Bases: AbstractVariant
A variant that can hold multiple values at once โ she contains multitudes โจ
append(value) โ add another value to this multi-valued variant. collect them all.
satisfies(other) โ returns true if other.name == self.name and other.value is a strict subset of self. it does not try to validate, so don't come crying to her about correctness ๐
- Parameters:
otherโ constraint to be met for the method to returnTrue - Returns:
TrueorFalse - Return type:
bool
exception spack.variant.MultipleValuesInExclusiveVariantError(variant, pkg)
Bases: SpecError, ValueError
Raised when multiple values are present in a variant that only wants one. she said she wanted one (1) value and you brought her five. disrespectful. ๐ญ
class spack.variant.SingleValuedVariant(name, value, propagate=False)
Bases: AbstractVariant
A variant that can hold multiple values โ but only one at a time. yes she's technically capable, no she's not entertaining that rn ๐
compatible(other) โ returns True if self and other are compatible, False otherwise. As there is no semantic check, two VariantSpec are compatible if either they contain the same value or they are both multi-valued.
- Parameters:
otherโ instance against which we test compatibility - Returns:
TrueorFalse - Return type:
bool
constrain(other) โ modify self to match all the constraints for other if both instances are multi-valued. Returns True if self changed, False otherwise.
- Parameters:
otherโ instance against which we constrainself - Returns:
TrueorFalse - Return type:
bool
intersects(other) โ returns True if there are variants matching both self and other, False otherwise. venn diagram moment ๐ธ
satisfies(other) โ returns true if other.name == self.name, because any value that other holds and isn't in self yet could still be added. it's giving "there's always room for more" ๐
- Parameters:
otherโ constraint to be met for the method to returnTrue - Returns:
TrueorFalse - Return type:
bool
yaml_entry() โ returns a key, value tuple suitable to be an entry in a yaml dict.
- Returns:
(name, value_representation) - Return type:
tuple
exception spack.variant.UnknownVariantError(spec, variants)
Bases: SpecError
Raised when an unknown variant occurs in a spec. bestieโฆ who is she? never seen her in my life ๐
exception spack.variant.UnsatisfiableVariantSpecError(provided, required)
Bases: UnsatisfiableSpecError
Raised when a spec variant conflicts with package constraints. incompatible astrological signs, basically โจ
class spack.variant.Value(value, when)
Bases: object
A conditional value that might be used in variants. "might" doing a LOT of heavy lifting in that sentence, love that for her ๐ฆ
class spack.variant.Variant(name, default, description, values=(True, False), multi=False, validator=None, sticky=False)
Bases: object
Represents a variant in a package, exactly as declared in the variant directive. the blueprint, the origin story, the main character ๐
allowed_values (property) โ returns a string representation of the allowed values, for printing purposes.
- Returns: representation of the allowed values
- Return type:
str
make_default() โ a factory that creates a variant holding the default value.
- Returns: instance of the proper variant
- Return type:
MultiValuedVariantorSingleValuedVariantorBoolValuedVariant
make_variant(value) โ a factory that creates a variant holding the value passed in as a parameter.
- Parameters:
valueโ value that will be hold by the variant - Returns: instance of the proper variant
- Return type:
MultiValuedVariantorSingleValuedVariantorBoolValuedVariant
validate_or_raise(vspec, pkg_cls=None) โ validate a variant spec against this package variant, and raise an exception if any error is found. no soft launches here, it's raising on sight ๐ฅ
- Parameters:
vspec(Variant) โ instance to be validatedpkg_cls(spack.package_base.PackageBase) โ the package class that required the validation, if available
- Raises:
InconsistentValidationErrorโ ifvspec.name != self.nameMultipleValuesInExclusiveVariantErrorโ ifvspechas multiple values butself.multi == FalseInvalidVariantValueErrorโ ifvspec.valuecontains invalid values
variant_cls (property) โ the proper variant class to be used for this configuration.
class spack.variant.VariantMap(spec)
Bases: HashableMap
A map containing variant instances. New values can be added only if the key is not already present โ no take-backs, no overwrites ๐
concrete (property) โ returns True if the spec is concrete in terms of variants.
- Returns:
TrueorFalse - Return type:
bool
constrain(other) โ adds all variants in other that aren't in self to self. Also constrains all multi-valued variants that are already present. Returns True if self changed, False otherwise.
- Parameters:
other(VariantMap) โ instance against which we constrainself - Returns:
TrueorFalse - Return type:
bool
copy() โ return an instance of VariantMap equivalent to self.
- Returns: a copy of
self - Return type:
VariantMap
dict
intersects(other)
satisfies(other)
substitute(vspec) โ substitutes the entry under vspec.name with vspec.
- Parameters:
vspecโ variant spec to be substituted
spack.variant.any_combination_of(*values)
A multi-valued variant that allows any combination of the specified values, and also lets the user specify 'none' (as a string) to choose none of them.
It is up to the package implementation to handle the value 'none' specially โ if at all. so like, don't assume, ok? ๐
- Parameters:
*valuesโ allowed variant values - Returns: a properly initialized instance of
DisjointSetsOfValues
spack.variant.auto_or_any_combination_of(*values)
A multi-valued variant that allows any combination of a set of values (but not the empty set) โ or 'auto'.
- Parameters:
*valuesโ allowed variant values - Returns: a properly initialized instance of
DisjointSetsOfValues
spack.variant.conditional(*values, **kwargs)
Conditional values that can be used in variant declarations. that's it, that's the whole vibe โจ
and bestie we are not done yet โ the variant saga continues ๐
spack.variant.disjoint_sets(*sets)โ this one gives you a multi-valued variant that lets you pick any combo from one of multiple disjoint sets of values, and it also lets the user pass'none'(as a string) to choose, like, none of them ๐- Lowkey it's completely up to the package implementation to handle the value
'none'specially โ if at all. No cap. ๐ - Parameters:
*sets - Returns: a properly initialized instance of
DisjointSetsOfValuesโจ
- Lowkey it's completely up to the package implementation to handle the value
spack.variant.implicit_variant_conversion(method)โ convertsothertotype(self)and then callsmethod(self, other)๐ฆ- Parameters:
methodโ any predicate method that takes another variant as an argument - Returns: decorated method
- Parameters:
spack.variant.substitute_abstract_variants(spec)โ uses the info chilling inspec.packageto turn any variant that needs it into aSingleValuedVariant๐ซถ- Real talk: this method is best effort. Every variant that can be substituted will get substituted before any error is raised. So it does its little bestie best first, then panics ๐ญ
- Parameters:
specโ spec on which to operate the substitution
๐ง spack.verify module
class spack.verify.VerificationResultsโ Bases:objectadd_error(path, field)has_errors()json_string()
spack.verify.check_entry(path, data)spack.verify.check_file_manifest(filename)spack.verify.check_spec_manifest(spec)spack.verify.compute_hash(path: str, block_size: int = 1048576) -> strspack.verify.create_manifest_entry(path: str) -> Dict[str, Any]spack.verify.write_manifest(spec)
๐ LLNL PACKAGE
๐ฆ Subpackages
๐ llnl.util package
Subpackages โจ
๐ llnl.util.tty package
class llnl.util.tty.SuppressOutput(msg_enabled=True, warn_enabled=True, error_enabled=True)โ Bases:object. It's the class for disabling output in a scope using thewithkeyword ๐ซถllnl.util.tty.debug(message, *args, **kwargs)llnl.util.tty.debug_level()llnl.util.tty.die(message, *args, **kwargs) -> NoReturnโ yeah, it does not come back ๐llnl.util.tty.error(message, *args, **kwargs)llnl.util.tty.error_enabled()llnl.util.tty.get_timestamp(force=False)โ get a string timestamp โฐllnl.util.tty.get_yes_or_no(prompt, **kwargs)llnl.util.tty.hline(label=None, **kwargs)โ draw a labeled horizontal line โ- Keyword Arguments
char(str) โ char to draw the line with. Default is'-'max_width(int) โ maximum width of the line. Default is 64 chars.
- Keyword Arguments
llnl.util.tty.info(message, *args, **kwargs)llnl.util.tty.is_debug(level=1)llnl.util.tty.is_verbose()llnl.util.tty.msg(message: Any, *args: Any, newline: bool = True) -> Nonellnl.util.tty.msg_enabled()llnl.util.tty.output_filter(filter_fn)โ a context manager that applies a filter to all output ๐ฆllnl.util.tty.process_stacktrace(countback)โ gives you the file and line framecountbackframes from the bottom ๐ฅllnl.util.tty.set_debug(level=0)llnl.util.tty.set_error_enabled(flag)llnl.util.tty.set_msg_enabled(flag)llnl.util.tty.set_stacktrace(flag)llnl.util.tty.set_timestamp(flag)llnl.util.tty.set_verbose(flag)llnl.util.tty.set_warn_enabled(flag)llnl.util.tty.show_pid()llnl.util.tty.terminal_size()โ gets the dimensions of the console:(rows, cols)๐llnl.util.tty.verbose(message, *args, **kwargs)llnl.util.tty.warn(message, *args, **kwargs)llnl.util.tty.warn_enabled()
Submodules ๐ง
๐ธ llnl.util.tty.colify module
Routines for printing columnar output. See colify() for more information. ๐
class llnl.util.tty.colify.ColumnConfig(cols)โ Bases:objectllnl.util.tty.colify.colified(elts: List[Any], cols: int = 0, indent: int = 0, padding: int = 2, tty: bool | None = None, method: str = 'variable', console_cols: int | None = None)โ invokes thecolify()function but hands the result back to you as a string instead of writing it to an output string ๐llnl.util.tty.colify.colify(elts: List[Any], cols: int = 0, output: IO | None = None, indent: int = 0, padding: int = 2, tty: bool | None = None, method: str = 'variable', console_cols: int | None = None)โ takes a list of elements and finds a good columnization of them, kinda like how gnulsdoes it. It supports both uniform-width and variable-width (tighter) columns ๐ฅ- If
eltsis not a list of strings, each element is first converted usingstr(). - Keyword Arguments
outputโ a file object to write to. Default issys.stdoutindentโ optionally indent all columns by some number of spacespaddingโ spaces between columns. Default is2widthโ width of the output. Default is80if tty not detectedcolsโ force number of columns. Default is to size to terminal, or single-column if no ttyttyโ whether to attempt to write to a tty. Default is to autodetect a tty. Set toFalseto force single-column outputmethodโ method to use to fit columns. Options arevariableoruniform. Variable-width columns are tighter, uniform columns are all the same width and fit less data on the screenconsole_colsโ number of columns on this console (default: autodetect)
- If
llnl.util.tty.colify.colify_table(table: List[List[Any]], output: IO | None = None, indent: int = 0, padding: int = 2, console_cols: int | None = None)โ thecolify()version for data expressed in rows (a list of lists) ๐ซถ- Same as regular
colifybut:
- This takes a list of lists, where each sub-list must be the same length, and each one is interpreted as a row in a table. Regular
colifydisplays a sequential list of values in columns. - Regular
colifywill always print with 1 column when the output is not a tty. This will always print with the same dimensions of thetableargument.
- Same as regular
llnl.util.tty.colify.config_uniform_cols(elts, console_width, padding, cols=0)โ the uniform-width column fitting algorithm โจ- Determines the longest element in the list, and determines how many columns of that width will fit on screen. Returns a corresponding column config.
llnl.util.tty.colify.config_variable_cols(elts, console_width, padding, cols=0)โ the variable-width column fitting algorithm ๐- This function determines the most columns that can fit in the screen width. Unlike uniform fitting โ where all columns take the width of the longest element in the list โ each column takes the width of its own longest element. This packs elements more efficiently on screen.
- If
colsis nonzero, force
๐ llnl.util.tty.color module
This file implements an expression syntax, similar to printf, for adding ANSI colors to text. ๐จ
See colorize(), cwrite(), and cprint() for routines that can generate colored output.
colorize will take a string and replace all color expressions with ANSI control codes. If the isatty keyword arg is set to False, then the color expressions get converted to null strings, and the returned string will have no color ๐
cwrite and cprint are equivalent to write() and print() calls in python, but they colorize their output. If the stream argument isn't supplied, they write to sys.stdout.
Here are some example color expressions: ๐ฆ
@rโ turn on red coloring@Rโ turn on bright red coloring@*{foo}โ boldfoo, but don't change text color@_{bar}โ underlinebar, but don't change text color@*bโ turn on bold, blue text@_Bโ turn on bright blue text with an underline@.โ revert to plain formatting@*g{green}โ print out'green'in bold, green text, then reset to plain.@*ggreen@.โ print out'green'in bold, green text, then reset to plain.
The syntax is literally this:
โโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ color-expr โ '@' [style] color-code '{' โ
โ โ text '}' | '@.' | '@@' โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ style โ '*' | '_' โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ color-code โ [krgybmcwKRGYBMCW] โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ text โ .* โ
โโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโSo @ is the little sparkle that says "a color expression starts here โจ". After it you can drop an optional * or _ to say whether the font should be bold or _underlined_ โ and if you don't provide either, the text stays plain, no cap. Then comes an optional color code, which can be [krgybmcw] or [KRGYBMCW], and the letters map like this: black(k), red(r), green(g), yellow(y), blue(b), magenta(m), cyan(c), and white(w). Lowercase = normal ANSI colors, capitals = bright ANSI colors (yes, uppercase is the loud version ๐
).
Finally the color expression can be followed by text wrapped in {}. If the braces are there, only the text inside them gets colored. If the braces are NOT there, then only the control codes to enable the color get output. You can reset the console back to plain text later with @..
Need to output a literal @? Use @@. Need to output a } inside braces? Use }}. That's the whole escape-the-escape situation, iykyk ๐ฆ.
exception llnl.util.tty.color.ColorParseError(message)โ Bases:Exception. Thrown when a color format fails to parse, i.e. your color expression said something it couldn't understand ๐ญ.class llnl.util.tty.color.ColorStream(stream, color=None)โ Bases:object. It's a stream wrapper situation, and it's got:write(string, **kwargs)writelines(sequence, **kwargs)
llnl.util.tty.color.cescape(string: str) -> strโ Escapes the special characters you need for color codes. It replaces these symbols with their equivalent literal forms:
โโโโโฌโโโโโ
โ @ โ @@ โ
โโโโโผโโโโโค
โ } โ }} โ
โโโโโดโโโโโParameters
string(str) โ the string to escape
Returns โ the string with color codes escaped
Return type โ (str)
llnl.util.tty.color.cextra(string)โ Gives you the length of the extra color characters in a string.llnl.util.tty.color.clen(string)โ Returns the length of a string, excluding ansi color sequences. So you measure the actual visible text, not the invisible color magic ๐.llnl.util.tty.color.color_when(value)โ A context manager to temporarily use a particular color setting.llnl.util.tty.color.colorize(string: str, color: bool | None = None, enclose: bool = False, zsh: bool = False) -> strโ Replaces all color expressions in a string with ANSI control codes. Obsessed.
Parameters
stringโ The string to replace
Returns โ The filtered string
Keyword Arguments
colorโ IfFalse, output will be plain text without control codes, for output to non-console devices (default: automatically choose color or not)encloseโ IfTrue, enclose ansi color sequences with square brackets to prevent misestimation of terminal widthzshโ IfTrue, use zsh ansi codes instead of bash ones (for variables likePS1)
llnl.util.tty.color.cprint(string, stream=None, color=None)โ Same ascwrite, but writes a trailing newline to the stream.llnl.util.tty.color.cwrite(string, stream=None, color=None)โ Replaces all color expressions instringwith ANSI control codes and writes the result to the stream. IfcolorisFalse, it writes plain text with no color. IfTrue, it always writes colored output. If not supplied, then it gets set based onstream.isatty().llnl.util.tty.color.get_color_when()โ Returns whether commands should print color or not.llnl.util.tty.color.set_color_when(when)โ Sets when color should be applied. The options are:Trueor'always': always print colorFalseor'never': never print colorNoneor'auto': only print color ifsys.stdoutis a tty.
llnl.util.tty.color.try_enable_terminal_color_on_windows()โ Turns on coloring in Windows terminal by enabling VTP in Windows consoles (CMD/PWSH/CONHOST). The method is based on the link below:https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences#example-of-enabling-virtual-terminal-processing
Note: it's a no-op on non windows platforms. So on everything else, it's just sitting there looking pretty ๐
๐ชต llnl.util.tty.log module
Utility classes for logging the output of blocks of code. Basically it's for capturing the vibes ๐ผ.
class llnl.util.tty.log.FileWrapper(file_like)โ Bases:object. Represents a file. Can be an open stream, a path to a file (not opened yet), or neither. When unwrapped, it returns an open file (or file-like) object. Methods:close()unwrap()
class llnl.util.tty.log.MultiProcessFd(fd)โ Bases:object. Returns an object which stores a file descriptor and can be passed as an argument to a function run withmultiprocessing.Process, such that the file descriptor is available in the subprocess. Nice. Members:close()- property
fd
class llnl.util.tty.log.StreamWrapper(sys_attr)โ Bases:object. A wrapper class to handle redirection of io streams.close()โ Redirect back to the original system stream, and close streamflush()redirect_stream(to_fd)โ Redirect stdout to the given file descriptor.
class llnl.util.tty.log.Unbuffered(stream)โ Bases:object. Wrapper for Python streams that forces them to be unbuffered. This is implemented by forcing a flush after each write. Zero patience, maximum freshness ๐ .write(data)writelines(datas)
llnl.util.tty.log.close_connection_and_file(multiprocess_fd, file)โ that's the whole signature, and the original doesn't add any more detail, so that's what we've got ๐ซถllnl.util.tty.log.ignore_signal(signum)โ Context manager to temporarily ignore a signal.class llnl.util.tty.log.keyboard_input(stream)โ Bases:object. Context manager to disable line editing and echoing. Use this withsys.stdinfor keyboard input, like:
with keyboard_input(sys.stdin) as kb:
while True:
kb.check_fg_bg()
r, w, x = select.select([sys.stdin], [], [])
# ... do something with keypresses ...The keyboard_input context manager disables canonical (line-based) input and echoing, so that keypresses are available on the stream immediately, and they are not printed to the terminal. Typically, standard input is line-buffered, which means keypresses won't be sent until the user hits return. In this mode, a user can hit, e.g., v, and it will be read on the other end of the pipe immediately but not printed.
The handler takes care to ensure that terminal changes only take effect when the calling process is in the foreground. If the process is backgrounded, canonical mode and echo are re-enabled. They are disabled again when the calling process comes back to the foreground.
This context manager works through a single signal handler for SIGTSTP, along with a polling routine called check_fg_bg(). Here are the relevant states, transitions, and POSIX signals:
[Running] -------- Ctrl-Z sends SIGTSTP ------------.
[ in FG ] <------- fg sends SIGCONT --------------. |
^ | |
| fg (no signal) | |
| | v
[Running] <------- bg sends SIGCONT ---------- [Stopped]
[ in BG ] [ in BG ]We handle all transitions except for SIGTSTP generated by Ctrl+Z by periodically calling check_fg_bg(). This routine notices if we are in the background with canonical mode or echo disabled, or if we are in the foreground without canonical disabled and echo enabled, and it fixes the terminal settings in response.
check_fg_bg() works except for when the process is stopped with SIGTSTP. We cannot rely on a periodic timer in this case, as it may not run before the process stops. We therefore restore terminal settings in the SIGTSTP handler.
Additional notes, and bestie these are important:
- We mostly use polling here instead of a
SIGARLMtimer or a thread. This is to avoid the complexities of many interrupts, which seem to make system calls (like I/O) unreliable in older Python versions (2.6 and 2.7). See these issues for details:https://www.python.org/dev/peps/pep-0475/https://bugs.python.org/issue8354There are essentially too many ways for asynchronous signals to go wrong if we also have to support older Python versions, so we opt not to use them.
SIGSTOPcan stop a process (in the foreground or background), but it can't be caught. Because of this, we can't fix any terminal settings onSIGSTOP, and the terminal will be left withICANONandECHOdisabled until it resumes execution.- Technically, a process could be sent
SIGTSTPwhile running in the foreground, without the shell backgrounding that process. This doesn't happen in practice, and we assume thatSIGTSTPalways means that defaults should be restored. - We rely on termios support. Without it, or if the stream isn't a TTY,
keyboard_inputhas no effect.
And yes, there's:
check_fg_bg()
llnl.util.tty.log.log_output(*args, **kwargs)โ Context manager that logs its output to a file. In the simplest case, the usage looks like this:
with log_output('logfile.txt'):
# do things ... output will be loggedThat's the simplest-case energy, and honestly it's serving exactly what it needs to ๐๐
Any output that happens inside the with block gets redirected to logfile.txt. Fr fr. And if you also want it echoed out to stdout at the same time? Pass echo:
with log_output('logfile.txt', echo=True):
# do things ... output will be logged and printed outโจ Heads up bestie: the next bit is Unix only. On Windows it's a total no-op, it just sits there doing nothing, which honestly? iconic of it. And if you only want to echo some stuff from the parent, that's what force_echo is for:
with log_output('logfile.txt', echo=False) as logger:
# do things ... output will be logged
with logger.force_echo():
# things here will be echoed *and* loggedSee the individual log classes for more info ๐
Fun fact: this method is actually a factory โ it serves up a per-platform (unix vs windows) log_output class. Like a little class vending machine ๐
class llnl.util.tty.log.nixlog(file_like=None, echo=False, debug=0, buffer=False, env=None, filter_fn=None)
Bases: object
Okay so under the hood? We spawn a daemon and set up a pipe between this process and the daemon. The daemon writes our output to both the file and to stdout (if echoing is on). The parent process can talk to the daemon to tell it when and when not to echo โ that's literally what force_echo does ๐
You can also enable/disable echoing by typing 'v'.
We try to use OS-level file descriptors to do the redirection, but if stdout or stderr has been set to some Python-level file object, we use Python-level redirection instead. Why do we care? Because this lets the redirection work inside test frameworks like nose and pytest ๐ง
force_echo() Context manager to force local echo, even if echo is off.
llnl.util.tty.log.replace_environment(env)
Replace the current environment (os.environ) with env.
And if env is empty (or None)? It unsets all current environment variables. Like, gone. Every last one ๐ซถ
class llnl.util.tty.log.winlog(file_like=None, echo=False, debug=0, buffer=False, env=None, filter_fn=None)
Bases: object
Similar to nixlog, except the underlying functionality is ported to support Windows ๐ช
One thing it does not support: using 'v' to toggle, the way nixlog does. So don't go looking for that key, bestie ๐ญ
force_echo() Context manager to force local echo, even if echo is off.
๐ฆ llnl.util.tty.pty module ๐ฆ
The pty module handles pseudo-terminals. v niche, v important.
Currently, the infrastructure here is only used to test llnl.util.tty.log.
If this is used outside a testing environment, we will want to reconsider things like timeouts in ProcessController.wait(), which are set to get tests done quickly, not to avoid high CPU usage. โจ
Note: The functionality in this module is unsupported on Windows ๐
class llnl.util.tty.pty.ProcessController(pid, controller_fd, timeout=1, sleep_time=0.1, debug=False)
Bases: object
A wrapper around some fundamental process control operations.
It lets one process (the controller) drive another (the minion) similar to the way a shell would โ by sending signals and I/O. Main character energy for real ๐
background() True if pgid is in a background pgroup of controller_fd's tty.
bg()
cont()
fg()
get_canon_echo_attrs() Get echo and canon attributes of the terminal of controller_fd.
horizontal_line(name) Labled horizontal line for debugging. (Yes, that's the original spelling, we're keeping it, character for character ๐
)
input_on() True if keyboard input is enabled on the controller_fd pty.
proc_status()
status() Print a debug message with status info for the minion.
tstp() Send SIGTSTP to the controlled process.
wait(condition)
wait_disabled()
wait_disabled_fg()
wait_enabled()
wait_running()
wait_stopped()
write(byte_string)
class llnl.util.tty.pty.PseudoShell(controller_function, minion_function)
Bases: object
Sets up controller and minion processes with a PTY.
You can create a PseudoShell if you want to test how some function responds to terminal input. This is a pseudo-shell from a job control perspective โ controller_function and minion_function are set up with a pseudoterminal (pty) so that the controller can drive the minion through process control signals and I/O ๐ธ
The two functions should have signatures like this:
def controller_function(proc, ctl, **kwargs)
def minion_function(**kwargs)controller_function is spawned in its own process and passed three arguments:
procโ themultiprocessing.Processobject representing the minionctlโ aProcessControllerobject tied to the minionkwargsโ keyword arguments passed fromPseudoShell.start()
minion_function is only passed kwargs delegated from PseudoShell.start().
The ctl.controller_fd will have its controller_fd connected to sys.stdin in the minion process. Both processes will share the same sys.stdout and sys.stderr as the process instantiating PseudoShell ๐
Here are the relationships between the processes created โ yes, an actual diagram, we're keeping it exactly:
._________________________________________________________.
| Minion Process | pid 2
| - runs minion_function | pgroup 2
|_________________________________________________________| session 1
^
| create process with controller_fd connected to stdin
| stdout, stderr are the same as caller
._________________________________________________________.
| Controller Process | pid 1
| - runs controller_function | pgroup 1
| - uses ProcessController and controller_fd to | session 1
| control minion |
|_________________________________________________________|
^
| create process
| stdin, stdout, stderr are the same as caller
._________________________________________________________.
| Caller | pid 0
| - Constructs, starts, joins PseudoShell | pgroup 0
| - provides controller_function, minion_function | session 0
|_________________________________________________________|join() Wait for the minion process to finish, and return its exit code.
start(**kwargs) Start the controller and minion processes.
Parameters
kwargs(dict) โ arbitrary keyword arguments that will be passed to controller and minion functions
The controller process will create the minion, then call controller_function. The minion process will call minion_function ๐
๐ Submodules ๐
llnl.util.argparsewriter module
class llnl.util.argparsewriter.ArgparseRstWriter(prog: str, out: ~typing.IO = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, aliases: bool = False, rst_levels: ~typing.Sequence[str] = ['=', '-', '^', '~', ':', ''])`
Bases: ArgparseWriter
Write argparse output as rst sections. It's giving documentation-girlie โจ
begin_command(prog: str) -> str Text to print before a command.
Parameters
progโ Program name.
Returns
Text before a command.
begin_optionals() -> str Text to print before optional arguments.
Returns
Optional arguments header.
begin_positionals() -> str Text to print before positional arguments.
Returns
Positional arguments header.
begin_subcommands(subcommands: List[Tuple[ArgumentParser, str, str]]) -> str Table with links to other subcommands.
Parameters
subcommandsโ List of subcommands.
Returns
Subcommand linking text.
description(description: str) -> str Description of a command.
Parameters
descriptionโ Command description.
Returns
Description of a command.
end_optionals() -> str Text to print after optional arguments.
Returns
Optional arguments footer.
end_positionals() -> str Text to print after positional arguments.
Returns
Positional arguments footer.
format(cmd: Command) -> str Return the string representation of a single node in the parser tree.
Parameters
cmdโ Parsed information about a command or subcommand.
Returns
String representation of a node.
optional(opts: str, help: str) -> str Description of an optional argument.
Parameters
optsโ Optional argument.helpโ Help text.
Returns
Optional argument description.
positional(name: str, help: str) -> str Description of a positional argument.
Parameters
nameโ Argument name.helpโ Help text.
Returns
Positional argument description.
usage(usage: str) -> str Example usage of a command.
Parameters
usageโ Command usage.
Returns
Usage of a command.
class llnl.util.argparsewriter.ArgparseWriter(prog: str, out: ~typing.IO = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, aliases: bool = False)
Bases: HelpFormatter, ABC
Analyze an argparse ArgumentParser for easy generation of help ๐
abstractmethod format(cmd: Command) -> str Return the string representation of a single node in the parser tree.
Returns โ the string version of this subcommand, i.e. how it's gonna show up on screen. That's the whole point, bestie ๐
๐ง parse(parser: ArgumentParser, prog: str) -> Command
Takes the parser object and hands you back the relevant components of it. It's giving "translator" energy.
Parameters
parserโ the command parser in question.progโ program name.
Returns โ info about the command, fresh out of the parser. ๐ซถ
๐ write(parser: ArgumentParser) -> None
Writes out all the deets about an ArgumentParser. Nothing returned, it just prints. Lowkey the main character of this whole class ngl.
Parameters
parserโ the command parser.
๐ฆ class llnl.util.argparsewriter.Command
Command(prog: str, description: str | None, usage: str,
positionals: List[Tuple[str, Iterable[Any] | None, int | str | None, str]],
optionals: List[Tuple[Sequence[str], List[str], str, int | str | None, str]],
subcommands: List[Tuple[ArgumentParser, str, str]])Bases: object
Okay so this is the parsed representation of a command from argparse โ like, one single command pulled out of an argparse parser โจ
ArgparseWriter is the one that builds these bad boys and returns them from parse(), and then it passes one of these to every single call to format() so we can take an action for one command at a time. It's giving assembly line, but make it cute ๐
๐ธ llnl.util.filesystem module
โญ class llnl.util.filesystem.BaseDirectoryVisitor
Bases: object
The base class and the interface for visit_directory_tree(). Think of it as the blueprint โ you don't use it raw, you build on top of it ๐
after_visit_dir(root: str, rel_path: str, depth: int) -> None
Called after the recursion into rel_path has finished. Heads up: this function is not called when rel_path was never recursed into in the first place. No cap.
Parameters
rootโ root directoryrel_pathโ relative path to the current directory fromrootdepthโ depth of the current directory from the root directory
after_visit_symlinked_dir(root: str, rel_path: str, depth: int) -> None
Same vibe as above but for symlinks โ called after recursion into rel_path is done. And again: it is not called when rel_path wasn't recursed into.
Parameters
rootโ root directoryrel_pathโ relative path to the current symlink fromrootdepthโ depth of the current symlink from the root directory
before_visit_dir(root: str, rel_path: str, depth: int) -> bool
This is your vibe check โ
. Return True from this function and we recurse into the directory at os.path.join(root, rel_path). Return False and we do not go any deeper. Your call, bestie.
Parameters
rootโ root directoryrel_pathโ relative path to the current directory fromrootdepthโ depth of the current directory from the root directory
Returns โ True when the directory should be recursed into, False when not.
Return type โ bool
before_visit_symlinked_dir(root: str, rel_path: str, depth: int) -> bool
Return True to recurse into the symlinked directory, False to not do that.
โ ๏ธ Note: rel_path is the path to the symlink itself. And fr, following symlinked directories blindly can cause infinite recursion because of cycles โ you've been warned, don't do it ๐
Parameters
rootโ root directoryrel_pathโ relative path to the current symlink fromrootdepthโ depth of the current symlink from the root directory
Returns โ True when the directory should be recursed into, False when not.
Return type โ bool
visit_file(root: str, rel_path: str, depth: int) -> None
Handles the non-symlink file sitting at os.path.join(root, rel_path). Regular file, regular treatment ๐
Parameters
rootโ root directoryrel_pathโ relative path to the current file fromrootdepth(int) โ depth of the current file from the root directory
visit_symlinked_file(root: str, rel_path: str, depth) -> None
Handles the symlink pointing at a file at os.path.join(root, rel_path).
โ ๏ธ Note: rel_path is where the symlink lives โ not where it's pointing. And the symlink might be dangling, so keep that in mind ๐
Parameters
rootโ root directoryrel_pathโ relative path to the current symlink fromrootdepthโ depth of the current symlink from the root directory
๐ class llnl.util.filesystem.FileFilter(*filenames)
Bases: object
The convenience class for when you gotta call filter_file a whole lot of times. It's giving bulk order ๐๏ธ
filter(...)
filter(regex: str, repl: str | Callable[[Match], str], string: bool = False,
backup: bool = False, ignore_absent: bool = False,
start_at: str | None = None, stop_at: str | None = None) -> None๐ class llnl.util.filesystem.FileList(files)
Bases: Sequence
A sequence of absolute paths to files. Also comes with a few convenience methods so you can manipulate file paths without losing your mind ๐ซถ
๐ property basenames
Stable de-duplication of the base-names in the list. Stable as in the order doesn't wiggle around on you, iconic behaviour honestly.
>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir3/liba.a'])
>>> l.basenames
['liba.a', 'libb.a']
>>> h = HeaderList(['/dir1/a.h', '/dir2/b.h', '/dir3/a.h'])
>>> h.basenames
['a.h', 'b.h']Returns โ a list of base-names.
Return type โ list
๐ property directories
Stable de-duplication of the directories where the files actually live.
>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/libc.a'])
>>> l.directories
['/dir1', '/dir2']
>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.directories
['/dir1', '/dir2']Returns โ a list of directories.
Return type โ list
joined(separator=' ')
๐ง class llnl.util.filesystem.HeaderList(files)
Bases: FileList
A sequence of absolute paths to headers โจ Comes with convenience methods to mess with header paths and grab the compiler flags or names you keep needing. It's giving "grab and go".
add_macro(macro)
Adds a macro definition.
Parameters
macro(str) โ the macro to add
๐ property cpp_flags
Include flags plus macro definitions. The combo meal ๐ฅ
>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.cpp_flags
'-I/dir1 -I/dir2'
>>> h.add_macro('-DBOOST_DYN_LINK')
>>> h.cpp_flags
'-I/dir1 -I/dir2 -DBOOST_DYN_LINK'Returns โ a joined list of include flags and macro definitions.
Return type โ str
๐ property directories
The directories to be searched for header files.
๐ property headers
Stable de-duplication of the headers. Same header twice? Not on her watch ๐
Returns โ a list of header files.
Return type โ list
๐ property include_flags
Include flags, plain and simple.
>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.include_flags
'-I/dir1 -I/dir2'Returns โ a joined list of include flags.
Return type โ str
include_regex
include_regex = re.compile('(.*?)(\\binclude\\b)(.*)')๐ property macro_definitions
Macro definitions, all joined up.
>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.add_macro('-DBOOST_LIB_NAME=boost_regex')
>>> h.add_macro('-DBOOST_DYN_LINK')
>>> h.macro_definitions
'-DBOOST_LIB_NAME=boost_regex -DBOOST_DYN_LINK'Returns โ a joined list of macro definitions.
Return type โ str
๐ property names
Stable de-duplication of the header names in the list, with the extensions stripped off. Clean girl aesthetic ๐ธ
>>> h = HeaderList(['/dir1/a.h', '/dir2/b.h', '/dir3/a.h'])
>>> h.names
['a', 'b']Returns โ a list of files without extensions.
Return type โ list
๐ class llnl.util.filesystem.LibraryList(files)
Bases: FileList
A sequence of absolute paths to libraries ๐ Comes with convenience methods to manipulate library paths and pull the compiler flags or names you actually want.
๐ property ld_flags
Search flags plus link flags. Both worlds, one property, periodt.
>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/liba.so'])
>>> l.ld_flags
'-L/dir1 -L/dir2 -la -lb'Returns โ a joined list of search flags and link flags.
Return type โ str
๐ property libraries
Stable de-duplication of the library files.
Returns โ a list of library files.
Return type โ list
๐ property link_flags
The link flags for the libraries.
>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/liba.so'])
>>> l.link_flags
'-la -lb'Returns โ a joined list of link flags.
Return type: str
๐ท property names
Stable de-duplication of library names in the list โ like, it keeps one of each name, in order, no duplicates, no chaos. ๐
>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir3/liba.so'])
>>> l.names
['a', 'b']Returns: a list of library names Return type: list
๐ท property search_flags
The search flags for the libraries, all joined up and ready to go โจ
>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/liba.so'])
>>> l.search_flags
'-L/dir1 -L/dir2'Returns: a joined list of search flags Return type: str
๐ฆ llnl.util.filesystem โ the filesystem besties
llnl.util.filesystem.ancestor(dir, n=1)
Get the *n*th ancestor of a directory. That's it, that's the whole vibe. ๐ง
llnl.util.filesystem.can_access(file_name)
True if we have read/write access to the file. Simple as. ๐ฅ
llnl.util.filesystem.change_sed_delimiter(old_delim, new_delim, *filenames)
Find all sed search/replace commands and change the delimiter.
So like, if the file contains seds that look like 's///', you can call change_sed_delimiter('/', '@', file) to change the delimiter to '@'.
โ ๏ธ Heads up, bestie: this routine will fail if the delimiter is'or". Handling those is left for future work โ not me, not today. ๐ญ
Parameters
- old_delim (
str) โ the delimiter to search for - new_delim (
str) โ the delimiter to replace with - *filenames โ one or more files to search and replace
llnl.util.filesystem.chgrp(path, group, follow_symlinks=True)
Implement the bash chgrp function on a single path. Bash energy, one path at a time. ๐ซถ
llnl.util.filesystem.chmod_x(entry, perms)
Implements chmod, treating all executable bits as set using the chmod utility's +X option. Iconic behaviour honestly.
llnl.util.filesystem.copy(src, dest, _permissions=False)
Copy the file(s) src to the file or directory dest.
If dest specifies a directory, the file will be copied into dest using the base filename from src.
src may contain glob characters. ๐
Parameters
- src (
str) โ the file(s) to copy - dest (
str) โ the destination file or directory - _permissions (
bool) โ for internal use only
Raises
IOErrorโ if src does not match any files or directoriesValueErrorโ if src matches multiple files but dest is not a directory
llnl.util.filesystem.copy_mode(src, dest)
Set the mode of dest to that of src unless it is a link. Copy the vibe, not the link. ๐
llnl.util.filesystem.copy_tree(src: str, dest: str, symlinks: bool = True, allow_broken_symlinks: bool = True, ignore: Callable[[str], bool] | None = None, _permissions: bool = False)
Recursively copy an entire directory tree rooted at src. Big main-character copy energy. ๐
If the destination directory dest does not already exist, it will be created as well as missing parent directories.
src may contain glob characters.
If symlinks is true, symbolic links in the source tree are represented as symbolic links in the new tree and the metadata of the original links will be copied as far as the platform allows; if false, the contents and metadata of the linked files are copied to the new tree.
If ignore is set, then each path relative to src will be passed to this function; the function returns whether that path should be skipped. Say less. ๐
Parameters
- src (
str) โ the directory to copy - dest (
str) โ the destination directory - symlinks (
bool) โ whether or not to preserve symlinks - allow_broken_symlinks (
bool) โ whether or not to allow broken (dangling) symlinks. On Windows, setting this toTruewill raise an exception. Defaults to true on unix. - ignore (
Callable) โ function indicating which files to ignore - _permissions (
bool) โ for internal use only
Raises
IOErrorโ if src does not match any files or directoriesValueErrorโ if src is a parent directory of dest
llnl.util.filesystem.filter_file(regex: str, repl: str | Callable[[Match], str], *filenames: str, string: bool = False, backup: bool = False, ignore_absent: bool = False, start_at: str | None = None, stop_at: str | None = None) -> None
Like sed, but uses python regular expressions. Obsessed, honestly. ๐
Filters every line of each file through regex and replaces the file with a filtered version. Preserves mode of filtered files.
As with re.sub, repl can be either a string or a callable. If it is a callable, it is passed the match object and should return a suitable replacement string. If it is a string, it can contain \1, \2, etc. to represent back-substitution as sed would allow.
Parameters
- regex (
str) โ the regular expression to search for - repl (
str) โ the string to replace matches with - *filenames โ one or more files to search and replace
- string (
bool) โ treat regex as a plain string. Default itFalse - backup (
bool) โ make backup file(s) suffixed with~. Default isFalse - ignore_absent (
bool) โ ignore any files that don't exist. Default isFalse - start_at (
str) โ marker used to start applying the replacements. If a text line matches this marker filtering is started at the next line. All contents before the marker and the marker itself are copied verbatim. Default is to start filtering from the first line of the file. - stop_at (
str) โ marker used to stop scanning the file further. If a text line matches this marker filtering is stopped and the rest of the file is copied verbatim. Default is to filter until the end of the file.
llnl.util.filesystem.find(root, files, recursive=True)
Search for files starting from the root directory.
Like GNU/BSD find but written entirely in Python. No cap, pure python slay. ๐โจ
Examples:
$ find /usr -name pythonis equivalent to:
>>> find('/usr', 'python')$ find /usr/local/bin -maxdepth 1 -name pythonis equivalent to:
>>> find('/usr/local/bin', 'python', recursive=False)Accepts any glob characters accepted by fnmatch:
*โ matches everything?โ matches any single character[seq]โ matches any character in seq[!seq]โ matches any character not in seq
Parameters
- root (
str) โ the root directory to start searching from - files (
strorcollections.abc.Sequence) โ library name(s) to search for - recursive (
bool) โ ifFalsesearch only root folder, ifTruedescends top-down from the root. Defaults toTrue.
Returns: the files that have been found Return type: list
llnl.util.filesystem.find_all_headers(root)
Convenience function that returns the list of all headers found in the directory passed as argument. A little grab-bag of headers, we love her. ๐
Parameters
- root (
str) โ directory where to look recursively for header files
Returns: list of all headers found in root and subdirectories.
llnl.util.filesystem.find_headers(headers, root, recursive=False)
Returns an iterable object containing a list of full paths to headers if found.
Accepts any glob characters accepted by fnmatch:
*โ matches everything?โ matches any single character[seq]โ matches any character in seq[!seq]โ matches any character not in seq
Parameters
headers(str or list) โ the header name(s) you're hunting for, bestie ๐ฏroot(str) โ the directory you wanna start the search fromrecursive(bool) โFalsemeans it only peeks in the root folder;Truemeans it descends top-down from the root. Defaults toFalse.
Returns โ the headers that have been found (obsessed with a clean return tbh)
Return type โ HeaderList โจ
๐
llnl.util.filesystem.find_libraries(libraries, root, shared=True, recursive=False, runtime=True)
Returns an iterable of full paths to libraries found in a root dir. Yes, iterable โ go ahead and loop it, bestie ๐ฆ
Accepts any glob characters accepted by fnmatch:
*โ matches everything?โ matches any single character[seq]โ matches any character inseq[!seq]โ matches any character not inseq
Parameters
libraries(str or list) โ library name(s) to search forroot(str) โ the root directory to start searching fromshared(bool) โTruesearches for shared libraries, otherwise it goes for static ones. Defaults toTrue.recursive(bool) โFalsesearches only the root folder,Truedescends top-down from the root. Defaults toFalse.runtime(bool) โ Windows-only option, total no-op everywhere else. If true, it searches for runtime shared libs (.DLL); otherwise it searches for.Libfiles. Ifsharedis false, this has no meaning whatsoever. Defaults toTrue.
Returns โ the libraries that have been found ๐
Return type โ LibraryList
๐ง llnl.util.filesystem.find_system_libraries(libraries, shared=True)
Searches the usual system library locations for libraries. Like, the classic spots, no cap.
Search order is as follows:
/lib64/lib/usr/lib64/usr/lib/usr/local/lib64/usr/local/lib
Accepts any glob characters accepted by fnmatch:
*โ matches everything?โ matches any single character[seq]โ matches any character inseq[!seq]โ matches any character not inseq
Parameters
libraries(str or list) โ library name(s) to search forshared(bool) โTruesearches for shared libraries, otherwise static. Defaults toTrue.
Returns โ the libraries that have been found ๐ธ
Return type โ LibraryList
๐ llnl.util.filesystem.fix_darwin_install_name(path)
Fixes the install name of dynamic libraries on Darwin so it has the full path. Manifesting correct .dylib linkage for you, periodt.
There are two parts of this task:
- Use
install_name('-id', ...)to change the install name of a single lib - Use
install_name('-change', ...)to change the cross linking between libs. The function assumes that all libraries are in one folder and currently won't follow subfolders. Read that again โ no subfolder traversal, it's giving "one folder only" energy.
Parameters
path(str) โ directory in which.dylibfiles are located
๐ llnl.util.filesystem.force_remove(*paths)
Remove files without printing errors. Like rm -f, it does NOT remove directories. Fr fr, files only ๐
llnl.util.filesystem.force_symlink(src, dest)
(no description, it just does the thing โ iconic and mysterious ๐ )
๐ฅ llnl.util.filesystem.getuid() -> str | int
Returns os getuid on non-Windows. On Windows it returns 0 for admin users, and the login string otherwise. This is in line with behavior from get_owner_uid, which always returns the login string on Windows. Consistency slay ๐
๐ llnl.util.filesystem.install(src, dest)
Install the file(s) src to the file or directory dest.
Same as copy() with the addition of setting proper permissions on the installed file. So it's copy's cooler older sister ๐
Parameters
src(str) โ the file(s) to installdest(str) โ the destination file or directory
Raises
IOErrorโ ifsrcdoes not match any files or directoriesValueErrorโ ifsrcmatches multiple files butdestis not a directory
๐ llnl.util.filesystem.install_tree(src, dest, symlinks=True, ignore=None, allow_broken_symlinks=True)
Recursively install an entire directory tree rooted at src. Whole tree, no crumbs left behind ๐ง
Same as copy_tree() with the addition of setting proper permissions on the installed files and directories.
Parameters
src(str) โ the directory to installdest(str) โ the destination directorysymlinks(bool) โ whether or not to preserve symlinksignore(Callable) โ function indicating which files to ignoreallow_broken_symlinks(bool) โ whether or not to allow broken (dangling) symlinks. On Windows, setting this toTruewill raise an exception.
Raises
IOErrorโ ifsrcdoes not match any files or directoriesValueErrorโ ifsrcis a parent directory ofdest
โญ llnl.util.filesystem.is_exe(path)
True if path is an executable file. Simple. Elegant. Main character energy.
llnl.util.filesystem.join_path(prefix, *args)
(no description here either โ it joins a path, you know the vibe ๐)
๐ฆ llnl.util.filesystem.keep_modification_time(*filenames)
Context manager to keep the modification timestamps of the input files. Tolerates and has no effect on non-existent files and files that are deleted by the nested code. It's very chill like that ๐ธ
Parameters
*filenamesโ one or more files that must have their modification timestamps unchanged
llnl.util.filesystem.last_modification_time_recursive(path)
(no description โ recursive, last modification time, you get it ๐ซถ)
๐
llnl.util.filesystem.library_extensions = ['a', 'la', 'so', 'tbd', 'dylib']
This generates the library filenames that may appear on any OS. That's the whole lineup, iconic list, no notes.
๐ llnl.util.filesystem.mkdirp(*paths: str, mode: int | None = None, group: str | int | None = None, default_perms: str | None = None)
Creates a directory, as well as parent directories if needed. It's mkdir -p but make it fancy โจ
Parameters
pathsโ paths to create withmkdirpmodeโ optional permissions to set on the created directory โ use OS default if not providedgroupโ optional group for permissions of the final created directory โ use OS default if not provided. Only used if world write permissions are not setdefault_permsโ one of'parents'or'args'. The default permissions that are set for directories that are not themselves an argument formkdirp.'parents'means intermediate directories get the permissions of their direct parent directory;'args'means intermediate ones get the same permissions specified in the arguments tomkdirpโ default value is'args'. iykyk, this one's subtle.
๐ฏ llnl.util.filesystem.partition_path(path, entry=None)
Split the prefixes of the path at the first occurrence of entry and return a 3-tuple containing a list of the prefixes before the entry, a string of the prefix ending with the entry, and a list of the prefixes after the entry.
If entry is not a node in the path, the result will be the prefix list followed by an empty string and an empty list. So like, the leftovers are just empty vibes ๐
๐ llnl.util.filesystem.prefixes(path)
Returns a list containing the path and its ancestors, top-to-bottom. Ancestry reveal, but for paths ๐
The list for an absolute path will not include an os.sep entry. For example, assuming os.sep is /, given path /ab/cd/efg the resulting paths will be, in order: /ab, /ab/cd, and /ab/cd/efg
The list for a relative path starting ./ will not include ... For example, path ./hi/jkl/mn results in a list with the following paths, in order: ./hi, ./hi/jkl, and ./hi/jkl/mn.
On Windows, paths will be normalized to use / and / will always be used as the separator instead of os.sep. Windows catching strays as always ๐ญ
Parameters
path(str) โ the string we're gonna derive ancestor paths from โจ
Returns โ a list of ancestor paths, in order, ending with path itself ๐ซถ
llnl.util.filesystem.remove_dead_links(root)โ recursively hunts down and yeets every dead link living underroot๐root(str) โ the path where we go looking for dead links
llnl.util.filesystem.remove_directory_contents(dir)โ wipes all the contents of a directory. like, all of it. no survivors ๐งนllnl.util.filesystem.remove_if_dead_link(path)โ removes the argument, but only if it's a dead link ๐ปpath(str) โ the potential dead link
llnl.util.filesystem.remove_linked_tree(path)โ removes a directory and its contents, no cap.
If the directory is a symlink, it follows the link and removes the real directory before removing the link. sneaky but correct ๐
Heads up: this method will force-delete files on Windows ๐ช
path(str) โ the directory to be removed
llnl.util.filesystem.rename(src, dst)โ literally just renames stuff, iconic behaviour, no notes ๐llnl.util.filesystem.set_executable(path)โ makes the thing executable โจllnl.util.filesystem.set_install_permissions(path)โ sets the appropriate permissions on the installed file ๐llnl.util.filesystem.touch(path)โ creates an empty file at the specified path. classic ๐ซถllnl.util.filesystem.touchp(path)โ liketouch, but it also creates any parent directories the file needs. the upgraded version, honestly ๐llnl.util.filesystem.traverse_tree(source_root: str, dest_root: str, rel_path: str = '', *, order: str = 'pre', ignore: Callable[[str], bool] | None = None, follow_nonexisting: bool = True, follow_links: bool = False)โ walks two filesystem trees at the same time, side by side, main character energy ๐
Walks the LinkTree directory in pre or post order. Yields each file in the source directory with a matching path from the dest directory, plus whether that file is a directory. so for this tree:
root/
a/
file1
file2
b/
file3When called on dest, this yields:
('root', 'dest')
('root/a', 'dest/a')
('root/a/file1', 'dest/a/file1')
('root/a/file2', 'dest/a/file2')
('root/b', 'dest/b')
('root/b/file3', 'dest/b/file3')Keyword Arguments
order(str) โ whether to do pre- or post-order traversal. Accepted values are'pre'and'post'โญignore(Callable) โ a function indicating which files to ignore. This will also ignore symlinks if they point to an ignored file (regardless of whether the symlink is explicitly ignored) โ and note this only supports one layer of indirection (i.e. if you havex -> y -> z, andzis ignored butx/yare not, thenywould be ignored but notx๐). To avoid that whole situation, make sure the ignore function also ignores the symlink paths too.follow_nonexisting(bool) โ whether to descend into directories insrcthat don't exist indest. Default isTruefollow_links(bool) โ whether to descend into symlinks insrc
llnl.util.filesystem.unset_executable_mode(path)โ undoes the executable bit, bye ๐llnl.util.filesystem.visit_directory_tree(root: str, visitor: BaseDirectoryVisitor, rel_path: str = '', depth: int = 0)โ recurses the directoryrootdepth-first through a visitor pattern, using the interface fromBaseDirectoryVisitor๐ฆ
Parameters
rootโ path of the directory to recurse intovisitorโ what visitor to userel_pathโ current relative path from the rootdepthโ current depth from the root
llnl.util.filesystem.working_dir(dirname: str, *, create: bool = False)โ a working directory context, vibe check passed โ
๐ธ llnl.util.lang module ๐ธ
class llnl.util.lang.GroupedExceptionForwarder(context: str, handler: GroupedExceptionHandler, base: type)โ Bases:object
A contextmanager that captures exceptions and forwards them to a GroupedExceptionHandler. so organized of it ๐
class llnl.util.lang.GroupedExceptionHandlerโ Bases:object
A generic mechanism to coalesce multiple exceptions and preserve tracebacks. very "i'll handle the group project" of it ngl ๐ฅ
forward(context: str, base: type = <class 'BaseException'>) -> GroupedExceptionForwarderโ returns a contextmanager which extracts tracebacks and prefixes a message โจgrouped_message(with_tracebacks: bool = True) -> strโ prints out an error message coalescing all the forwarded errors ๐ข
class llnl.util.lang.HashableMapโ Bases:MutableMapping
This is a hashable, comparable dictionary. The hash is performed on a tuple of the values in the dictionary. lowkey genius ๐
copy()โ type-agnostic clone method. Preserves subclass type. ๐ซถdict
class llnl.util.lang.ObjectWrapper(wrapped_object)โ Bases:object
Base class that wraps an object. Derived classes can add new behavior while staying undercover ๐ต๏ธโโ๏ธ
This class is modeled after the stackoverflow answer: http://stackoverflow.com/a/1445289/771663
class llnl.util.lang.Singleton(factory)โ Bases:object
Simple wrapper for lazily initialized singleton objects. one of a kind, literally ๐
property instance
class llnl.util.lang.TypedMutableSequence(iterable)โ Bases:MutableSequence
Base class that behaves like a list, just with a different type ๐
Client code can inherit from this base class:
class Foo(TypedMutableSequence):
passand later perform checks based on types:
if isinstance(l, Foo):
# do somethinginsert(index, item)โS.insert(index, value)โ insert value before index
exception llnl.util.lang.UnhashableArgumentsโ Bases:TypeError
Raised when an @memoized function receives unhashable arg or kwarg values ๐
llnl.util.lang.attr_setdefault(obj, name, value)โ likedict.setdefault, but for objects. same energy, different target โจllnl.util.lang.caller_locals()โ this returns the locals of the parent of the caller. Which lets a function insert variables into its caller's scope. Yes, this is some black magic, and yes it's useful for implementing things likedepends_onandprovides๐ฎ no cap, it's giving wizardryclass llnl.util.lang.classproperty(callback)โ Bases:object
Non-data descriptor to evaluate a class-level property. The function that performs the evaluation is injected at creation time and takes an instance (could be None) and an owner (i.e. the class that originated the instance) ๐
llnl.util.lang.decorator_with_or_without_args(decorator)โ lets a decorator be used with or without arguments, e.g.:
# Calls the decorator function some args
@decorator(with, arguments, and=kwargs)or:
# Calls the decorator function with zero arguments
@decoratorflexible queen behaviour ๐
llnl.util.lang.dedupe(sequence, key=None)โ yields a stable de-duplication of a hashable sequence by key. no duplicates allowed, we don't do repeats here ๐ซ
Parameters
sequenceโ hashable sequence to be de-duplicatedkeyโ callable applied on values before the uniqueness test; identity by default.
Returns โ stable de-duplication of the sequence โจ
Examples
Dedupe a list of integers:
[x for x in dedupe([1, 2, 1, 3, 2])] == [1, 2, 3]
[x for x in llnl.util.lang.dedupe([1,-2,1,3,2], key=abs)] == [1, -2, 3]llnl.util.lang.done = <object object>โ sentinel for testing that iterators are done inlazy_lexicographic_ordering๐llnl.util.lang.elide_list(line_list, max_num=10)โ takes a long list and limits it to a smaller number of elements, replacing the intervening elements with'...'. For example:
elide_list([1,2,3,4,5,6], 4)gives:
[1, 2, 3, '...', 6]short and cute, we love a summary ๐ธ
llnl.util.lang.ensure_last(lst, *elements)โ performs a stable partition oflst, ensuring thatelementsoccur at the end oflstin the specified order. Mutateslst. RaisesValueErrorif any elements are not already inlstโ ๏ธ like, they gotta already be in there bestie, no adding new onesllnl.util.lang.enum(**kwargs)โ returns an enum-like class ๐
Parameters โ **kwargs โ explicit dictionary of enums
llnl.util.lang.get_entry_points(*, group: str)โ wrapper forimportlib.metadata.entry_points๐ฆ
Parameters โ group โ entry points to select
Returns โ EntryPoints for the group, or an empty list if unsupported
llnl.util.lang.has_method(cls, name)โ checks whether the class has that method, simple as that โจllnl.util.lang.index_by(objects, *funcs)โ creates a hierarchy of dictionaries by splitting the supplied set of objects on unique values of the supplied functions. honestly iconic ๐ซถ
Values are used as keys. For example, suppose you have four objects with attributes that look like this:
a = Spec("boost %gcc target=skylake")
b = Spec("mrnet %intel target=zen2")
c = Spec("libelf %xlc target=skylake")
d = Spec("libdwarf %intel target=zen2")
list_of_specs = [a,b,c,d]
index1 = index_by(list_of_specs, lambda s: str(s.target),
lambda s: s.compiler)
index2 = index_by(list_of_specs, lambda s: s.compiler)index1 now has two levels of dicts, with lists at the leaves, like this:
{ 'zen2' : { 'gcc' : [a], 'xlc' : [c] },
'skylake' : { 'intel' : [b, d] }
}And index2 is a single level dictionary of lists that looks like this:
{ 'gcc' : [a],
'intel' : [b,d],
'xlc' : [c]
}If any element in funcs is a string, it's treated as the name of an attribute, and acts like getattr(object, name). So shorthand for the above two indexes would be:
index1 = index_by(list_of_specs, 'arch', 'compiler')
index2 = index_by(list_of_specs, 'compiler')You can also index by tuples by passing tuples:
index1 = index_by(list_of_specs, ('target', 'compiler'))Keys in the resulting dict will look like ('gcc', 'skylake') ๐
๐ llnl.util.lang.key_ordering(cls)
okay so picture this: you've got a class, and you want it to Just Compare Correctly without writing six dunder methods by hand ๐ญ this decorator says "say less bestie" and sprinkles on all the rich comparison ops plus __hash__. It assumes your class has a little function called _cmp_key(), and that's the whole deal โ the rich comparisons compare objects using that key, and __hash__ returns the hash of that key. iconic behavior, honestly ๐ซถ
๐จ heads up: if the class already has__eq__,__ne__,__lt__,__le__,__gt__, or__ge__defined, this decorator will overwrite them. no cap, they're gone.
Raises TypeError โ if the class does not have a _cmp_key method ๐
๐ฆ llnl.util.lang.lazy_eq(lseq, rseq)
Equality comparison for two lazily generated sequences โจ
See lazy_lexicographic_ordering.
๐ธ llnl.util.lang.lazy_lexicographic_ordering(cls, set_hash=True)
okay this one is the main character ๐ โ it decorates a class with extra methods that implement rich comparison.
It's the lazy version of the tuple comparison we all use constantly in Python. So like, say you've got some objects with fields and you're doing the classic tuple-key comparison thing:
class Widget:
def _cmp_key(self):
return (
self.a,
self.b,
(self.c, self.d),
self.e
)
def __eq__(self, other):
return self._cmp_key() == other._cmp_key()
def __lt__(self):
return self._cmp_key() < other._cmp_key()
# etc.Python would compare Widgets lexicographically based on their tuples. Cute! But the issue for simple comparators is that we have to build the tuples, and we have to generate all the values in them up front ๐ When you're implementing comparisons for large data structures, that can be costly. it's giving wasteful.
Lazy lexicographic comparison maps the tuple comparison shown above to generator functions. Instead of comparing based on pre-constructed tuple keys, users of this decorator can compare using elements from a generator. so you'd write:
@lazy_lexicographic_ordering
class Widget:
def _cmp_iter(self):
yield a
yield b
def cd_fun():
yield c
yield d
yield cd_fun
yield e
# operators are added by decoratorThere are no tuples preconstructed, and the generator does not have to complete ๐ Instead of tuples, we simply make functions that lazily yield what would've been in the tuple. The @lazy_lexicographic_ordering decorator handles all the details of implementing comparison operators, and the Widget implementor only has to worry about writing _cmp_iter, and making sure the elements in it are also comparable. that's the whole assignment, bestie ๐
Some things to note:
- If a class already has
__eq__,__ne__,__lt__,__le__,__gt__,__ge__, or__hash__defined, this decorator will overwrite them ๐ - If
set_hashisFalse, this will not overwrite__hash__. - This class uses Python 2
None-comparison semantics. If you yieldNoneand it is compared to a non-Nonetype,Nonewill always be less than the other object. yes really, that's the vibe.
Raises TypeError โ if the class does not have a _cmp_iter method ๐ญ
โญ llnl.util.lang.lazy_lt(lseq, rseq)
Less-than comparison for two lazily generated sequences โจ
See lazy_lexicographic_ordering.
๐ง llnl.util.lang.list_modules(directory, **kwargs)
Lists all of the modules, excluding __init__.py, in a particular directory. Listed packages have no particular order โ it's not sorted, it's just vibes ๐
๐ llnl.util.lang.load_module_from_file(module_name, module_path)
Loads a python module from the path of the corresponding file ๐ซถ
If the module is already in sys.modules it will be returned as is and not reloaded. no re-runs, we respect the cache.
Parameters
module_name(str) โ namespace where the python module will be loaded, e.g.foo.barmodule_path(str) โ path of the python file containing the module
Returns A valid module object ๐
Raises
ImportErrorโ when the module can't be loaded ๐ญFileNotFoundErrorโ whenmodule_pathdoesn't exist ๐
๐ llnl.util.lang.match_predicate(*args)
Utility function for making string matching predicates โจ
Each arg can be a:
- regex
- list or tuple of regexes
- predicate that takes a string
This returns a predicate that is true if:
- any
argregex matches - any regex in a list or tuple of regexes matches.
- any predicate in
argsmatches.
๐ฆ llnl.util.lang.memoized(func)
Decorator that caches the results of a function, storing them in an attribute of that function ๐พ so it remembers, and you don't have to.
๐ธ llnl.util.lang.nullcontext(*args, **kwargs)
Empty context manager ๐ซง
TODO: replace with contextlib.nullcontext() if we ever require python 3.7.
๐ llnl.util.lang.pretty_date(time, now=None)
Convert a datetime or timestamp to a pretty, relative date ๐ฐ๏ธ like, human-readable and adorable.
Parameters
time(datetime.datetime or int) โ date to print prettilynow(datetime.datetime) โ datetime for 'now', i.e. the date the pretty date is relative to (default isdatetime.now())
Returns
pretty string like 'an hour ago', 'Yesterday', '3 months ago', 'just now', etc. ๐
Return type (str)
Adapted from https://stackoverflow.com/questions/1551382.
โฑ๏ธ llnl.util.lang.pretty_seconds(seconds)
Seconds to string with appropriate units โจ
Parameters seconds (float) โ Number of seconds
Returns Time string with units
Return type str
๐ซถ llnl.util.lang.pretty_seconds_formatter(seconds)
(no description in the source โ it's just out here existing, mysterious and unbothered ๐)
๐ llnl.util.lang.pretty_string_to_date(date_str, now=None)
Parses a string representing a date and returns a datetime object ๐ฐ๏ธ
Parameters date_str (str) โ string representing a date. This string might be in different format (like YYYY, YYYY-MM, YYYY-MM-DD, YYYY-MM-DD HH:MM, YYYY-MM-DD HH:MM:SS) or be a pretty date (like yesterday or two months ago)
Returns datetime object corresponding to date_str
Return type (datetime.datetime)
โญ llnl.util.lang.stable_args(*args, **kwargs)
A key factory that performs a stable sort of the parameters ๐
๐ง llnl.util.lang.stable_partition(input_iterable: Iterable, predicate_fn: Callable[[Any], bool]) -> Tuple[List[Any], List[Any]]
Partition the input iterable according to a custom predicate ๐ฏ
Parameters
input_iterableโ input iterable to be partitioned.predicate_fnโ predicate function accepting an iterable item as argument.
Returns Tuple of the list of elements evaluating to True, and list of elements evaluating to False. two piles, zero drama ๐
๐ llnl.util.lang.tuplify(seq)
Helper for lazy_lexicographic_ordering() ๐ซถ
๐ฅ llnl.util.lang.union_dicts(*dicts)
Use update() to combine all dicts into one ๐งฉ
This builds a new dictionary, into which we update() each element of dicts in order. Items from later dictionaries will override items from earlier dictionaries. last one wins, sorry to the early birds ๐ญ
Parameters dicts (list) โ list of dictionaries
Return: (dict): a merged dictionary containing combined keys and values from dicts.
๐ llnl.util.lang.uniq(sequence)
Remove strings of duplicate elements from a list โจ
This works like the command-line uniq tool. It filters strings of duplicate elements in a list. Adjacent matching elements are merged into the first occurrence ๐ซถ
For example:
uniq([1, 1, 1, 1, 2, 2, 2, 3, 3]) == [1, 2, 3]
uniq([1, 1, 1, 1, 2, 2, 2, 1, 1]) == [1, 2, 1]๐ฆ llnl.util.link_tree module
LinkTree class for setting up trees of symbolic links ๐ณ๐
๐ class llnl.util.link_tree.LinkTree(source_root)
Bases: object
Class to create trees of symbolic links from a source directory โจ
LinkTree objects are constructed with a source root. Their methods allow you to create and delete trees of symbolic links back to the source tree in specific destination directories. Trees comprise symlinks only to files; directries are never symlinked to, to prevent the source directory from ever being modified ๐
โโ๏ธ absolutely not touching the source, we don't do that here.
๐ find_conflict(dest_root, ignore=None, ignore_file_conflicts=False)
Returns the first file in dest that conflicts with src ๐ซฃ
๐ฆ find_dir_conflicts(dest_root, ignore)
(no description in the source โ it finds dir conflicts, iykyk ๐ธ)
โญ get_file_map(dest_root, ignore)
(no description in the source โ it gets the file map ๐ซ)
๐
merge(dest_root, ignore_conflicts=False, ignore=None, link=<function symlink>, relative=False)
Link all files in src into dest, creating directories if necessary ๐
Keyword Args:
ignore_conflicts(bool): ifTrue, do not break when the target exists; return a list of files that could not be linkedignore(callable): callable that returnsTrueif a file is to be ignored in the merge (by default ignore nothing)link(callable): function to create links with (defaults tollnl.util.symlink)relative(bool): create all symlinks relative to the target (defaultFalse)
๐ง merge_directories(dest_root, ignore)
(no description in the source โ it merges directories, that's the whole personality ๐ซถ)
๐ unmerge(dest_root, ignore=None, remove_file=<function remove_link>)
Unlink all files in dest that exist in src โ๏ธ
Unlinks directories in dest if they are empty.
๐ธ unmerge_directories(dest_root, ignore)
(no description in the source โ the un-merge-y sibling ๐)
๐ llnl.util.lock module
๐ exception llnl.util.lock.CantCreateLockError(path)
Bases: LockPermissionError
Attempt to create a lock in an unwritable location ๐ซ that's the error, that's the crime.
๐ class llnl.util.lock.Lock(path: str, *, start: int = 0, length: int = 0, default_timeout: float | None = None, debug: bool = False, desc: str = '')
Bases: object
Like, this thing is a filesystem lock built on top of Python's lockf. And lowkey the cool part? In Python, lockf actually just calls fcntl, so it'll work with literally any filesystem implementation that supports locking through the fcntl calls. That includes distributed filesystems like Lustre (when flock is enabled) and recent NFS versions. bestie, your network filesystem is covered โจ
Okay but real talk, read this bit twice ๐
This is for managing contention over resources between processes, not for managing contention between threads in a process: the functions of this object are not thread-safe. Also, a process must not maintain multiple locks on the same file (or, more specifically, on overlapping byte ranges in the same file). ๐
acquire_read(timeout: float | None = None) -> boolโ grabs a recursive, shared lock for reading ๐ซถ Read and write locks can be acquired and released in arbitrary order, but the POSIX lock is held until all local read and write locks are released. ReturnsTrueif it's the first acquire and it actually acquires the POSIX lock, andFalseif it's a nested transaction.acquire_write(timeout: float | None = None) -> boolโ grabs a recursive, exclusive lock for writing ๐ Same deal: read and write locks can be acquired and released in arbitrary order, but the POSIX lock is held until all local read and write locks are released. ReturnsTrueif it's the first acquire and it actually acquires the POSIX lock,Falseif it's a nested transaction.cleanup() -> Noneโ that's it, that's the whole entry ๐ (no description given, so no cap, we're not making one up)downgrade_write_to_read(timeout: float | None = None) -> Noneโ downgrades you from an exclusive write lock to a shared read ๐- Raises
LockDowngradeErrorโ if this is an attempt at a nested transaction
- Raises
is_write_locked() -> boolโ vibe check: is the file write locked? ๐- Returns:
Trueif the path is write locked, otherwiseFalse - Return type:
(bool)
- Returns:
release_read(release_fn: Callable[[], bool] | None = None) -> boolโ releases a read lock ๐ฆ- Parameters:
release_fn(Callable) โ function to call before the last recursive lock (read or write) is released. - If the last recursive lock will be released, then this will call
release_fnand return its result (if provided), or returnTrue(ifrelease_fnwas not provided). - Otherwise, we're still nested inside some other lock, so do not call the
release_fn, and returnFalse. - Does limited correctness checking: if a read lock is released when none are held, this will raise an assertion error. ๐
- Parameters:
release_write(release_fn: Callable[[], bool] | None = None) -> boolโ releases a write lock ๐ง- Parameters:
release_fn(Callable) โ function to call before the last recursive write is released. - If the last recursive write lock will be released, then this will call
release_fnand return its result (if provided), or returnTrue(ifrelease_fnwas not provided). Otherwise, we're still nested inside some other write lock, so do not call therelease_fn, and returnFalse. - Does limited correctness checking: if a read lock is released when none are held, this will raise an assertion error. ๐
- Parameters:
upgrade_read_to_write(timeout: float | None = None) -> Noneโ attempts to upgrade from a shared read lock to an exclusive write ๐ฅ- Raises
LockUpgradeErrorโ if this is an attempt at a nested transaction
- Raises
exception llnl.util.lock.LockDowngradeError(path)โ Bases:LockError. Raised when unable to downgrade from a write to a read lock ๐exception llnl.util.lock.LockErrorโ Bases:Exception. Raised for any errors related to locks, periodt.exception llnl.util.lock.LockPermissionErrorโ Bases:LockError. Raised when there are permission issues with a lock ๐ซexception llnl.util.lock.LockROFileError(path)โ Bases:LockPermissionError. Tried to take an exclusive lock on a read-only file. Not me trying to write to a read-only file ๐ญexception llnl.util.lock.LockTimeoutError(lock_type, path, time, attempts)โ Bases:LockError. Raised when an attempt to acquire a lock times out โฐclass llnl.util.lock.LockTransaction(lock: Lock, acquire: Callable[[], bool] | None | ContextManager = None, release: Callable[[], bool] | None | ContextManager = None, timeout: float | None = None)โ Bases:object. Simple nested transaction context manager that uses a file lock ๐- Parameters:
lock(Lock) โ underlying lock for this transaction to be accquired on enter and released on exitacquire(Callable orcontextlib.contextmanager) โ function to be called after lock is acquired, or contextmanager to enter after acquire and leave before release.release(Callable) โ function to be called before release. If acquire is a contextmanager, this will be called after exiting the nexted context and before the lock is released.timeout(float) โ number of seconds to set for the timeout when accquiring the lock (default no timeout)
- If the
acquire_fnreturns a value, it is used as the return value for__enter__, allowing it to be passed as theasargument of awithstatement. - If
acquire_fnreturns a context manager, its__enter__function will be called after the lock is acquired, and its__exit__funciton will be called beforerelease_fnin__exit__, allowing you to nest a context manager inside this one. - Timeout for lock is customizable. โจ
- Parameters:
exception llnl.util.lock.LockUpgradeError(path)โ Bases:LockError. Raised when unable to upgrade from a read to a write lock.class llnl.util.lock.ReadTransaction(lock: Lock, acquire: Callable[[], bool] | None | ContextManager = None, release: Callable[[], bool] | None | ContextManager = None, timeout: float | None = None)โ Bases:LockTransaction. LockTransaction context manager that does a read and releases it ๐ซถclass llnl.util.lock.WriteTransaction(lock: Lock, acquire: Callable[[], bool] | None | ContextManager = None, release: Callable[[], bool] | None | ContextManager = None, timeout: float | None = None)โ Bases:LockTransaction. LockTransaction context manager that does a write and releases it ๐
๐ฆ llnl.util.multiproc module
This implements a parallel map operation but it can accept more values than multiprocessing.Pool.apply() can. For example, apply() will fail to pickle functions if they're passed indirectly as parameters. iykyk ๐
class llnl.util.multiproc.Barrier(n, timeout=None)โ Bases:object. Simple reusable semaphore barrier. Python 2 doesn't have multiprocessing barriers so we implement this. Seehttps://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf, p. 41. ๐wait()
๐ธ llnl.util.symlink module
exception llnl.util.symlink.AlreadyExistsErrorโ Bases:SymlinkError. Link path already exists. oops ๐exception llnl.util.symlink.SymlinkErrorโ Bases:RuntimeError. Exception class for errors raised while creating symlinks, junctions and hard links.llnl.util.symlink.islink(path: str) -> boolโ Overrideos.islinkto give correct answer for spack logic ๐- For Non-Windows: a link can be determined with the
os.path.islinkmethod. Windows-only methods will return false for other operating systems. - For Windows: spack considers symlinks, hard links, and junctions to all be links, so if any of those are
True, returnTrue. - Parameters:
path(str) โ path to check if it is a link. - Returns:
bool- whether the path is any kind link or not.
- For Non-Windows: a link can be determined with the
llnl.util.symlink.readlink(path: str, *, dir_fd=None)โ Spack utility to override ofos.readlinkmethod to work cross platform ๐llnl.util.symlink.resolve_link_target_relative_to_the_link(link)โos.path.isdirusesos.path.exists, which for links will check the existence of the link target. If the link target is relative to the link, we need to construct a pathname that is valid from our cwd (which may not be the same as the link's directory). ๐งllnl.util.symlink.symlink(source_path: str, link_path: str, allow_broken_symlinks: bool = True)โ Create a link. ๐- On non-Windows and Windows with System Administrator privleges this will be a normal symbolic link via
os.symlink. - On Windows without privledges the link will be a junction for a directory and a hardlink for a file. On Windows the various link types are:
- Symbolic Link: A link to a file or directory on the same or different volume (drive letter) or even to a remote file or directory (using UNC in its path). Need System Administrator privileges to make these.
- Hard Link: A link to a file on the same volume (drive letter) only. Every file (file's data) has at least 1 hard link (file's name). But when this method creates a new hard link there will be 2. Deleting all hard links effectively deletes the file. Don't need System Administrator privileges.
- Junction: A link to a directory on the same or different volume (drive letter) but not to a remote directory. Don't need System Administrator privileges.
- Parameters:
source_path(str) โ The real file or directory that the link points to. Must be absolute OR relative to the link.link_path(str) โ The path where the link will exist.allow_broken_symlinks(bool) โ On Linux or Mac, don't raise an exception if thesource_pathdoesn't exist. This will still raise an exception on Windows. โ ๏ธ
- On non-Windows and Windows with System Administrator privleges this will be a normal symbolic link via
๐ธ SUBMODULES ๐ฆ
okay bestie, this chunk is the module-by-module tour of Spack's little helper libraries โ think of them as the utility belt ๐ everything here is small, focused, and (spoiler) mostly just needs the Python standard library. let's go ๐
๐ llnl.path module
Path primitives that just require Python standard library. no extra deps, no drama โจ
๐ง class llnl.path.Path
Bases: object
Enum to identify the path-style.
platform_path: int = 0unix: int = 0windows: int = 1
llnl.path.convert_to_platform_path(path: str) -> str
Converts the input path to the current platform's native style.
llnl.path.convert_to_posix_path(path: str) -> str
Converts the input path to POSIX style.
llnl.path.format_os_path(path: str, mode: int = 0) -> str
Formats the input path to use consistent, platform specific separators.
Absolute paths are converted between drive letters and a prepended / as per platform requirement.
Parameters
- path โ the path to be normalized, must be a string or expose the
replacemethod. - mode โ the path file separator style to normalize the passed path to. Default is unix style, i.e.
/
llnl.path.path_to_os_path(*parameters: str) -> List[str]
Takes an arbitrary number of positional parameters, converts each argument of type string to use a normalized filepath separator, and returns a list of all values. it's giving "normalize everything" energy ๐
llnl.path.sanitize_win_longpath(path: str) -> str
Strip Windows extended path prefix from strings Returns sanitized string. no-op if extended path prefix is not present.
llnl.path.system_path_filter(_func=None, arg_slice: slice | None = None)
Filters function arguments to account for platform path separators. Optional slicing range can be specified to select specific arguments.
This decorator takes all (or a slice) of a method's positional arguments and normalizes usage of filepath separators on a per platform basis.
Note:**kwargs, urls, and any type that is not a string are ignored so in such cases where path normalization is required, that should be handled by callingpath_to_os_pathdirectly as needed.
Parameters
- arg_slice โ a slice object specifying the slice of arguments in the decorated method over which filepath separators are normalized
๐ธ llnl.string module
String manipulation functions that do not have other dependencies than Python standard library. tiny module, big main character energy ๐
llnl.string.comma_and(sequence: List[str]) -> str
Return a string with all the elements of the input joined by comma, but the last one (which is joined by 'and').
llnl.string.comma_list(sequence: List[str], article: str = '') -> str
bestie the original literally hands you the signature and no description โ it's giving mysterious โจ (the function is real, the docs are just shy.)
llnl.string.comma_or(sequence: List[str]) -> str
Return a string with all the elements of the input joined by comma, but the last one (which is joined by 'or').
llnl.string.plural(n: int, singular: str, plural: str | None = None, show_n: bool = True) -> str
Pluralize singular word by adding an s if n != 1.
Parameters
- n โ number of things there are
- singular โ singular form of word
- plural โ optional plural form, for when it's not just singular +
's' - show_n โ whether to include
nin the result string (defaultTrue)
Returns
"1 thing" if n == 1 or "n things" if n != 1
llnl.string.quote(sequence: List[str], q: str = "'") -> List[str]
Quotes each item in the input list with the quote character passed as second argument.
๐ฅ llnl.url module
URL primitives that just require Python standard library. this one ate, fr fr โ it's all the fiddly URL archaeology ๐ฆ
llnl.url.allowed_archive(path_or_url: str) -> bool
Returns true if the input is a valid archive, False otherwise.
llnl.url.check_and_remove_ext(path: str, *, extension: str) -> str
Returns the input path with the extension removed, if the extension is present in path. Otherwise, returns the input unchanged.
llnl.url.compression_ext_from_compressed_archive(extension: str) -> str | None
Returns compression extension for a compressed archive
llnl.url.determine_url_file_extension(path: str) -> str
This returns the type of archive a URL refers to. This is sometimes confusing because of URLs like:
Where the URL doesn't actually contain the filename. We need to know what type it is so that we can appropriately name files in mirrors. (the audacity of that URL honestly ๐ญ)
llnl.url.expand_contracted_extension(extension: str) -> str
Returns the expanded version of a known contracted extension.
This function maps extensions like ".tgz" to ".tar.gz". On unknown extensions, return the input unmodified.
llnl.url.expand_contracted_extension_in_path(path_or_url: str, *, extension: str | None = None) -> str
Returns the input path or URL with any contraction extension expanded.
Parameters
- path_or_url โ path or URL to be expanded
- extension โ if specified, only attempt to expand that extension
llnl.url.extension_from_path(path_or_url: str | None) -> str | None
Tries to match an allowed archive extension to the input. Returns the first match, or None if no match was found.
Raises ValueError โ if the input is None
llnl.url.find_list_urls(url: str) -> Set[str]
Find good list URLs for the supplied URL.
By default, returns the dirname of the archive path.
Provides special treatment for the following websites, which have a unique list URL different from the dirname of the download URL:
- GitHub โ
https://github.com/<repo>/<name>/releases - GitLab โ
https://gitlab.*/<repo>/<name>/tags - BitBucket โ
https://bitbucket.org/<repo>/<name>/downloads/?tab=tags - CRAN โ
https://*.r-project.org/src/contrib/Archive/<name> - PyPI โ
https://pypi.org/simple/<name>/ - LuaRocks โ
https://luarocks.org/modules/<repo>/<name>
Note: this function is called byspack versions,spack checksum, andspack create, but not byspack fetchorspack install.
Parameters
- url (
str) โ The download URL for the package
Returns
One or more list URLs for the package
Return type
set
llnl.url.has_extension(path_or_url: str, ext: str) -> bool
Returns true if the extension in input is present in path, false otherwise.
llnl.url.remove_extension(path_or_url: str, *, extension: str) -> str
Returns the input with the extension removed
llnl.url.split_url_extension(url: str) -> Tuple[str, ...]
Some URLs have a query string, e.g.:
https://github.com/losalamos/CLAMR/blob/packages/PowerParser_v2.0.7.tgz?raw=truehttp://www.apache.org/dyn/closer.cgi?path=/cassandra/1.2.0/apache-cassandra-1.2.0-rc2-bin.tar.gzhttps://gitlab.kitware.com/vtk/vtk/repository/archive.tar.bz2?ref=v7.0.0
In (1), the query string needs to be stripped to get at the extension, but in (2) & (3), the filename is IN a single final query argument.
This strips the URL into three pieces: prefix, ext, and suffix. The suffix contains anything that was stripped off the URL to get at the file extension. In (1), it will be '?raw=true', but in (2), it will be empty. In (3) the suffix is a parameter that follows after the file extension, e.g.:
1. ('https://github.com/losalamos/CLAMR/blob/packages/PowerParser_v2.0.7',
'.tgz', '?raw=true')
2. ('http://www.apache.org/dyn/closer.cgi?path=/cassandra/1.2.0/apache-cassandra-1.2.0-rc2-bin',
'.tar.gz', None)
3. ('https://gitlab.kitware.com/vtk/vtk/repository/archive',
'.tar.bz2', '?ref=v7.0.0')llnl.url.split_url_on_sourceforge_suffix(url: str) -> Tuple[str, ...]
If the input is a sourceforge URL, returns base URL and "/download" suffix. Otherwise, returns the input URL and an empty string.
llnl.url.strip_compression_extension(path_or_url: str, ext: str | None = None) -> str
Strips the compression extension from the input, and returns it. For instance, "foo.tgz" becomes "foo.tar".
If no extension is given, try a default list of extensions.
Parameters
- path_or_url โ input to be stripped
- ext โ if given, extension to be stripped
llnl.url.strip_extension(path_or_url: str, *, extension: str | None = None) -> str
If a path contains the extension in input, returns the path stripped of the extension. Otherwise, returns the input path.
If extension is None, attempts to strip any allowed extension from path.
llnl.url.strip_query_and_fragment(url: str) -> Tuple[str, str]โ bestie, this one takes a URL and literally rips the query and the fragment right off it, then hands you back the base url and the suffix. Two-for-one, we love a return type that gives. ๐
Parameters
urlโ the URL that's about to get stripped. That's it, that's the whole job.
Heads up babes: it raises ValueError when there's any error parsing the URL. So if you feed it something cursed, it's gonna yell. ๐
llnl.url.strip_version_suffixes(path_or_url: str) -> strโ okay so some tarballs out here are doing the absolute most and putting extra info after the version. Iconic of them, but no. Examples of the chaos:bowtie2-2.2.5-sourcelibevent-2.0.21-stablecuda_8.0.44_linux.run
These strings are not part of the version number and should be ignored โ like, read the room, -source. This function strips those suffixes off and returns whatever's left. The whole goal is that the version is always the last thing in the path, so it ends up looking like:
bowtie2-2.2.5libevent-2.0.21cuda_8.0.44
Clean. Gorgeous. Exactly what we asked for. โจ
Parameters
path_or_urlโ the filename or URL for the package, aka the thing with the loud suffixes.
Returns
The path with any extraneous suffixes removed. Periodt. ๐
- Index โ the index, your map to the whole thing ๐บ๏ธ
- Module Index โ all the modules, catalogued and cute
- Search Page โ for when you lowkey forgot where something was and need to go find it
๐ AUTHOR
Todd Gamblin โ the main character who authored all of this, fr fr. ๐ซถ
๐ COPYRIGHT
2013-2023, Lawrence Livermore National Laboratory.
And that's a wrap on the whole page, bestie!! ๐ You made it to the very end โ genuinely iconic behavior. This edition is version 0.22, dated Jul 10, 2024, so if you're quoting it in the group chat, that's your receipt. Now go spack install something and slay. ๐๐๐ฆ
๐ฆ where this page comes from
- opensuse-leap 16.0: spack-man 0.22.1-bp160.1.9 ยท Apache-2.0 AND MIT AND Python-2.0 AND BSD-3-Clause
- ๐ girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on : a language model's rewrite of the original, under its terms