INTAKE(1)

📦

version 0.6.6 · 2 versions

0.6.6 · debian bookworm (latest)
0.6.5 · ubuntu jammy

🦋 INTAKE(1) — the `intake` manual, bestie edition 🦋

okay so this is the manual page for intake (that's the command, the package, the whole vibe). Buckle up, it's giving documentation, but make it cute. 💅

🌸 NAME 🌸

intake — Intake Documentation

Taking the pain out of data access and distribution ✨

Intake is a lightweight package for finding, investigating, loading and disseminating data. It will appeal to different groups for some of the reasons below, but is useful for all and acts as a common platform that everyone can use to smooth the progression of data from developers and providers to users.

Intake contains the following main components. You do not need to use them all! The library is modular, only use the parts you need:

A set of data loaders (Drivers) with a common interface, so that you can investigate or load anything, from local or remote, with the exact same call, and turning into data structures that you already know how to manipulate, such as arrays and data-frames.
A Cataloging system (Catalogs) for listing data sources, their metadata and parameters, and referencing which of the Drivers should load each. The catalogs for a hierarchical, searchable structure, which can be backed by files, Intake servers or third-party data services
Sets of convenience functions to apply to various data sources, such as data-set persistence, automatic concatenation and metadata inference and the ability to distribute catalogs and data sources using simple packaging abstractions.
A GUI layer accessible in the Jupyter notebook or as a standalone webserver, which allows you to find and navigate catalogs, investigate data sources, and plot either predefined visualisations or interactively find the right view yourself
A client-server protocol to allow for arbitrary data cataloging services or to serve the data itself, with a pluggable auth model.

💜 DATA USER 💜

Intake loads the data for a range of formats and types (see Plugin Directory) into containers you already use, like Pandas dataframes, Python lists, NumPy arrays, and more
Intake loads, then gets out of your way
GUI search and introspect data-sets in Catalogs: quickly find what you need to do your work
Install data-sets and automatically get requirements
Leverage cloud resources and distributed computing.

See the executable tutorial:

https://mybinder.org/v2/gh/intake/intake-examples/master?filepath=tutorial%2Fdata_scientist.ipynb

🌈 DATA PROVIDER 🌈

Simple spec to define data sources
Single point of truth, no more copy&paste
Distribute data using packages, shared files or a server
Update definitions in-place
Parametrise user options
Make use of additional functionality like filename parsing and caching.

See the executable tutorial:

https://mybinder.org/v2/gh/intake/intake-examples/master?filepath=tutorial%2Fdata_engineer.ipynb

🔥 IT 🔥

Create catalogs out of established departmental practices
Provide data access credentials via Intake parameters
Use server-client architecture as gatekeeper:
- add authentication methods
- add monitoring point; track the data-sets being accessed.
Hook Intake into proprietary data access systems.

👑 DEVELOPER 👑

Turn boilerplate code into a reusable Driver
Pluggable architecture of Intake allows for many points to add and improve
Open, simple code-base -- come and get involved on github!

See the executable tutorial:

https://mybinder.org/v2/gh/intake/intake-examples/master?filepath=tutorial%2Fdev.ipynb

The Start here document contains the sections that all users new to Intake should read through. Use Cases - I want to... shows specific problems that Intake solves. For a brief demonstration, which you can execute locally, go to Quickstart. For a general description of all of the components of Intake and how they fit together, go to Overview. Finally, for some notebooks using Intake and articles about Intake, go to Examples and intake-examples. These and other documentation pages will make reference to concepts that are defined in the Glossary.

✨ START HERE ✨

These documents will familiarise you with Intake, show you some basic usage and examples, and describe Intake's place in the wider python data world. 🫶

🧚 Quickstart 🧚

This guide will show you how to get started using Intake to read data, and give you a flavour of how Intake feels to the Data User. It assumes you are working in either a conda or a virtualenv/pip environment. For notebooks with executable code, see the Examples. This walk-through can be run from a notebook or interactive python session.

🍓 Installation 🍓

If you are using Anaconda or Miniconda, install Intake with the following commands:

conda install -c conda-forge intake

If you are using virtualenv/pip, run the following command:

pip install intake

Note that this will install with the minimum of optional requirements. If you want a more complete install, use intake[complete] instead. 💖

🌸 Creating Sample Data 🌸

Let's begin by creating a sample data set and catalog. At the command line, run the intake example command. This will create an example data Catalog and two CSV data files. These files contains some basic facts about the 50 US states, and the catalog includes a specification of how to load them.

💫 Loading a Data Source 💫

Data sources can be created directly with the open_*() functions in the intake module. To read our example data:

>>> import intake
>>> ds = intake.open_csv('states_*.csv')
>>> print(ds)
<intake.source.csv.CSVSource object at 0x1163882e8>

Each open function has different arguments, specific for the data format or service being used.

🌈 Reading Data 🌈

Intake reads data into memory using containers you are already familiar with:

Tables: Pandas DataFrames
Multidimensional arrays: NumPy arrays
Semistructured data: Python lists of objects (usually dictionaries)

To find out what kind of container a data source will produce, inspect the container attribute:

>>> ds.container
'dataframe'

The result will be dataframe, ndarray, or python. (New container types will be added in the future.)

For data that fits in memory, you can ask Intake to load it directly:

>>> df = ds.read()
>>> df.head()
        state        slug code                nickname  ...
0     Alabama     alabama   AL      Yellowhammer State
1      Alaska      alaska   AK       The Last Frontier
2     Arizona     arizona   AZ  The Grand Canyon State
3    Arkansas    arkansas   AR       The Natural State
4  California  california   CA            Golden State

Many data sources will also have quick-look plotting available. The attribute .plot will list a number of built-in plotting methods, such as .scatter(), see Plotting.

Intake data sources can have partitions. A partition refers to a contiguous chunk of data that can be loaded independent of any other partition. The partitioning scheme is entirely up to the plugin author. In the case of the CSV plugin, each .csv file is a partition.

To read data from a data source one chunk at a time, the read_chunked() method returns an iterator:

>>> for chunk in ds.read_chunked(): print('Chunk: %d' % len(chunk))
...
Chunk: 24
Chunk: 26

💅 Working with Dask 💅

Working with large datasets is much easier with a parallel, out-of-core computing library like Dask. Intake can create Dask containers (like dask.dataframe) from data sources that will load their data only when required:

>>> ddf = ds.to_dask()
>>> ddf
Dask DataFrame Structure:
            admission_date admission_number capital_city capital_url    code constitution_url facebook_url landscape_background_url map_image_url nickname population population_rank skyline_background_url    slug   state state_flag_url state_seal_url twitter_url website
npartitions=2
                object            int64       object      object  object           object       object                   object        object   object      int64           int64                 object  object  object         object         object      object  object
                    ...              ...          ...         ...     ...              ...          ...                      ...           ...      ...        ...             ...                    ...     ...     ...            ...            ...         ...     ...
                    ...              ...          ...         ...     ...              ...          ...                      ...           ...      ...        ...             ...                    ...     ...     ...            ...            ...         ...     ...
Dask Name: from-delayed, 4 tasks

The Dask containers will be partitioned in the same way as the Intake data source, allowing different chunks to be processed in parallel. Please read the Dask documentation to understand the differences when working with Dask collections (Bag, Array or Data-frames).

🎀 Opening a Catalog 🎀

A Catalog is an inventory of data sources, with the type and arguments prescribed for each, and arbitrary metadata about each source. In the simplest case, a catalog can be described by a file in YAML format, a "‐ Catalog file". In real usage, catalogues can be defined in a number of ways, such as remote files, by connecting to a third-party data service (e.g., SQL server) or through an Intake Server protocol, which can implement any number of ways to search and deliver data sources.

The intake example command, above, created a catalog file with the following YAML-syntax content:

sources:
  states:
    description: US state information from [CivilServices](https://civil.services/)
    driver: csv
    args:
      urlpath: '{{ CATALOG_DIR }}/states_*.csv'
    metadata:
      origin_url: 'https://github.com/CivilServiceUSA/us-states/blob/v1.0.0/data/states.csv'

To load a Catalog from a Catalog file:

>>> cat = intake.open_catalog('us_states.yml')
>>> list(cat)
['states']

This catalog contains one data source, called states. It can be accessed by attribute:

>>> cat.states.to_dask()[['state','slug']].head()
        state        slug
0     Alabama     alabama
1      Alaska      alaska
2     Arizona     arizona
3    Arkansas    arkansas
4  California  california

Placing your data source specs into a catalog like this? chef's kiss 💅 It means you get to declare your data sets in one canonical place, and you never have to copy-paste boilerplate code into every single notebook or script that touches the data. Catalogs can also reference each other, get stored remotely, and carry extra metadata — like a set of named quick-look plots that make sense for that particular data source. And bestie, catalogs are not restricted to living in YAML files, it's just that YAML happens to be the simplest way to show them off. ✨

Many catalog entries will also contain "user_parameter" blocks, which are basically the catalog author saying "these are the options I explicitly allow" — or they're there for validation, or for the values you pass. You customise how a data source gets accessed by giving values for those user_parameters, overriding the arguments specified in the entry, or passing extra keyword arguments straight through to the driver. Important: the keywords you're allowed to pass are limited to the user_parameters that are defined plus whatever inputs the specific driver expects — and this usage is really only for people who already know the ins and outs of that format. In the example below, the user overrides the "csv_kwargs" keyword, which is described in the docs for CSVSource and gets passed down to the CSV reader:

# pass extra kwargs understood by the csv driver
>>> intake.cat.states(csv_kwargs={'header': None, 'skiprows': 1}).read().head()
           0           1   ...                                17
0     Alabama     alabama  ...    https://twitter.com/alabamagov
1      Alaska      alaska  ...        https://twitter.com/alaska

And ngl, if you're the one making these catalogs, you'll probably start by trying the open_csv command from above, then hitting it with print(ds.yaml()). Do that right now and you'll see the output looks very similar to the catalog file we gave you. 🎀

💖 Installing Data Source Packages 💖

Intake makes it possible to create Data packages (pip or conda) that install data sources into a global catalog. So like, watch — we can install a data package containing the same data we've been playing with:

conda install -c intake data-us-states

Conda drops the catalog file from this package into $CONDA_PREFIX/share/intake/us_states.yml. Now when we import intake, the data from this package just appears as part of a global catalog called intake.cat. In this specific case we use Dask to do the reading (it handles larger-than-memory data and parallel processing, slay), but read() would work too:

>>> import intake
>>> intake.cat.states.to_dask()[['state','slug']].head()
        state        slug
0     Alabama     alabama
1      Alaska      alaska
2     Arizona     arizona
3    Arkansas    arkansas
4  California  california

The global catalog is a union of all the catalogs installed in your conda/virtualenv environment, plus any catalogs installed in user-specific locations. 🌈

🦋 Adding Data Source Packages using the Intake path 🦋

Intake checks the Intake config file for catalog_path, or the environment variable "INTAKE_PATH" for a colon separated list of paths (semicolon on windows) to search for catalog files. Then when you import intake, you'll see all the entries from all of the catalogues referenced as part of a global catalog called intake.cat. periodt. 💅

✨ Using the GUI ✨

There's a graphical data browser available in the Jupyter notebook environment or as a standalone web-server. It shows the contents of any installed catalogs, plus it lets you select local and remote catalogs to browse and pick entries from. See GUI. 👑

🌸 Use Cases - I want to... 🌸

Here's a list of specific things people might want to get done, and the deets on how Intake can help. How to actually achieve each one lives in the rest of the detailed documentation. 💜

Avoid copy&paste of blocks of code for accessing data

This is such a common pattern: you want some specific data, so you find someone — maybe a colleague — who's accessed it before, and you copy that code. That practice is extremely error prone, and it causes a whole proliferation of copies of code, each one evolving over time, with various versions all in use at the same time. 😭

Intake separates the concerns of data-source specification from code. The specs get stored separately, and every user can reference the one and only authoritative definition — whether that's a shared file, a service everyone can see, or the Intake server. Update the spec and everyone gets the current version instead of relying on outdated code. 🫶

Version control data sources

Version control (e.g., using git) is an essential practice in modern software engineering and data science. It makes sure the change history is recorded — with times, descriptions and authors — alongside the changes themselves.

When data is specified using a well-structured syntax like YAML, you can check it into a version-controlled repository in the usual fashion. So you get to bring rigorous practices to your data as well as your code. ✨

And if you're using conda packages to distribute data specifications, those come with a natural internal version numbering system, so users just need to run conda update ... to get the latest version.

Install data

Often, finding and grabbing data is a major hurdle to productivity. People get stuck downloading artifacts from various places or digging through storage systems to find the exact thing they're after. One-line commands that can retrieve data-source specifications or the files themselves can be a massive time-saver. Plus, each data-set will typically need its own code to access it, and probably extra software dependencies too. 💀

Intake lets you build conda packages that can include catalog files referencing online resources, or include the data files directly in the package. Whether uploaded to anaconda.org or hosted on a private enterprise channel, getting the data becomes a single conda install ... command, and after that it'll show up as an entry in intake.cat. The conda package brings versioning and dependency declaration for free, and you can toss any code that data-set needs right into the package too. 🎀

Update data specifications in-place

Individual data-sets are often static, but commonly the "best" data to get a job done changes over time as new facts emerge. And conversely, the very same data might be better stored in a different format — one that's, say, better-suited to parallel access in the cloud. In those situations you really don't want to force all the data scientists relying on it to have their code temporarily broken and be forced to change it. 😭

By working with a catalog file/service in a fixed shared location, you can update the data source specs in-place. When users run their code now, they get the latest version. And because all Intake drivers share the same API, the code using the data is identical and doesn't need to change — even when the format has been updated to something more optimised. 🔥

Access data stored on cloud resources

Services like AWS S3, GCS and Azure Datalake (or private enterprise variants of those) are increasingly popular places to amass large amounts of data. Not only are they relatively cheap per GB, but they provide long-term resilience, metadata services, complex access control patterns, and can have very large data throughput when accessed in parallel by machines on the same architecture. ☁️

Intake comes with integration to cloud-based storage out-of-the-box for most of the file-based data formats, so you can access the data directly in-place and in parallel. For the few remaining cases where direct access isn't feasible, the caching system in Intake lets you download files on first use, so all further access is much faster. 🌈

Work with Big Data

The era of Big Data is here, bestie! The term means different things to different people, but it definitely implies that an individual data-set is too large to fit into the memory of a typical workstation computer (>>10GB). And yet, most data-loading examples out there use functions in packages like pandas and expect to produce in-memory representations of the whole data. That's clearly a problem, and "get more memory in your machine" is not a real answer. 🙈

Intake integrates with Dask and Spark, which both offer out-of-core computation (loading the data in chunks that fit in memory and aggregating the result) or can spread their work over a cluster of machines — effectively using the shared memory resources of the whole cluster. Dask integration is built into the majority of the drivers and exposed with the .to_dask() method, and Spark integration is available for a small number of drivers with a similar .to_spark() method, as well as directly with the intake-spark package. 🔥

Intake also integrates with many data services that can do big-data computations themselves, only extracting the smaller aggregated data-sets that do fit into memory for further analysis. Services like SQL systems, solr, elastic-search, splunk, accumulo and hbase can all distribute the work required to fulfill a query across many nodes of a cluster. 👑

Find the right data-set

Browsing for the data-set that'll solve a particular problem can be hard, even when the data has been curated and stored in a single, well-structured system. You do not want to rely on word-of-mouth to figure out which data is right for which job. 💅

Intake catalogs allow for self-description of data-sets, with simple text and arbitrary metadata, plus a consistent access pattern. Not only can you list the data available to you, but you can find out what exactly that data represents, and the form the data would take if loaded (table versus list of items, for example). This extra metadata is also searchable: you can descend through a hierarchy of catalogs with a single search and find all the entries containing some particular keywords. 🦋

You can use the Intake GUI to graphically browse through your available data-sets, or point it at catalogs available to you, look through the entries listed there and get info about each one — or even show a sample of the data or quick-look plots. The GUI can also execute searches and browse file-systems to find data artifacts of interest. And this same functionality is available via a command-line interface or programmatically too. periodt. ✨

✨ Work remotely ✨

Okay so cloud storage is literally the bestie of convenience, right? But you are NOT about to download a whole gigabyte situation onto your lil laptop or workstation just to do some analysis. That's giving chaos energy. 💀

Intake is lowkey made for the remote-execution lifestyle — think jupyter, Anaconda Enterprise, and all those other in-browser technologies. It's living its best life there. For example, you can run the Intake GUI either as a stand-alone application for browsing data-sets, or right inside a notebook for full analytics, and have all the runtime live on a remote machine — or maybe a whole cluster that's co-located with the data storage. Iconic. 👑

And when you pair that with cloud-optimised data formats such as parquet? That's the ideal set-up for processing data at web scale. Slay. 💅

So there's a MASSIVE amount of data out there living in human-readable formats like JSON, XML and CSV. Cute, but here's the tea: those are not very efficient in terms of space usage, and they need to be parsed on load just to turn into arrays or tables. So much extra work. 😭

Way faster processing times can be had with modern compact, optimised formats — such as parquet.

Intake has a "persist" mechanism to transform any input data-source into the format most appropriate for that type of data — e.g., parquet for tabular data. And the way that works? The persisted data gets used in preference at analysis time, and the schedule for updating from the original source is configurable. 🎀

The location of these persisted data-sets can be shared with others so they can also gain the benefits. OR — and this is the fun part — the "export" variant can be used to produce an independent version in the same format, together with a spec to reference it by. Then you just... share that spec with others. Periodt. 🦋

🔒 Access data without leaking credentials 🔒

Security is important, bestie. Users' identity and authority to view specific data should be established before handing over any sensitive bytes. It is, unfortunately, all too common for data scientists to include their username, passwords or other credentials directly in code — so that it can run automatically — thus presenting a potential security gap. 🙈

Intake does not manage credentials or user identities directly, but it does provide hooks for fetching details from the environment or other service, and using the values in templating at the time of reading the data. So the details are not included in the code, but every access still requires for them to be present. Vibe check: passed. ✨

In other cases, you may want to require the user to provide their credentials every single time, rather than automatically establish them — and "user parameters" can be specified in Intake to cover this case.

🛡️ Establish a data gateway 🛡️

The Intake server protocol allows you fine-grained control over the set of data sources that are listed, and exactly what to return to a user when they want to read some of that data. Which is honestly such a moment. Ideal opportunity to include authorisation checks, audit logging, and any more complicated access patterns, as required. 👑

By streaming the data through a single channel on the server, rather than allowing users direct access to the data storage backend, you can log and verify all access to your data.

👩‍🔬 Clear distinction between data curator and analyst roles 👩‍💻

It is desirable to separate out two tasks:

the definition of data-source specifications
accessing and using data

This is so that those who understand the origins of the data and the implications of various formats and other storage options (such as chunk-size) should make those decisions and encode what they have done into specs. It leaves the data users, e.g., data scientists, free to find and use the data-sets appropriate for their work and simply get on with their job — without having to learn about various storage formats and access APIs. Main character energy for everyone involved, honestly. 💅

This separation is at the very core of what Intake was designed to do. Fr fr.

🧠 Users to be able to access data without learning every backend API 🧠

Data formats and services are a wide mess of many libraries and APIs. A large amount of time can be wasted in the life of a data scientist or engineer just finding out the details of the ones required by their work. Not the vibe. 😭

Intake wraps these various libraries, REST APIs and similar, to provide a consistent experience for the data user. source.read() will simply get all of the data into memory in the container type for that source — no further parameters or knowledge required. That's it. That's the whole thing. 🎀

Even for the curator of data catalogs or data driver authors, the framework established by Intake provides a lot of convenience and simplification which allows each person to deal with only the specifics of their job.

🦋 Data sources to be self-describing 🦋

Having a bunch of files in some directory is a very common pattern for data storage in the wild. There may or may not be a README file co-located giving some information in a human-readable form, but generally not structured — such files are usually different in every case. It's giving chaos. 🌈

When a data source is encoded into a catalog, the spec offers a natural place to describe what that data is, along with the possibility to provide an arbitrary amount of structured metadata and to describe any parameters that are to be exposed for user choice.

Furthermore, Intake data sources each have a particular container type, so that users know whether to expect a dataframe, array, etc., and simple introspection methods like describe and discover which return basic information about the data without having to load all of it into memory first. Chef's kiss. 💖

🌳 A data source hierarchy for natural structuring 🌳

Usually, the set of data sources held by an organisation have relationships to one another, and would be poorly served to be provided as a simple flat list of everything available. Imagine that. Not cute.

Intake allows catalogs to refer to other catalogs. This means you can group data sources by various facets (type, department, time...) and establish hierarchical data-source trees within which to find the particular data most likely to be of interest. Since the catalogs live outside and separate from the data files themselves, as many hierarchy structures as thought useful could be created. ✨

For even more complicated data source meta-structures, it is possible to store all the details and even metadata in some external service (e.g., traditional SQL tables) with which Intake can interact to perform queries and return particular subsets of the available data sources.

🌈 Expose several data collections under a single system 🌈

There are already several catalog-like data services in existence in the world, and some organisation may have several of these in-house for various different purposes. For example:

an SQL server may hold details of customer lists and transactions
but historical time-series and reference data may be held separately in archival data formats like parquet on a file-storage system
while real-time system monitoring is done by a totally unrelated system such as Splunk or elastic search

Of course, Intake can read from various file formats and data services. However, it can also interpret the internal conception of data catalogs that some data services may have. For example, all of the tables known to the SQL server, or all of the pre-defined queries in Splunk can be automatically included as catalogs in Intake, and take their place amongst the regular YAML-specified data sources — with exactly the same usage for all of them. Obsessed. 💜

These data sources and their hierarchical structure can then be exposed via the graphical data browser, for searching, selecting and visualising data-sets.

📊 Modern visualisations for all data-sets 📊

Intake is integrated with the comprehensive holoviz suite, particularly hvplot, to bring simple yet powerful data visualisations to any Intake data source by using just one single method for everything. One. Method. For. Everything. 🔥

These plots are interactive, and can include server-side dynamic aggregation of very large data-sets to display more data points than the browser can handle. Big data, tiny vibes. 💅

You can specify specific plot types right in the data source definition, to have these customised visualisations available to the user as simple one-liners known to reveal the content of the data — or even view the same visuals right in the graphical data source browser application. Thus, Intake is already an all-in-one data investigation and dashboarding app. Ate. 👑

⚡ Update data specifications in real time ⚡

Intake data catalogs are not limited to reading static specification from files. They can also execute queries on remote data services and return lists of data sources dynamically at runtime.

New data sources may appear, for example, as directories of data files are pushed to a storage service, or new tables are created within a SQL server. It's giving ✨ alive ✨.

📦 Distribute data in a custom format 📦

Sometimes, the well-known data formats are just not right for a given data-set, and a custom-built format is required. In such cases, the code to read the data may not exist in any library. And that's okay, bestie. 🙈

Intake allows for code to be distributed along with data source specs/catalogs or even files in a single conda package. That encapsulates everything needed to describe and use that particular data, and can then be distributed as a single entity, and installed with a one-liner. Slay.

Furthermore, should the few builtin container types (sequence, array, dataframe) not be sufficient, you can supply your own, and then build drivers that use it. This was done, for example, for xarray-type data, where multiple related N-D arrays share a coordinate system and metadata. By creating this container, a whole world of scientific and engineering data was opened up to Intake. 🦋

Creating new containers is not hard, though, and we foresee more coming, such as machine-learning models and streaming/real-time data.

🎀 Create Intake data-sets from scratch 🎀

If you have a set of files or a data service which you wish to make into a data-set, so that you can include it in a catalog, you should use the set of functions intake.open_*, where you need to pick the function appropriate for your particular data. You can use tab-completion to list the set of data drivers you have installed, and find others you may not yet have installed at Plugin Directory.

Once you have determined the right set of parameters to load the data in the manner you wish, you can use the source's .yaml() method to find the spec that describes the source, so you can insert it into a catalog (with appropriate description and metadata). Alternatively, you can open a YAML file as a catalog with intake.open_catalog and use its .add() method to insert the source into the corresponding file. 💖

If, instead, you have data in your session in one of the containers supported by Intake (e.g., array, data-frame), you can use the intake.upload() function to save it to files in an appropriate format and a location you specify, and give you back a data-source instance, which, again, you can use with .yaml() or .add(), as above. Periodt. ✨

✨ OVERVIEW ✨

💬 Introduction

Okay so, real talk — this page is the technical design of Intake, and it also gives you the brief ✨sparkle✨ version of what the project is even trying to do, plus a little tour of the library's components. That's it, that's the assignment. 🫶

🦋 Why Intake?

ngl, Intake exists because a whole set of problems kept showing up and being annoying, and here they are:

The Python API standards for loading data (think DB-API 2.0 and friends) are optimized for transactional databases and for query results that get processed one row at a time. Which is… not it, for a lot of us.
Libraries that do load data in bulk? Each one has its own whole API for it, so every time you switch data formats you're back to square one and it's giving friction 💀.
Loading data into a distributed data structure (like the ones Dask and Spark have) often means writing a completely separate loader. Every. Single. Time. 😭
Abstractions tend to hyperfixate on exactly one data model — tabular, n-dimensional array, or semi-structured — when so many projects genuinely need to work with multiple kinds of data at once.

And here's the thing, bestie: Intake has the explicit goal of not defining a computational expression system. Not its job, not its vibe. Intake plugins load the data into containers (like arrays or data-frames) and those containers bring their own data processing features to the party. Which means making a brand new Intake plugin is genuinely easy and takes a relatively small amount of Python. Iconic behavior honestly. 👑

🧚 Structure

Intake is a Python library for accessing data in a simple and uniform way, and it's built from three parts:

A lightweight plugin system for adding data loader drivers for new file formats and servers (like databases, REST endpoints, or other cataloging services).
A cataloging system for specifying these sources in simple YAML syntax — or with plugins that read source specs from some external data service.
A server-client architecture that can share data catalog metadata over the network, or even stream the data directly to clients if needed.

Intake supports loading data into standard Python containers. The list can be easily extended, but rn the currently supported list is:

Pandas Dataframes — tabular data
NumPy Arrays — tensor data
Python lists of dictionaries — semi-structured data

And on top of that, Intake can also load data into distributed data structures. Currently it supports Dask, a flexible parallel computing library with distributed containers like dask.dataframe, dask.array, and dask.bag. In the future, other distributed computing systems could use Intake to create similar data structures. 👀

💅 Concepts

Intake is built out of four core concepts, and yes, you need all four:

Data Source classes — the "driver" plugins, where each one implements loading of some specific type of data into python, with plugin-specific arguments.
Data Source — an object that represents a reference to a data source. Data source instances have methods for loading the data into standard containers, like Pandas DataFrames, but do not load any data until specifically requested. Lazy queen behavior, we love her. 💖
Catalog — an inventory of catalog entries, each of which defines a Data Source. Catalog objects can be created from local YAML definitions, by connecting to remote servers, or by some driver that knows how to query an external data service.
Catalog Entry — a named data source held internally by catalog objects, which generate data source instances when accessed. The catalog entry includes metadata about the source, plus the name of the driver and its arguments. And here's the fun part: arguments can be parameterized, so one single entry can return different subsets of data depending on what the user requests. ✨

So the business of a plugin is basically: go from some data format (a bunch of files, or some remote service) to a "Container" of the data (e.g., a data-frame) — a thing you can then do further analysis on. Drivers can be used directly by the user, or indirectly through data catalogs. And data sources can be pickled, sent over the network to other hosts, and reopened — assuming the remote system has access to the required files or servers, obvi.

🌈 Future Directions

Ongoing work for enhancements, plus requests for plugins and all that good stuff, lives over at the issue tracker. And if you want the general mid- and long-term goals, go check out the Roadmap. 🗺️

🎀 Examples

Here we list links to notebooks and other code demonstrating the use of Intake in various scenarios. The first section is of general interest to various users, and the sections that follow tend to be more specific about particular features and workflows.

Many of the entries here include a link to Binder, which is a service that lets you execute code live in a notebook environment. This is a great way to experience using Intake. Fair warning though — it can take a while, sometimes, for Binder to come up, so please have patience bestie, it's doing its best. 🙈

See also the examples repository, containing data-sets which can be built and installed as conda packages.

💜 General

Basic Data scientist workflow: using Intake [Static] [Executable].
Workflow for creating catalogs: a Data Engineer's approach to Intake [Static] [Executable].

🔥 Developer

Tutorials delving deeper into the Internals of Intake, for those who wish to contribute:

How you would go about writing a new plugin [Static] [Executable].

⭐ Features

More specific examples of Intake functionality:

Caching:
- New-style data package creation [Static]
- Using automatically cached data-files [Static] [Executable]
- Earth science demonstration of cached dataset [Static] [Executable]
File-name pattern parsing:
- Satellite imagery, science workflow [Static] [Executable]
- How to set up pattern parsing [Static] [Executable]
Custom catalogs:
- A custom intake plugin that adapts DCAT catalogs [Static] [Executable]

🍓 Data

Anaconda package data, originally announced in this blog
Planet Four Catalog, originally from https://www.planetfour.org/results
The official Intake examples

🌸 Blogs

These are Intake-related articles that may be of interest:

Discovering and Exploring Data in a Graphical Interface
Taking the Pain out of Data Access
Caching Data on First Read Makes Future Analysis Faster
Parsing Data from Filenames and Paths
Intake for cataloguing Spark
Intake released on Conda-Forge

🦋 Talks

__init__ podcast interview (May 2019)
AnacondaCon (March 2019)
PyData DC (November 2018)
PyData NYC (October 2018)
ESIP tech dive (November 2018)

👑 News

See our Wiki page.

🌟 Deployment Scenarios

In the following sections, we will describe some of the ways in which Intake is used in real production systems. And bestie, these go well beyond the typical YAML files you saw in the quickstart and examples sections — those are necessarily short and simple, and honestly they do not demonstrate the full power of Intake. Consider this the glow-up. 💅

This is the simplest scenario, and it's amply described in these documents. The primary advantage is simplicity: it is enough to put a file in an accessible place (even a gist or a repo), in order for someone else to be able to discover and load that data. Furthermore, such files can easily refer to one-another, to build up a full tree of data assets with minimum pain. And since YAML files are text, this also lends itself to working well with version control systems. ✨ Furthermore — say it with me — all sources can describe themselves as YAML, and the export and upload commands can produce an efficient format (possibly remote) together with a YAML definition in a single step. Ate.

🌍 Pangeo

The Pangeo collaboration uses Intake to catalog their data holdings, which are generally in various forms of netCDF-compliant formats — massive multi-dimensional arrays with data relating to earth and climate science and meteorology. On their cloud-based platform, containers start up jupyter-lab sessions which have Intake installed, and therefore can simply pick and load the data that each researcher needs — often requiring large Dask clusters to actually do the processing. Big data, big energy. 🔥

A static rendering of the catalog contents is available, so that users can browse the holdings without even starting a python session. This rendering is produced by CI on the repo whenever new definitions are added, and it also checks (using Intake, ofc) that each definition is indeed loadable.

Pangeo also developed intake-stac, which can talk to STAC servers to make real-time queries and parse the results into Intake data sources. This is a standard for spatio-temporal data assets, and it indexes massive amounts of cloud-stored data. ☁️

🎀 Anaconda Enterprise

Intake will be the basis of the data access and cataloging service within Anaconda Enterprise, running as a micro-service in a container, and offering data source definitions to users. The access control, who gets to see which data-set, and serving of credentials to be able to read from the various data storage services — all of that will be handled by the platform and be fully configurable by admins. Very much permissions queen energy.

🧊 National Center for Atmospheric Research

NCAR has developed intake-esm, a mechanism for creating file-based Intake catalogs for climate data from project efforts such as the Coupled Model Intercomparison Project (CMIP) and the Community Earth System Model (CESM) Large Ensemble Project. These projects produce a huge amount of climate data persisted on tape, disk storage components across multiple (of the order ~300,000) netCDF files. Finding, investigating, and loading these files into data array containers such as xarray can be a daunting task due to the large number of files a user may be interested in. 😭 Intake-esm addresses this issue in three steps:

Dataset Catalog Curation_ in form of YAML files. These YAML files provide information about data locations, access pattern, directory structure, etc. intake-esm uses these YAML files in conjunction with file name templates to construct a local database. Each row in this database consists of a set of metadata such as experiment, modeling realm, frequency corresponding to data contained in one netCDF file.

cat = intake.open_esm_metadatastore(catalog_input_definition="GLADE-CMIP5")

Search and Discovery: once the database is built, intake-esm can be used for searching and discovering of climate datasets by eliminating the need for the user to know specific locations (file path) of their data set of interest:

sub_cat = cat.search(variable=['hfls'], frequency='mon', modeling_realm='atmos', institute=['CCCma', 'CNRM-CERFACS'])

Access: when the user is satisfied with the results of their query, they can ask intake-esm to load the actual netCDF files into xarray datasets:

dsets = cat.to_xarray(decode_times=True, chunks={'time': 50})

🦋 Brookhaven Archive

The Bluesky project uses Intake to dynamically query a MongoDB instance, which holds the details of experimental and simulation data catalogs, to return a custom Catalog for every query. Data-sets can then be loaded into python, or the original raw data can be accessed ...

Zillow

okay so Zillow is out here literally building Intake to serve their datalake access layer (DAL), because their data is extremely hierarchical and they needed something to wrap it up all cute. The thing they care about most? Being able to hand out different versions (testing vs production, and different storage formats, bestie — the whole vibe) of the same logical dataset, depending on whether you're reading it on your laptop or on the production infrastructure ... 💅

Intake Server

The server protocol (go peek at Server Protocol) is simple enough that honestly anyone can write their own implementation, with full customisation and behaviour of their own. In particular, auth and monitoring would be essential for a production-grade deployment. fr fr. 🔐

🌸 USER GUIDE 🌸

More detailed info about specific parts of Intake — like how to author catalogs, how to use the graphical interface, plotting, etc. ✨

🎀 GUI 🎀

Using the GUI

Note: the GUI requires panel and bokeh to be available in the current environment. 👀

The Intake top-level singleton intake.gui gives you access to a graphical data browser right inside your Jupyter notebook. To expose it, literally just enter it into a code cell (Jupyter automatically displays the last object in a code cell). 📸 [image]

New instances of the GUI are also available by instantiating intake.interface.gui.GUI, where you can specify a list of catalogs to initially include. so many options, we love that for us 💖

The GUI contains three main areas:

a list of catalogs. The "builtin" catalog, displayed by default, includes data-sets installed in the system, the same as intake.cat.
a list of sources within the currently selected catalog.
a description of the currently selected source.

Catalogs

Selecting a catalog from the list will display nested catalogs below the parent and display source entries from the catalog in the list of sources. 🦋

Below the lists of catalogs is a row of buttons used for adding, removing and searching-within catalogs:

Add: opens a sub-panel for adding catalogs to the interface, by either browsing for a local YAML file or by entering a URL for a catalog, which can be a remote file or Intake server
Remove: deletes the currently selected catalog from the list
Search: opens a sub-panel for finding entries in the currently selected catalog (and its sub-catalogs)

Add Catalogs

The Add button (+) exposes a sub-panel with two main ways to add catalogs to the interface: 📸 [image]

This panel has a tab to load files from local; from that you can navigate around the filesystem using the arrow or by editing the path directly. Use the home button to get back to the starting place. Select the catalog file you need. Use the "Add Catalog" button to add the catalog to the list above. 📸 [image]

Another tab loads a catalog from remote. Any URL is valid here, including cloud locations, "gcs://bucket/...", and intake servers, "intake://server:port". Without a protocol specifier, this can be a local path. Again, use the "Add Catalog" button to add the catalog to the list above. 📸 [image]

Finally, you can add catalogs to the interface in code, using the .add() method, which can take filenames, remote URLs or existing Catalog instances. 💅

Remove Catalogs

The Remove button (-) deletes the currently selected catalog from the list. Important though, bestie: this action does not have any impact on files, it only affects what shows up in the list. your files are safe, breathe 🫶 📸 [image]

Search

The sub-panel opened by the Search button (🔍) allows the user to search within the selected catalog 📸 [image]

From the Search sub-panel the user enters for free-form text. Since some catalogs contain nested sub-catalogs, the Depth selector allows the search to be limited to the stated number of nesting levels. This may be necessary, since, in theory, catalogs can contain circular references, and therefore allow for infinite recursion. 😭 (we do not want infinite recursion, no ma'am) 📸 [image]

Upon execution of the search, the currently selected catalog will be searched. Entries will be considered to match if any of the entered words is found in the description of the entry (this is case-insensitive). If any matches are found, a new entry will be made in the catalog list, with the suffix "_search". 📸 [image]

Sources

Selecting a source from the list updates the description text on the left-side of the gui.

Below the list of sources is a row of buttons for inspecting the selected data source:

Plot: opens a sub-panel for viewing the pre-defined (specified in the yaml) plots for the selected source.

Plot

The Plot button (📊) opens a sub-panel with an area for viewing pre-defined plots. 📸 [image]

These plots are specified in the catalog yaml and that yaml can be displayed by checking the box next to "show yaml". 📸 [image]

The holoviews object can be retrieved from the gui using intake.interface.source.plot.pane.object, and you can then use it in Python or export it to a file. iconic ✨

Interactive Visualization

If you have installed the optional extra packages dfviz and xrviz, you can interactively plot your dataframe or array data, respectively. 📸 [image]

The button "customize" will be available for data sources of the appropriate type. Click this to open the interactive interface. If you have not selected a predefined plot (or there are none), then the interface will start without any prefilled values, but if you do first select a plot, then the interface will have its options pre-filled from the options

For specific instructions on how to use the interfaces (which can also be used independently of the Intake GUI), please navigate to the linked documentation.

Note that the final parameters that are sent to hvPlot to produce the output each time a plot if updated, are explicitly available in YAML format, so that you can save the state as a "predefined plot" in the catalog. The same set of parameters can also be used in code, with datasource.plot(...). 📸 [image]

Using the Selection

Once catalogs are loaded and the desired sources has been identified and selected, the selected sources will be available at the .sources attribute (intake.gui.sources). Each source entry has informational methods available and can be opened as a data source, as with any catalog entry:

In [ ]: source_entry = intake.gui.sources[0]
        source_entry
Out   :
name: sea_ice_origin
container: dataframe
plugin: ['csv']
description: Arctic/Antarctic Sea Ice
direct_access: forbid
user_parameters: []
metadata:
args:
  urlpath: https://timeseries.weebly.com/uploads/2/1/0/8/21086414/sea_ice.csv

In [ ]: data_source = source_entry()  # may specify parameters here
        data_source.read()
Out   : < some data >

In [ ]: source_entry.plot()  # or skip data source step
Out   : < graphics>

Catalogs

Data catalogs provide an abstraction that allows you to externally define, and optionally share, descriptions of datasets, called catalog entries. A catalog entry for a dataset includes information like: 👑

The name of the Intake driver that can load the data
Arguments to the __init__() method of the driver
Metadata provided by the catalog author (such as field descriptions and types, or data provenance)

In addition, Intake allows the arguments to data sources to be templated, with the variables explicitly expressed as "user parameters". The given arguments are rendered using jinja2, the values of named user parameterss, and any overrides. The parameters are also offer validation of the allowed types and values, for both the template values and the final arguments passed to the data source. The parameters are named and described, to indicate to the user what they are for. This kind of structure can be used to, for example, choose between two parts of a given data source, like "latest" and "stable", see the entry1_part entry in the example below. 🧚

The user of the catalog can always override any template or argument value at the time that they access a give source. periodt. 💅

The Catalog class

In Intake, a Catalog instance is an object with one or more named entries. The entries might be read from a static file (e.g., YAML, see the next section), from an Intake server or from any other data service that has a driver. Drivers which create catalogs are ordinary DataSource classes, except that they have the container type "catalog", and do not return data products via the read() method.

For example, you might choose to instantiate the base class and fill in some entries explicitly in your code

from intake.catalog import Catalog
from intake.catalog.local import LocalCatalogEntry
mycat = Catalog.from_dict({
    'source1': LocalCatalogEntry(name, description, driver, args=...),
    ...
    })

Alternatively, subclasses of Catalog can define how entries are created from whichever file format or service they interact with, examples including RemoteCatalog and SQLCatalog. These generate entries based on their respective targets; some provide advanced search capabilities executed on the server. 🔥

YAML Format

Intake catalogs can most simply be described with YAML files. This is very common in the tutorials and this documentation, because it simple to understand, but demonstrate the many features of Intake. Note that YAML files are also the easiest way to share a catalog, simply by copying to a publicly-available location such as a cloud storage bucket. Here is an example:

metadata:
  version: 1
  parameters:
    file_name:
      type: str
      description: default file name for child entries
      default: example_file_name
sources:
  example:
    description: test
    driver: random
    args: {}

  entry1_full:
    description: entry1 full
    metadata:
      foo: 'bar'
      bar: [1, 2, 3]
    driver: csv
    args: # passed to the open() method
      urlpath: '{{ CATALOG_DIR }}/entry1_*.csv'

  entry1_part:
    description: entry1 part
    parameters: # User parameters
      part:
        description: section of the data
        type: str
        default: "stable"
        allowed: ["latest", "stable"]
    driver: csv
    args:
      urlpath: '{{ CATALOG_DIR }}/entry1_{{ part }}.csv'

  entry2:
    description: entry2
    driver: csv
    args:
      # file_name parameter will be inherited from file-level parameters, so will
      # default to "example_file_name"
      urlpath: '{{ CATALOG_DIR }}/entry2/{{ file_name }}.csv`

✨ Metadata ✨

So you wanna shove some random descriptive extra info in there? Go off, bestie — the metadata section is literally made for that 💖

Some fields get claimed for internal use, and some might be locked down to local reading only, but for now? The only field anyone's expecting is version, and that gets bumped whenever a breaking change happens to the file format. Every catalog is gonna have .metadata and .version attributes available, no cap 👑

Also heads up: each source has its own metadata too. Don't forget it exists 🎀

Oh and — the metadata section can also hold parameters, and those get inherited by the sources in the file. Lowkey important: those sources can totally augment those parameters, or straight up override them with their own parameters. Main character energy, honestly 💅

✨ Extra drivers ✨

The driver: entry of a data source spec can just be a driver name, like all the examples we've been vibing with so far. But it can also be an absolute class path to use for the data source — and then there's zero ambiguity about how to load the data. That's the preferred way to be explicit when the driver name alone isn't enough (peep Driver Selection, below 🦋).

plugins:
  source:
    - module: intake.catalog.tests.example1_source
sources:
  ...

But fr fr, you don't generally need to do this, because the driver: field of each source can also explicitly refer to the plugin class anyway. Slay 💖

✨ Sources ✨

The majority of a catalog file is composed of data sources — these are named data sets that can be loaded for the user 🌸 Catalog authors describe the contents of a data set, how to load it, and optionally offer some customization of the returned data. Each data source has several attributes:

name — the canonical name of the source. Best practice? Compose source names from valid Python identifiers, because that lets Intake support things like tab completion of data source names on catalog objects ✨ For example, monthly_downloads is a good source name. Chef's kiss.
description — human readable description of the source. To help catalog browsing tools, this should be Markdown 📝
driver — name of the Intake Driver to use with this source. It must either already be installed in the current Python environment (i.e. with conda or pip) or loaded in the plugin section of the file. Can be a simple driver name like "csv" or the full path to the implementation class like "package.module.Class".
args — keyword arguments to the init method of the driver. Arguments may use template expansion 🎀
metadata — any metadata keys that should be attached to the data source when opened. These will be supplemented by additional metadata provided by the driver. Catalog authors can use whatever key names they want, with one exception: keys starting with a leading underscore are reserved for future internal use by Intake 🙈
direct_access — control whether the data is directly accessed by the client, or proxied through a catalog server. See Server Catalogs for more deets.
parameters — a dictionary of data source parameters. More details below 👇

✨ Caching Source Files Locally ✨

⚠️ This method of defining the cache with a dedicated block is deprecated — see the Remote Access section, below 💀

To enable caching on the first read of remote data source files, add the cache section with these attributes:

argkey — the args section key which contains the URL(s) of the data to be cached.
type — one of the keys in the cache registry [intake.source.cache.registry], referring to an implementation of caching behaviour. The default is "file" for the caching of one or more files.

Example:

test_cache:
  description: cache a csv file from the local filesystem
  driver: csv
  cache:
    - argkey: urlpath
      type: file
  args:
    urlpath: '{{ CATALOG_DIR }}/cache_data/states.csv'

The cache_dir defaults to ~/.intake/cache, and can be specified in the intake configuration file or the INTAKE_CACHE_DIR environment variable, or at runtime using the "cache_dir" key of the configuration 💖 The special value "catdir" implies that cached files will appear in the same directory as the catalog file in which the data source is defined, inside a directory named "intake_cache". Heads up though: those will not appear in the cache usage reported by the CLI 🙈

Optionally, the cache section can have a regex attribute that modifies the path of the cache on the disk ✨ By default, the cache path is made by concatenating cache_dir, dataset name, hash of the url, and the url itself (without the protocol). The regex attribute lets you remove part of the url (the matching part). Iconic.

Caching can be disabled at runtime for all sources, regardless of the catalog specification:

from intake.config import conf

conf['cache_disabled'] = True

By default, progress bars are shown during downloads if the package tqdm is available 🦋 but this can be disabled (e.g., for consoles that don't support complex text) with

conf['cache_download_progress'] = False

or, equivalently, the environment parameter INTAKE_CACHE_PROGRESS.

The "types" of caching that are supported are listed in intake.source.cache.registry — peep the docstrings of each for the specific parameters that should appear in the cache block 👑

It is possible to work with compressed source files by setting type: compression in the cache specification. By default the compression type is inferred from the file extension, otherwise it can be set by assigning the decomp variable to any of the options listed in intake.source.decompress.decomp 🌸 This will extract all the file(s) in the compressed file referenced by urlpath and store them in the cache directory.

In cases where miscellaneous files are present in the compressed file, a regex_filter parameter can be used — only the extracted filenames that match the pattern will be loaded ✨ The cache path is appended to the filename, so it is necessary to include a wildcard at the beginning of the pattern. Don't forget, bestie:

Example:

test_compressed:
  driver: csv
  args:
    urlpath: 'compressed_file.tar.gz'
  cache:
    - type: compressed
      decomp: tgz
      argkey: urlpath
      regex_filter: '.*data.csv'

✨ Templating ✨

Intake catalog files support Jinja2 templating for driver arguments 💖 Any occurrence of a substring like {{field}} will be replaced by the value of the user parameters with that same name, or the value explicitly provided by the user. Wanna know how to specify those user parameters? Next section, babe 👇

Some additional values are available for templating. This one is always available: CATALOG_DIR, the full path to the directory containing the YAML catalog file ✨ This is especially useful for constructing paths relative to the catalog directory to locate data files and custom drivers. For example, the search for CSV files for the two "entry1" blocks, above, will happen in the same directory as where the catalog file was found 🎀

The following functions may be available. Since these execute code, the user of a catalog may decide whether they trust those functions or not 🙈

env("USER") — look in the set environment variables for the named variable.
client_env("USER") — exactly the same, except that when using a client-server topology, the value will come from the environment of the client.
shell("get_login thisuser -t") — execute the command, and use the output as the value. The output will be trimmed of any trailing whitespace.
client_shell("get_login thisuser -t") — exactly the same, except that when using a client-server topology, the value will come from the system of the client.

The reason for the "client" versions of the functions is to prevent leakage of potentially sensitive information between client and server by controlling where lookups happen 🔒 When working without a server, only the ones without "client" are used.

An example:

sources:
  personal_source:
    description: This source needs your username
    args:
      url: "http://server:port/user/{{env(USER)}}"

Here, if the user is named "blogs", the url argument will resolve to "http://server:port/user/blogs"; if the environment variable is not defined, it will resolve to "http://server:port/user/" 🌈

✨ Parameter Definition ✨

🌸 Source parameters 🌸

A source definition can contain a "parameters" block. Expressed in YAML, a parameter may look as follows:

parameters:
  name:
    description: name to use  # human-readable text for what this parameter means
    type: str  # one of bool, str, int, float, list[str | int | float], datetime, mlist
    default: normal  # optional, value to assume if user does not override
    allowed: ["normal", "strange"]  # optional, list of values that are OK, for validation
    min: "n"  # optional, minimum allowed, for validation
    max: "t"  # optional, maximum allowed, for validation

A parameter — and do not confuse it with an argument, bestie — can have one of two uses:

to provide values for variables to be used in templating the arguments. If the pattern "{{name}}" exists in any of the source arguments, it will be replaced by the value of the parameter. If the user provides a value (e.g., source = cat.entry(name='something")), that will be used, otherwise the default value. If there is no user input or default, the empty value appropriate for type is used. The default field allows for the same function expansion as listed for arguments, above ✨
If an argument with the same name as the parameter exists, its value, after any templating, will be coerced to the given type of the parameter and validated against the allowed/max/min. It is therefore possible to use the string templating system (e.g., to get a value from the environment), but pass the final value as, for example, an integer 🦋 It makes no sense to provide a default for this case (the argument already has a value), but providing a default will not raise an exception.
the "mlist" type is special: it means that the input must be a list, whose values are chosen from the allowed list 💅 This is the only type where the parameter value is not the same type as the allowed list's values, e.g., if a list of str is set for allowed, a list of str must also be the final value.

Note: the datetime type accepts multiple values: Python datetime, ISO8601 string, Unix timestamp int, "now" and "today" 👑

🎀 Catalog parameters 🎀

You can also define user parameters at the catalog level. This applies the parameter to all entries within that catalog, without having to define it for each and every entry — imagine the time saved, we love efficiency ✨ Furthermore, catalogs nested within the catalog will also inherit the parameter(s) 🌈

For example, bestie, with this spec right here:

metadata:
  version: 1
  parameters:
    bucket:
      type: str
      description: description
      default: test_bucket
sources:
  param_source:
    driver: parquet
    description: description
    args:
      urlpath: s3://{{bucket}}/file.parquet
  subcat:
    driver: yaml_file
    path: "{{CATALOG_DIR}}/other.yaml"

If cat is the corresponding catalog instance, then the URL of source cat.param_source will evaluate to "s3://test_bucket/file.parquet" by default — cute, right? 💅 But plot twist: you can override that parameter with cat.param_source(bucket="other_bucket"). Also, any entries of subcat — that's another catalog referenced from here — would also have the "bucket"-named parameter attached to all sources. And of course, those sources do no need to make use of the parameter. It's just vibing there, available. 🦋

To change the default, we can generate a new instance:

cat2 = cat(bucket="production")  # sets default value of "bucket" for cat2
subcat = cat.subcat(bucket="production")  # sets default only for the nested catalog

Of course, in these situations you can still override the value of the parameter for any source, or pass explicit values for the arguments of the source, as normal. No rules broken, no cap. ✨

For cases where the catalog is not defined in a YAML spec, the argument user_parameters to the constructor takes the same form as parameters above: a dict of user parameters, either as UserParameter instances or as a dictionary spec for each one.

✨ Templating parameters ✨

Template functions can also be used in parameters (see Templating, above) — but here's the slay part: you can use the available functions directly without the extra {{...}}. 🎀

For example, this catalog entry uses the env("HOME") functionality as described to set a default based on the user's home directory.

sources:
  variabledefault:
    description: "This entry leads to an example csv file in the user's home directory by default, but the user can pass root=\"somepath\" to override that."
    driver: csv
    args:
      path: "{{root}}/example.csv"
    parameters:
      root:
        description: "root path"
        type: str
        default: "env(HOME)"

🌈 Driver Selection 🌈

In some cases, it may be possible that multiple backends are capable of loading from the same data format or service. Sometimes that means two drivers with unique names, or a single driver with a parameter to choose between the different backends. It's giving "too many options, all of them valid." 🙈

However, it is possible that multiple drivers for reading a particular type of data also share the same driver name: for example, both the intake-iris and the intake-xarray packages contain drivers with the name "netcdf", which are capable of reading the same files, but with different backends. Here we will describe the various possibilities of coping with this situation. Intake's plugin system makes it easy to encode such choices. 👑

It may be acceptable to use any driver which claims to handle that data type, or to give the option of which driver to use to the user, or it may be necessary to specify which precise driver(s) are appropriate for that particular data. Intake allows all of these possibilities, even if the backend drivers require extra arguments.

Specifying a single driver explicitly, rather than using a generic name, would look like this:

sources:
  example:
    description: test
    driver: package.module.PluginClass
    args: {}

It is also possible to describe a list of drivers with the same syntax. The first one found will be the one used. Note that the class imports will only happen at data source instantiation, i.e., when the entry is selected from the catalog. So it's lazy, it's chill, it's not doing work it doesn't have to. 💅

sources:
  example:
    description: test
    driver:
      - package.module.PluginClass
      - another_package.PluginClass2
    args: {}

These alternative plugins can also be given data-source specific names, allowing the user to choose at load time with driver= as a parameter. Additional arguments may also be required for each option (which, as usual, may include user parameters); however, the same global arguments will be passed to all of the drivers listed.

sources:
  example:
    description: test
    driver:
      first:
        class: package.module.PluginClass
        args:
          specific_thing: 9
      second:
        class: another_package.PluginClass2
    args: {}

🦋 Remote Access 🦋

(see also Remote Data for the implementation details)

Many drivers support reading directly from remote data sources such as HTTP, S3 or GCS. In these cases, the path to read from is usually given with a protocol prefix such as gcs://. Additional dependencies will typically be required (requests, s3fs, gcsfs, etc.), any data package should specify these. Further parameters may be necessary for communicating with the storage backend and, by convention, the driver should take a parameter storage_options containing arguments to pass to the backend. Some remote backends may also make use of environment variables or config files to determine their default behaviour. 🤝

The special template variable "CATALOG_DIR" may be used to construct relative URLs in the arguments to a source. In such cases, if the filesystem used to load that catalog contained arguments, then the storage_options of that file system will be extracted and passed to the source. Therefore, all sources which can accept general URLs (beyond just local paths) must make sure to accept this argument.

As an example of using storage_options, the following two sources would allow for reading CSV data from S3 and GCS backends without authentication (anonymous access), respectively:

sources:
  s3_csv:
    driver: csv
    description: "Publicly accessible CSV data on S3; requires s3fs"
    args:
      urlpath: s3://bucket/path/*.csv
      storage_options:
        anon: true
  gcs_csv:
    driver: csv
    description: "Publicly accessible CSV data on GCS; requires gcsfs"
    args:
      urlpath: gcs://bucket/path/*.csv
      storage_options:
        token: "anon"

💜 Using S3 Profiles 💜

An AWS profile may be specified as an argument under storage_options via the following format:

args:
  urlpath: s3://bucket/path/*.csv
  storage_options:
    profile: aws-profile-name

🍓 Caching 🍓

URLs interpreted by fsspec offer automatic caching. For example, to enable file-based caching for the first source above, you can do:

sources:
  s3_csv:
    driver: csv
    description: "Publicly accessible CSV data on S3; requires s3fs"
    args:
      urlpath: simplecache::s3://bucket/path/*.csv
      storage_options:
        s3:
          anon: true

Here we have added the "simplecache" to the URL (this caching backend does not store any metadata about the cached file) and specified that the "anon" parameter is meant as an argument to s3, not to the caching mechanism. As each file in s3 is accessed, it will first be downloaded and then the local version used instead. Sneaky? No. Efficient? Absolutely. ⭐

You can tailor how the caching works. In particular the location of the local storage can be set with the cache_storage parameter (under the "simplecache" group of storage_options, of course) — otherwise they are stored in a temporary location only for the duration of the current python session. The cache location is particularly useful in conjunction with an environment variable, or relative to "{{CATALOG_DIR}}", wherever the catalog was loaded from.

Please see the fsspec documentation for the full set of cache types and their various options.

🧚 Local Catalogs 🧚

A Catalog can be loaded from a YAML file on the local filesystem by creating a Catalog object:

from intake import open_catalog
cat = open_catalog('catalog.yaml')

Then sources can be listed:

list(cat)

and data sources are loaded via their name:

data = cat.entry_part1

and you can optionally configure new instances of the source to define user parameters or override arguments by calling either of:

data = cat.entry_part1.configure_new(part='1')
data = cat.entry_part1(part='1')  # this is a convenience shorthand

Intake also supports loading a catalog from all of the files ending in .yml and .yaml in a directory, or by using an explicit glob-string. Note that the URL provided may refer to a remote storage systems by passing a protocol specifier such as s3://, gcs://.:

cat = open_catalog('/research/my_project/catalog.d/')

Intake Catalog objects will automatically reload changes or new additions to catalog files and directories on disk. These changes will not affect already-opened data sources. Periodt. 💖

🌸 Catalog Nesting 🌸

A catalog is just another type of data source for Intake. For example, you can print a YAML specification corresponding to a catalog as follows:

cat = intake.open_catalog('cat.yaml')
print(cat.yaml())

results in:

sources:
  cat:
    args:
      path: cat.yaml
    description: ''
    driver: intake.catalog.local.YAMLFileCatalog
    metadata: {}

The point here, is that this can be included in another catalog. (It would, of course, be better to include a description and the full path of the catalog file here.) If the entry above were saved to another file, "root.yaml", and the original catalog contained an entry, data, you could access it as:

root = intake.open_catalog('root.yaml')
root.cat.data

It is, therefore, possible to build up a hierarchy of catalogs referencing each other. These can, of course, include remote URLs and indeed catalog sources other than simple files (all the tables on a SQL server, for instance). Plus, since the argument and parameter system also applies to entries such as the example above, it would be possible to give the user a runtime choice of multiple catalogs to pick between, or have this decision depend on an environment variable. Catalog-ception, and honestly? Iconic. 👑

🎀 Server Catalogs 🎀

Intake also includes a server which can share an Intake catalog over HTTP (or HTTPS with the help of a TLS-enabled reverse proxy). From the user perspective, remote catalogs function identically to local catalogs:

cat = open_catalog('intake://catalog1:5000')
list(cat)

The difference is that operations on the catalog translate to requests sent to the catalog server. Catalog servers provide access to data sources in one of two modes:

Direct access — in this mode, the catalog server tells the client how to load the data, but the client uses its local drivers to make the connection. This requires the client has the required driver already installed and has direct access to the files or data servers that the driver will connect to. no driver, no vibe check passed 💅
Proxied access — in this mode, the catalog server uses its local drivers to open the data source and stream the data over the network to the client. The client does not need any special drivers to read the data, and can read data from files and data servers that it cannot access, as long as the catalog server has the required access. server said "I gotchu bestie, you just sit there and look cute" 🫶

Whether a particular catalog entry supports direct or proxied access is determined by the direct_access option:

forbid (default) — force all clients to proxy data through the catalog server. no exceptions, everyone goes through the server 🚦
allow — if the client has the required driver, access the source directly, otherwise proxy the data through the catalog server. chill and flexible, we love a diplomat ✨
force — force all clients to access the data directly. If they do not have the required driver, an exception will be raised. 😭 no driver? no data. them's the rules.

⚠️ lowkey important: note that when the client is loading a data source via direct access, the catalog server will need to send the driver arguments to the client. Do not include sensitive credentials in a data source that allows direct access. fr fr, keep the passwords out of there 🔥

🌸 Client Authorization Plugins 🌸

Intake servers can check if clients are authorized to access the catalog as a whole, or individual catalog entries. Typically a matched pair of server-side plugin (called an "auth plugin") and a client-side plugin (called a "client auth plugin") need to be enabled for authorization checks to work. it's giving "you two are nothing without each other" 👯

This feature is still in early development, but see module intake.auth.secret for a demonstration pair of server and client classes implementation auth via a shared secret. See Authorization Plugins. iykyk 🔐

✨ Command Line Tools ✨

The package installs two executable commands: one for starting the catalog server; and a client for accessing catalogs and manipulating the configuration. two icons, one install 👑

🎀 Configuration 🎀

A file-based configuration service is available to Intake. This file is by default sought at the location ~/.intake/conf.yaml, but either of the environment variables INTAKE_CONF_DIR or INTAKE_CONF_FILE can be used to specify another directory or file. If both are given, the latter takes priority. (yes, INTAKE_CONF_FILE wins the argument. periodt. 💅)

At present, the configuration file might look as follows:

auth:
  cls: "intake.auth.base.BaseAuth"
port: 5000
catalog_path:
  - /home/myusername/special_dir

These are the defaults, and any parameters not specified will take the values above 🌈

the Intake Server will listen on port 5000 (this can be overridden on the command line, see below)
and the auth system used will be the fully qualified class given (which, for BaseAuth, always allows access). For further information on securing the Intake Server, see the Authorization Plugins.

See intake.config.defaults for a full list of keys and their default values.

🦋 Log Level 🦋

The logging level is configurable using Python's built-in logging module.

The config option 'logging' holds the current level for the intake logger, and can take values such as 'INFO' or 'DEBUG'. This can be set in the conf.yaml file of the config directory (e.g., ~/.intake/), or overridden by the environment variable INTAKE_LOG_LEVEL. ✨

Furthermore, the level and settings of the logger can be changed programmatically in code:

import logging
logger = logging.getLogger('intake')
logger.setLevel(logging.DEBUG)
logget.addHandler(..)

(that last line is literally written like that, bestie — not my typo, I promise 💀)

🧚 Intake Server 🧚

The server takes one or more catalog files as input and makes them available on port 5000 by default.

You can see the full description of the server command with:

>>> intake-server --help

usage: intake-server [-h] [-p PORT] [--list-entries] [--sys-exit-on-sigterm]
                     [--flatten] [--no-flatten] [-a ADDRESS]
                     FILE [FILE ...]

Intake Catalog Server

positional arguments:
  FILE                  Name of catalog YAML file

optional arguments:
  -h, --help            show this help message and exit
  -p PORT, --port PORT  port number for server to listen on
  --list-entries        list catalog entries at startup
  --sys-exit-on-sigterm
                        internal flag used during unit testing to ensure
                        .coverage file is written
  --flatten
  --no-flatten
  -a ADDRESS, --address ADDRESS
                        address to use as a host, defaults to the address in
                        the configuration file, if provided otherwise localhost
  usage: intake-server [-h] [-p PORT] [--list-entries] [--sys-exit-on-sigterm]
               [--flatten] [--no-flatten] [-a ADDRESS]
               FILE [FILE ...]

To start the server with a local catalog file, use the following:

>>> intake-server intake/catalog/tests/catalog1.yml
Creating catalog from:
  - intake/catalog/tests/catalog1.yml
catalog_args ['intake/catalog/tests/catalog1.yml']
Entries: entry1,entry1_part,use_example1
Listening on port 5000

You can use the catalog client (defined below) using:

$ intake list intake://localhost:5000
entry1
entry1_part
use_example1

👑 Intake Client 👑

While the Intake data sources will typically be accessed through the Python API, you can use the client to verify a catalog file.

Unlike the server command, the client has several subcommands to access a catalog. You can see the list of available subcommands with:

>>> intake --help
usage: intake {list,describe,exists,get,discover} ...

We go into further detail in the following sections. let's gooo 🚀

📋 List 📋

This subcommand lists the names of all available catalog entries. This is useful since other subcommands require these names.

If you wish to see the details about each catalog entry, use the --full flag. This is equivalent to running the intake describe subcommand for all catalog entries.

>>> intake list --help
usage: intake list [-h] [--full] URI

positional arguments:
  URI         Catalog URI

optional arguments:
  -h, --help  show this help message and exit
  --full

>>> intake list intake/catalog/tests/catalog1.yml
entry1
entry1_part
use_example1
>>> intake list --full intake/catalog/tests/catalog1.yml
[entry1] container=dataframe
[entry1] description=entry1 full
[entry1] direct_access=forbid
[entry1] user_parameters=[]
[entry1_part] container=dataframe
[entry1_part] description=entry1 part
[entry1_part] direct_access=allow
[entry1_part] user_parameters=[{'default': '1', 'allowed': ['1', '2'], 'type': u'str', 'name': u'part', 'description': u'part of filename'}]
[use_example1] container=dataframe
[use_example1] description=example1 source plugin
[use_example1] direct_access=forbid
[use_example1] user_parameters=[]

🔍 Describe 🔍

Given the name of a catalog entry, this subcommand lists the details of the respective catalog entry.

>>> intake describe --help
usage: intake describe [-h] URI NAME

positional arguments:
  URI         Catalog URI
  NAME        Catalog name

optional arguments:
  -h, --help  show this help message and exit

>>> intake describe intake/catalog/tests/catalog1.yml entry1
[entry1] container=dataframe
[entry1] description=entry1 full
[entry1] direct_access=forbid
[entry1] user_parameters=[]

🦋 Discover 🦋

Given the name of a catalog entry, this subcommand returns a key-value description of the data source. The exact details are subject to change. (so don't get too attached, bestie 🙈)

>>> intake discover --help
usage: intake discover [-h] URI NAME

positional arguments:
  URI         Catalog URI
  NAME        Catalog name

optional arguments:
  -h, --help  show this help message and exit

>>> intake discover intake/catalog/tests/catalog1.yml entry1
{'npartitions': 2, 'dtype': dtype([('name', 'O'), ('score', '<f8'), ('rank', '<i8')]), 'shape': (None,), 'datashape':None, 'metadata': {'foo': 'bar', 'bar': [1, 2, 3]}}

✅ Exists ✅

Given the name of a catalog entry, this subcommand returns whether or not the respective catalog entry is valid. it's giving true/false girlie 💅

>>> intake exists --help
usage: intake exists [-h] URI NAME

positional arguments:
  URI         Catalog URI
  NAME        Catalog name

optional arguments:
  -h, --help  show this help message and exit

>>> intake exists intake/catalog/tests/catalog1.yml entry1
True
>>> intake exists intake/catalog/tests/catalog1.yml entry2
False

🍓 Get 🍓

Given the name of a catalog entry, this subcommand outputs the entire data source to standard output.

>>> intake get --help
usage: intake get [-h] URI NAME

positional arguments:
  URI         Catalog URI
  NAME        Catalog name

optional arguments:
  -h, --help  show this help message and exit

>>> intake get intake/catalog/tests/catalog1.yml entry1
       name  score  rank
0    Alice1  100.5     1
1      Bob1   50.3     2
2  Charlie1   25.0     3
3      Eve1   25.0     3
4    Alice2  100.5     1
5      Bob2   50.3     2
6  Charlie2   25.0     3
7      Eve2   25.0     3

look at that dataframe, she ate and left no crumbs 🧚

🗂️ Config and Cache 🗂️

CLI functions starting with intake cache and intake config are available to provide information about the system: the locations and value of configuration parameters, and the state of cached files. very "let me check my notes" energy 📝

💾 Persisting Data 💾

(this is an experimental new feature, expect enhancements and changes) — so like, it's still in her soft launch era 🙈

🌟 Introduction 🌟

As defined in the glossary, to Persist is to convert data into the storage format most appropriate for the container type, and save a copy of this for rapid lookup in the future. This is of great potential benefit where the creation or transfer of the original data source takes some time.

This is not to be confused with the file Cache. two different things, bestie, do not mix them up 💀

🌈 Usage 🌈

Any Data Source has a method .persist(). The only option that you will need to pick is a TTL, the number of seconds that the persisted version lasts before expiry (leave as None for no expiry). This creates a local copy in the persist directory, which may be in "~/.intake/persist, but can be configured. ✨

Each container type (dataframe, array, ...) gets its own persistence implementation, and its own particular file storage format that goes with it. The call to .persist() can take arguments to tune how the local files get created, and in some cases it might need extra optional packages installed first. 💅

Example, bestie:

cat = intake.open_catalog('mycat.yaml')  # load a remote cat
source = cat.csvsource()  # source pointing to remote data
source.persist()

source = cat.csvsource()  # future use now gives local intake_parquet.ParquetSource

To control whether a catalog will automatically hand you the persisted version of a source like this, use the argument persist_mode — like, if you wanted to straight up ignore locally persisted versions, you could've done:

cat = intake.open_catalog('mycat.yaml', persist_mode='never')
or
source = cat.csvsource(persist_mode='never')

Heads up: if you give a TTL (in seconds), then the original source gets accessed and a new persisted version written transparently once the old persisted version has expired. ✨

Also: after persisting, the original source will have source.has_been_persisted == True, and the persisted source (aka the one loaded from local files) will have source.is_persisted == True. Two different flags, two different sources, don't mix them up. 🫶

🎀 Export 🎀

A similar concept to Persist — Export lets you make a copy of some data source, in the format that's appropriate for its container, and drop that data-set wherever suits you, including remote locations. This functionality (source.export()) does not touch the persist store, okay? Instead, it returns a YAML text representation of the output, so you can put it into a catalog of your own. And it's that catalog you'd share with other people. Iconic. 👑

Note that "exported" data-sources like this do contain the info of the original source they were made from in their metadata, so you can recreate the original source if you want to, and read from there. 🦋

🌈 Persisting to Remote 🌈

If you're typically running your code inside of ephemoral containers, then persisting data-sets might be something you actually want to do (because the original source is slow, or parsing is CPU/memory intensive) — but the local storage is just not useful. In some cases you'll have access to some shared network storage mounted on the instance, but in other cases you'll want to persist to a remote store.

The config value 'persist_path', which can also be set by the environment variable INTAKE_PERSIST_PATH, can be a remote location such as s3://mybucket/intake-persist. You'll need to install the appropriate package to talk to the external storage (e.g., s3fs, gcsfs, pyarrow), but otherwise everything should work as before, and you can access the persisted data from any container. No cap. ✨

💖 The Persist Store 💖

You can interact directly with the class implementing persistence:

from intake.container.persist import store

This singleton instance acts like a catalog — it lets you query the contents of the instance store, and add and remove entries. It also lets you find the original source for any given persisted source, and refresh the persisted version on demand.

For details on the methods of the persist store, see the API documentation: intake.container.persist.PersistStore(). Sources in the store carry a lot of information about the sources they were made from, so they can be remade successfully. All of that shows up in the source metadata. The sources use the "token" of the original data source as their key in the store — a value you can find by dask.base.tokenize(source) for the original source, or you can grab it from the metadata of a persisted source. 🍓

Note that all of the information about persisted sources is held in a single YAML file in the persist directory (typically /persisted/cat.yaml within the config directory, but see intake.config.conf['persist_path']). This file can be edited by hand if you wanted to — for example, to set some persisted source not to expire. This is only recommended for experts. 💀

👑 Future Enhancements 👑

CLI functionality to investigate and alter the state of the persist store.
Time check-pointing of persisted data, such that you can not only get the "most recent" but any version in the time-series.
(eventually) pipeline functionality, whereby a persisted data source depends on another persisted data source, and the whole train can be refreshed on a schedule or on demand.

📊 Plotting 📊

Intake provides a plotting API based on the hvPlot library, which closely mirrors the pandas plotting API but generates interactive plots using HoloViews and Bokeh. 🌈

The hvPlot website provides comprehensive documentation on using the plotting API to quickly visualize and explore small and large datasets. The main features offered by the plotting API include:

Support for tabular data stored in pandas and dask dataframes
Support for gridded data stored in xarray backed nD-arrays
Support for plotting large datasets with datashader

Using Intake alongside hvPlot allows declaratively persisting plot declarations and default options in the regular catalog.yaml files. ✨

🧚 Setup 🧚

For detailed installation instructions see the getting started section in the hvPlot documentation. To start with, install hvplot using conda:

conda install -c conda-forge hvplot

or using pip:

pip install hvplot

💜 Usage 💜

The plotting API is designed to work well in and outside the Jupyter notebook, however when using it in JupyterLab the PyViz lab extension must be installed first:

jupyter labextension install @pyviz/jupyterlab_pyviz

For detailed instructions on displaying plots in the notebook and from the Python command prompt see the hvPlot user guide.

🎀 Python Command Prompt & Scripts 🎀

Assuming the US Crime dataset has been installed (in the intake-examples repo, or from conda with conda install -c intake us_crime):

Once installed the plot API can be used, by using the .plot method on an intake DataSource:

import intake
import hvplot as hp

crime = intake.cat.us_crime
columns = ['Burglary rate', 'Larceny-theft rate', 'Robbery rate', 'Violent Crime rate']

violin = crime.plot.violin(y=columns, group_label='Type of crime',
                           value_label='Rate per 100k', invert=True)
hp.show(violin)

[image]

🦋 Notebook 🦋

Inside the notebook plots will display themselves, however the notebook extension must be loaded first. The extension may be loaded by importing the hvplot.intake module, or explicitly loading the holoviews extension, or by calling intake.output_notebook():

# To load the extension run this import
import hvplot.intake

# Or load the holoviews extension directly
import holoviews as hv
hv.extension('bokeh')

# convenience function
import intake
intake.output_notebook()

crime = intake.cat.us_crime
columns = ['Violent Crime rate', 'Robbery rate', 'Burglary rate']
crime.plot(x='Year', y=columns, value_label='Rate (per 100k people)')

⭐ Predefined Plots ⭐

Some catalogs will define plots appropriate to a specific data source. These will be specified such that the user gets the right view with the right columns and labels, without having to investigate the data in detail — this is ideal for quick-look plotting when browsing sources. 🫶

import intake
intake.us_crime.plots

Returns ['example']. This works whether accessing the entry object or the source instance. To visualise:

intake.us_crime.plot.example()

🌸 Persisting metadata 🌸

Intake allows catalog yaml files to declare metadata fields for each data source, which are made available alongside the actual dataset. The plotting API reserves certain fields to define default plot options, to label and annotate the data fields in a dataset, and to declare pre-defined plots. 💅

💖 Declaring defaults 💖

The first set of metadata used by the plotting API is the plot field in the metadata section. Any options found in the metadata field will apply to all plots generated from that data source, allowing the definition of plotting defaults. For example, when plotting a fairly large dataset such as the NYC Taxi data, it might be desirable to enable datashader by default, ensuring that any plot that supports it is datashaded. The syntax to declare default plot options is as follows:

sources:
  nyc_taxi:
    description: NYC Taxi dataset
    driver: parquet
    args:
      urlpath: 's3://datashader-data/nyc_taxi_wide.parq'
    metadata:
      plot:
        datashade: true

✨ Declaring data fields ✨

The columns of a CSV or parquet file, or the coordinates and data variables in a NetCDF file, often have shortened or cryptic names with underscores. They also do not provide additional information about the units of the data or the range of values — therefore the catalog yaml specification also provides the ability to define additional information about the fields in a dataset.

Valid attributes that may be defined for the data fields include:

label: A readable label for the field which will be used to label axes and widgets
unit: A unit associated with the values inside a data field
range: A range associated with a field declaring limits which will override those computed from the data

Just like the default plot options, the fields may be declared under the metadata section of a data source:

sources:
  nyc_taxi:
    description: NYC Taxi dataset
    driver: parquet
    args:
      urlpath: 's3://datashader-data/nyc_taxi_wide.parq'
    metadata:
      fields:
        dropoff_x:
          label: Longitude
        dropoff_y:
          label: Latitude
        total_fare:
          label: Fare
          unit: $

🌈 Declaring custom plots 🌈

As shown in the hvPlot user guide, the plotting API provides a variety of plot types, which can be declared using the kind argument or via convenience methods on the plotting API, e.g. cat.source.plot.scatter(). In addition to declaring default plot options and field metadata, data sources may also declare custom plot, which will be made available as methods on the plotting API. In this way a catalogue may declare any number of custom plots alongside a datasource. 👑

To make this more concrete, consider the following custom plot declaration on the plots field in the metadata section:

          sources:
            nyc_taxi:
              description: NYC Taxi dataset
              driver: parquet
              args:
                urlpath: 's3://datashader-data/nyc_taxi_wide.parq'
              metadata:
                plots:
                  dropoff_scatter:
                    kind: scatter
                    x: dropoff_x
                    y: dropoff_y
                    datashade: True
                    width: 800
                    height: 600

okay so this little declarative spec right here? It's manifesting a whole new custom plot called dropoff_scatter 💖 and it'll be sitting pretty on the catalog under cat.nyc_taxi.plot.dropoff_scatter(). Call that method on the plot API and boom — you get a datashaded scatter plot of the dropoff locations in the NYC taxi dataset, automatically, no extra steps, no cap. ✨

And ofc bestie, those three metadata fields can absolutely be used together — declaring global defaults under the plot field, annotations for the data fields under the fields key, and custom plots via the plots field. All three, one config, iconic behavior. 🌈

🧩 Plugin Directory 🧩

This is the known-projects list, aka every project that installs driver plugins for Intake, with the named drivers each one contains chilling in parentheses:

builtin to Intake — (catalog, csv, intake_remote, ndzarr, numpy, textfiles, yaml_file_cat, yaml_files_cat, zarr_cat, json, jsonl)
intake-astro — Table and array loading of FITS astronomical data (fits_array, fits_table)
intake-accumulo — Apache Accumulo clustered data storage (accumulo)
intake-avro: Apache Avro data serialization format (avro_table, avro_sequence)
intake-bluesky: search and retrieve data in the bluesky data model
intake-dcat — Browse and load data from DCAT catalogs. (dcat)
intake-dynamodb — link to Amazon DynamoDB (dynamodb)
intake-elasticsearch: Elasticsearch search and analytics engine (elasticsearch_seq, elasticsearch_table)
intake-esm: Plugin for building and loading intake catalogs for earth system data sets holdings, such as CMIP (Coupled Model Intercomparison Project) and CESM Large Ensemble datasets.
intake-geopandas: load from ESRI Shape Files, GeoJSON, and geospatial databases with geopandas (geojson, postgis, shapefile, spatialite) and regionmask for opening shapefiles into regionmask.
intake-google-analytics: run Google Analytics queries and load data as a DataFrame (google_analytics_query)
intake-hbase: Apache HBase database (hbase)
intake-iris — load netCDF and GRIB files with IRIS (grib, netcdf)
intake-metabase: Generate catalogs and load tables as DataFrames from Metabase (metabase_catalog, metabase_table)
intake-mongo: MongoDB noSQL query (mongo)
intake-nested-yaml-catalog: Plugin supporting a single YAML hierarchical catalog to organize datasets and avoid a data swamp. (nested_yaml_cat)
intake-netflow: Netflow packet format (netflow)
intake-notebook: Experimental plugin to access parameterised notebooks through intake and executed via papermill (ipynb)
intake-odbc: ODBC database (odbc)
intake-parquet: Apache Parquet file format (parquet)
intake-pattern-catalog: Plugin for specifying a file-path pattern which can represent a number of different entries (pattern_cat)
intake-pcap: PCAP network packet format (pcap)
intake-postgres: PostgreSQL database (postgres)
intake-s3-manifests (s3_manifest)
intake-salesforce: Generate catalogs and load tables as DataFrames from Salesforce (salesforce_catalog, salesforce_table)
intake-sklearn: Load scikit-learn models from Pickle files (sklearn)
intake-solr: Apache Solr search platform (solr)
intake-stac: Intake Driver for SpatioTemporal Asset Catalogs (STAC).
intake-stripe: Generate catalogs and load tables as DataFrames from Stripe (stripe_catalog, stripe_table)
intake-spark: data processed by Apache Spark (spark_cat, spark_rdd, spark_dataframe)
intake-sql: Generic SQL queries via SQLAlchemy (sql_cat, sql, sql_auto, sql_manual)
intake-sqlite: Local caching of remote SQLite DBs and queries via SQLAlchemy (sqlite_cat, sqlite, sqlite_auto, sqlite_manual)
intake-splunk: Splunk machine data query (splunk)
intake-streamz: real-time event processing using Streamz (streamz)
intake-thredds: Intake interface to THREDDS data catalogs (thredds_cat, thredds_merged_source)
intake-xarray: load netCDF, Zarr and other multi-dimensional data (xarray_image, netcdf, grib, opendap, rasterio, remote-xarray, zarr)

The status of these projects is available at Status Dashboard. 👑

Not seeing your fave format up in here? 😭 Peep Making Drivers for how to create new plugins, bestie — you can literally just make one.

Quick note tho — if you want your plugin listed in this lil directory, open an issue in the Intake issue repository and add an entry to the status dashboard repository. We also have a plugin wishlist Github issue that shows the breadth of plugins we hope to see for Intake. 🫶

🛰️ Server Protocol 🛰️

This page is giving deep lore on how the Intake server is actually implemented, fr fr. If you're just here to run and configure a server, go look at the Command Line Tools section instead — that's your vibe.

Communication between the intake client and server happens exclusively over HTTP, with all parameters passed using msgpack UTF8 encoding. The server side is implemented by the module intake.cli.server. Currently, only the following two routes are available:

The server may be configured to use auth services, which, when passed the header of the incoming call, can determine whether the given request is allowed. See Authorization Plugins. 🔐

📥 GET /info 📥

Retrieve information about the data-sets available on this server. The list of data-sets may be paginated, in order to avoid excessively long transactions. Notice that the catalog for which a listing is being requested can itself be a data-source (when source-id is passed) — this is how nested sub-catalogs are handled on the server. 🪆

Parameters

page_size, int or none (optional): to enable pagination, set this value. The number of entries returned will be this value at most. If None, returns all entries. This is passed as a query parameter.
page_offset, int (optional): when paginating, start the list from this numerical offset. The order of entries is guaranteed if the base catalog has not changed. This is passed as a query parameter.
source-id, uuid string (optional): when the catalog being accessed is not the route catalog, but an open data-source on the server, this is its unique identifier. See POST /source for how these IDs are generated. If the catalog being accessed is the root Catalog, this parameter should be omitted. This is passed as an HTTP header.

Returns

version, string: the server's Intake version
sources, list of objects: the main payload, where each object contains a name, and the result of calling .describe() on the corresponding data-source, i.e., the container type, description, metadata.
metadata, object: any metadata associated with the whole catalog

🎯 GET /source 🎯

Fetch information about a specific source. This is the random-access variant of the GET /info route, by which a particular data-source can be accessed without paginating through all of the sources. So much faster, honestly. 💅

Parameters

name, string (required): the data source name being accessed, one of the members of the catalog. This is passed as a query parameter.
source-id, uuid string (optional): when the catalog being accessed is not the root catalog, but an open data-source on the server, this is its unique identifier. See POST /source for how these IDs are generated. If the catalog being accessed is the root Catalog, this parameter should be omitted. This is passed as an HTTP header.

Returns

Same as one of the entries in sources for GET /info: the result of .describe() on the given data-source in the server ✨

🔎 POST /source, action="search" 🔎

Searching a Catalog returns search results in the form of a new Catalog. This "results" Catalog is cached on the server the same as any other Catalog. 🗂️

Parameters

source-id, uuid string (optional): When the catalog being searched is not the root catalog, but a subcatalog on the server, this is its unique identifier. If the catalog being searched is the root Catalog, this parameter should be omitted. This is passed as an HTTP header.
query: tuple of (args, kwargs): These will be unpacked into Catalog.search on the server to create the "results" Catalog. This is passed in the body of the message.

Returns

source_id, uuid string: the identifier of the results Catalog in the server's source cache

🚪 POST /source, action="open" 🚪

This one is a more involved processing of a data-source, and, if successful, returns one of two possible scenarios:

direct-access, in which all the details required for reading the data directly from the client are passed, and the client then creates a local copy of the data source and needs no further involvement from the server in order to fetch the data
remote-access, in which the client is unable or unwilling to create a local version of the data-source, and instead created a remote data-source which will fetch the data for each partition from the server.

The set of parameters supplied and the server/client policies will define which method of access is employed. In the case of remote-access, the data source is instantiated on the server, and .discover() run on it. The resulting information is passed back, and must be enough to instantiate a subclass of intake.container.base.RemoteSource appropriate for the container of the data-set in question (e.g., RemoteArray when container="ndarray"). In this case, the response also includes a UUID string for the open instance on the server, referencing the cache of open sources maintained by the server. 🧠

Note: "opening" a data entry which is itself is a catalog implies instantiating that catalog object on the server and returning its UUID, such that a listing can be made using GET/ info or GET /source.

Parameters

name, string (required): the data source name being accessed, one of the members of the catalog. This is passed in the body of the request.
source-id, uuid string (optional): when the catalog being accessed is not the root catalog, but an open data-source on the server, this is its unique identifier. If the catalog being accessed is the root Catalog, this parameter should be omitted. This is passed as an HTTP header.
available_plugins, list of string (optional): the set of named data drivers supported by the client. If the driver required by the data-source is not supported by the client, then the source must be opened remote-access. This is passed in the body of the request.
parameters, object (optional) — bestie, these are the user parameters you pass to the data-source when it's instantiating. Whether direct-access is even possible might, in principle, depend on these params, but honestly it's unlikely. Also note that some parameter default value functions are designed to be evaluated on the server, which may have access to, like, some credentials service (check out Parameter Definition). This whole thing gets passed in the body of the request. 💅

💖 Returns 💖

If direct-access: the driver plugin name and set of arguments for instantiating the data-source in the client.

If remote-access: the data-source container, schema and source-ID so that further reads can be made from the server. 🫶

✨ POST /source, action="read" ✨

This route fetches data from the server once a data-source has been opened in remote-access mode. 🔥

✨ Parameters ✨

source-id, uuid string (required) — the identifier of the data-source in the server's source cache. This is returned when action="open". Passed in the body of the request. 💖
partition, int or tuple (optional, but necessary for some sources) — section/chunk of the data to fetch. When the data-source is partitioned, the client fetches the data one partition at a time, so it appears partitioned the same way on the client side for iteration or passing to Dask. Some data-sources don't support partitioning, and then this parameter is not required/ignored. Passed in the body of the request. 🌸
accepted_formats, accepted_compression, list of strings (required) — to specify how serialization of data happens. This is an expert feature, see docs in the module intake.container.serializer. Passed in the body of the request. 🧚

🌈 Dataset Transforms 🌈

aka. derived datasets.

WARNING: experimental feature, the API may change. The data sources in intake.source.derived are not yet declared as top-level named drivers in the package entrypoints. 💀

Intake allows for the definition of data sources which take as their input another source in the same directory, so that you have the opportunity to present processing to the user of the catalog. ✨

The "target" of a derived data source will normally be a string. In the simple case, it is the name of a data source in the same catalog. However, we use the syntax "catalog:source" to refer to sources in other catalogs, where the part before ":" will be passed to intake.open_catalog(), together with any keyword arguments from cat_kwargs. 👑

This can be done by defining classes which inherit from intake.source.derived.DerivedSource, or using one of the pre-defined classes in the same module, which usually need to be passed a reference to a function in a python module. We will demonstrate both. 🫶

🎀 Example 🎀

Consider the following target dataset, which loads some simple facts about US states from a CSV file. This example is taken from the Intake test suite. 📊

We now show two ways to apply a super-simple transform to this data, which selects two of the dataframe's columns. 💅

💜 Class Example 💜

The first version uses an approach in which the transform is derived in a data source class, and the parameters passed are specific to the transform type. Note that the driver is referred to by its fully-qualified name in the Intake package. 🌟

The source class for this is included in the Intake codebase, but the important part is:

class Columns(DataFrameTransform):
    ...

    def pick_columns(self, df):
        return df[self._params["columns"]]

We see that this specific class inherits from DataFrameTransform, with transform=self.pick_columns. We know that the inputs and outputs are both dataframes. This allows for some additional validation and an automated way to infer the output dataframe's schema that reduces the number of lines of code required. 😭

The given method does exactly what you might imagine: it takes an input dataframe and applies a column selection to it. 🎯

Running cat.derive_cols.read() will indeed, as expected, produce a version of the data with only the selected columns included. It does this by defining the original dataset, applying the selection, and then getting Dask to generate the output. For some datasets, this can mean that the selection is pushed down to the reader, and the data for the dropped columns is never loaded. The user may choose to do .to_dask() instead, and manipulate the lazy dataframe directly, before loading. 🔥

🦋 Functional Example 🦋

This second version of the same output uses the more generic and flexible intake.source.derived.DataFrameTransform. ✨

derive_cols_func:
  driver: intake.source.derived.DataFrameTransform
  args:
    targets:
      - input_data
    transform: "intake.source.tests.test_derived._pick_columns"
    transform_kwargs:
      columns: ["state", "slug"]

In this case, we pass a reference to a function defined in the Intake test suite. Normally this would be declared in user modules, where perhaps those declarations and catalog(s) are distributed together as a package. 📦

def _pick_columns(df, columns):
    return df[columns]

This is, of course, very similar to the method shown in the previous section, and again applies the selection in the given named argument to the input. Note that Intake does not support including actual code in your catalog, since we would not want to allow arbitrary execution of code on catalog load, as opposed to execution. 🚫

Loading this data source proceeds exactly the same way as the class-based approach, above. Both Dask and in-memory (Pandas, via .read()) methods work as expected. The declaration in YAML, above, is slightly more verbose, but the amount of code is smaller. This demonstrates a tradeoff between flexibility and concision. If there were validation code to add for the arguments or input dataset, it would be less obvious where to put these things. ⚖️

💀 Barebone Example 💀

The previous two examples both did dataframe to dataframe transforms. However, totally arbitrary computations are possible. Consider the following:

barebones:
  driver: intake.source.derived.GenericTransform
  args:
    targets:
      - input_data
    transform: builtins.len
    transform_kwargs: {}

This applies len to the input dataframe. cat.barebones.describe() gives the output container type as "other", i.e., not specified. The result of read() on this gives the single number 50, the number of rows in the input data. This class, and DerivedDataSource, are included with the intent as superclasses, and probably will not be used directly often. 🔢

⚡ Execution engine ⚡

None of the above examples specified explicitly where the compute implied by the transformation will take place. However, most Intake drivers support in-memory containers and Dask; remembering that the input dataset here is a dataframe. However, the behaviour is defined in the driver class itself - so it would be fine to write a driver in which we make different assumptions. Let's suppose, for instance, that the original source is to be loaded from spark (see the intake-spark package), the driver could explicitly call .to_spark on the original source, and be assured that it has a Spark object to work with. It should, of course, explain in its documentation what assumptions are being made and that, presumably, the user is expected to also call .to_spark if they wished to directly manipulate the spark object. 🚀

🌸 Plugin examples 🌸

call .sel on xarray datasets xarray-plugin-transform

👑 API 👑

intake.source.derived.DerivedSource(*args, ...) — Base source deriving from another source in the same catalog
intake.source.derived.GenericTransform(...) — (no description in the original)
intake.source.derived.DataFrameTransform(...) — Transform where the input and output are both Dask-compatible dataframes
intake.source.derived.Columns(*args, **kwargs) — Simple dataframe transform to pick columns

`class intake.source.derived.DerivedSource(*args, **kwargs)`

Base source deriving from another source in the same catalog

Target picking and parameter validation are performed here, but you probably want to subclass from one of the more specific classes like DataFrameTransform. 💅

__init__(targets, target_chooser=<function first>,
         target_kwargs=None, cat_kwargs=None, container=None,
         metadata=None, **kwargs)

Parameters

targets: list of string or DataSources — If string(s), refer to entries of the same catalog as this Source
target_chooser: function to choose between targets — function(targets, cat) -> source, or a fully-qualified dotted string pointing to it
target_kwargs: dict of dict with keys matching items of targets
cat_kwargs: to pass to intake.open_catalog, if the target is in another catalog
container: str (optional) — Assumed output container, if known/different from input

🧬 `GenericTransform` — the blueprint bestie

class intake.source.derived.GenericTransform(*args, **kwargs)

__init__(targets, target_chooser=<function first>, target_kwargs=None, cat_kwargs=None, container=None, metadata=None, **kwargs)

Parameters

targets: list of string or DataSources — if it's string(s), they refer to entries of the same catalog as this Source. No wandering off to other catalogs unannounced, okay? 💅
target_chooser: function to choose between targets — literally function(targets, cat) -> source, or a fully-qualified dotted string pointing to it 🎯
target_kwargs: dict of dict with keys matching items of targets
cat_kwargs: to pass to intake.open_catalog, if the target is in another catalog
container: str (optional) — assumed output container, if known / different from input

[Note: the exact form of target_kwargs and cat_kwargs may be subject to change]

🦋 `DataFrameTransform` — glow-up edition

class intake.source.derived.DataFrameTransform(*args, **kwargs)

A Transform where the input and the output are both Dask-compatible dataframes. She's versatile, she contains multitudes. ✨

This derives from GenericTransform, and you must supply transform and any transform_kwargs. Not optional. It's giving required. 🫶

__init__(targets, target_chooser=<function first>, target_kwargs=None, cat_kwargs=None, container=None, metadata=None, **kwargs)

Parameters

targets: list of string or DataSources — if it's string(s), they refer to entries of the same catalog as this Source
target_chooser: function to choose between targets — literally function(targets, cat) -> source, or a fully-qualified dotted string pointing to it 🎯
target_kwargs: dict of dict with keys matching items of targets
cat_kwargs: to pass to intake.open_catalog, if the target is in another catalog
container: str (optional) — assumed output container, if known / different from input

[Note: the exact form of target_kwargs and cat_kwargs may be subject to change]

🎀 `Columns` — the little demo that could

class intake.source.derived.Columns(*args, **kwargs)

A simple dataframe transform to pick columns. Basically the "hello world" of dataframe transforms, and honestly? She ate. 🔥

She's given here as an example of how to make a specific dataframe transform. And bestie, lowkey important: you could totally use DataFrameTransform directly instead, by writing a function to choose the columns rather than a method like this one. Both paths are valid, iykyk. 😌

__init__(columns, **kwargs)

columns: list of labels (usually str) or slice — the columns to choose from the target dataframe 🌸

💖 REFERENCE 💖

📚 API

⭐ Auto-generated reference

Auto-generated reference. 👑

🌸 End User 🌸

These are reference class and function definitions likely to be useful to everyone. Literally everyone, bestie. No gatekeeping here, this is the stuff you'll actually reach for. ✨

intake.open_catalog([uri]) — Create a Catalog object 👑
intake.registry — Dict of driver: DataSource class
intake.register_driver(name, value[, ...]) — Add runtime driver definition
intake.upload(data, path, **kwargs) — Given a concrete data object, store it at given location return Source
intake.source.csv.CSVSource(*args, **kwargs) — Read CSV files into dataframes
intake.source.textfiles.TextFilesSource(...) — Read textfiles as sequence of lines
intake.source.jsonfiles.JSONFileSource(...) — Read JSON files as a single dictionary or list
intake.source.jsonfiles.JSONLinesFileSource(...) — Read a JSONL (https://jsonlines.org/) file and return a list of objects, each being valid json object (e.g.
intake.source.npy.NPySource(*args, **kwargs) — Read numpy binary files into an array
intake.source.zarr.ZarrArraySource(*args, ...) — Read Zarr format files into an array
intake.catalog.local.YAMLFileCatalog(*args, ...) — Catalog as described by a single YAML file
intake.catalog.local.YAMLFilesCatalog(*args, ...) — Catalog as described by a multiple YAML files
intake.catalog.zarr.ZarrGroupCatalog(*args, ...) — A catalog of the members of a Zarr group. 🦋

ok real talk, that last JSONLinesFileSource description genuinely trails off at "(e.g." on the page. We're not inventing the rest for you, we're just reporting what's there. 💀

🎀 `intake.open_catalog(uri=None, **kwargs)` 🎀

Create a Catalog object

Can load YAML catalog files, connect to an intake server, or create any arbitrary Catalog subclass instance. In the general case, the user should supply driver= with a value from the plugins registry which has a container type of catalog. File locations can generally be remote, if specifying a URL protocol. So many doors, so little time.

The default behaviour if not specifying the driver is as follows:

if uri is a a single string ending in "yml" or "yaml", open it as a catalog file
if uri is a list of strings, a string containing a glob character ("*") or a string not ending in "y(a)ml", open as a set of catalog files. In the latter case, assume it is a directory.
if uri beings with protocol "intake:", connect to a remote Intake server
if uri is None or missing, create a base Catalog object without entries.

(and yes bestie, the page really does say "a a" and "beings" — we copy the lore exactly as it was written, no cap. 🙈)

Parameters

uri: str or pathlib.Path — Designator for the location of the catalog.
kwargs: — passed to subclass instance, see documentation of the individual catalog classes. For example, yaml_files_cat (when specifying multiple uris or a glob string) takes the additional parameter flatten=True|False, specifying whether all data sources are merged in a single namespace, or each file becomes a sub-catalog. 🧚

SEE ALSO:

intake.open_yaml_files_cat, intake.open_yaml_file_cat
intake.open_intake_remote

💜 `intake.registry` 💜

Mapping from plugin names to the DataSource classes that implement them. These are the names that should appear in the driver: key of each source definition in a catalog. See Plugin Directory for more details. It's the plug, fr fr.

⭐ `intake.open_` ⭐

Set of functions, one for each plugin, for direct opening of a data source. The names are derived from the names of the plugins in the registry at import time. Dynamic queen behaviour, honestly.

🫶 `intake.upload(data, path, **kwargs)` 🫶

Given a concrete data object, store it at given location return Source

Use this function to publicly share data which you have created in your python session. Intake will try each of the container types, to see if one of them can handle the input data, and write the data to the path given, in the format most appropriate for the data type, e.g., parquet for pandas or dask data-frames.

With the DataSource instance you get back, you can add this to a catalog, or just get the YAML representation for editing (.yaml()) and sharing. Sharing is caring and also the whole point.

Parameters

data: instance — The object to upload and store. In many cases, the dask or in-memory variant are handled equivalently.
path: str — Location of the output files; can be, for instance, a network drive for sharing over a VPC, or a bucket on a cloud storage service
kwargs — passed to the writer for fine control. UNINDENT

Returns

DataSource instance

that UNINDENT up there? Not us. That's the page. We keep the page's little quirks because the page is our friend and we love it. 💅

🌈 Source classes 🌈

🍓 `class intake.source.csv.CSVSource(*args, **kwargs)` 🍓

Read CSV files into dataframes

Prototype of sources reading dataframe data

💖 `init(urlpath, csv_kwargs=None, metadata=None, storage_options=None, path_as_pattern=True)`

bestie this is the constructor, it's where you tell your CSV source where the data at and how to read it. let's break it down 🫶

Parameters

urlpath — str or iterable, location of data. can be a local path, or a remote path if you include a protocol specifier like 's3://'. it can also include glob wildcards or format pattern strings. some examples so you can see the vibe:
- {{ CATALOG_DIR }}data/precipitation.csv
- s3://data/*.csv
- s3://data/precipitation_{state}_{zip}.csv
- s3://data/{year}/{month}/{day}/precipitation.csv
- {{ CATALOG_DIR }}data/precipitation_{date:%Y-%m-%d}.csv
csv_kwargs — dict. any further arguments to pass to Dask's read_csv (like block size) or to the CSV parser in pandas (like which columns to use, encoding, data-types). literally just yeet your extra kwargs here.
storage_options — dict. any parameters that need to be passed to the remote data backend, such as credentials. 👑
path_as_pattern — bool or str, optional. whether to treat the path as a pattern (ie. data_{field}.csv) and create new columns in the output corresponding to pattern fields. if it's a str, it's treated as the pattern to match on. default is True. periodt ✨

🔍 `discover()`

open resource and populate the source attributes. that's it, that's the whole job. she's simple, she's iconic. 💅

📦 `export(path, **kwargs)`

save this data for sharing with other people 🫶

creates a copy of the data in a format appropriate for its container, in the location specified (which can be remote, e.g., s3).

returns the resultant source object, so that you can, for instance, add it to a catalog (catalog.add(source)) or get its YAML representation (.yaml()).

💾 `persist(ttl=None, **kwargs)`

save data from this source to local persistent storage. no cap, this one's handy.

Parameters

ttl: numeric, optional — time to live in seconds. if provided, the original source will be accessed and a new persisted version written transparently when more than ttl seconds have passed since the old persisted version was written.
kargs: passed to the _persist method on the base container.

📖 `read()`

load entire dataset into a container and return it. the main character moment, fr fr.

🍰 `read_partition(i)`

return a part of the data corresponding to i-th partition.

by default, assumes i should be an integer between zero and npartitions; override for more complex indexing schemes.

🔥 `to_dask()`

return a dask container for this data source. slay.

🌈 class `intake.source.zarr.ZarrArraySource(*args, **kwargs)`

read Zarr format files into an array ✨

Zarr is a numerical array storage format which works particularly well with remote and parallel access. for specifics of the format, see https://zarr.readthedocs.io/en/stable/

🎀 `init(urlpath, storage_options=None, component=None, metadata=None, **kwargs)`

the parameters dtype and shape will be determined from the first file, if not given. cute, right?

Parameters

urlpath — str. location of data file(s), possibly including protocol information.
storage_options — dict. passed on to storage backend for remote files.
component — str or None. if None, assume the URL points to an array. if given, assume the URL points to a group, and descend the group to find the array at this location in the hierarchy.
kwargs — passed on to dask.array.from_zarr.

🔍 `discover()`

open resource and populate the source attributes.

📦 `export(path, **kwargs)`

save this data for sharing with other people

creates a copy of the data in a format appropriate for its container, in the location specified (which can be remote, e.g., s3).

returns the resultant source object, so that you can, for instance, add it to a catalog (catalog.add(source)) or get its YAML representation (.yaml()).

💾 `persist(ttl=None, **kwargs)`

save data from this source to local persistent storage

Parameters

ttl: numeric, optional — time to live in seconds. if provided, the original source will be accessed and a new persisted version written transparently when more than ttl seconds have passed since the old persisted version was written.
kargs: passed to the _persist method on the base container.

📖 `read()`

load entire dataset into a container and return it

🍰 `read_partition(i)`

return a part of the data corresponding to i-th partition.

by default, assumes i should be an integer between zero and npartitions; override for more complex indexing schemes.

🔥 `to_dask()`

return a dask container for this data source

🌸 class `intake.source.textfiles.TextFilesSource(*args, **kwargs)`

read textfiles as sequence of lines 🦋

prototype of sources reading sequential data.

takes a set of files, and returns an iterator over the text in each of them. the files can be local or remote. extra parameters for encoding, etc., go into storage_options.

🎀 `init(urlpath, text_mode=True, text_encoding='utf8', compression=None, decoder=None, read=True, metadata=None, storage_options=None)`

Parameters

urlpath — str or list(str). target files. can be a glob-path (with "*") and include protocol specified (e.g., "s3://"). can also be a list of absolute paths.
text_mode — bool. whether to open the file in text mode, recoding binary characters on the fly.
text_encoding — str. if text_mode is True, apply this encoding. UTF* is by far the most common.
compression — str or None. if given, decompress the file with the given codec on load. can be something like "gzip", "bz2", or to try to guess from the filename, 'infer'.
decoder — function, str or None. use this to decode the contents of files. if None, you will get a list of lines of text/bytes. if a function, it must operate on an open file-like object or a bytes/str instance, and return a list.
read — bool. if decoder is not None, this flag controls whether bytes/str get passed to the function indicated (True) or the open file-like object (False).
storage_options: dict — options to pass to the file reader backend, including text-specific encoding arguments, and parameters specific to the remote file-system driver, if using.

🔍 `discover()`

open resource and populate the source attributes.

📦 `export(path, **kwargs)`

save this data for sharing with other people

creates a copy of the data in a format appropriate for its container, in the location specified (which can be remote, e.g., s3).

returns the resultant source object, so that you can, for instance, add it to a catalog (catalog.add(source)) or get its YAML representation (.yaml()).

💾 `persist(ttl=None, **kwargs)`

save data from this source to local persistent storage

Parameters

ttl: numeric, optional — time to live in seconds. if provided, the original source will be accessed and a new persisted version written transparently when more than ttl seconds have passed since the old persisted version was written.
kargs: passed to the _persist method on the base container.

📖 `read()`

load entire dataset into a container and return it

🍰 `read_partition(i)`

return a part of the data corresponding to i-th partition.

by default, assumes i should be an integer between zero and npartitions; override for more complex indexing schemes.

🔥 `to_dask()`

return a dask container for this data source

⭐ class `intake.source.jsonfiles.JSONFileSource(*args, **kwargs)`

read JSON files as a single dictionary or list 🍓

These files can be local or remote, bestie 💖 — no biggie either way. Anything extra you need for encoding and all that jazz just goes straight into storage_options.

__init__(urlpath: str, text_mode: bool = True, text_encoding: str = 'utf8', compression: Optional[str] = None, read: bool = True, metadata: Optional[dict] = None, storage_options: Optional[dict] = None)

Parameters

urlpath — str. The target file, babe. It can totally have the protocol baked right in, like "s3://".
text_mode — bool. Do we open the file in text mode? If yes, binary characters get recoded on the fly ✨
text_encoding — str. If text_mode is True, this is the encoding we apply. UTF* is, like, by far the most common one, no cap.
compression — str or None. If you give one, the file gets decompressed with that codec on load. Could be something like "zip", "gzip", "bz2", or — my fave — 'infer' to try and guess it from the filename 🕵️
storage_options — dict. Options passed to the file reader backend, including text-specific encoding arguments, plus any parameters specific to the remote file-system driver, if you're using one.

discover() — Opens the resource and populates the source attributes. Little glow-up moment for your metadata 💅
export(path, **kwargs) — Saves this data so you can share it with other people 🫶

It makes a copy of the data in a format that fits its container, at the location you specify (which, yes, can be remote — s3 and friends, we see you).

It returns the resulting source object, so you can, for instance, add it to a catalog (catalog.add(source)) or grab its YAML representation (.yaml()).

persist(ttl=None, **kwargs) — Saves data from this source into local persistent storage.

Parameters

ttl — numeric, optional. Time to live, in seconds. If you provide it, the original source gets accessed and a fresh persisted version gets written transparently whenever more than ttl seconds have passed since the old persisted version was written. So iconic ⏰
kargs — passed to the _persist method on the base container.

read() — Loads the entire dataset into a container and returns it. All of it. The whole thing. Periodt.

🌈 class `intake.source.jsonfiles.JSONLinesFileSource(*args, **kwargs)`

Reads a JSONL file (https://jsonlines.org/) and hands you back a list of objects, where each one is a valid json object — like a dictionary or a list, iykyk.

__init__(urlpath: str, text_mode: bool = True, text_encoding: str = 'utf8', compression: Optional[str] = None, read: bool = True, metadata: Optional[dict] = None, storage_options: Optional[dict] = None)

Parameters

urlpath — str. The target file. Can include the protocol specified, e.g., "s3://".
text_mode — bool. Whether to open the file in text mode, recoding binary characters on the fly.
text_encoding — str. If text_mode is True, apply this encoding. UTF* is by far the most common 💖
compression — str or None. If given, decompress the file with the given codec on load. Can be something like "zip", "gzip", "bz2", or — to try to guess from the filename — 'infer'.
storage_options — dict. Options to pass to the file reader backend, including text-specific encoding arguments, and parameters specific to the remote file-system driver, if using.

discover() — Opens the resource and populates the source attributes ✨
export(path, **kwargs) — Saves this data for sharing with other people.

Creates a copy of the data in a format appropriate for its container, in the location specified (which can be remote, e.g., s3).

Returns the resultant source object, so that you can, for instance, add it to a catalog (catalog.add(source)) or get its YAML representation (.yaml()).

head(nrows: int = 100) — Returns the first nrows lines from the file. A lil peek, bestie 👀
persist(ttl=None, **kwargs) — Saves data from this source to local persistent storage.

Parameters

ttl — numeric, optional. Time to live in seconds. If provided, the original source will be accessed and a new persisted version written transparently when more than ttl seconds have passed since the old persisted version was written.
kargs — passed to the _persist method on the base container.

read() — Loads the entire dataset into a container and returns it.

🦋 class `intake.source.npy.NPySource(*args, **kwargs)`

Reads numpy binary files into an array, and honestly? Slay.

Prototype source showing an example of working with arrays.

Each file becomes one or more partitions, but partitioning within a file is only along the largest dimension, to ensure contiguous data. That part matters, don't skip it 💀

__init__(path, dtype=None, shape=None, chunks=None, storage_options=None, metadata=None)

The parameters dtype and shape will be determined from the first file, if not given.

Parameters

path — str or list of str. Location of data file(s), possibly including glob and protocol information.
dtype — str dtype spec. If known, the dtype (e.g., "int64" or "f4").
shape — tuple of int. If known, the length of each axis.
chunks — int. Size of chunks within a file along the biggest dimension — it need not be an exact factor of the length of that dimension.
storage_options — dict. Passed to the file-system backend.

discover() — Opens the resource and populates the source attributes.
export(path, **kwargs) — Saves this data for sharing with other people.

Creates a copy of the data in a format appropriate for its container, in the location specified (which can be remote, e.g., s3).

Returns the resultant source object, so that you can, for instance, add it to a catalog (catalog.add(source)) or get its YAML representation (.yaml()).

persist(ttl=None, **kwargs) — Saves data from this source to local persistent storage.

Parameters

ttl — numeric, optional. Time to live in seconds. If provided, the original source will be accessed and a new persisted version written transparently when more than ttl seconds have passed since the old persisted version was written.
kargs — passed to the _persist method on the base container.

read() — Loads the entire dataset into a container and returns it.
read_partition(i) — Returns a part of the data corresponding to the i-th partition.

By default, it assumes i should be an integer between zero and npartitions; override it for more complex indexing schemes.

to_dask() — Returns a dask container for this data source. Dask girlies rise 🙌

🎀 class `intake.catalog.local.YAMLFileCatalog(*args, **kwargs)`

A catalog as described by a single YAML file. So neat, so tidy, so cute 🍓

__init__(path=None, text=None, autoreload=True, **kwargs)

Parameters

path — str. Location of the file to parse (can be remote).
text — str. The YAML contents of the catalog — this takes precedence over path.
reload — bool. Whether to watch the source file for changes; make it False if you want an editable Catalog.

export(path, **kwargs) — Saves this data for sharing with other people.

Creates a copy of the data in a format appropriate for its container, in the location specified (which can be remote, e.g., s3).

Returns the resultant source object, so that you can, for instance, add it to a catalog (catalog.add(source)) or get its YAML representation (.yaml()).

persist(ttl=None, **kwargs) — Saves data from this source to local persistent storage.

Parameters

ttl — numeric, optional. Time to live in seconds. If provided, the original source will be accessed and a new persisted version written transparently when more than ttl seconds have passed since the old persisted version was written.
kargs — passed to the _persist method on the base container.

reload() — Reloads the catalog if sufficient time has passed ⏳
walk(sofar=None, prefix=None, depth=2) — Gets all entries in this catalog and sub-catalogs.

Parameters

sofar — dict or None. Within recursion, use this dict for output.

prefix: list of str or None — the names of the levels we already slid into, bestie, so we don't get lost on the way back up 🌀
- depth: int — how many levels down we're willing to go; this is the thing that stops circular references from sending us into an infinite spiral and keeps the output looking clean 💅

Returns

A Dict where the keys are the entry names in dotted syntax, and the

values are entry instances. Periodt.

💾 `class intake.catalog.local.YAMLFilesCatalog(*args, **kwargs)`

A catalog as described by a multiple YAML files situation — yes plural, we contain multitudes ✨

`init(path, flatten=True, **kwargs)`

Parameters

path: str — where the files to parse live (and yeah, it can be remote, we're not judging). It can include possible glob (*) character(s). It can also be a list of paths, but those ones can't have glob characters, okay? Rules are rules 🎀
flatten: bool (True) — do we list all the entries in the cats right at the top level (True), or do we make sub-cats from each file (False)? Your call, main character 💖

`export(path, **kwargs)`

Save this data for sharing with other people 🫶

It creates a copy of the data in a format that's appropriate for its container, dropped into the location you specified (which can be remote, e.g., s3).

It returns the resultant source object, so that you can, for instance, add it to a catalog (catalog.add(source)) or get its YAML representation (.yaml()). Obsessed.

`persist(ttl=None, **kwargs)`

Save data from this source to local persistent storage 💾✨

Parameters

ttl: numeric, optional — time to live, in seconds. If you provide it, the original source gets accessed and a brand-new persisted version is written transparently once more than ttl seconds have passed since the old persisted version was written. It just... handles it. Iconic.
kargs: passed to the _persist method on the base container.

`reload()`

Reload catalog if sufficient time has passed ⏳

`walk(sofar=None, prefix=None, depth=2)`

Get all entries in this catalog and sub-catalogs — the full tour, no skips 🌈

Parameters

sofar: dict or None — within recursion, use this dict for output.
prefix: list of str or None — names of levels already visited.
depth: int — number of levels to descend; needed to truncate circular references and for cleaner output.

Returns

A Dict where the keys are the entry names in dotted syntax, and the

values are entry instances. Same energy as before, no cap 💜

🦓 `class intake.catalog.zarr.ZarrGroupCatalog(*args, **kwargs)`

A catalog of the members of a Zarr group. It's giving structured ✨

`init(urlpath, storage_options=None, component=None, metadata=None, consolidated=False, name=None)`

Parameters

urlpath — str. Location of data file(s), possibly including protocol information.
storage_options — dict, optional. Passed on to the storage backend for remote files.
component — str, optional. If None, build a catalog from the root group. If given, build the catalog from the group at this location in the hierarchy. Choose your fighter 💅
metadata — dict, optional. Catalog metadata. If you don't provide it, it'll be populated from Zarr group attributes automatically. It does the work for you, lowkey.
consolidated — bool, optional. If True, assume Zarr metadata has been consolidated.

`export(path, **kwargs)`

Save this data for sharing with other people 🫶

It creates a copy of the data in a format appropriate for its container, in the location you specified (which can be remote, e.g., s3).

Returns the resultant source object, so that you can, for instance, add it to a catalog (catalog.add(source)) or get its YAML representation (.yaml()).

`persist(ttl=None, **kwargs)`

Save data from this source to local persistent storage 💾

Parameters

ttl: numeric, optional — time to live in seconds. If provided, the original source will be accessed and a new persisted version written transparently when more than ttl seconds have passed since the old persisted version was written.
kargs: passed to the _persist method on the base container.

`reload()`

Reload catalog if sufficient time has passed ⏳

`walk(sofar=None, prefix=None, depth=2)`

Get all entries in this catalog and sub-catalogs 🌸

Parameters

sofar: dict or None — within recursion, use this dict for output.
prefix: list of str or None — names of levels already visited.
depth: int — number of levels to descend; needed to truncate circular references and for cleaner output.

Returns

A Dict where the keys are the entry names in dotted syntax, and the

values are entry instances. Chef's kiss 👑

✨ Base Classes ✨

Okay bestie, this next bit is a reference API class listing, and it's mainly for developers. So if you're just here to load a CSV, you can vibe past this — but if you're building drivers, this is your moment. 💅

intake.source.base.DataSourceBase(*args, ...) — An object which can produce data 🫶
intake.source.base.DataSource(*args, **kwargs) — A Data Source will all optional functionality ✨
intake.source.base.PatternMixin() — Helper class to provide file-name parsing abilities to a driver class 🎀
intake.container.base.RemoteSource(*args, ...) — Base class for all DataSources living on an Intake server 🌐
intake.catalog.Catalog(*args, **kwargs) — Manages a hierarchy of data sources as a collective unit. 👑
intake.catalog.entry.CatalogEntry(*args, ...) — A single item appearing in a catalog 🦋
intake.catalog.local.UserParameter(*args, ...) — A user-settable item that is passed to a DataSource upon instantiation. 🌸
intake.auth.base.BaseAuth(*args, **kwargs) — Base class for authorization 🔐
intake.source.cache.BaseCache(driver, spec) — Provides utilities for managing cached data files. 💾
intake.source.base.Schema(**kwargs) — Holds details of data description for any type of data-source 🗂️
intake.container.persist.PersistStore(*args, ...) — Specialised catalog for persisted data-sources 💜

🌟 `class intake.source.base.DataSource(*args, **kwargs)`

A Data Source will all optional functionality

Not me reading "will all" — yes bestie, that's literally what it says, we're keeping it fr fr. 😭

When subclassed, child classes will have the base data source functionality, plus caching, plotting and persistence abilities. It's giving "everything you need in one inheritance tree" energy. 🌈

plot — Accessor for HVPlot methods. See Plotting for more details. 📊

👑 `class intake.catalog.Catalog(*args, **kwargs)`

Manages a hierarchy of data sources as a collective unit.

A catalog is a set of available data sources for an individual entity (remote server, local file, or a local directory of files). This can be expanded to include a collection of subcatalogs, which are then managed as a single unit. So like, one catalog to rule them all, lowkey. 🫶

A catalog is created with a single URI or a group of URIs. A URI can either be a URL or a file path. Periodt. ✨

Each catalog in the hierarchy is responsible for caching the most recent refresh time to prevent overeager queries. (No thirsty reloading, bestie — the catalog knows when it last refreshed. 💅)

🌸 Attributes

metadata — dict Arbitrary information to carry along with the data source specs. 🎀

🦋 `configure_new(**kwargs)`

Create a new instance of this source with altered arguments.

Enables the picking of options and re-evaluating templates from any user-parameters associated with this source, or overriding any of the init arguments.

Returns a new data source instance. The instance will be recreated from the original entry definition in a catalog if this source was originally created from a catalog. 🔁

✨ `discover()`

Open resource and populate the source attributes. 💖

🔍 `filter(func)`

Create a Catalog of a subset of entries based on a condition.

WARNING: This function operates on CatalogEntry objects not DataSource objects. Do not come for me, I'm just the messenger. 💀

NOTE: Note that, whatever specific class this is performed on, the return instance is a Catalog. The entries are passed unmodified, so they will still reference the original catalog instance and include its details such as directory,.

Parameters

func — function This should take a CatalogEntry and return True or False. Those items returning True will be included in the new Catalog, with the same entry names 👑

Returns

Catalog — New catalog with Entries that still refer to their parents 🫶

⚡ `force_reload()`

Imperative reload data now. No patience, just vibes. 😤

🧚 `classmethod from_dict(entries, **kwargs)`

Create Catalog from the given set of entries.

Parameters

entries — dict-like A mapping of name:entry which supports dict-like functionality, e.g., is derived from collections.abc.Mapping. 🗂️
kwargs — passed on the constructor Things like metadata, name; see __init__. ✨

Returns

Catalog instance 💜

🌈 `get(**kwargs)`

Create a new instance of this source with altered arguments.

Enables the picking of options and re-evaluating templates from any user-parameters associated with this source, or overriding any of the init arguments.

Returns a new data source instance. The instance will be recreated from the original entry definition in a catalog if this source was originally created from a catalog. 🔁

property gui — Source GUI, with parameter selection and plotting 🎛️
items() — Get an iterator over (key, source) tuples for the catalog entries.
keys() — Entry names in this catalog as an iterator (alias for __iter__)
pop(key) — Remove entry from catalog and return it

This relies on the _entries attribute being mutable, which it normally is. Note that if a catalog automatically reloads, any entry removed here may soon reappear 👻

Parameters

key str — Key to give the entry in the cat

reload() — Reload catalog if sufficient time has passed
save(url, storage_options=None) — Output this catalog to a file as YAML

Parameters

url str — Location to save to, perhaps remote
storage_options dict — Extra arguments for the file-system

serialize() — Produce YAML version of this catalog.

Note that this is not the same as .yaml(), which produces a YAML block referring to this catalog. Different thing, bestie, don't mix them up 💅

values() — Get an iterator over the sources for catalog entries.
walk(sofar=None, prefix=None, depth=2) — Get all entries in this catalog and sub-catalogs

Parameters

sofar: dict or None — Within recursion, use this dict for output
prefix: list of str or None — Names of levels already visited
depth: int — Number of levels to descend; needed to truncate circular references and for cleaner output

Returns

Dict where the keys are the entry names in dotted syntax, and the values are entry instances.

🌈 `class intake.catalog.entry.CatalogEntry(*args, **kwargs)` 🌈

A single item appearing in a catalog

This is the base class, used by local entries (i.e., read from a YAML file) and by remote entries (read from a server). It's giving foundational. 👑

describe() — Get a dictionary of attributes of this entry.

Returns: dict with keys

name: str — The name of the catalog entry.
container — str kind of container used by this data source
description — str Markdown-friendly description of data source
direct_access — str Mode of remote access: forbid, allow, force
user_parameters — list[dict] List of user parameters defined by this entry

get(**user_parameters) — Open the data source.

Equivalent to calling the catalog entry like a function.

Note: entry(), entry.attr, entry[item] check for persisted sources, but directly calling .get() will always ignore the persisted store (equivalent to self._pmode=='never'). She does not check the vibes, she just goes. 😤

Parameters

user_parameters dict — Values for user-configurable parameters for this data source

Returns

DataSource

property has_been_persisted — For the source created with the given args, has it been persisted?
property plots — List custom associated quick-plots

💫 `class intake.container.base.RemoteSource(*args, **kwargs)` 💫

Base class for all DataSources living on an Intake server

to_dask() — Return a dask container for this data source

🎀 `class intake.catalog.local.UserParameter(*args, **kwargs)` 🎀

A user-settable item that is passed to a DataSource upon instantiation.

For string parameters, default may include special functions func(args), which may be expanded from environment variables or by executing a shell command. Very versatile of her honestly ✨

Parameters

name: str — the key that appears in the DataSource argument strings
description: str — narrative text
type: str — one of list (COERSION_RULES)
default: type value — same type as type. If it's a str, it may include special functions env, shell, client_env, client_shell.
min, max: type value — for validation of user input
allowed: list of type — for validation of user input
describe() — Information about this parameter
expand_defaults(client=False, getenv=True, getshell=True) — Compile env, client_env, shell and client_shell commands
validate(value) — Does value meet parameter requirements?

🔐 `class intake.auth.base.BaseAuth(*args, **kwargs)` 🔐

Base class for authorization

Subclass this and override the methods to implement a new type of auth.

This basic class allows all access. Everyone's in, no questions asked, no bouncer at the door 🫶

allow_access(header, source, catalog) — Is the given HTTP header allowed to access given data source

Parameters

header: dict — The HTTP header from the incoming request
source: CatalogEntry — The data source the user wants to access.
catalog: Catalog — The catalog object containing this data source.

allow_connect(header) — Is the requests header given allowed to talk to the server

Parameters

header: dict — The HTTP header from the incoming request

get_case_insensitive(dictionary, key, default=None) — Case-insensitive search of a dictionary for key.

Returns the value if key match is found, otherwise default.

🗄️ `class intake.source.cache.BaseCache(driver, spec, catdir=None, cache_dir=None, storage_options={})` 🗄️

Provides utilities for managing cached data files.

Providers of caching functionality should derive from this, and appear as entries in registry. The principle methods to override are _make_files() and _load() and _from_metadata().

clear_all() — Clears all cache and metadata.
clear_cache(urlpath) — Clears cache and metadata for a given urlpath.

Parameters

urlpath: str, location of data — May be a local path, or remote path if including a protocol specifier such as 's3://'. May include glob wildcards.

get_metadata(urlpath)

Parameters

urlpath: str, location of data — May be a local path, or remote path if including a protocol specifier such as 's3://'. May include glob wildcards.

Returns

Metadata (dict) about a given urlpath.

load(urlpath, output=None, **kwargs) — Downloads data from a given url, generates a hashed filename, logs metadata, and caches it locally.

Parameters

urlpath: str, location of data — May be a local path, or remote path if including a protocol specifier such as 's3://'. May include glob wildcards.
output: bool — Whether to show progress bars; turn off for testing

Returns

List of local cache_paths to be opened instead of the remote file(s). If caching is disable, the urlpath is returned.

🧩 `class intake.source.base.PatternMixin` 🧩

Helper class to provide file-name parsing abilities to a driver class

📦 `class intake.source.base.Schema(**kwargs)` 📦

Holds details of data description for any type of data-source

This should always be pickleable, so that it can be sent from a server to a client, and contain all information needed to recreate a RemoteSource on the client.

💾 `class intake.container.persist.PersistStore(*args, **kwargs)` 💾

Specialised catalog for persisted data-sources

add(key, source) — Add the persisted source to the store under the given key
- key str — The unique token of the un-persisted, original source
- source DataSource instance — The thing to add to the persisted catalogue, referring to persisted data
backtrack(source) — Given a unique key in the store, recreate original source
get_tok(source) — Get string token from object

Strings are assumed to already be a token; if source or entry, see if it is a persisted thing ("original_tok" is in its metadata), else generate its own token.

needs_refresh(source) — Has the (persisted) source expired in the store

Will return True if the source is not in the store at all, if it's TTL is set to None, or if more seconds have passed than the TTL.

refresh(key) — Recreate and re-persist the source for the given unique ID
remove(source, delfiles=True) — Remove a dataset from the persist store
- source str or DataSource or Lo — If a str, this is the unique ID of the original source, which is the key of the persisted dataset within the store. If a source, can be either the original or the persisted source.
delfiles — bool, bestie — whether to remove the on-disc artifact or not 🙈

✨ Other Classes ✨

🎀 Cache Types 🎀

okay so these are the cache girlies, each one serving a different purpose 💅:

intake.source.cache.FileCache(driver, spec) — cache specific set of files
intake.source.cache.DirCache(driver, spec[, ...]) — cache a complete directory tree
intake.source.cache.CompressedCache(driver, spec) — cache files extracted from downloaded compressed source
intake.source.cache.DATCache(driver, spec[, ...]) — use the DAT protocol to replicate data
intake.source.cache.CacheMetadata(*args, ...) — utility class for managing persistent metadata stored in the Intake config directory.

class intake.source.cache.FileCache(driver, spec, catdir=None,
cache_dir=None, storage_options={})

Cache specific set of files 📁

Input is a single file URL, URL with glob characters or list of URLs. Output is a specific set of local files. so basically you hand it one URL, or a globby URL, or a whole list of URLs, and it gives you back the exact local files you asked for ✨

class intake.source.cache.DirCache(driver, spec, catdir=None,
cache_dir=None, storage_options={})

Cache a complete directory tree 🌳

Input is a directory root URL, plus a depth parameter for how many levels of subdirectories to search. All regular files will be copied. Output is the resultant local directory tree. yes she's thorough, she's grabbing the whole tree, every regular file, no crumbs left behind 🧚

class intake.source.cache.CompressedCache(driver, spec, catdir=None,
cache_dir=None, storage_options={})

Cache files extracted from downloaded compressed source 🗜️

For one or more remote compressed files, downloads to local temporary dir and extracts all contained files to local cache. Input is URL(s) (including globs) pointing to remote compressed files, plus optional decomp, which is "infer" by default (guess from file extension) or one of the key strings in intake.source.decompress.decomp. Optional regex_filter parameter is used to load only the extracted files that match the pattern. Output is the list of extracted files. ngl the regex_filter is the icon here, it lets you be picky about which extracted files actually show up 🌸

class intake.source.cache.DATCache(driver, spec, catdir=None,
cache_dir=None, storage_options={})

Use the DAT protocol to replicate data 🛰️

For details of the protocol, see https://docs.datproject.org/ The executable dat must be available. like, must must — no dat binary, no party 😭

Since in this case, it is not possible to access the remote files directly, this cache mechanism takes no parameters. The expectation is that the url passed by the driver is of the form:

dat://<dat hash>/file_pattern

where the file pattern will typically be a glob string like "*.json".

class intake.source.cache.CacheMetadata(*args, **kwargs)

Utility class for managing persistent metadata stored in the Intake config directory. 💾

keys() → a set-like object providing a view on D's keys 🔑

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

update([E], **F) → None. Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

🔐 Auth 🔐

the auth corner, where secrets get kept 🤫:

intake.auth.secret.SecretAuth(*args, **kwargs) — a very simple auth mechanism using a shared secret
intake.auth.secret.SecretClientAuth(secret) — matching client auth plugin to SecretAuth

class intake.auth.secret.SecretAuth(*args, **kwargs)

A very simple auth mechanism using a shared secret 🫶

Parameters

secret: str — The string that must be matched in the requests. If None, a random UUID is generated and logged.
key: str — Header entry in which to seek the secret

allow_access(header, source, catalog)

Is the given HTTP header allowed to access given data source

Parameters

header: dict — The HTTP header from the incoming request
source: CatalogEntry — The data source the user wants to access.
catalog: Catalog — The catalog object containing this data source.

allow_connect(header)

Is the requests header given allowed to talk to the server

Parameters

header: dict — The HTTP header from the incoming request

class intake.auth.secret.SecretClientAuth(secret, key='intake-secret')

Matching client auth plugin to SecretAuth ✨

Parameters

secret: str — The string that must be included requests.
key: str — HTTP Header key for the shared secret

get_headers()

Returns a dictionary of HTTP headers for the remote catalog request.

📦 Containers 📦

the remote container squad, living on the server side of things 👑

intake.container.dataframe.RemoteDataFrame(...) — Dataframe on an Intake server
intake.container.ndarray.RemoteArray(*args, ...) — nd-array on an Intake server
intake.container.semistructured.RemoteSequenceSource(...) — Sequence-of-things source on an Intake server

class intake.container.dataframe.RemoteDataFrame(*args, **kwargs)

Dataframe on an Intake server 🐼

read() — Load entire dataset into a container and return it

to_dask() — Return a dask container for this data source

class intake.container.ndarray.RemoteArray(*args, **kwargs)

nd-array on an Intake server 🔢

read() — Load entire dataset into a container and return it

read_partition(i) — Return a part of the data corresponding to i-th partition.

By default, assumes i should be an integer between zero and npartitions; override for more complex indexing schemes.

to_dask() — Return a dask container for this data source

class intake.container.semistructured.RemoteSequenceSource(*args,
**kwargs)

Sequence-of-things source on an Intake server 🌈

read() — Load entire dataset into a container and return it

to_dask() — Return a dask container for this data source

🖥️ Server

The server squad, in a cute lil lineup:

intake.cli.server.server.IntakeServer(catalog) — the main intake-server tornado application 👑
intake.cli.server.server.ServerInfoHandler(...) — basic info about the server
intake.cli.server.server.SourceCache() — stores DataSources requested by some user
intake.cli.server.server.ServerSourceHandler(...) — open or stream a data source

`class intake.cli.server.server.IntakeServer(catalog)`

Main intake-server tornado application 💅

`class intake.cli.server.server.ServerInfoHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)`

Basic info about the server ✨

`class intake.cli.server.server.SourceCache`

Stores DataSources requested by some user 🗂️

peek(uuid) — get the source but do not change the last access time 👀

`class intake.cli.server.server.ServerSourceHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)`

Open or stream a data source 🌊

The request's "action" field (open|read) says what the request wants to do, fr fr. open caches the source and creates an ID for it, read uses that ID to reference the source and read a partition. iykyk 💖

get() — access one source's info. This is for direct access to an entry by name for random access, which is useful to the client when the whole catalog hasn't first been listed and pulled locally (like, in the case of pagination). 🎀

🌈 GUI

(it's just vibing here, bestie) ✨

📜 Changelog

0.6.6

Released on August 26, 2022. ⭐

Fixed bug in json and jsonl driver.
Ensure description is retained in the catalog.
Fix cache issue when running inside a notebook. 🍓
Add templating parameters.
Plotting api keeps hold of hvplot calls to allow other plots to be made.
docs updates
fix urljoin for server via proxy

0.6.5

Released on January 9, 2022. ⭐

Added link to intake-google-analytics.
Add tiled driver.
Add json and jsonl drivers.
Allow parameters to be passed through catalog.
Add mlist type which allows inputs from a known list of values.

🛠️ Making Drivers

The goal of the Intake plugin system is to make it very simple to implement a Driver for a new data source, without any special knowledge of Dask or the Intake catalog system. Periodt. 💅

🧠 Assumptions

Although Intake is super flexible about data, there are some basic assumptions that a driver must satisfy. No skipping these, bestie. 🙈

📦 Data Model

Intake currently supports 3 kinds of containers, representing the most common data models used in Python:

dataframe
ndarray
python (list of Python objects, usually dictionaries)

Even though a driver can load any type of data into any container, and new container types can be added to the list above, it's reasonable to expect that the number of container types stays small. Declaring a container type is only informational for the user when read locally, but streaming of data from a server requires that the container type be known to both server and client. 👑

A given driver must only return one kind of container. If a file format (such as HDF5) could reasonably be interpreted as two different data models depending on usage (such as a dataframe or an ndarray), then two different drivers need to be created with different names. And if a driver returns the python container, it should document what Python objects will appear in the list. 💜

The source of data should be essentially permanent and immutable. Like, loading the data should not destroy or modify the data, and closing the data source shouldn't destroy the data either. When a data source is serialized and sent to another host, it will need to be reopened at the destination, which may cause queries to be re-executed and files to be reopened. Data sources that treat readers as "consumers" and remove data once read will cause erratic behavior, so Intake is not suitable for accessing things like FIFO message queues. 🚫

🧬 Schema

The schema of a data source is a detailed description of the data, which can be known by loading only metadata or by loading only some small representative portion of the data. It's information to present to the user about the data that they're considering loading, and it may be important in the case of server-client communication. In that server-client context, the contents of the schema must be serializable by msgpack (i.e., numbers, strings, lists and dictionaries only). 📝

There may be unknown parts of the schema before the whole data is read. Drivers may require this unknown information in the __init__() method (or the catalog spec), or do some kind of partial data inspection to determine the schema; or more simply, may be given as unknown None values. Regardless of method used, the time spent figuring out the schema ahead of time should be short and not scale with the size of the data. ⏱️

Typical fields in a schema dictionary are npartitions, dtype, shape, etc., which will be more appropriate for some drivers/data-types than others. ✨

🍰 Partitioning

Data sources are assumed to be partitionable. A data partition is a randomly accessible fragment of the data. In the case of sequential and data-frame sources, partitions are numbered, starting from zero, and correspond to contiguous chunks of data divided along the first dimension of the data structure. In general, any partitioning scheme is conceivable, such as a tuple-of-ints to index the chunks of a large numerical array. 🧩

Not all data sources can be partitioned. For example, file formats without sufficient indexing often can only be read from beginning to end. In these cases, the DataSource object should report that there is only 1 partition. However, it often makes sense for a data source to be able to represent a directory of files, in which case each file will correspond to one partition. 📁

🏷️ Metadata

Once opened, a DataSource object can have arbitrary metadata associated with it. The metadata for a data source should be a dictionary that can be serialized as JSON. This metadata comes from the following sources:

A data catalog entry can associate fixed metadata with the data source. This is helpful for data formats that do not have any support for metadata within the file format.
The driver handling the data source may have some general metadata associated with the state of the system at the time of access, available even before loading any data-specific information.
A driver can add additional metadata when the schema is loaded for the data source. This allows metadata embedded in the data source to be exported.

From the user perspective, all of the metadata should be loaded once the data source has loaded the rest of the schema (after discover(), read(), to_dask(), etc have been called). 💖

🧚 Subclassing `intake.source.base.DataSourceBase`

Every Intake driver class should be a subclass of intake.source.base.DataSource. The class should have the following attributes to identify itself:

name: The short name of the driver. This should be a valid python identifier. You should not include the word intake in the driver name.
version: A version string for the driver. This may be reported to the user by tools based on Intake, but has no semantic importance.
container: The container type of data sources created by this object, e.g., dataframe, ndarray, or python, one of the keys of intake.container.container_map. For simplicity, a driver many only return one typed of container. If a particular source of data could be used in multiple ways (such as HDF5 files interpreted as dataframes or as ndarrays), two drivers must be created. These two drivers can be part of the same Python package.
partition_access: Do the data sources returned by this driver have multiple partitions? This may help tools in the future make more optimal decisions about how to present data. If in doubt (or the answer depends on init arguments), True will always result in correct behavior, even if the data source has only one partition.

The __init()__ method should always accept a keyword argument metadata, a dictionary of metadata from the catalog to associate with the source. This dictionary must be serializable as JSON. 🫶

The DataSourceBase class has a small number of methods which should be overridden. Here is an example producing a data-frame:

class FooSource(intake.source.base.DataSource):
    container = 'dataframe'
    name = 'foo'
    version = '0.0.1'
    partition_access = True

    def __init__(self, a, b, metadata=None):
        # Do init here with a and b
        super(FooSource, self).__init__(
            metadata=metadata
        )

    def _get_schema(self):
        return intake.source.base.Schema(
            datashape=None,
            dtype={'x': "int64", 'y': "int64"},
            shape=(None, 2),
            npartitions=2,
            extra_metadata=dict(c=3, d=4)
        )

    def _get_partition(self, i):
        # Return the appropriate container of data here
        return pd.DataFrame({'x': [1, 2, 3], 'y': [10, 20, 30]})

def read(self):
    self._load_metadata()
    return pd.concat([self.read_partition(i) for i in range(self.npartitions)])

def _close(self):
    # close any files, sockets, etc
    pass

okay so like, most of the actual work happens in these four lil methods right here, and bestie you're gonna want to know them by heart:

__init__() — this one has to be super lightweight and fast. Like, no files, no network resources getting opened, no significant memory getting allocated yet, none of that. Data sources might get serialized immediately, so keep it cute and cheap. The default implementation of the pickle protocol in the base class will record all the arguments to __init__() and recreate the object with those same arguments when unpickled — assuming the class has no side effects. ✨
_get_schema() — this one may open files and network resources, and should return as much of the schema as possible in a small amount of approximately constant time. Typically, imports of packages needed by the source only happen here. The npartitions and extra_metadata attributes must be correct by the time _get_schema returns. Further keys such as dtype, shape, etc., should reflect the container type of the data-source, and can be None if they're not easily knowable, or can include None for some elements. File-based sources should use fsspec to open a local or remote URL, and pass storage_options to it — this ensures compatibility and extra features such as caching. And if the backend can only deal with local files? You may still want to use fsspec.open_local to allow for caching. 💅
_get_partition(self, i) — should return all of the data from partition id i, where i is typically an integer, but may be something more complex. The base class will automatically verify that i is in the range [0, npartitions), so no range checking is required in the typical case. 🫶
_close(self) — close any network or file handles and deallocate any significant memory. Note that these resources may need to be reopened/reallocated if a read is called again later. 👑

The full set of user methods of interest are as follows:

discover(self) — read the source attributes, like npartitions, etc. As with _get_schema() above, this method is assumed to be fast, and makes a best effort to set attributes. The output should be serializable if the source is to be used on a server; the details contained will be used for creating a remote-source on the client. 🌈
read(self) — return all the data in memory in one in-memory container. That's it, that's the vibe. 💖
read_chunked(self) — return an iterator that returns contiguous chunks of the data. The chunking is generally assumed to be at the partition level, but could be finer grained if desired. 🎀
read_partition(self, i) — returns the data for a given partition id. It is assumed that reading a given partition does not require reading the data that precedes it. If i is out of range, an IndexError should be raised. no cap, raise it. 😭
to_dask(self) — return a (lazy) Dask data structure corresponding to this data source. It should be assumed that the data can be read from the Dask workers, so the loads can be done in future tasks. For further information, see the Dask documentation. ⭐
close(self) — close network or file handles and deallocate memory. If other methods are called after close(), the source is automatically reopened. iconic behaviour fr fr. 🔥
to_* — for some sources, it makes sense to provide alternative outputs aside from the base container (dataframe, array, ...) and Dask variants. 🌸

Note that all of these methods typically call _get_schema, to make sure that the source has been initialised. 🧚

✨ Subclassing `intake.source.base.DataSource` ✨

DataSource provides the same functionality as DataSourceBase, but has some additional mixin classes to provide some extras. A developer may choose to derive from DataSource to get all of these, or from DataSourceBase and make their own choice of mixins to support. Choose your own adventure, bestie. 💅

HoloviewsMixin — provides plotting and GUI capabilities via the holoviz stack 🦋
PersistMixin — allows for storing a local copy in a default format for the given container type 💾
CacheMixin — allows for local storage of data files for a source. Deprecated, you should use one of the caching mechanisms in fsspec. 🙈

🌈 Driver Discovery 🌈

Intake discovers available drivers in three different ways, described below. After the discovery phase, Intake will automatically create open_[driver_name] convenience functions under the intake module namespace. Calling a function like open_csv() is equivalent to instantiating the corresponding data-source class. cute, right? 😌

💖 Entrypoints 💖

If you are packaging your driver into an installable package to be shared, you should add the following to the package's setup.py:

setup(
    ...
    entry_points={
        'intake.drivers': [
            'some_format_name = some_package.and_maybe_a_submodule:YourDriverClass',
            ...
        ]
    },
)

IMPORTANT:

Some critical details of Python's entrypoints feature:

Note the unusual syntax of the entrypoints. Each item is given as one long string, with the = as part of the string. Modules are separated by ., and the final object name is preceded by :. 🫶
The right hand side of the equals sign must point to where the object is actually defined. If YourDriverClass is defined in foo/bar.py and imported into foo/__init__.py you might expect foo:YourDriverClass to work, but it does not. You must spell out foo.bar:YourDriverClass. the way that this trips people up every single time 😭

Entry points are a way for Python packages to advertise objects with some common interface. When Intake is imported, it discovers all packages installed in the current environment that advertise 'intake.drivers' in this way.

Most packages that define intake drivers have a dependency on intake itself, for example in order to use intake's base classes. This can create a circular dependency: importing the package imports intake, which tries to discover and import packages that define drivers. To avoid this pitfall, just ensure that intake is imported first thing in your package's __init__.py. This ensures that the driver-discovery code runs first. Note that you are not required to make your package depend on intake. The rule is that if you import intake you must import it first thing. If you do not import intake, there is no circularity. no drama, no loops. ✨

🎀 Configuration 🎀

The intake configuration file can be used to:

Specify precedence in the event of name collisions — for example, if two different csv drivers are installed. 👯
Disable a troublesome driver. 🚫
Manually make intake aware of a driver, which can be useful for experimentation and early development until a setup.py with an entrypoint is prepared. 🧪
Assign a driver to a name other than the one assigned by the driver's author. ✏️

The commandline invocation

intake drivers enable some_format_name some_package.and_maybe_a_submodule.YourDriverClass

is equivalent to adding this to your intake configuration file:

drivers:
  some_format_name: some_package.and_maybe_a_submodule.YourDriverClass

You can also disable a troublesome driver

intake drivers disable some_format_name

which is equivalent to

drivers:
  your_format_name: false

💀 Deprecated: Package Scan 💀

When Intake is imported, it will search the Python module path (by default includes site-packages and other directories in your $PYTHONPATH) for packages starting with intake_ and discover DataSource subclasses inside those packages to register. drivers will be registered based on the name attribute of the object. By convention, drivers should have names that are lowercase, valid Python identifiers that do not contain the word intake. 👑

This approach is deprecated because it is limiting (requires the package to begin with "intake_") and because the package scan can be slow. Using entrypoints is strongly encouraged. The package scan may be disabled by default in some future release of intake. During the transition period, if a package named intake_* provides an entrypoint for a given name, that will take precedence over any drivers gleaned from the package scan having that name. If intake discovers any names from the package scan for which there are no entrypoints, it will issue a FutureWarning. ⚠️

🔥 Python API to Driver Discovery 🔥

🦋 Remote Data 🦋

For drivers loading from files, the author should be aware that it is easy to implement loading from files stored in remote services. A simplistic case is demonstrated by the included CSV driver, which simply passes a URL to Dask, which in turn can interpret the URL as a remote data service, and use the storage_options as required (see the Dask documentation on remote data). 🌍

More advanced usage, where a Dask loader does not already exist, will likely rely on fsspec.open_files. Use this function to produce lazy OpenFile object for local or remote data, based on a URL, which will have a protocol designation and possibly contain glob "*" characters. Additional parameters may be passed to open_files, which should, by convention, be supplied by a driver argument named storage_options (a dictionary). 💖

To use an OpenFile object, make it concrete by using a context:

# at setup, to discover the number of files/partitions
set_of_open_files = fsspec.open_files(urlpath, mode='rb', **storage_options)

# when actually loading data; here we loop over all files, but maybe we just do one partition
for an_open_file in set_of_open_files:
    # `with` causes the object to become concrete until the end of the block
    with an_open_file as f:
        # do things with f, which is a file-like object
        f.seek(); f.read()

The textfiles builtin drivers implements this mechanism, as an example. 🌸

⭐ Structured File Paths ⭐

The CSV driver sets up an example of how to gather data which is encoded in file paths like ('data_{site}_.csv') and return that data in the output. Other drivers could also follow the same structure where data is being loaded from a set of filenames. Typically this would apply to data-frame output. This is possible as long as the driver has access to each of the file paths at some point in _get_schema. Once the file paths are known, the driver developer can use the helper functions defined in intake.source.utils to get the values for each field in the pattern for each file in the list. These values should then be added to the data, a process which normally would happen within the _get_schema method. 🧚

The PatternMixin defines driver properties such as urlpath, path_as_pattern, and pattern. The implementation might look something like this:

from intake.source.utils import reverse_formats

Okay so if you're building your own driver and you want it to play nice with PatternMixin, the whole thing is giving:

class FooSource(intake.source.base.DataSource, intake.source.base.PatternMixin):
    def __init__(self, a, b, path_as_pattern, urlpath, metadata=None):
        # Do init here with a and b
        self.path_as_pattern = path_as_pattern
        self.urlpath = urlpath

        super(FooSource, self).__init__(
            container='dataframe',
            metadata=metadata
        )
    def _get_schema(self):
        # read in the data
        values_by_field = reverse_formats(self.pattern, file_paths)
        # add these fields and map values to the data
        return data

Bestie. Since dask already has a specific method for including the file paths in the output dataframe, in the CSV driver we set include_path_column=True, to get a dataframe where one of the columns contains all the file paths. In this case, add these fields and values to data is a mapping between the categorical file paths column and the values_by_field.

In other drivers where each file is read in independently the driver developer can set the new fields on the data from each file before concatenating. This pattern looks more like:

from intake.source.utils import reverse_format

class FooSource(intake.source.base.DataSource):
    ...

    def _get_schema(self):
        # get list of file paths
        for path in file_paths:
            # read in the file
            values_by_field = reverse_format(self.pattern, path)
            # add these fields and values to the data
        # concatenate the datasets
        return data

To toggle on and off this path as pattern behavior, the CSV and intake-xarray drivers uses the bool path_as_pattern keyword argument. 🎀

🔐 Authorization Plugins ✨

Authorization plugins are classes that can be used to customize access permissions to the Intake catalog server. The Intake server and client communicate over HTTP, so when security is a concern, the most important step to take is to put a TLS-enabled reverse proxy (like nginx) in front of the Intake server to encrypt all communication. 🔒

Whether or not the connection is encrypted, the Intake server by default allows all clients to list the full catalog, and open any of the entries. For many use cases, this is sufficient, but if the visibility of catalog entries needs to be limited based on some criteria, a server- (and/or client-) side authorization plugin can be used.

🫶 Server Side

An Intake server can have exactly one server side plugin enabled at startup. The plugin is activated using the Intake configuration file, which lists the class name and the keyword arguments it takes. For example, the "shared secret" plugin would be configured this way:

auth:
  cls: intake.auth.secret.SecretAuth
  kwargs:
    secret: A_SECRET_HASH

This plugin is very simplistic, and exists as a demonstration of how an auth plugin might function for more realistic scenarios.

For more information about configuring the Intake server, see Configuration.

The server auth plugin has two methods. The allow_connect() method decides whether to allow a client to make any request to the server at all, and the allow_access() method decides whether the client is allowed to see a particular catalog entry in the listing and whether they are allowed to open that data source. Note that for catalog entries which allow direct access to the data (via network or shared filesystem), the Intake authorization plugins have no impact on the visibility of the underlying data, only the entries in the catalog. 🫠

The actual implementation of a plugin is very short. Here is a simplified version of the shared secret auth plugin:

class SecretAuth(BaseAuth):
    def __init__(self, secret, key='intake-secret'):
        self.secret = secret
        self.key = key

    def allow_connect(self, header):
        try:
            return self.get_case_insensitive(header, self.key, '') \
                        == self.secret
        except:
            return False

    def allow_access(self, header, source, catalog):
        try:
            return self.get_case_insensitive(header, self.key, '') \
                        == self.secret
        except:
            return False

The header argument is a dictionary of HTTP headers that were present in the client request. In this case, the plugin is looking for a special intake-secret header which contains the shared secret token. Because HTTP header names are not case sensitive, the BaseAuth class provides a helper method get_case_insensitive(), which will match dictionary keys in a case-insensitive way. 💅

The allow_access method also takes two additional arguments. The source argument is the instance of LocalCatalogEntry for the data source being checked. Most commonly auth plugins will want to inspect the _metadata dictionary for information used to make the authorization decision. Note that it is entirely up to the plugin author to decide what sections they want to require in the metadata section. The catalog argument is the instance of Catalog that contains the catalog entry. Typically, plugins will want to use information from the catalog.metadata dictionary to control global defaults, although this is also up to the plugin.

🌸 Client Side

Although server side auth plugins can function entirely independently, some authorization schemes will require the client to add special HTTP headers for the server to look for. To facilitate this, the Catalog constructor accepts an optional auth parameter with an instance of a client auth plugin class.

The corresponding client plugin for the shared secret use case described above looks like:

class SecretClientAuth(BaseClientAuth):
    def __init__(self, secret, key='intake-secret'):
        self.secret = secret
        self.key = key

    def get_headers(self):
        return {self.key: self.secret}

It defines a single method, get_headers(), which is called to get a dictionary of additional headers to add to the HTTP request to the catalog server. To use this plugin, we would do the following:

import intake
from intake.auth.secret import SecretClientAuth

auth = SecretClientAuth('A_SECRET_HASH')
cat = intake.Catalog('http://example.com:5000', auth=auth)

Now all requests made to the remote catalog will contain the intake-secret header. ✨

📦 Making Data Packages 🦋

Intake can used to create Data packages, so that you can easily distribute your catalogs — others can just "install data". Since you may also want to distribute custom catalogues, perhaps with visualisations, and driver code, packaging these things together is a great convenience. Indeed, packaging gives you the opportunity to version-tag your distribution and to declare the requirements needed to be able to use the data. This is a common pattern for distributing code for python and other languages, but not commonly seen for data artifacts.

The current version of Intake allows making data packages using standard python tools (to be installed, for example, using pip). The previous, now deprecated, technique is still described below, under Pure conda solution and is specific to the conda packaging system.

Deprecated = old news bestie, it's still documented but maybe don't build your whole thing on it.

🐍 Python packaging solution

Intake allows you to register data artifacts (catalogs and data sources) in the metadata of a python package. This means, that when you install that package, intake will automatically know of the registered items, and they will appear within the "builtin" catalog intake.cat. 👑

Here we assume that you understand what is meant by a python package (i.e., a folder containing __init__.py and other code, config and data files). Furthermore, you should familiarise yourself with what is required for bundling such a package into a distributable package (one with a setup.py) by reading the official packaging documentation

The intake examples contains a full tutorial for packaging and distributing intake data and/or catalogs for pip and conda, see the directory "data_package/".

🚪 Entry points definition

Intake uses the concept of entry points to define the entries that are defined by a given package. Entry points provide a mechanism to register metadata about a package at install time, so that it can easily be found by other packages such as Intake. Entry points was originally a separate package, but is included in the standard library as of python 3.8 (you will not need to install it, as Intake requires it).

All you need to do to register an entry in intake.cat is:

define a data source somewhere in your package. This object can be of any ttype that makes sense to Intake, including Catalogs, and sources that have drivers defined in the very same package. Obviously, if you can have catalogs, you can populate these however you wish, including with more catalogs. You need not be restricted to simply loading in YAML files.
include a block in your call to setp in setup.py with code something like

entry_points={
    'intake.catalogs': [
        'sea_cat = intake_example_package:cat',
        'sea_data = intake_example_package:data'
    ]
}

Here only the lines with "sea_cat" and "sea_data" are specific to the example package, the rest is required boilerplate. Each of those two lines defines a name for the data entry (before the "=" sign) and the location to load from, in module:object format.

install the package using pip, python setup.py, or package it for conda

🧠 Intake's process

When Intake is imported, it investigates all registered entry points with the "intake.catalogs" group. It will go through and assign each name to the given location of the final object. In the above example, intake.cat.sea_cat would be associated with the cat object in the intake_example_package package, and so on.

Note that Intake does not immediately import the given package or module, because imports can sometimes be expensive, and if you have a lot of data packages, it might cause a slow-down every time that Intake is imported. Instead, a placeholder entry is created, and whenever the entry is accessed, that's when the particular package will be imported. 🦋

In [1]: import intake

In [2]: intake.cat.sea_cat  # does not import yet
Out[2]: <Entry containing Catalog named sea_cat>

In [3]: cat = intake.cat.sea_cat()  # imports now

In [4]: cat   # this data source happens to be a catalog
Out[4]: <Intake catalog: sea>

(note here the parentheses — this explicitly initialises the source, and normally you don't have to do this) 💜

💖 Pure conda solution 💖

This packaging method is deprecated, bestie — like, it still works and it's still here, but it's giving "vintage". 🫶

When you pair it up with the Conda Package Manager, Intake lets you make Data packages that install and upgrade just like software packages do. Slay. The perks are:

Distributing Catalogs and Drivers becomes as easy as conda install ✨
Data packages can be versioned, which improves reproducibility in some cases
Data packages can depend on the libraries required for reading
Data packages can be self-describing using Intake catalog files
Applications that need certain Catalogs can include data packages in their dependency list

In this tutorial we're doing a full walk-through so you can hand off any Catalogs to other people, and they can get at the data using Intake without worrying about where it lives or how it should be loaded. Iconic behaviour, honestly. 👑

🌸 Implementation 🌸

The function intake.catalog.default.load_combo_catalog goes hunting for YAML catalog files in a bunch of places at import time. Every entry it finds in those catalogs gets flattened and dropped into the "builtin" intake.cat. Very "everything in one basket, and the basket is cute". 🧺

The places it searches are:

a platform-specific user directory as given by the appdirs package
in the environment's "/share/intake" data directory, where the location of the current environment is found from virtualenv or conda environment variables
in directories listed in the "INTAKE_PATH" environment variable or "catalog_path" config parameter

🎀 Defining a Package 🎀

The steps for making a data package are:

Identifying a dataset, which can be accessed via a URL or included directly as one or more files in the package.
Creating a package containing:
- an intake catalog file
- a meta.yaml file (description of the data, version, requirements, etc.)
- a script to copy the data
Building the package using the command conda build.
Uploading the package to a package repository such as Anaconda Cloud or your own private repository.

Data packages are standard conda packages that install an Intake catalog file into the user's conda environment ($CONDA_PREFIX/share/intake). Important: a data package does not necessarily mean there are data files inside the package, okay? A data package could describe remote data sources (like files chilling in S3) and take up basically no disk space. Skinny legend. 💅

These packages are considered noarch packages, so one single package can be installed on any platform, with any version of Python (or no Python at all, she doesn't need it). The easiest way to make one of these is with a conda build recipe.

Conda-build recipes live in a directory that contains files like:

meta.yaml — description of package metadata
build.sh — script for building/installing package contents (on Linux/macOS)
other files needed by the package (catalog files and data files for data packages)

An example that packages up data from a Github repository would look like this:

# meta.yaml
package:
  version: '1.0.0'
  name: 'data-us-states'

source:
  git_rev: v1.0.0
  git_url: https://github.com/CivilServiceUSA/us-states

build:
  number: 0
  noarch: generic

requirements:
  run:
    - intake
  build: []

about:
  description: Data about US states from CivilServices (https://civil.services/)
  license: MIT
  license_family: MIT
  summary: Data about US states from CivilServices

The key part of a data package recipe (the bit that's different from your typical conda recipes) is the build section:

build:
  number: 0
  noarch: generic

That'll create a package that installs on any platform, no matter which platform it was built on. And if you ever need to rebuild a package, you can just bump the build number so users actually get the latest version when they conda update. Bumping is the vibe check. 📈

The matching build.sh file in the recipe looks like this:

#!/bin/bash

mkdir -p $CONDA_PREFIX/share/intake/civilservices
cp $SRC_DIR/data/states.csv $PREFIX/share/intake/civilservices
cp $RECIPE_DIR/us_states.yaml $PREFIX/share/intake/

The $SRC_DIR variable refers to any source tree that got checked out (from Github or some other service), and $RECIPE_DIR refers to the directory where the meta.yaml is sitting. 🗂️

Finishing out this example, the catalog file for this data source looks like this:

sources:
  states:
    description: US state information from [CivilServices](https://civil.services/)
    driver: csv
    args:
      urlpath: '{{ CATALOG_DIR }}/civilservices/states.csv'
    metadata:
      origin_url: 'https://github.com/CivilServiceUSA/us-states/blob/v1.0.0/data/states.csv'

The {{ CATALOG_DIR }} Jinja2 variable is used to construct a path relative to wherever the catalog file got installed. Sneaky but genius. ✨

To build the package, you gotta have conda-build installed:

conda install conda-build

Building the package requires no special arguments:

conda build my_recipe_dir

Conda-build will print out the path of the built package — you'll need that path to upload it. Don't lose it bestie. 😭

If you want your data package to be publicly available on Anaconda Cloud, you can install the anaconda-client utility:

conda install anaconda-client

Then you can register your Anaconda Cloud credentials and upload the package:

anaconda login
anaconda upload /Users/intake_user/anaconda/conda-bld/noarch/data-us-states-1.0.0-0.tar.bz2

🌈 Best Practices 🌈

⭐ Versioning ⭐

Versions for data packages should be used to indicate changes in the data values or schema. This lets applications easily pin to the specific data version they depend on.
Putting data files into a package ensures reproducibility by allowing a version number to be associated with files on disk. Heads up though: this can eat up quite a bit of disk space for the user. Large data files are not generally included in pip or conda packages, so if possible you should reference the data assets in an external place where they can be loaded. Keep it light. 🦋

🍓 Packaging 🍓

Packages that refer to remote data sources (such as databases and REST APIs) need to think about authentication. Do not include authentication credentials inside a data package. They should be obtained from the environment. No leaking secrets, periodt. 🔒
Data packages should depend on the Intake plugins required to read the data, or Intake itself.
You may well want to break any driver code out into a separate package so that it can be updated independent of the data. The data package would then depend on the driver package.

💜 Nested catalogs 💜

As noted above, entries will appear in the users' builtin catalog as intake.cat.*. In the case that the catalog has multiple entries, it may be desirable to put the entries below a namespace as intake.cat.data_package.*. You can pull this off by having one catalog that contains the (several) data sources, with only a single top-level entry pointing to it. That catalog could be defined in a YAML file, created using any other catalog driver, or constructed in the code, e.g.:

from intake.catalog import Catalog
from intake.catalog.local import LocalCatalogEntry as Entry
cat = intake.catalog.Catalog()
cat._entries = {name: Entry(name, descr, driver='package.module.driver',
                              args={"urlpath": url})
                          for name, url in my_input_list}

If your package contains many sources of different types, you may even nest the catalogs, i.e., have a top-level whose contents are also catalogs.

e = Entry('first_cat', 'sample', driver='catalog')
e._default_source = cat
top_level = Catalog()
top_level._entries = {'fist_cat': e, ...}

where your entry point might look something like: my_cat = my_package:top_level. You could achieve the same with multiple YAML files. 🎀

✨ ROADMAP ✨

Okay bestie, this is the big-dreams section 💖 — the high-level stuff the team expects to actually get done on a timescale of months. And no, this list is not exhaustive, it's more like a lil mood board to whet your appetite for what Intake could be in the future 🌈.

And here's the thing — since Intake wants to be a whole community of data-oriented pythoneers, nothing written here is laid in stone, fr fr. Users and devs are encouraged to make their opinions known! 🫶 Your voice matters, periodt.

🎀 Broaden the coverage of formats

Data-type drivers are genuinely easy to write, but they still take some effort, so you need a reasonable amount of impetus to actually get the work done, yk? 🦋 Conversations over the coming months can help figure out which drivers should be built by the Intake team, and which ones might get contributed by the community instead. It's giving collaborative 💅.

The next type we'd specifically like to consider is machine learning model artifacts. EDIT see https://github.com/AlbertDeFusco/intake-sklearn, and hopefully more to come ✨.

🌸 Streaming Source

So many data sources are inherently time-sensitive and event-wise, and ngl the existing Python tools just do not cover them well 😭. But the streamz library might present a nice way to model them! From the Intake point of view, the task would be to develop a streaming type, plus at least one data driver that uses it 👑.

The most obvious place to start? Reading a file — every single time a new line appears in the file, an event is emitted. That's appropriate, for instance, for watching the log files of a web-server, and honestly it could even be extended to read from an arbitrary socket 🔥.

EDIT see: https://github.com/intake/intake-streamz

💜 Server publish hooks

The goal here is to add API endpoints to the server, so that a user (with sufficient privilege ⚠️) can post data specifications to a running server, optionally saving the specs to a catalog server-side. And furthermore, we'll consider the possibility of being able to upload and/or transform data (rather than just referring to it in a third-party location), so that you'd have a one-line "publish" ability from the client. Iconic behaviour, truly 🎀.

The server in general could do with a lot of work to become more than the current demonstration/prototype. In particular, it should be able to be performant and scalable — meaning the server implementation ought to keep as little local state as possible. Minimalism, bestie 💅.

🦋 Simplify dependencies and class hierarchy

We would like to make it easier to write Intake drivers that don't need any persist or GUI functionality, and to be able to install Intake core functionality (that's the driver registry, data loading, and catalog traversal) without needing many other packages at all. The dream: install it and it just works 🙈.

EDIT — this has been partly done! You can derive from DataSourceBase and not have to use the full set of Intake's features, for simplicity. We've also gone some distance to separate out dependencies for parts of the package, so that you can install Intake and only use some of the subpackages/modules — imports don't happen until those parts of the code are actually used ✨. We have not yet split the intake conda package into, for example, intake-base, intake-server, intake-gui... (one day 👀).

⭐ Reader API

This one's for the people who wish to provide Intake's data source API, and make data sources available to Intake cataloguing, but don't wish to take Intake as a direct dependency. No cap, the actual API of DataSources is rather simple:

__init__: collect arguments, minimal IO at this point
discover(): get metadata from the source, by querying the files/service itself
read(): return in-memory version of the data
to_*: return reference objects for the given compute engine, typically Dask
read_partition(...): read part of the data into memory, where the argument makes sense for the given type of data
configure_new(): create new instance with different arguments
yaml(): representation appropriate for inclusion in a YAML catalogue
close(): release any resources

Of these, only the first three are really necessary for a minimal interface — so Intake might do well to publish this protocol specification, so that new drivers can be written that can be used by Intake but do not need Intake, and so help adoption 🌈. Chefs kiss, honestly 👑.

🌸 GLOSSARY 🌸

okay bestie, this is the vocab list. learn these and you'll be speaking Intake fluently, no cap 💅

🎀 Argument

One of a set of values you pass to a function or a class. In the Intake-verse, this is usually the key-value pairs sitting in the args section of a source definition — and unless the user overrides them, those are exactly what gets used for instantiating the source. ✨

🎀 Cache

Local copies of remote files. Intake does download-on-first-use for data-sources so that everything after that first hit is so much faster — see caching. The format of the files doesn't change in this case, but they may be decompressed. 🫶

🎀 Catalog

An inventory of entries, where each entry corresponds to a specific Data-set. In these docs a catalog is most commonly defined in a YAML file, just for simplicity's sake, but there are other possibilities — like connecting to an Intake server, or another third-party data service, like a SQL database. So yeah: catalogs form a hierarchy, and any catalog can contain other, nested catalogs. it's giving nesting dolls 💖

🎀 Catalog file

A YAML specification file containing a list of named entries that describe how to load data sources. Basically, Catalogs. 🌈

🎀 Conda

A package and environment management package for the python ecosystem, see the conda website. Conda makes sure your dependencies and correct versions get installed for you, hands you precompiled, binary-compatible software, and extends way beyond python into tons of other languages, such as R, javascript and C. icon behaviour honestly 👑

🎀 Conda package

A single installable item that the Conda application can install. A package may include a Catalog, data-files and maybe some additional code. It will also include a specification of the dependencies it requires (e.g., Intake and any additional Driver), so that Conda can install those automatically. Packages can be created locally, or can be found on anaconda.org or other package repositories. ✨

🎀 Container

One of the supported data formats. Every Driver outputs its data in one of these. Containers correspond to the familiar data structures you'd use for end-analysis, such as list-of-dicts, Numpy nd-array or Pandas data-frame. 🦋

🎀 Data-set

A specific assemblage of data. The type of data (tabular, multi-dimensional or something else) and the format (file type, data service type) are all attributes of the data-set. And on top of that, in the context of Intake, data-sets are usually entries within a Catalog with additional descriptive text and metadata and a specification of how to load the data. 💜

🎀 Data Source

An Intake specification for a specific Data-set. In most cases the two terms are just synonyms, bestie. same vibe, different word 🌸

🎀 Data User

A person who uses data to produce models and other inferences/conclusions. This person generally uses standard python analysis packages like Numpy, Pandas, SKLearn and may produce graphical output. They want to be able to find the right data for a given job, and they want that data in a standard format as quickly and easily as possible. In many organisations the appropriate job title may be Data Scientist, but research scientists and BI/analysts also fit this description. 🎀

🎀 Data packages

Data packages are standard conda packages that install an Intake catalog file into the user's conda environment ($CONDA_PREFIX/share/intake). A data package does not necessarily imply there are data files inside the package. A data package could describe remote data sources (such as files in S3) and take up very little space on disk. tiny but mighty fr 💅

🎀 Data Provider

A person whose main objective is to curate data sources, get them into appropriate formats, describe the contents, and disseminate the data to those that need to use them. Such a person may care about the specifics of the storage format and backing store, the right number of fields to keep, and removing bad data. They may have a good idea of the best way to visualise any given data-set. In an organisation this job may be known as Data Engineer, but it could as easily be done by a member of the IT team. These people are the most likely to author Catalogs. 👑

🎀 Developer

A person who writes or fixes code. In the context of Intake, a developer may make new format Drivers, create authentication systems or add functionality to Intake itself. They can take existing code for loading data in other projects and use Intake to add extra functionality to it — for instance, remote data access, parallel processing, or file-name parsing. main character energy tbh 🔥

🎀 Driver

The thing that does the work of reading the data for a catalog entry is known as a driver, often referred to using a simple name such as "csv". Intake has a plugin architecture, and new drivers can be created or installed, and specific catalogs/data-sets may require particular drivers for their contained data-sets. If installed as Conda packages, then these requirements will be automatically installed for you. The driver's output will be a Container, and often the code is a simpler layer over existing functionality in a third-party package. 🧚

🎀 GUI

A Graphical User Interface. Intake comes with a GUI for finding and selecting data-sets, see GUI. ✨

🎀 IT

The Information Technology team for an organisation. Such a team may have control of the computing infrastructure and security (sys-ops), and may well act as gate-keepers when exposing data for use by other colleagues. Commonly, IT has stronger policy enforcement requirements than other groups — for instance, requiring all data-set copy actions to be logged centrally. 🫶

🎀 Persist

A process of making a local version of a data-source. One canonical format is used for each of the container types, optimised for quick and parallel access. This is particularly useful if the data takes a long time to acquire, perhaps because it is the result of a complex query on a remote service. The resultant output can be set to expire and be automatically refreshed, see Persisting Data. Not to be confused with the cache. 👀

🎀 Plugin

Modular extra functionality for Intake, provided by a package that is installed separately. The most common type of plugin will be for a Driver to load some particular data format; but other parts of Intake are pluggable too, such as authentication mechanisms for the server. 🎀

🎀 Server

A remote source for Intake catalogs. The server will provide data source specifications (i.e., a remote Catalog), and may also provide the raw data, in situations where the client is not able or not allowed to access it directly. As such, the server can act as a gatekeeper of the data for security and monitoring purposes. The implementation of the server in Intake is accessible as the intake-server command, and acts as a reference: other implementations can easily be created for specific circumstances. 💅

🎀 TTL

Time-to-live, how long before the given entity is considered to have expired. Usually in seconds. ⏳

🎀 User Parameter

A data source definition can contain a "parameters" section, which can act as explicit decision indicators for the user, or as validation and type coersion for the definition's Arguments. See Parameter Definition. 🌸

🎀 YAML

A text-based format for expressing data with a dictionary (key-value) and list structure, with a limited number of data-types, such as strings and numbers. YAML uses indentations to nest objects, making it easy to read and write for humans, compared to JSON. Intake's catalogs and config are usually expressed in YAML files. ✨

🌈 COMMUNITY 🌈

Intake is used and developed by individuals at a variety of institutions. It is open source (license) and sits within the broader Python numeric ecosystem commonly referred to as PyData or SciPy. it's giving we're all in this together 💖

💬 Discussion

Conversation happens in the following places:

Usage questions are directed to Stack Overflow with the #intake tag. Intake developers monitor this tag.
Bug reports and feature requests are managed on the GitHub issue tracker. Individual intake plugins are managed in separate repositories each with its own issue tracker. Please consult the Plugin Directory for a list of available plugins.
Chat occurs on at gitter.im/ContinuumIO/intake. Note that because gitter chat is not searchable by future users we discourage usage questions and bug reports on gitter and instead ask people to use Stack Overflow or GitHub.
Monthly community meeting happens the first Thursday of the month at 9:00 US Central Time. See https://github.com/intake/intake/issues/596, with a reminder sent out on the gitter channel. Strictly informal chatter. 🎉

🫶 Asking for help

We welcome usage questions and bug reports from all users, even those who are new to using the project. There are a few things you can do to improve the likelihood of quickly getting a good answer.

Ask questions in the right place: We strongly prefer the use of Stack Overflow or GitHub issues over Gitter chat. GitHub and Stack Overflow are more easily searchable by future users, and therefore is more efficient for everyone's time. Gitter chat is strictly reserved for developer and community discussion.

If you have a general question about how something should work or want best practices then use Stack Overflow. If you think you have found a bug then use GitHub

Ask only in one place: Please restrict yourself to posting your question in only one place (likely Stack Overflow or GitHub) and don't post in both
Create a minimal example: It is ideal to create minimal, complete, verifiable examples. This significantly reduces the time that answerers spend understanding your situation, resulting in higher quality answers more quickly. 🧚

Index
Module Index
Search Page

✨ AUTHOR ✨

Anaconda

💖 COPYRIGHT 💖

2022, Anaconda

and that's the whole page bestie, you made it to the end!! 🎀 this was INTAKE(1), version 0.6.6, dated Sep 22, 2022, and it was written by Anaconda with the copyright 2022, Anaconda — slay, go build something iconic with it 💅✨🦋

📦 where this page comes from

debian bookworm: python3-intake 0.6.6-1 · BSD-2-Clause
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-10-13: a language model's rewrite of the original, under its terms

INTAKE(1)

🦋 INTAKE(1) — the intake manual, bestie edition 🦋

🌸 NAME 🌸

💜 DATA USER 💜

🌈 DATA PROVIDER 🌈

🔥 IT 🔥

👑 DEVELOPER 👑

✨ START HERE ✨

🧚 Quickstart 🧚

🍓 Installation 🍓

🌸 Creating Sample Data 🌸

💫 Loading a Data Source 💫

🌈 Reading Data 🌈

💅 Working with Dask 💅

🎀 Opening a Catalog 🎀

💖 Installing Data Source Packages 💖

🦋 Adding Data Source Packages using the Intake path 🦋

✨ Using the GUI ✨

🌸 Use Cases - I want to... 🌸

✨ Work remotely ✨

🌸 Transform data to efficient formats for sharing 🌸

🔒 Access data without leaking credentials 🔒

🛡️ Establish a data gateway 🛡️

👩‍🔬 Clear distinction between data curator and analyst roles 👩‍💻

🧠 Users to be able to access data without learning every backend API 🧠

🦋 Data sources to be self-describing 🦋

🌳 A data source hierarchy for natural structuring 🌳

🌈 Expose several data collections under a single system 🌈

📊 Modern visualisations for all data-sets 📊

⚡ Update data specifications in real time ⚡

📦 Distribute data in a custom format 📦

🎀 Create Intake data-sets from scratch 🎀

✨ OVERVIEW ✨

💬 Introduction

🦋 Why Intake?

🧚 Structure

💅 Concepts

🌈 Future Directions

🎀 Examples

💜 General

🔥 Developer

⭐ Features

🍓 Data

🌸 Blogs

🦋 Talks

👑 News

🌟 Deployment Scenarios

💖 Sharing YAML files

🌍 Pangeo

🎀 Anaconda Enterprise

🧊 National Center for Atmospheric Research

🦋 Brookhaven Archive

Zillow

Intake Server

🌸 USER GUIDE 🌸

🎀 GUI 🎀

Using the GUI

Catalogs

Add Catalogs

Remove Catalogs

Search

Sources

Plot

Interactive Visualization

Using the Selection

Catalogs

The Catalog class

YAML Format

✨ Metadata ✨

✨ Extra drivers ✨

✨ Sources ✨

✨ Caching Source Files Locally ✨

✨ Templating ✨

✨ Parameter Definition ✨

🌸 Source parameters 🌸

🎀 Catalog parameters 🎀

✨ Templating parameters ✨

🌈 Driver Selection 🌈

🦋 Remote Access 🦋

💜 Using S3 Profiles 💜

🦋 INTAKE(1) — the `intake` manual, bestie edition 🦋

`class intake.source.derived.DerivedSource(*args, **kwargs)`

🧬 `GenericTransform` — the blueprint bestie

🦋 `DataFrameTransform` — glow-up edition

🎀 `Columns` — the little demo that could

🎀 `intake.open_catalog(uri=None, **kwargs)` 🎀

💜 `intake.registry` 💜

⭐ `intake.open_` ⭐

🫶 `intake.upload(data, path, **kwargs)` 🫶