SQLITE-UTILS(1)
💖 NAME 💖
sqlite-utils — sqlite-utils documentation
okay so hi bestie 💅 you've stumbled into the manual page for sqlite-utils and honestly? you're gonna be obsessed. it's a CLI tool and a Python library, and it's about to make your SQLite life so much easier. let's get into it 🫶
✨ CLI TOOL AND PYTHON LIBRARY FOR MANIPULATING SQLITE DATABASES ✨
This library and command-line utility helps create SQLite databases from an existing collection of data.
Most of the functionality is available as either a Python API or through the sqlite-utils command-line tool.
sqlite-utils is not intended to be a full ORM: the focus is utility helpers to make creating the initial database and populating it with data as productive as possible.
It is designed as a useful complement to Datasette <https://datasette.io/>.
Cleaning data with sqlite-utils and Datasette <https://datasette.io/tutorials/clean-data> provides a tutorial introduction (and accompanying ten minute video) about using this tool. like, actual homework but the fun kind 🎀
📚 Contents 📚
🧸 Installation 🧸
sqlite-utils is tested on Linux, macOS and Windows. all three, no cap 🌈
🍎 Using Homebrew 🍎
The sqlite-utils command-line tool <#cli> can be installed on macOS using Homebrew:
brew install sqlite-utilsIf you have it installed and want to upgrade to the most recent release, you can run:
brew upgrade sqlite-utilsThen run sqlite-utils --version to confirm the installed version.
🐍 Using pip 🐍
The sqlite-utils package <https://pypi.org/project/sqlite-utils/> on PyPI includes both the sqlite_utils Python library <#python-api> and the sqlite-utils command-line tool. You can install them using pip like so:
pip install sqlite-utils🎁 Using pipx 🎁
pipx <https://pypi.org/project/pipx/> is a tool for installing Python command-line applications in their own isolated environments. You can use pipx to install the sqlite-utils command-line tool like this:
pipx install sqlite-utils🧐 Alternatives to sqlite3 🧐
By default, sqlite-utils uses the sqlite3 package bundled with the Python standard library.
Depending on your operating system, this may come with some limitations. lowkey annoying but we cope 🙈
On some platforms the ability to load additional extensions (via conn.load_extension(...) or --load-extension=/path/to/extension) may be disabled.
You may also see the error sqlite3.OperationalError: table sqlite_master may not be modified when trying to alter an existing table.
You can work around these limitations by installing the pysqlite3 <https://pypi.org/project/pysqlite3/> package, which provides a drop-in replacement for the standard library sqlite3 module but with a recent version of SQLite and full support for loading extensions.
To install pysqlite3 run the following:
sqlite-utils install pysqlite3pysqlite3 does not provide an implementation of the .iterdump() method. To use that method (see Dumping the database to SQL <#python-api-itedump>) or the sqlite-utils dump command you should also install the sqlite-dump package:
sqlite-utils install sqlite-dump🐚 Setting up shell completion 🐚
You can configure shell tab completion for the sqlite-utils command using these commands.
For bash:
eval "$(_SQLITE_UTILS_COMPLETE=bash_source sqlite-utils)"For zsh:
eval "$(_SQLITE_UTILS_COMPLETE=zsh_source sqlite-utils)"Add this code to ~/.zshrc or ~/.bashrc to automatically run it when you start a new shell.
See the Click documentation <https://click.palletsprojects.com/en/8.1.x/shell-completion/> for more details.
👑 sqlite-utils command-line tool 👑
The sqlite-utils command-line tool can be used to manipulate SQLite databases in a number of different ways.
Once installed <#installation> the tool should be available as sqlite-utils. It can also be run using python -m sqlite_utils.
okay this next bit is the whole menu of what it can do, and it's a lot, so buckle up bestie 💀
- Running SQL queries
- Returning JSON
- Newline-delimited JSON
- JSON arrays
- Unicode characters in JSON
- Binary data in JSON
- Nested JSON values
- Returning CSV or TSV
- Table-formatted output
- Returning raw data, such as binary content
- Using named parameters
UPDATE,INSERTandDELETE- Defining custom SQL functions
- SQLite extensions
- Attaching additional databases
- Returning JSON
- Querying data directly using an in-memory database
- Running queries directly against CSV or JSON
- Explicitly specifying the format
- Joining in-memory data against existing databases using
--attach --schema,--analyze,--dumpand--save
- Returning all rows in a table
- Listing tables
- Listing views
- Listing indexes
- Listing triggers
- Showing the schema
- Analyzing tables
- Saving the analyzed table details
- Creating an empty database
- Running migrations
- Inserting JSON data
- Inserting binary data
- Inserting newline-delimited JSON
- Flattening nested JSON objects
- Inserting CSV or TSV data
- Overriding column types
- Alternative delimiters and quote characters
- CSV files without a header row
- Inserting unstructured data with
--linesand--text - Applying conversions while inserting data
--convertwith--lines--convertwith--text
- Inserting rows generated by Python code
- Insert-replacing data
- Upserting data
- Executing SQL in bulk
- Inserting data from files
- Converting data in columns
- Importing additional modules
- Using the debugger
- Defining a
convert()function sqlite-utils convertrecipes- Saving the result to a different column
- Converting a column into multiple columns
- Creating tables
- Renaming a table
- Duplicating tables
- Dropping tables
- Transforming tables
- Adding a primary key to a rowid table
- Extracting columns into a separate table
- Creating views
- Dropping views
- Adding columns
- Adding columns automatically on insert/update
- Adding foreign key constraints
- Adding multiple foreign keys at once
- Adding indexes for all foreign keys
- Setting defaults and not null constraints
- Creating indexes
- Dropping indexes
- Configuring full-text search
- Executing searches
- Enabling cached counts
- Optimizing index usage with
ANALYZE - Vacuum
- Optimize
- WAL mode
- Dumping the database to SQL
- Loading SQLite extensions
- SpatiaLite helpers
- Adding spatial indexes
- Installing packages
- Uninstalling packages
🔍 Running SQL queries 🔍
The sqlite-utils query command lets you run queries directly against a SQLite database file. This is the default subcommand, so the following two examples work the same way:
sqlite-utils query dogs.db "select * from dogs"
sqlite-utils dogs.db "select * from dogs"Note:
In Python:db.query()<#python-api-query> CLI reference:sqlite-utils query<#cli-ref-query>
Pass - as the SQL query to read the query from standard input. This is useful for longer queries that would otherwise require careful shell escaping, or for piping in SQL generated by another tool:
echo "select * from dogs" | sqlite-utils query dogs.db -
sqlite-utils query dogs.db - < query.sql🌸 Returning JSON 🌸
The default format returned for queries is JSON:
sqlite-utils dogs.db "select * from dogs"
[{"id": 1, "age": 4, "name": "Cleo"},
{"id": 2, "age": 2, "name": "Pancakes"}]If the query returns more than one column with the same name, later occurrences are renamed with a numeric suffix — select 1 as id, 2 as id returns [{"id": 1, "id_2": 2}]. This only applies to JSON output: CSV and TSV and table output keep the duplicate column headers unchanged. iykyk 🙈
🆕 Newline-delimited JSON 🆕
Use --nl to get back newline-delimited JSON objects:
sqlite-utils dogs.db "select * from dogs" --nl
{"id": 1, "age": 4, "name": "Cleo"}
{"id": 2, "age": 2, "name": "Pancakes"}📦 JSON arrays 📦
You can use --arrays to request arrays instead of objects:
sqlite-utils dogs.db "select * from dogs" --arrays
[[1, 4, "Cleo"],
[2, 2, "Pancakes"]]You can also combine --arrays and --nl:
sqlite-utils dogs.db "select * from dogs" --arrays --nl
[1, 4, "Cleo"]
[2, 2, "Pancakes"]If you want to pretty-print the output further, you can pipe it through python -mjson.tool:
sqlite-utils dogs.db "select * from dogs" | python -mjson.tool
[
{
"id": 1,
"age": 4,
"name": "Cleo"
},
{
"id": 2,
"age": 2,
"name": "Pancakes"
}
]🈶 Unicode characters in JSON 🈶
JSON output includes unicode characters directly, without escaping them:
sqlite-utils dogs.db "select '日本語' as text"
[{"text": "日本語"}]Use --ascii to escape non-ASCII characters as \uXXXX sequences instead:
sqlite-utils dogs.db "select '日本語' as text" --ascii
[{"text": "\u65e5\u672c\u8a9e"}]The --ascii option can help on systems that cannot display or process UTF-8, such as Windows consoles using a legacy code page. On Windows, setting the PYTHONUTF8=1 environment variable is an alternative fix for UnicodeEncodeError crashes when redirecting output to a file.
🔢 Binary data in JSON 🔢
Binary strings are not valid JSON, so BLOB columns containing binary data will be returned as a JSON object containing base64 encoded data, that looks like this:
sqlite-utils dogs.db "select name, content from images" | python -mjson.tool
[
{
"name": "transparent.gif",
"content": {
"$base64": true,
"encoded": "R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"
}
}
]🪆 Nested JSON values 🪆
If one of your columns contains JSON, by default it will be returned as an escaped string:
sqlite-utils dogs.db "select * from dogs" | python -mjson.tool
[
{
"id": 1,
"name": "Cleo",
"friends": "[{\"name\": \"Pancakes\"}, {\"name\": \"Bailey\"}]"
}
]You can use the --json-cols option to automatically detect these JSON columns and output them as nested JSON data:
sqlite-utils dogs.db "select * from dogs" --json-cols | python -mjson.tool
[
{
"id": 1,
"name": "Cleo",
"friends": [
{
"name": "Pancakes"
},
{
"name": "Bailey"
}
]
}
]🌸 Returning CSV or TSV 🌸
okay so bestie, if you want your query results to come back as CSV, you use the --csv option and it literally just does that:
sqlite-utils dogs.db "select * from dogs" --csv
id,age,name
1,4,Cleo
2,2,Pancakesby default it's giving header row energy — the column names show up first 💅. Wanna yeet those headers? Use --no-headers:
sqlite-utils dogs.db "select * from dogs" --csv --no-headers
1,4,Cleo
2,2,Pancakesand if you want tabs instead of commas, swap --csv for --tsv:
sqlite-utils dogs.db "select * from dogs" --tsv
id age name
1 4 Cleo
2 2 Pancakes✨ Table-formatted output ✨
you can use the --table option (or the -t shortcut, iconic) to output your query results as a table:
sqlite-utils dogs.db "select * from dogs" --table
id age name
---- ----- --------
1 4 Cleo
2 2 Pancakesand if you're not feeling that table format, the --fmt option lets you pick a different one — like rst for reStructuredText:
sqlite-utils dogs.db "select * from dogs" --fmt rst
==== ===== ========
id age name
==== ===== ========
1 4 Cleo
2 2 Pancakes
==== ===== ========the --fmt options you can pick from are, in order:
asciidoccolon_griddouble_griddouble_outlinefancy_gridfancy_outlinegithubgridheavy_gridheavy_outlinehtmljiralatexlatex_booktabslatex_longtablelatex_rawmediawikimixed_gridmixed_outlinemoinmoinorgtbloutlinepipeplainprestoprettypsqlrounded_gridrounded_outlinerstsimplesimple_gridsimple_outlinetextiletsvunsafehtmlyoutrack
you can also pull up this whole list yourself by running sqlite-utils query --help 🫶
💖 Returning raw data, such as binary content 💖
if your table is holding binary data in a BLOB, the --raw option lets you send specific columns straight to standard out. no extra layers, just vibes.
so for example, to grab a binary image out of a BLOB column and stash it in a file, you'd do:
sqlite-utils photos.db "select contents from photos where id=1" --raw > myphoto.jpgand if you want the first column of every result as raw data, separated by newlines, that's --raw-lines:
sqlite-utils photos.db "select caption from photos" --raw-lines > captions.txt🎀 Using named parameters 🎀
you can pass named parameters into the query with -p name value, like:
sqlite-utils query dogs.db "select :num * :num2" -p num 5 -p num2 6
[{":num * :num2": 30}]these get correctly quoted and escaped in the SQL query, which makes this a safe way to combine other values with your SQL. no cap 🔥
👑 UPDATE, INSERT and DELETE 👑
if you run an UPDATE, INSERT or DELETE query, the command hands back the number of rows that got affected:
sqlite-utils dogs.db "update dogs set age = 5 where name = 'Cleo'"
[{"rows_affected": 1}]🌈 Defining custom SQL functions 🌈
the --functions option lets you pass in a block of Python code that defines extra functions your SQL query can then call. so slay.
this example defines a function that pulls the domain out of a URL:
sqlite-utils query sites.db "select url, domain(url) from urls" --functions '
from urllib.parse import urlparse
def domain(url):
return urlparse(url).netloc
'every callable object defined in that block gets registered as a SQL function with the same name — the only exception being functions whose names begin with an underscore (those ones get left out).
you can also just hand it the path to a Python file with your function definitions in it:
sqlite-utils query sites.db "select url, domain(url) from urls" --functions functions.pyand --functions can be used multiple times, so you can load functions from heaps of different sources:
sqlite-utils query sites.db "select url, domain(url), extract_path(url) from urls" \
--functions domain_funcs.py \
--functions 'def extract_path(url):
from urllib.parse import urlparse
return urlparse(url).path'Note: In Python: db.register_function() <#python-api-register-function>
💅 SQLite extensions 💅
you can load SQLite extension modules using the --load-extension option — see Loading SQLite extensions for the full story.
sqlite-utils dogs.db "select spatialite_version()" --load-extension=spatialite
[{"spatialite_version()": "4.3.0a"}]🦋 Attaching additional databases 🦋
SQLite supports cross-database SQL queries, which means you can join data from tables living in more than one database file. teamwork, honestly.
you attach one or more extra databases with the --attach option — you give it an alias to use for that database plus the path to the SQLite file on disk.
this example attaches the books.db database under the alias books, then runs a query that smushes data from that database together with the default dogs.db database:
sqlite-utils dogs.db --attach books books.db \
'select * from sqlite_master union all select * from books.sqlite_master'Note: In Python: db.attach() <#python-api-attach>
🍓 Querying data directly using an in-memory database 🍓
the sqlite-utils memory command works a lot like sqlite-utils query, except it lets you run queries against an in-memory database.
and bestie, you can also pass this command CSV or JSON files, which get loaded into a temporary in-memory table — so you can run SQL against that data without a whole separate step where you convert it to SQLite first. main character efficiency.
with no extra arguments, this command runs SQL against the in-memory database directly:
sqlite-utils memory 'select sqlite_version()'
[{"sqlite_version()": "3.35.5"}]it takes all the same output formatting options as sqlite-utils query: --csv and --csv and --table and --nl:
sqlite-utils memory 'select sqlite_version()' --csv
sqlite_version()
3.35.5sqlite-utils memory 'select sqlite_version()' --fmt grid
+--------------------+
| sqlite_version() |
+====================+
| 3.35.5 |
+--------------------+✨ Running queries directly against CSV or JSON ✨
if your data lives in CSV or JSON format, you can load it into an in-memory SQLite database and query it directly, all in one single command, using sqlite-utils memory like this:
sqlite-utils memory data.csv "select * from data"you can pass multiple files to the command if you want to run joins across data from different files:
sqlite-utils memory one.csv two.json \
"select * from one join two on one.id = two.other_id"if your data is JSON, it should be in the same format that the sqlite-utils insert command supports — so either a single JSON object (treated as a single row) or a list of JSON objects.
CSV data can be comma- or tab- delimited.
the in-memory tables get named after the files, minus their extensions. the tool also sets up aliases for those tables (using SQL views) as t1, t2 and so on — or you can use the alias t to refer to the first table:
sqlite-utils memory example.csv "select * from t"and if two files have the same name, they get assigned a numeric suffix:
sqlite-utils memory foo/data.csv bar/data.csv "select * from data_2"to read from standard input, use either - or stdin as the filename — then use stdin or t or t1 as the table name:
cat example.csv | sqlite-utils memory - "select * from stdin"incoming CSV data is assumed to use utf-8. if your data uses a different character encoding you can specify it with --encoding:
cat example.csv | sqlite-utils memory - "select * from stdin" --encoding=latin-1if you're joining across multiple CSV files, they all have to use the same encoding.
column types get automatically detected in CSV or TSV data, as described in Inserting CSV or TSV data. you can pass the --no-detect-types option to turn that automatic type detection off and instead treat all CSV and TSV columns as TEXT.
💜 Explicitly specifying the format 💜
by default, sqlite-utils memory will try to detect the incoming data format (JSON, TSV or CSV) all on its own.
but you can skip the guessing and specify an explicit format by sticking a :csv, :tsv, :json or :nl (for newline-delimited JSON) suffix onto the filename. for example:
sqlite-utils memory one.dat:csv two.dat:nl \
"select * from one union select * from two"here the contents of one.dat get treated as CSV and the contents of two.dat get treated as newline-delimited JSON.
to explicitly specify the format for data piped into the tool on standard input, use stdin:format — like:
cat one.dat | sqlite-utils memory stdin:csv "select * from stdin"🧚 Joining in-memory data against existing databases using --attach 🧚
the attach option can be used to attach database files to the in-memory connection, which unlocks joins between in-memory data loaded from a file and tables in existing SQLite database files. example time:
echo "id\n1\n3\n5" | sqlite-utils memory - --attach trees trees.db \
"select * from trees.trees where rowid in (select id from stdin)"here the --attach trees trees.db option makes the trees.db database available under the alias trees.
select * from trees.trees where ... can then query the trees table inside that database.
the CSV data that got piped into the script is sitting in the stdin table, so ... where rowid in (select id from stdin) can be used to return rows from the trees table that match the IDs that were piped in as CSV content. obsessed 🌸
⭐ --schema, --analyze, --dump and --save ⭐
to see the in-memory database schema that would be used for a file, or for multiple files, use --schema:
sqlite-utils memory dogs.csv --schema
CREATE TABLE "dogs" (
"id" INTEGER,
"age" INTEGER,
"name" TEXT
);
CREATE VIEW "t1" AS select * from "dogs";
CREATE VIEW "t" AS select * from "dogs";you can run the equivalent of the analyze-tables command using --analyze:
sqlite-utils memory dogs.csv --analyze
dogs.id: (1/3)
Total rows: 2
Null rows: 0
Blank rows: 0
Distinct values: 2
dogs.name: (2/3)
Total rows: 2
Null rows: 0
Blank rows: 0
Distinct values: 2
dogs.age: (3/3)
Total rows: 2
Null rows: 0
Blank rows: 0
Distinct values: 2and you can output SQL that will both create the tables and insert the full data used to populate the in-memory database, using --dump:
sqlite-utils memory dogs.csv --dumpBEGIN TRANSACTION;
CREATE TABLE "dogs" (
"id" INTEGER,
"age" INTEGER,
"name" TEXT
);
INSERT INTO "dogs" VALUES('1','4','Cleo');
INSERT INTO "dogs" VALUES('2','2','Pancakes');
CREATE VIEW "t1" AS select * from "dogs";
CREATE VIEW "t" AS select * from "dogs";
COMMIT;Passing --save other.db will instead use that SQL to populate a new database file:
sqlite-utils memory dogs.csv --save dogs.dbThese features are mainly intended as debugging tools — for much more finely grained control over how data is inserted into a SQLite database file see Inserting JSON data and Inserting CSV or TSV data. 🫶
✨ Returning all rows in a table ✨
You can return every single row in a specified table using the rows command:
sqlite-utils rows dogs.db dogs
[{"id": 1, "age": 4, "name": "Cleo"},
{"id": 2, "age": 2, "name": "Pancakes"}]This command accepts the same output options as query — so you can pass --nl, --csv, --tsv, --no-headers, --table and --fmt. No cap. 💅
You can use the -c option to specify a subset of columns to return:
sqlite-utils rows dogs.db dogs -c age -c name
[{"age": 4, "name": "Cleo"},
{"age": 2, "name": "Pancakes"}]You can filter rows using a where clause with the --where option:
sqlite-utils rows dogs.db dogs -c name --where 'name = "Cleo"'
[{"name": "Cleo"}]Or pass named parameters using --where in combination with -p:
sqlite-utils rows dogs.db dogs -c name --where 'name = :name' -p name Cleo
[{"name": "Cleo"}]You can define a sort order using --order column or --order 'column desc'. 📊
Use --limit N to only return the first N rows. Use --offset N to return rows starting from the specified offset.
Note 💖 In Python:table.rows <#python-api-rows>— CLI reference:sqlite-utils rows <#cli-ref-rows>
✨ Listing tables ✨
You can list the names of tables in a database using the tables command:
sqlite-utils tables mydb.db
[{"table": "dogs"},
{"table": "cats"},
{"table": "chickens"}]You can output this list in CSV using the --csv or --tsv options:
sqlite-utils tables mydb.db --csv --no-headers
dogs
cats
chickensIf you just want to see the FTS4 tables, you can use --fts4 (or --fts5 for FTS5 tables):
sqlite-utils tables docs.db --fts4
[{"table": "docs_fts"}]Use --counts to include a count of the number of rows in each table:
sqlite-utils tables mydb.db --counts
[{"table": "dogs", "count": 12},
{"table": "cats", "count": 332},
{"table": "chickens", "count": 9}]Use --columns to include a list of columns in each table:
sqlite-utils tables dogs.db --counts --columns
[{"table": "Gosh", "count": 0, "columns": ["c1", "c2", "c3"]},
{"table": "Gosh2", "count": 0, "columns": ["c1", "c2", "c3"]},
{"table": "dogs", "count": 2, "columns": ["id", "age", "name"]}]Use --schema to include the schema of each table:
sqlite-utils tables dogs.db --schema --table
table schema
------- -----------------------------------------------
Gosh CREATE TABLE Gosh (c1 text, c2 text, c3 text)
Gosh2 CREATE TABLE Gosh2 (c1 text, c2 text, c3 text)
dogs CREATE TABLE "dogs" (
"id" INTEGER,
"age" INTEGER,
"name" TEXT)The --nl, --csv, --tsv, --table and --fmt options are also available. ⭐
Note 🎀 In Python:db.tables or db.table_names() <#python-api-tables>— CLI reference:sqlite-utils tables <#cli-ref-tables>
✨ Listing views ✨
The views command shows any views defined in the database:
sqlite-utils views sf-trees.db --table --counts --columns --schema
view count columns schema
--------- ------- -------------------- --------------------------------------------------------------
demo_view 189144 ['qSpecies'] CREATE VIEW demo_view AS select qSpecies from Street_Tree_List
hello 1 ['sqlite_version()'] CREATE VIEW hello as select sqlite_version()It takes the same options as the tables command:
--columns--schema--counts--nl--csv--tsv--table
Note 🦋 In Python:db.views or db.view_names() <#python-api-views>— CLI reference:sqlite-utils views <#cli-ref-views>
✨ Listing indexes ✨
The indexes command lists any indexes configured for the database:
sqlite-utils indexes covid.db --table
table index_name seqno cid name desc coll key
-------------------------------- ------------------------------------------------------ ------- ----- ----------------- ------ ------ -----
johns_hopkins_csse_daily_reports idx_johns_hopkins_csse_daily_reports_combined_key 0 12 combined_key 0 BINARY 1
johns_hopkins_csse_daily_reports idx_johns_hopkins_csse_daily_reports_country_or_region 0 1 country_or_region 0 BINARY 1
johns_hopkins_csse_daily_reports idx_johns_hopkins_csse_daily_reports_province_or_state 0 2 province_or_state 0 BINARY 1
johns_hopkins_csse_daily_reports idx_johns_hopkins_csse_daily_reports_day 0 0 day 0 BINARY 1
ny_times_us_counties idx_ny_times_us_counties_date 0 0 date 1 BINARY 1
ny_times_us_counties idx_ny_times_us_counties_fips 0 3 fips 0 BINARY 1
ny_times_us_counties idx_ny_times_us_counties_county 0 1 county 0 BINARY 1
ny_times_us_counties idx_ny_times_us_counties_state 0 2 state 0 BINARY 1It shows indexes across all tables. To see indexes for specific tables, list those after the database:
sqlite-utils indexes covid.db johns_hopkins_csse_daily_reports --tableThe command defaults to only showing the columns that are explicitly part of the index. To also include auxiliary columns use the --aux option — these columns will be listed with a key of 0. 🧚
The command takes the same format options as the tables and views commands.
Note 💜 In Python:table.indexes <#python-api-introspection-indexes>— CLI reference:sqlite-utils indexes <#cli-ref-indexes>
✨ Listing triggers ✨
The triggers command shows any triggers configured for the database:
sqlite-utils triggers global-power-plants.db --table
name table sql
--------------- --------- -----------------------------------------------------------------
plants_insert plants CREATE TRIGGER "plants_insert" AFTER INSERT ON "plants"
BEGIN
INSERT OR REPLACE INTO "_counts"
VALUES (
'plants',
COALESCE(
(SELECT count FROM "_counts" WHERE "table" = 'plants'),
0
) + 1
);
ENDIt defaults to showing triggers for all tables. To see triggers for one or more specific tables pass their names as arguments:
sqlite-utils triggers global-power-plants.db plantsThe command takes the same format options as the tables and views commands.
Note 🌸 In Python:table.triggers or db.triggers <#python-api-introspection-triggers>— CLI reference:sqlite-utils triggers <#cli-ref-triggers>
✨ Showing the schema ✨
The sqlite-utils schema command shows the full SQL schema for the database:
sqlite-utils schema dogs.db
CREATE TABLE "dogs" (
"id" INTEGER PRIMARY KEY,
"name" TEXT
);This will show the schema for every table and index in the database. To view the schema just for a specified subset of tables pass those as additional arguments:
sqlite-utils schema dogs.db dogs chickensNote 🎀 In Python:table.schema <#python-api-introspection-schema>ordb.schema <#python-api-schema>— CLI reference:sqlite-utils schema <#cli-ref-schema>
✨ Analyzing tables ✨
When working with a new database it can be useful to get an idea of the shape of the data. The sqlite-utils analyze-tables command inspects specified tables (or all tables) and calculates some useful details about each of the columns in those tables.
To inspect the tags table in the github.db database, run the following:
sqlite-utils analyze-tables github.db tags
tags.repo: (1/3)
Total rows: 261
Null rows: 0
Blank rows: 0
Distinct values: 14
Most common:
88: 107914493
75: 140912432
27: 206156866
Least common:
1: 209590345
2: 206649770
2: 303218369
tags.name: (2/3)
Total rows: 261
Null rows: 0
Blank rows: 0
Distinct values: 175
Most common:
10: 0.2
9: 0.1
7: 0.3
Least common:
1: 0.1.1
1: 0.11.1
1: 0.1a2
tags.sha: (3/3)
Total rows: 261
Null rows: 0
Blank rows: 0
Distinct values: 261For each column this tool displays the number of null rows, the number of blank rows (rows that contain an empty string), the number of distinct values and, for columns that are not entirely distinct, the most common and least common values. 👑
If you do not specify any tables every table in the database will be analyzed:
sqlite-utils analyze-tables github.dbIf you wish to analyze one or more specific columns, use the -c option:
sqlite-utils analyze-tables github.db tags -c shaTo show more than 10 common values, use --common-limit 20. To skip the most common or least common value analysis, use --no-most or --no-least:
sqlite-utils analyze-tables github.db tags --common-limit 20 --no-leastNote 🔥 In Python:table.analyze_column() <#python-api-analyze-column>— CLI reference:sqlite-utils analyze-tables <#cli-ref-analyze-tables>
✨ Saving the analyzed table details ✨
analyze-tables can take quite a while to run for large database files. You can save the results of the analysis to a database table called _analyze_tables_ using the --save option:
sqlite-utils analyze-tables github.db --saveThe _analyze_tables_ table has the following schema:
CREATE TABLE "_analyze_tables_" (
"table" TEXT,
"column" TEXT,
"total_rows" INTEGER,
"num_null" INTEGER,
"num_blank" INTEGER,
"num_distinct" INTEGER,
"most_common" TEXT,
"least_common" TEXT,
PRIMARY KEY ("table", "column")
);The most_common and least_common columns are gonna hold nested JSON arrays of the most common and least common values, and they look like this:
[
["Del Libertador, Av", 5068],
["Alberdi Juan Bautista Av.", 4612],
["Directorio Av.", 4552],
["Rivadavia, Av", 4532],
["Yerbal", 4512],
["Cosquín", 4472],
["Estado Plurinacional de Bolivia", 4440],
["Gordillo Timoteo", 4424],
["Montiel", 4360],
["Condarco", 4288]
]🌱 Creating an empty database 🌱
Okay so you can spin up a brand new empty database file with the create-database command, it's literally one line:
sqlite-utils create-database empty.dbWanna turn on WAL mode on your freshly made db? Add the --enable-wal option, bestie:
sqlite-utils create-database empty.db --enable-walAnd if you want SpatiaLite metadata on a newly created database, add the --init-spatialite flag:
sqlite-utils create-database empty.db --init-spatialiteThat's gonna go hunting for SpatiaLite in a bunch of predictable locations. Want it from somewhere else instead? Use the --load-extension option:
sqlite-utils create-database empty.db --init-spatialite --load-extension /path/to/spatialite.so🧚 Running migrations 🧚
The migrate command applies pending Python migrations to a database. For the full migration file format and the Python API, go peek at Database migrations #migrations.
sqlite-utils migrate creatures.db path/to/migrations.pyIf you just... don't give it a migration path? It'll go search the current directory and all its subdirectories for files called migrations.py:
sqlite-utils migrate creatures.dbUse --list to list applied and pending migrations without actually running them:
sqlite-utils migrate creatures.db --listUse --stop-before to stop before a named migration. You can pass this option more than once, and you can target a specific migration set using migration_set:migration_name:
sqlite-utils migrate creatures.db path/to/migrations.py \
--stop-before creatures:add_weight \
--stop-before sales:drop_index🍓 Inserting JSON data 🍓
Got your data as JSON? Then sqlite-utils insert tablename is your girl. She'll insert it into a database and create the table with the correct columns (automatically detected, no cap) if it doesn't already exist.
You can hand her a single JSON object or a whole list of JSON objects — either as a filename or piped straight into standard-in by using - as the filename.
Here's the simplest possible example, look how cute:
echo '{"name": "Cleo", "age": 4}' | sqlite-utils insert dogs.db dogs -Wanna pick a column as the primary key? Use --pk=column_name.
Wanna make a compound primary key across more than one column? Just use --pk multiple times. Periodt.
And if you feed her a JSON list she'll insert multiple records. Say dogs.json looks like this:
[
{
"id": 1,
"name": "Cleo",
"age": 4
},
{
"id": 2,
"name": "Pancakes",
"age": 2
},
{
"id": 3,
"name": "Toby",
"age": 6
}
]Then you can import all three records into an automatically created dogs table and set the id column as the primary key like so:
sqlite-utils insert dogs.db dogs dogs.json --pk=idPass --pk multiple times to define a compound primary key.
You can skip inserting any records that have a primary key that already exists using --ignore:
sqlite-utils insert dogs.db dogs dogs.json --pk=id --ignoreYou can delete all the existing rows in the table before inserting the new records using --truncate:
sqlite-utils insert dogs.db dogs dogs.json --truncateAnd you can add the --analyze option to run ANALYZE against the table after the rows have been inserted.
Note: In Python:table.insert_all()#python-api-bulk-inserts— CLI reference:sqlite-utils insert#cli-ref-insert
💅 Inserting binary data 💅
You can insert binary data into a BLOB column by first encoding it using base64 and then structuring it like this:
[
{
"name": "transparent.gif",
"content": {
"$base64": true,
"encoded": "R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"
}
}
]🦋 Inserting newline-delimited JSON 🦋
You can also import newline-delimited JSON (see JSON Lines https://jsonlines.org/) using the --nl option:
echo '{"id": 1, "name": "Cleo"}
{"id": 2, "name": "Suna"}' | sqlite-utils insert creatures.db creatures - --nlNewline-delimited JSON is exactly what it sounds like — full JSON objects separated by newlines. Iconic, honestly.
If you're processing data using jq you can use the jq -c option to output valid newline-delimited JSON.
Since Datasette https://datasette.io/ can export newline-delimited JSON, you can combine Datasette and sqlite-utils like so:
curl -L "https://latest.datasette.io/fixtures/facetable.json?_shape=array&_nl=on" \
| sqlite-utils insert nl-demo.db facetable - --pk=id --nlYou can also pipe sqlite-utils together to create a new SQLite database file containing the results of a SQL query against another database:
sqlite-utils sf-trees.db \
"select TreeID, qAddress, Latitude, Longitude from Street_Tree_List" --nl \
| sqlite-utils insert saved.db trees - --nl
sqlite-utils saved.db "select * from trees limit 5" --csv
TreeID,qAddress,Latitude,Longitude
141565,501X Baker St,37.7759676911831,-122.441396661871
232565,940 Elizabeth St,37.7517102172731,-122.441498017841
119263,495X Lakeshore Dr,,
207368,920 Kirkham St,37.760210314285,-122.47073935813
188702,1501 Evans Ave,37.7422086702947,-122.387293152263✨ Flattening nested JSON objects ✨
sqlite-utils insert and sqlite-utils memory both expect incoming JSON data to consist of an array of JSON objects, where the top-level keys of each object become columns in the created database table.
If your data is nested you can use the --flatten option to create columns that are derived from the nested data.
Consider this example document, in a file called log.json:
{
"httpRequest": {
"latency": "0.112114537s",
"requestMethod": "GET",
"requestSize": "534",
"status": 200
},
"insertId": "6111722f000b5b4c4d4071e2",
"labels": {
"service": "datasette-io"
}
}Inserting this into a table using sqlite-utils insert logs.db logs log.json will create a table with the following schema:
CREATE TABLE "logs" (
"httpRequest" TEXT,
"insertId" TEXT,
"labels" TEXT
);With the --flatten option, columns will be created using topkey_nextkey column names — so running sqlite-utils insert logs.db logs log.json --flatten will create the following schema instead:
CREATE TABLE "logs" (
"httpRequest_latency" TEXT,
"httpRequest_requestMethod" TEXT,
"httpRequest_requestSize" TEXT,
"httpRequest_status" INTEGER,
"insertId" TEXT,
"labels_service" TEXT
);🎀 Inserting CSV or TSV data 🎀
Is your data in CSV format? Then insert it using the --csv option, easy:
sqlite-utils insert dogs.db dogs dogs.csv --csvFor tab-delimited data, use --tsv:
sqlite-utils insert dogs.db dogs dogs.tsv --tsvData is expected to be encoded as Unicode UTF-8. If your data is in another character encoding you can specify it using the --encoding option:
sqlite-utils insert dogs.db dogs dogs.tsv --tsv --encoding=latin-1Wanna stop inserting after a specified number of records? Super useful for getting a faster preview of a large file — just use the --stop-after option:
sqlite-utils insert dogs.db dogs dogs.csv --csv --stop-after=10A progress bar shows up when you're inserting data from a file. You can hide the progress bar using the --silent option.
By default, column types are automatically detected for CSV or TSV files — which gives you a mix of TEXT, INTEGER and REAL columns. To disable type detection and treat all columns as TEXT, use the --no-detect-types option.
Detected types are only applied when the table is created by the command, okay? Inserting CSV or TSV data into a table that already exists leaves the existing column types completely unchanged — values get inserted using the table's existing schema.
For example, given a creatures.csv file containing this:
name,age,weight
Cleo,6,45.5
Dori,1,3.5The following command:
sqlite-utils insert creatures.db creatures creatures.csv --csvWill produce this schema, with automatically detected types:
sqlite-utils schema creatures.db
CREATE TABLE "creatures" (
"name" TEXT,
"age" INTEGER,
"weight" REAL
);👑 Overriding column types 👑
Use --type column-name type to override the type automatically chosen when the table is created. You can use this option more than once, and it works with both insert and upsert:
sqlite-utils insert places.db places places.csv --csv \
--type zipcode text \
--type score realThis is lowkey so useful for values like ZIP codes, which may look like integers but should be stored as TEXT to preserve leading zeros.
The column type should be one of TEXT, INTEGER, FLOAT, REAL, BLOB or ANY. Column types are matched case-insensitively.
ANY is especially useful with --strict. An ANY column in a strict table preserves values without coercion, so text such as 000123 stays text instead of getting converted to an integer:
sqlite-utils insert events.db events events.csv --csv --strict \
--type payload anyAs with detected column types, --type only affects tables created by the command. If the table already exists, its existing column types are left unchanged.
To disable type detection and treat all columns as TEXT, use --no-detect-types:
sqlite-utils insert creatures.db creatures creatures.csv --csv --no-detect-typesIf a CSV or TSV file includes empty cells, like this one:
name,age,weight
Cleo,6,
Dori,,3.5They will be imported into SQLite as empty string values, "".
To import them as NULL values instead, use the --empty-null option:
sqlite-utils insert creatures.db creatures creatures.csv --csv --empty-nullAlternative delimiters and quote characters 🎀
Okay so bestie, what if your file is ✨not✨ using a , as its delimiter, or it's using a quote character that isn't "? You have two options: let the tool try to figure it out, or just tell it directly. Both valid, both iconic.
The --sniff option makes it attempt to detect the delimiters for you:
sqlite-utils insert dogs.db dogs dogs.csv --sniffOrrr you can just be a decisive queen and specify them yourself with the --delimiter and --quotechar options.
Here's a CSV file that uses ; for delimiters and the | symbol for quote characters:
name;description
Cleo;|Very fine; a friendly dog|
Pancakes;A local corgiYou can import that using:
sqlite-utils insert dogs.db dogs dogs.csv --delimiter=";" --quotechar="|"Heads up bestie: passing--delimiter,--quotecharor--sniffimplies--csv, so you can totally omit the--csvoption. Sneaky convenience, we love. 💅
CSV files without a header row 🙈
The first row of any CSV or TSV file is expected to contain the names of the columns in that file. That's the vibe. That's the expectation.
But if your file does NOT include that row, no worries — use the --no-headers option to specify that the tool should not use that first row as headers.
If you do this, the table will be created with column names called untitled_1 and untitled_2 and so on. Very demure, very placeholder-core. You can then rename them using the sqlite-utils transform ... --rename command — see Transforming tables. ✨
Inserting unstructured data with --lines and --text 🦋
Got an unstructured file and just want to yeet its contents into a table? Say less. Use --lines to insert the contents into a table with a single line column containing each line from the file. This is so useful if you intend to further analyze those lines later using SQL string functions or sqlite-utils convert:
sqlite-utils insert logs.db loglines logfile.log --linesThis will produce the following schema:
CREATE TABLE "loglines" (
"line" TEXT
);You can also insert the entire contents of the file into a single column called text using --text:
sqlite-utils insert content.db content file.txt --textThe schema here will be:
CREATE TABLE "content" (
"text" TEXT
);It's giving "one column, endless possibilities." 💖
Applying conversions while inserting data 🌸
The --convert option can be used to apply a Python conversion function to imported data before it is inserted into the database. It works in a similar way to sqlite-utils convert.
Your Python function will be passed a dictionary called row for each item that is being imported. You can modify that dictionary and return it — or return a fresh dictionary — to change the data that will be inserted.
Given a JSON file called dogs.json containing this:
[
{"id": 1, "name": "Cleo"},
{"id": 2, "name": "Pancakes"}
]The following command will insert that data and add an is_good column set to 1 for each dog:
sqlite-utils insert dogs.db dogs dogs.json --convert 'row["is_good"] = 1'The --convert option also works with the --csv, --tsv and --nl insert options.
As with sqlite-utils convert you can use --import to import additional Python modules, see Importing additional modules for details.
You can also pass code that runs some initialization steps and defines a convert(value) function, see Defining a convert() function.
--convert with --lines 🌈
Things work slightly differently when combined with the --lines or --text options.
With --lines, instead of being passed a row dictionary your function will be passed a line string representing each line of the input. Given a file called access.log containing the following:
INFO: 127.0.0.1:60581 - GET / HTTP/1.1 200 OK
INFO: 127.0.0.1:60581 - GET /foo/-/static/app.css?cead5a HTTP/1.1 200 OKYou could convert it into structured data like so:
sqlite-utils insert logs.db loglines access.log --convert '
type, source, _, verb, path, _, status, _ = line.split()
return {
"type": type,
"source": source,
"verb": verb,
"path": path,
"status": status,
}' --linesThe resulting table would look like this:
┌───────┬─────────────────┬──────┬──────────────────────────────┬────────┐
│ type │ source │ verb │ path │ status │
├───────┼─────────────────┼──────┼──────────────────────────────┼────────┤
│ INFO: │ 127.0.0.1:60581 │ GET │ / │ 200 │
├───────┼─────────────────┼──────┼──────────────────────────────┼────────┤
│ INFO: │ 127.0.0.1:60581 │ GET │ /foo/-/static/app.css?cead5a │ 200 │
└───────┴─────────────────┴──────┴──────────────────────────────┴────────┘Structured data out of chaos? We ate. 👑
--convert with --text 💜
With --text the entire input to the command will be made available to the function as a variable called text.
The function can return a single dictionary which will be inserted as a single row, or it can return a list or iterator of dictionaries, each of which will be inserted. Queen behavior, truly.
Here's how to use --convert and --text to insert one record per word in the input:
echo 'A bunch of words' | sqlite-utils insert words.db words - \
--text --convert '({"word": w} for w in text.split())'The result looks like this:
sqlite-utils dump words.db
BEGIN TRANSACTION;
CREATE TABLE "words" (
"word" TEXT
);
INSERT INTO "words" VALUES('A');
INSERT INTO "words" VALUES('bunch');
INSERT INTO "words" VALUES('of');
INSERT INTO "words" VALUES('words');
COMMIT;Inserting rows generated by Python code 🧚
Instead of providing a FILE to import, you can use the --code option to pass a block of Python code that generates the rows to insert. This is the command-line equivalent of calling db["creatures"].insert_all(rows()) from the Python API <#python-api>.
Your code should define either a rows() function that returns or yields dictionaries, or a rows iterable such as a list of dictionaries:
sqlite-utils insert data.db creatures --code '
def rows():
yield {"id": 1, "name": "Cleo"}
yield {"id": 2, "name": "Suna"}
' --pk id--code can also be given a path to a Python .py file.
The --code option works with both sqlite-utils insert and sqlite-utils upsert, and composes with table options such as --pk, --replace, --alter, --not-null and --default. It cannot be combined with a FILE argument or with input format options such as --csv or --convert.
Insert-replacing data 🔥
The --replace option to insert causes any existing records with the same primary key to be replaced entirely by the new records. Like, gone-gone. Full makeover.
To replace a dog with in ID of 2 with a new record, run the following:
echo '{"id": 2, "name": "Pancakes", "age": 3}' | \
sqlite-utils insert dogs.db dogs - --pk=id --replaceNote: In Python:table.insert(..., replace=True)<#python-api-insert-replace> CLI reference:sqlite-utils insert<#cli-ref-insert>
Upserting data ⭐
Upserting is update-or-insert. If a row exists with the specified primary key the provided columns will be updated. If no row exists that row will be created. Best of both worlds, no cap. 💅
Unlike insert --replace, an upsert will ignore any column values that exist but are not present in the upsert document.
For example:
echo '{"id": 2, "age": 4}' | \
sqlite-utils upsert dogs.db dogs - --pk=idThis will update the dog with an ID of 2 to have an age of 4, creating a new record (with a null name) if one does not exist. If a row DOES exist the name will be left as-is.
If the table already exists and has a primary key, you can omit the --pk option and sqlite-utils will use that existing primary key.
The command will fail if you reference columns that do not exist on the table. To automatically create missing columns, use the --alter option.
Note:upsertinsqlite-utils1.x worked likeinsert ... --replacedoes in 2.x. See issue #66 <https://github.com/simonw/sqlite-utils/issues/66> for details of this change.
Note: In Python:table.upsert()<#python-api-upsert> CLI reference:sqlite-utils upsert<#cli-ref-upsert>
Executing SQL in bulk 🌊
If you have a JSON, newline-delimited JSON, CSV or TSV file you can execute a bulk SQL query using each of the records in that file using the sqlite-utils bulk command.
The command takes the database file, the SQL to be executed and the file containing records to be used when evaluating the SQL query.
The SQL query should include :named parameters that match the keys in the records. iykyk. 🫶
For example, given a chickens.csv CSV file containing the following:
id,name
1,Blue
2,Snowy
3,Azi
4,Lila
5,Suna
6,CardiYou could insert those rows into a pre-created chickens table like so:
sqlite-utils bulk chickens.db \
'insert into chickens (id, name) values (:id, :name)' \
chickens.csv --csvThis command takes the same options as the sqlite-utils insert command — so it defaults to expecting JSON but can accept other formats using --csv or --tsv or --nl or other options described above.
By default all of the SQL queries will be executed in a single transaction. To commit every 20 records, use --batch-size 20.
Inserting data from files 🍓
The insert-files command can be used to insert the content of files, along with their metadata, into a SQLite table.
Here's an example that inserts all of the GIF files in the current directory into a gifs.db database, placing the file contents in an images table:
sqlite-utils insert-files gifs.db images *.gifYou can also pass one or more directories, in which case every file in those directories will be added recursively:
sqlite-utils insert-files gifs.db images path/to/my-gifsBy default this command will create a table with the following schema:
CREATE TABLE "images" (
"path" TEXT PRIMARY KEY,
"content" BLOB,
"size" INTEGER
);Content will be treated as binary by default and stored in a BLOB column. You can use the --text option to store that content in a TEXT column instead.
You can customize the schema using one or more -c options. For a table schema that includes just the path, MD5 hash and last modification time of the file, you would use this:
sqlite-utils insert-files gifs.db images *.gif -c path -c md5 -c mtime --pk=pathThat gives you this schema, bestie:
CREATE TABLE "images" (
"path" TEXT PRIMARY KEY,
"md5" TEXT,
"mtime" REAL
);Real talk: there is no content column in there at all! Like, it's just not invited to the party. So if you go custom with -c, and you actually want that column, you have to explicitly include -c content. Otherwise she's simply not showing up. 🙈
You can also rename one of these columns with a -c colname:coldef parameter. Wanna rename mtime to last_modified? Bet:
sqlite-utils insert-files gifs.db images *.gif \
-c path -c md5 -c last_modified:mtime --pk=pathAnd if you try to insert a file whose primary key already exists, you get to decide the drama with --replace or --upsert. ✨ Pass --alter and any missing columns just get added to the table, no questions asked.
🌈 The full column definition menu 🌈
Here's literally every column definition you can use, no gatekeeping:
name— the name of the file, e.g.cleo.jpgpath— the path to the file relative to the root folder, e.g.pictures/cleo.jpgfullpath— the fully resolved path to the image, e.g./home/simonw/pictures/cleo.jpgsha256— the SHA256 hash of the file contentsmd5— the MD5 hash of the file contentsmode— the permission bits of the file, as an integer — you may want to convert this to octalcontent— the binary file contents, which will be stored as aBLOBcontent_text— the text file contents, which will be stored asTEXTmtime— the modification time of the file, as floating point seconds since the Unix epochctime— the creation time of the file, as floating point seconds since the Unix epochmtime_int— the modification time as an integer rather than a floatctime_int— the creation time as an integer rather than a floatmtime_iso— the modification time as an ISO timestamp, e.g.2020-07-27T04:24:06.654246ctime_iso— the creation time is an ISO timestampsize— the integer size of the file in bytesstem— the filename without the extension — forfile.txt.gzthis would befile.txtsuffix— the file extension — forfile.txt.gzthis would be.gz
You can pipe data in from standard input too, like this:
cat dog.jpg | sqlite-utils insert-files dogs.db pics - --name=dog.jpgThat lil' - says "read the data from standard input, bestie". And whatever string you hand over via the --name option gets used for the name and path values. 🎀
Heads up though: when you're inserting from standard input, only these column definitions are supported:name,path,content,content_text,sha256,md5andsize. That's the whole list. Don't come asking formtime_iso, she's off duty. 💅
💖 Converting data in columns 💖
The convert command is your little glow-up station for the data sitting in one specific column. Think: parsing a date string into an ISO timestamp, or splitting a string of tags into a JSON array. It's giving makeover. ✨
The command takes a database, a table, one or more columns, and a string of Python code that gets executed against the values from those columns. This example right here would replace the values in the headline column in the articles table with an upper-case version:
sqlite-utils convert content.db articles headline 'value.upper()'The Python code is passed as a string, and inside that string the value variable is the value of the current column. Iconic, honestly.
Your code gets compiled into a function that takes value as a single argument. And if you break your function body into multiple lines, the last line should be a return statement:
sqlite-utils convert content.db articles headline '
value = str(value)
return value.upper()'Your code gets auto-wrapped in a function for you, but plot twist — you can also define a function called convert(value) and that's the one that'll be called, if it's available:
sqlite-utils convert content.db articles headline '
def convert(value):
return value.upper()'Wanna read your code from standard input? Just use a CODE value of -:
cat mycode.py | sqlite-utils convert content.db articles headline -Where mycode.py contains a fragment of Python code that looks like this:
def convert(value):
return value.upper()The conversion gets applied to every single row in the specified table. But you can limit that to just the rows matching a WHERE clause with --where:
sqlite-utils convert content.db articles headline 'value.upper()' \
--where "headline like '%cat%'"And you can put named parameters in your where clause, populating them with one or more --param options:
sqlite-utils convert content.db articles headline 'value.upper()' \
--where "headline like :query" \
--param query '%cat%'The --dry-run option gives you a lil' preview of the conversion against the first ten rows, without touching the database at all. Look, don't touch. 👀
Note: in Python it'stable.convert()<#python-api-convert>, and the CLI reference lives atsqlite-utils convert<#cli-ref-convert>.
🦋 Importing additional modules 🦋
You can specify Python modules that should be imported and made available to your code using one or more --import options. This example uses the textwrap module to wrap the content column at 100 characters:
sqlite-utils convert content.db articles content \
'"\n".join(textwrap.wrap(value, 100))' \
--import=textwrapAnd nested imports? Supported, bestie. For example, to use ElementTree <https://docs.python.org/3/library/xml.etree.elementtree.html>:
sqlite-utils convert content.db articles content \
'xml.etree.ElementTree.fromstring(value).attrib["title"]' \
--import=xml.etree.ElementTree🎀 Using the debugger 🎀
If an error pops off while your conversion operation is running, you might see a message like this:
user-defined function raised exceptionAdd the --pdb option to catch that error and open the Python debugger right there at the scene of the crime. The conversion operation will exit after you type q in the debugger. 😭
Here's an example debugging session. First, create an articles table with invalid XML in the content column:
echo '{"content": "This is not XML"}' | sqlite-utils insert content.db articles -Now run the conversion with the --pdb option:
sqlite-utils convert content.db articles content \
'xml.etree.ElementTree.fromstring(value).attrib["title"]' \
--import=xml.etree.ElementTree \
--pdbWhen the error occurs, the debugger opens:
Exception raised, dropping into pdb...: syntax error: line 1, column 0
> .../python3.11/xml/etree/ElementTree.py(1338)XML()
-> parser.feed(text)
(Pdb) args
text = 'This is not XML'
parser = <xml.etree.ElementTree.XMLParser object at 0x102c405e0>
(Pdb) qargs here shows the arguments to the current function in the stack. And the Python pdb documentation <https://docs.python.org/3/library/pdb.html#debugger-commands> has the full details on all the other available commands, if you wanna go deeper. 📚
💜 Defining a convert() function 💜
Instead of handing over a single line of code to be executed against each value, you can define a function called convert(value).
Real ones know this mechanism can be used to execute one-off initialization code that runs once at the start of the conversion run. Main character energy, honestly.
The following example adds a new score column, then updates it to list a random number — after first seeding the random number generator to make sure multiple runs produce the same results:
sqlite-utils add-column content.db articles score float --not-null-default 1.0
sqlite-utils convert content.db articles score '
import random
random.seed(10)
def convert(value):
return random.random()
'🌸 sqlite-utils convert recipes 🌸
There are various built-in recipe functions available for common operations. These are them, and yes, they're all iconic:
r.jsonsplit(value, delimiter=',', type=<class 'str'>)— convert a string likea,b,cinto a JSON array["a", "b", "c"]
The delimiter parameter can be used to specify a different delimiter.
The type parameter can be set to float or int to produce a JSON array of different types. For example, if the column's string value was 1.2,3,4.5, then this:
r.jsonsplit(value, type=float)would produce an array like this: [1.2, 3.0, 4.5]
r.parsedate(value, dayfirst=False, yearfirst=False, errors=None)— parse a date and convert it to ISO date format:yyyy-mm-dd
In the case of dates such as 03/04/05, U.S. MM/DD/YY format is assumed — you can use dayfirst=True or yearfirst=True to change how these ambiguous dates are interpreted.
Use the errors= parameter to specify what should happen if a value cannot be parsed.
By default, if any value cannot be parsed an error will be occurred and all values will be left as they were.
Set errors=r.IGNORE to ignore any values that cannot be parsed, leaving them unchanged.
Set errors=r.SET_NULL to set any values that cannot be parsed to null.
r.parsedatetime(value, dayfirst=False, yearfirst=False, errors=None)— parse a datetime and convert it to ISO datetime format:yyyy-mm-ddTHH:MM:SS
These recipes can be used in the code passed to sqlite-utils convert like this:
sqlite-utils convert my.db mytable mycolumn \
'r.jsonsplit(value)'You can also pass the recipe function directly, without the (value) part — sqlite-utils will detect that it's a callable and use it automatically:
sqlite-utils convert my.db mytable mycolumn r.parsedateAnd this shorter syntax works for any callable, including functions from imported modules:
sqlite-utils convert my.db mytable mycolumn json.loads --import jsonTo use any of the documented parameters though, you gotta use the full function call syntax:
sqlite-utils convert my.db mytable mycolumn \
'r.jsonsplit(value, delimiter=":")'⭐ Saving the result to a different column ⭐
The --output and --output-type options can be used to save the result of the conversion to a separate column — and that column gets created if it doesn't already exist. So generous of her, fr fr.
sqlite-utils convert content.db articles headline 'value.upper()' \
--output headline_upperThe type of the created column defaults to text, but you can specify a different column type with --output-type. This example creates a new floating point column called id_as_a_float with a copy of each item's ID increased by 0.5:
sqlite-utils convert content.db articles id 'float(value) + 0.5' \
--output id_as_a_float \
--output-type floatAnd if the original column has gotta go at the end of the operation, just add --drop. 💅
✨ Converting a column into multiple columns ✨
okay so sometimes you've got ONE column and you're like... babe this is literally several columns wearing a trenchcoat 🧥. Like a location column stuffed with latitude,longitude values and you're dying for them to be their own separate latitude and longitude situations.
The --multi option on sqlite-utils convert is your girl. She expects your Python code to hand back a Python dictionary — and boom, shiny new columns get created and filled in for every single key in that dict. 💅
For the latitude,longitude era, you'd do:
sqlite-utils convert demo.db places location \
'bits = value.split(",")
return {
"latitude": float(bits[0]),
"longitude": float(bits[1]),
}' --multiThe type of the values you return gets taken into account when those new columns are born, bestie. So in this case the resulting schema is giving:
CREATE TABLE "places" (
"location" TEXT,
"latitude" REAL,
"longitude" REAL
);Also your code function can return None, and if it does, its output just gets ignored, no drama. And if you wanna ✨ yeet ✨ the original column once the op is done, just add --drop.
🛠️ Creating tables
Most of the time, creating tables by inserting example data is genuinely the quickest route — it's the low-effort high-reward path, fr. But if you need an empty table ready and waiting before you insert anything, create-table has you:
sqlite-utils create-table mydb.db mytable id integer name text --pk=idThis makes a table called mytable with two columns — an integer id column and a text name column — and sets id as the primary key. Iconic.
You can pass as many column-name column-type pairs as your heart desires. Valid types are integer, text, float and blob.
Pass --pk more than once if you want a compound primary key covering multiple columns. Not me making a whole squad of keys 🔑🔑.
You can mark columns that should be NOT NULL with --not-null colname, and set default values with --default colname defaultvalue.
sqlite-utils create-table mydb.db mytable \
id integer \
name text \
age integer \
is_good integer \
--not-null name \
--not-null age \
--default is_good 1 \
--pk=id
sqlite-utils tables mydb.db --schema -t
table schema
------- --------------------------------
mytable CREATE TABLE "mytable" (
"id" INTEGER PRIMARY KEY,
"name" TEXT NOT NULL,
"age" INTEGER NOT NULL,
"is_good" INTEGER DEFAULT '1'
)You can also set up foreign key relationships between the tables you're creating with --fk colname othertable othercolumn:
sqlite-utils create-table books.db authors \
id integer \
name text \
--pk=id
sqlite-utils create-table books.db books \
id integer \
title text \
author_id integer \
--pk=id \
--fk author_id authors id
sqlite-utils tables books.db --schema -t
table schema
------- -------------------------------------------------
authors CREATE TABLE "authors" (
"id" INTEGER PRIMARY KEY,
"name" TEXT
)
books CREATE TABLE "books" (
"id" INTEGER PRIMARY KEY,
"title" TEXT,
"author_id" INTEGER REFERENCES "authors"("id")
)Wanna go full SQLite STRICT mode? Use --strict — here's the link if you need the lore: <https://www.sqlite.org/stricttables.html>:
sqlite-utils create-table mydb.db mytable id integer name text --strictUse the any type for a strict column that should accept integers, floating point values, text, binary data or null without coercion — she's flexible like that 💜:
sqlite-utils create-table events.db events id integer payload any --strict
sqlite-utils tables mydb.db --schema -t
table schema
------- ------------------------
mytable CREATE TABLE "mytable" (
"id" INTEGER,
"name" TEXT
) STRICTIf a table with the same name already exists, you're getting an error, periodt. But you can choose to silently ignore it with --ignore, or replace the existing table with a brand new empty one using --replace. 🫶
You can also pass --transform to transform the existing table so it matches the new schema. See Explicitly creating a table <#python-api-explicit-create> in the Python library documentation for the deets on how that option works.
Note: In Python:table.create()<#python-api-explicit-create> — CLI reference:sqlite-utils create-table<#cli-ref-create-table>
✏️ Renaming a table
You can rename a table with the rename-table command:
sqlite-utils rename-table mydb.db oldname newnamePass --ignore to ignore any errors caused by the table not existing, or by the new name already being in use. She's chill like that 😌.
Note: In Python:db.rename_table()<#python-api-rename-table> — CLI reference:sqlite-utils rename-table<#cli-ref-rename-table>
👯 Duplicating tables
The duplicate command duplicates a table — creating a new table with the same schema AND a copy of all the rows, a whole twin flame situation:
sqlite-utils duplicate books.db authors authors_copyNote: In Python:table.duplicate()<#python-api-duplicate> — CLI reference:sqlite-utils duplicate<#cli-ref-duplicate>
🗑️ Dropping tables
Drop a table with the drop-table command:
sqlite-utils drop-table mydb.db mytableUse --ignore to ignore the error if the table doesn't exist. No tears, no errors 💀.
Note: In Python:table.drop()<#python-api-drop> — CLI reference:sqlite-utils drop-table<#cli-ref-drop-table>
🔄 Transforming tables
The transform command lets you apply complex transformations to a table that a regular SQLite ALTER TABLE command simply cannot pull off. See Transforming a table <#python-api-transform> for how this all works. By default, the transform command preserves a table's STRICT mode.
sqlite-utils transform mydb.db mytable \
--drop column1 \
--rename column2 column_renamedEvery option for this table (with the one exception of --pk-none) can be specified multiple times. Here's the full rundown:
--type column-name new-type— Change the type of the specified column. Valid types areinteger,text,float,real,blobandany. Changing aTEXTcolumn toINTEGER,FLOATorREALconverts exact empty-string values toNULL.--drop column-name— Drop the specified column.--rename column-name new-name— Rename this column to a new name.--column-order column— Use this multiple times to specify a new order for your columns. The-oshortcut is also available.--not-null column-name— Set this column asNOT NULL.--not-null-false column-name— For a column that's currently set asNOT NULL, remove theNOT NULL.--pk column-name— Change the primary key column for this table. Pass--pkmultiple times if you want a compound primary key.--pk-none— Remove the primary key from this table, turning it into a rowid table.--default column-name value— Set the default value of this column.--default-none column— Remove the default value for this column.--drop-foreign-key column— Drop the specified foreign key.--add-foreign-key column other_table other_column— Add a foreign key constraint tocolumnpointing toother_table.other_column.--strict— Convert the table to a SQLite STRICT table <https://www.sqlite.org/stricttables.html>. The command fails if the available SQLite version doesn't support strict tables. If existing rows contain values that are incompatible with their declared column types, the transformation fails and the original table is left unchanged.--no-strict— Convert a strict table back to a regular non-strict table.
If you wanna see the SQL that's gonna be executed to make the change without actually executing it, add the --sql flag. Peek behind the curtain, bestie 👀:
sqlite-utils transform fixtures.db roadside_attractions \
--rename pk id \
--default name Untitled \
--column-order id \
--column-order longitude \
--column-order latitude \
--drop address \
--sql
CREATE TABLE "roadside_attractions_new_4033a60276b9" (
"id" INTEGER PRIMARY KEY,
"longitude" FLOAT,
"latitude" FLOAT,
"name" TEXT DEFAULT 'Untitled'
);
INSERT INTO "roadside_attractions_new_4033a60276b9" ("longitude", "latitude", "id", "name")
SELECT "longitude", "latitude", "pk", "name" FROM "roadside_attractions";
DROP TABLE "roadside_attractions";
PRAGMA legacy_alter_table=ON;
ALTER TABLE "roadside_attractions_new_4033a60276b9" RENAME TO "roadside_attractions";
PRAGMA legacy_alter_table=OFF;Tables that are referenced by views can be transformed too — the view definitions are just left unchanged. See Tables referenced by views <#python-api-transform-views> for the details.
Note: In Python:table.transform()<#python-api-transform> — CLI reference:sqlite-utils transform<#cli-ref-transform>
👑 Adding a primary key to a rowid table
SQLite tables created without an explicit primary key are created as rowid tables <https://www.sqlite.org/rowidtable.html>. They still have a numeric primary key chillin' in the rowid column — but that column is not included in the output of select *. Here's an example, no cap:
echo '[{"name": "Azi"}, {"name": "Suna"}]' | \
sqlite-utils insert chickens.db chickens -
sqlite-utils schema chickens.db
CREATE TABLE "chickens" (
"name" TEXT
);
sqlite-utils chickens.db 'select * from chickens'
[{"name": "Azi"},
{"name": "Suna"}]
sqlite-utils chickens.db 'select rowid, * from chickens'
[{"rowid": 1, "name": "Azi"},
{"rowid": 2, "name": "Suna"}]You can use sqlite-utils transform ... --pk id to add a primary key column called id to the table. The primary key gets created as an INTEGER PRIMARY KEY, the existing rowid values get copied over into it, and it'll automatically increment as new rows get added. She's self-sufficient 💅:
sqlite-utils transform chickens.db chickens --pk id
sqlite-utils schema chickens.db
CREATE TABLE "chickens" (
"id" INTEGER PRIMARY KEY,
"name" TEXT
);
sqlite-utils chickens.db 'select * from chickens'
[{"id": 1, "name": "Azi"},
{"id": 2, "name": "Suna"}]okay so picking up right where we left off — that insert into chickens.db was only half the story, bestie. here's the rest of it:
echo '{"name": "Cardi"}' | sqlite-utils insert chickens.db chickens -
sqlite-utils chickens.db 'select * from chickens'and that last one hands you back:
[{"id": 1, "name": "Azi"},
{"id": 2, "name": "Suna"},
{"id": 3, "name": "Cardi"}]three chickens. all named. all serving. 🐔✨ no notes.
🎀 Extracting columns into a separate table 🎀
the sqlite-utils extract command does exactly what it says on the tin — it takes the columns you point at and pulls them out into their own little table, so your main table stops being a messy multi-column situation. glow up fr.
if you want the full tea — including what your table schemas look like before and after an extraction — the Python API docs have a whole section on Extracting columns into a separate table (#python-api-extract). go read it, it's thorough.
one lil detail that matters: rows where every extracted column is null do not get extracted. those rows just get a null value in their new foreign key column, and no record gets created for them in the lookup table. no orphan entries, no drama. 🫶
so you give it a database, a table, and one or more columns to extract. want the species column out of the trees table? boom:
sqlite-utils extract my.db trees speciesand that leaves you with this schema:
CREATE TABLE "trees" (
"id" INTEGER PRIMARY KEY,
"TreeAddress" TEXT,
"species_id" INTEGER,
FOREIGN KEY(species_id) REFERENCES species(id)
);
CREATE TABLE "species" (
"id" INTEGER PRIMARY KEY,
"species" TEXT
);
CREATE UNIQUE INDEX "idx_species_species"
ON "species" ("species");look at that normalization. chef's kiss. 👑
these are the options it takes:
--table TEXT— the name of the lookup table you're extracting those columns into. if you don't say anything, it just defaults to using the name of the columns being extracted. lowkey convenient.--fk-column TEXT— the name of the foreign key column it adds to your original table. defaults tocolumnname_id.--rename <TEXT TEXT>— use this when you want to rename the columns that get created in the new lookup table.--silent— don't show the progress bar. for when you want peace and quiet while it works. 🙈
🦋 a more complex example (yes, it eats) 🦋
here's the glow-up version that actually uses all those options. it grabs this [CSV file](https://github.com/wri/global-power-plant-database/blob/232a666653e14d803ab02717efc01cdd437e7601/output_database/global_power_plant_database.csv) — full of global power plants — turns it into SQLite, then extracts the country and country_long columns into a separate countries table:
wget 'https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true'
sqlite-utils insert global.db power_plants \
'global_power_plant_database.csv?raw=true' --csv
# Extract those columns:
sqlite-utils extract global.db power_plants country country_long \
--table countries \
--fk-column country_id \
--rename country_long nameand once that's done, running sqlite-utils schema global.db reveals this absolutely gorgeous schema:
CREATE TABLE "countries" (
"id" INTEGER PRIMARY KEY,
"country" TEXT,
"name" TEXT
);
CREATE TABLE "power_plants" (
"country_id" INTEGER,
"name" TEXT,
"gppd_idnr" TEXT,
"capacity_mw" TEXT,
"latitude" TEXT,
"longitude" TEXT,
"primary_fuel" TEXT,
"other_fuel1" TEXT,
"other_fuel2" TEXT,
"other_fuel3" TEXT,
"commissioning_year" TEXT,
"owner" TEXT,
"source" TEXT,
"url" TEXT,
"geolocation_source" TEXT,
"wepp_id" TEXT,
"year_of_capacity_data" TEXT,
"generation_gwh_2013" TEXT,
"generation_gwh_2014" TEXT,
"generation_gwh_2015" TEXT,
"generation_gwh_2016" TEXT,
"generation_gwh_2017" TEXT,
"generation_data_source" TEXT,
"estimated_generation_gwh" TEXT,
FOREIGN KEY("country_id") REFERENCES "countries"("id")
);
CREATE UNIQUE INDEX "idx_countries_country_name"
ON "countries" ("country", "name");okay but the FOREIGN KEY line and that CREATE UNIQUE INDEX?? we love a clean relational moment. 😭💖
🐍 In Python: it'stable.extract()— anchor#python-api-extract· CLI reference:sqlite-utils extract— anchor#cli-ref-extract
🌈 Creating views 🌈
you can spin up a view with the create-view command:
sqlite-utils create-view mydb.db version "select sqlite_version()"
sqlite-utils mydb.db "select * from version"
[{"sqlite_version()": "3.31.1"}]that's it. that's the whole vibe check.
want it to replace a view with the same name that already exists? use --replace. want it to just do nothing if a view already exists there? use --ignore. two moods, pick yours. 💅
🐍 In Python:db.create_view()— anchor#python-api-create-view· CLI reference:sqlite-utils create-view— anchor#cli-ref-create-view
💀 Dropping views 💀
bye bye view, use the drop-view command:
sqlite-utils drop-view myviewif the view might not even exist and you don't want to hear about it, use --ignore to ignore the error. self care tbh.
🐍 In Python:view.drop()— anchor#python-api-drop· CLI reference:sqlite-utils drop-view— anchor#cli-ref-drop-view
✨ Adding columns ✨
new column, who dis. use the add-column command:
sqlite-utils add-column mydb.db mytable nameofcolumn textthat last argument is the type of the column you're creating, and it can be any of these:
textorstrintegerorintfloatbloborbytes
it's optional btw — if you don't pass a type it just defaults to text. easy.
you can also add a column that's a foreign key reference to another table using --fk:
sqlite-utils add-column mydb.db dogs species_id --fk speciesand it'll automatically sniff out the name of the primary key on the species table and use that (plus its type) for your brand new column. no manual labor required. 🔥
if you feel like being explicit about which column you're referencing, use --fk-col:
sqlite-utils add-column mydb.db dogs species_id --fk species --fk-col refand if you want to slap a NOT NULL DEFAULT 'x' constraint on that new column, --not-null-default is your girl:
sqlite-utils add-column mydb.db dogs friends_count integer --not-null-default 0🐍 In Python:table.add_column()— anchor#python-api-add-column· CLI reference:sqlite-utils add-column— anchor#cli-ref-add-column
🌸 Adding columns automatically on insert/update 🌸
the --alter option is the "just handle it" option — it automatically adds new columns if the data you're inserting or upserting has a different shape than what's already there:
sqlite-utils insert dogs.db dogs new-dogs.json --pk=id --alterno schema panic, it adapts. main character energy honestly.
🐍 In Python:table.insert(..., alter=True)— anchor#python-api-add-column-alter
💜 Adding foreign key constraints 💜
here's the thing — SQLite's own ALTER TABLE command does not support adding foreign key references. so sqlite-utils does it for you with add-foreign-key. iconic behavior, truly.
to point the books.author_id column at authors.id in another table:
sqlite-utils add-foreign-key books.db books author_id authors idand if you leave out the other table and other column, sqlite-utils will attempt to guess them — so that same example could just be:
sqlite-utils add-foreign-key books.db books author_idadd --ignore if you want it to ignore an already-existing foreign key instead of throwing an error at you:
sqlite-utils add-foreign-key books.db books author_id --ignorethere's a whole Adding foreign key constraints section (#python-api-add-foreign-key) in the Python API docs if you want the details, including how that automatic table-guessing mechanism actually works. go look, it's worth it.
🐍 In Python:table.add_foreign_key()— anchor#python-api-add-foreign-key· CLI reference:sqlite-utils add-foreign-key— anchor#cli-ref-add-foreign-key
👑 Adding multiple foreign keys at once 👑
real talk: adding a foreign key requires a VACUUM. on big databases that is a genuinely expensive operation. so if you're adding a bunch of foreign keys, combine them into one operation (and therefore one VACUUM) using add-foreign-keys:
sqlite-utils add-foreign-keys books.db \
books author_id authors id \
authors country_id countries idheads up though — when you use this command, each foreign key has to be defined in full, as four arguments: the table, column, other table and other column. no shortcuts here, bestie. write them all out.
🐍 In Python:db.add_foreign_keys()— anchor#python-api-add-foreign-keys· CLI reference:sqlite-utils add-foreign-keys— anchor#cli-ref-add-foreign-keys
🧚 Adding indexes for all foreign keys 🧚
if you want to make sure every foreign key column in your database has a corresponding index, it's literally one command:
sqlite-utils index-foreign-keys books.dbdo it. your query performance will thank you. ⭐
🐍 In Python:db.index_foreign_keys()— anchor#python-api-index-foreign-keys· CLI reference:sqlite-utils index-foreign-keys— anchor#cli-ref-index-foreign-keys
🍓 Setting defaults and not null constraints 🍓
you can use the --not-null and --default options (on both insert and upsert) to say which columns should be NOT NULL, or to set database defaults for one or more specific columns:
sqlite-utils insert dogs.db dogs_with_scores dogs-with-scores.json \
--not-null=age \
--not-null=name \
--default age 2 \
--default score 5setting your boundaries and your defaults in one line. growth. 🌱
🐍 In Python: thenot_null=anddefaults=arguments — anchor#python-api-defaults-not-null
🎀 Creating indexes 🎀
you can add an index to an existing table with the create-index command:
sqlite-utils create-index mydb.db mytable col1 [col2...]which works for a single column or multiple columns, your call.
the name of the index gets automatically derived from the table and the columns. want a different name? pass --name=name_of_index.
use --unique if you want to create a unique index.
use --if-not-exists so it doesn't even try to create the index if one with that name is already hanging around. very demure.
now — to add an index on a column in descending order, you prefix the column with a hyphen. and since that can absolutely get confused for a command-line option, you have to construct it like this:
sqlite-utils create-index mydb.db mytable -- col1 -col2 col3which creates an index on that table on (col1, col2 desc, col3). note the -- doing the heavy lifting there. don't skip it. 💅
warning, fr fr: if your column names are already prefixed with a hyphen, you'll need to manually run a CREATE INDEX SQL statement to add indexes to them, rather than using this tool. no workaround, that's just the deal.
add the --analyze option and it'll run ANALYZE against the index after it's been created.
🐍 In Python:table.create_index()— anchor#python-api-create-index· CLI reference:sqlite-utils create-index— anchor#cli-ref-create-index
💖 Dropping indexes 💖
drop an index from an existing table with the drop-index command:
sqlite-utils drop-index mydb.db mytable idx_mytable_col1use --ignore to ignore the error if that index doesn't exist on that table. avoid the sad little error message, live your life. 😌
🐍 In Python:table.drop_index()— anchor#python-api-create-index· CLI reference:sqlite-utils drop-index— anchor#cli-ref-drop-index
⭐ Configuring full-text search ⭐
you can enable SQLite full-text search on a table and a set of columns like this:
sqlite-utils enable-fts mydb.db documents title summaryand that's exactly where we're leaving it for now — more coming, keeping the momentum going 💖
This one's riding with SQLite's FTS5 module by default, no cap ✨. But if your heart is set on FTS4, just say so with --fts4:
sqlite-utils enable-fts mydb.db documents title summary --fts4And bestie, enable-fts does not leave your old docs stranded — it populates the shiny new index with every existing document. The thing is, if you add more documents later, you gotta run populate-fts so they get indexed too:
sqlite-utils populate-fts mydb.db documents title summaryLowkey the better solution here is database triggers 🎀. Set them up with the --create-triggers option on your very first enable-fts run and the full-text index just updates itself like it's got main character energy:
sqlite-utils enable-fts mydb.db documents title summary --create-triggersWant a custom FTS tokenizer — say, to unlock Porter stemming? That's --tokenize=:
sqlite-utils populate-fts mydb.db documents title summary --tokenize=porterTo yeet the FTS tables and the triggers you created, use disable-fts:
sqlite-utils disable-fts mydb.db documentsTo rebuild one or more FTS tables (peep Rebuilding a full-text search table <#python-api-fts-rebuild>), use rebuild-fts:
sqlite-utils rebuild-fts mydb.db documentsAnd you can rebuild every FTS table by running rebuild-fts without passing any table names at all:
sqlite-utils rebuild-fts mydb.db📝 Note: In Python:table.enable_fts()<#python-api-fts-enable>• CLI reference:sqlite-utils enable-fts<#cli-ref-enable-fts>
🔍 Executing searches 🔍
Once your table is all set up with full-text search, you can search it with sqlite-utils search:
sqlite-utils search mydb.db documents searchtermThis command takes the exact same output options as sqlite-utils query: --table, --csv, --tsv, --nl, etc. 💅
By default it serves you the most relevant matches first. Want a different sort order? Use the -o option — it takes a column, or a column followed by desc:
# Sort by rowid
sqlite-utils search mydb.db documents searchterm -o rowid
# Sort by created in descending order
sqlite-utils search mydb.db documents searchterm -o 'created desc'SQLite advanced search syntax (https://www.sqlite.org/fts5.html#full_text_query_syntax) is enabled by default, queen. If you want to run a search with automatic quoting slapped onto your terms — so nothing gets accidentally interpreted as advanced search syntax — use the --quote option.
You can pick a subset of columns to get back with the -c option, one or more times:
sqlite-utils search mydb.db documents searchterm -c title -c createdBy default, all the search results come back. Use --limit 20 to get just the first 20.
And use the --sql option to see the SQL that would run, instead of actually running the query:
sqlite-utils search mydb.db documents searchterm --sql
with original as (
select
rowid,
*
from "documents"
)
select
"original".*
from
"original"
join "documents_fts" on "original".rowid = "documents_fts".rowid
where
"documents_fts" match :query
order by
"documents_fts".rank📝 Note: In Python:table.search()<#python-api-fts-search>• CLI reference:sqlite-utils search<#cli-ref-search>
🔢 Enabling cached counts 🔢
select count(*) queries can take forever against big tables 😭. sqlite-utils can speed them up by adding triggers that maintain a _counts table — see Cached table counts using triggers <#python-api-cached-table-counts> for the deets.
The sqlite-utils enable-counts command configures those triggers, either for every table in the database or for specific tables:
# Configure triggers for every table in the database
sqlite-utils enable-counts mydb.db
# Configure triggers just for specific tables
sqlite-utils enable-counts mydb.db table1 table2If the _counts table ever drifts out-of-sync with the real table counts, you can repair it with the reset-counts command:
sqlite-utils reset-counts mydb.db📝 Note: In Python:table.enable_counts()<#python-api-cached-table-counts>• CLI reference:sqlite-utils enable-counts<#cli-ref-enable-counts>
📊 Optimizing index usage with ANALYZE 📊
The SQLite ANALYZE command (https://www.sqlite.org/lang_analyze.html) builds a whole table of statistics that the query planner can use to make better decisions about which indexes to reach for on a given query.
You should run ANALYZE if your database is big and you don't think your indexes are being used efficiently.
To run ANALYZE against every index in a database, do this:
sqlite-utils analyze mydb.dbYou can also run it against specific tables, or specific named indexes, by passing them as optional arguments:
sqlite-utils analyze mydb.db mytable idx_mytable_nameAnd you can run ANALYZE as part of another command using the --analyze option — it's supported by the create-index, insert and upsert commands. 🫶
📝 Note: In Python:db.analyze()<#python-api-analyze>• CLI reference:sqlite-utils analyze<#cli-ref-analyze>
🧹 Vacuum 🧹
You can run VACUUM to optimize your database like so:
sqlite-utils vacuum mydb.db📝 Note: In Python:db.vacuum()<#python-api-vacuum>• CLI reference:sqlite-utils vacuum<#cli-ref-vacuum>
✨ Optimize ✨
The optimize command can dramatically shrink the size of your database if you're using SQLite full-text search 🔥. It runs OPTIMIZE against all of your FTS4 and FTS5 tables, then runs VACUUM.
If you just want the OPTIMIZE without the VACUUM, use the --no-vacuum flag:
# Optimize all FTS tables and then VACUUM
sqlite-utils optimize mydb.db
# Optimize but skip the VACUUM
sqlite-utils optimize --no-vacuum mydb.dbTo optimize specific tables instead of every FTS table, pass those tables as extra arguments:
sqlite-utils optimize mydb.db table_1 table_2📝 Note: In Python:table.optimize()<#python-api-fts-optimize>• CLI reference:sqlite-utils optimize<#cli-ref-optimize>
📝 WAL mode 📝
You can enable Write-Ahead Logging (https://www.sqlite.org/wal.html) for a database file with the enable-wal command:
sqlite-utils enable-wal mydb.dbAnd you can turn WAL mode back off with disable-wal:
sqlite-utils disable-wal mydb.dbBoth of these commands accept one or more database files as arguments. 💜
📝 Note: In Python:db.enable_wal()anddb.disable_wal()<#python-api-wal>• CLI reference:sqlite-utils enable-wal<#cli-ref-enable-wal>
💾 Dumping the database to SQL 💾
The dump command spits out a SQL dump of the schema plus the full contents of the database file you name:
sqlite-utils dump mydb.db
BEGIN TRANSACTION;
CREATE TABLE ...
...
COMMIT;📝 Note: In Python:db.iterdump()<#python-api-itedump>• CLI reference:sqlite-utils dump<#cli-ref-dump>
🧩 Loading SQLite extensions 🧩
A bunch of these commands can load extra SQLite extensions using the --load-extension=/path/to/extension option — run --help to check if it's supported, e.g. sqlite-utils rows --help.
This option can be applied multiple times, so you can load multiple extensions. Greedy and valid. 👑
Since SpatiaLite (https://www.gaia-gis.it/fossil/libspatialite/index) is commonly used with SQLite, the value spatialite is special: it'll search for SpatiaLite in the most common installation locations, saving you from having to remember exactly where that module lives:
sqlite-utils memory "select spatialite_version()" --load-extension=spatialite
[{"spatialite_version()": "4.3.0a"}]🌍 SpatiaLite helpers 🌍
SpatiaLite (https://www.gaia-gis.it/fossil/libspatialite/home) adds geographic capability to SQLite — kinda like how PostGIS builds on PostgreSQL, iykyk. The SpatiaLite cookbook (http://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/index.html) is a great resource for learning what's possible with it. 🗺️
You can convert an existing table into a geographic table by adding a geometry column — use the sqlite-utils add-geometry-column command:
sqlite-utils add-geometry-column spatial.db locations geometry --type POLYGON --srid 4326The table (locations in the example above) has to already exist before you add a geometry column. So run sqlite-utils create-table first, then add-geometry-column.
Use the --type option to specify a geometry type. By default, add-geometry-column uses a generic GEOMETRY, which works with any type — though it may not be supported by some desktop GIS applications.
Eight types are allowed (case-insensitive):
POINTLINESTRINGPOLYGONMULTIPOINTMULTILINESTRINGMULTIPOLYGONGEOMETRYCOLLECTIONGEOMETRY
📝 Note: In Python:table.add_geometry_column()<#python-api-gis-add-geometry-column>• CLI reference:sqlite-utils add-geometry-column<#cli-ref-add-geometry-column>
🚀 Adding spatial indexes 🚀
Once you've got a geometry column, you can speed up bounding box queries by adding a spatial index:
sqlite-utils create-spatial-index spatial.db locations geometrySee this SpatiaLite Cookbook recipe (http://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/cookbook_topics.03.html#topic_Wonderful_RTree_Spatial_Index) for examples of how to actually use a spatial index. ✨
📝 Note: In Python:table.create_spatial_index()<#python-api-gis-create-spatial-index>• CLI reference:sqlite-utils create-spatial-index<#cli-ref-create-spatial-index>
📦 Installing packages 📦
The convert command, plus the insert --convert and query --functions options, can be handed a Python script that imports additional modules from the sqlite-utils environment.
You can install packages from PyPI straight into the correct environment using sqlite-utils install <package>. It's a wrapper around pip install, bestie:
sqlite-utils install beautifulsoup4Use -U to upgrade a package you already have.
🗑️ Uninstalling packages 🗑️
You can uninstall packages that were installed with sqlite-utils install using sqlite-utils uninstall <package>:
sqlite-utils uninstall beautifulsoup4Use -y to skip the confirmation request — no more "are you sure" jumpscare 🙈.
🐍 sqlite_utils Python library 🐍
- Getting started
- Connecting to or creating a database
- Closing a database
- Attaching additional databases
- Tracing queries
- Executing queries
db.query(sql, params)db.execute(sql, params)- Passing parameters
- Transactions and saving your changes
- Grouping changes with
db.atomic() - Raw SQL writes with
db.execute() - Managing transactions yourself
- Supported connection modes
- Grouping changes with
- Accessing tables
- Accessing views
- Listing tables
- Listing views
- Listing rows
- Counting rows
- Listing rows with their primary keys
- Retrieving a specific record
- Showing the schema
- Creating tables
- Custom column order and column types
- Explicitly creating a table
- Compound primary keys
- Specifying foreign keys
- Compound foreign keys
- Table configuration options
- Setting defaults and not null constraints
- Renaming a table
- Duplicating tables
- Bulk inserts
- Inserting data from a list or tuple iterator
- Insert-replacing data
- Updating a specific record
- Deleting a specific record
- Deleting multiple records
- Upserting data
- Alternative upserts using INSERT OR IGNORE
- Converting data in columns
- Working with lookup tables
- Creating lookup tables explicitly
- Populating lookup tables automatically during insert/upsert
- Working with many-to-many relationships
- Using m2m and lookup tables together
- Analyzing a column
- Adding columns
- Adding columns automatically on insert/update
- Adding foreign key constraints
- Adding multiple foreign key constraints at once
- Adding indexes for all foreign keys
- Dropping a table or view
- Transforming a table
- Altering column types
- Changing strict mode
- Renaming columns
- Dropping columns
- Changing primary keys
- Changing not null status
- Altering column defaults
- Changing column order
- Adding foreign key constraints
- Replacing foreign key constraints
- Dropping foreign key constraints
- CHECK constraints
- Tables referenced by views
- Custom transformations with
.transform_sql() - Foreign keys and transactions
- Extracting columns into a separate table
- Setting an ID based on the hash of the row contents
- Creating views
- Storing JSON
- Converting column values using SQL functions
- Checking the SQLite version
- Dumping the database to SQL
- Introspecting tables and views
.exists().count.columns.columns_dict.default_values.pks.use_rowid.checks.column_checks.table_checks.foreign_keys.schema.strict.indexes.xindexes.triggers.triggers_dict.detect_fts().virtual_table_using.has_counts_triggersdb.supports_strict
- Full-text search
- Enabling full-text search for a table
- Quoting characters for use in search
- Searching with
table.search() - Building SQL queries with
table.search_sql()
- Rebuilding a full-text search table
- Optimizing a full-text search table
- Cached table counts using triggers
- Creating indexes
- Optimizing index usage with ANALYZE
- Vacuum
- WAL mode
- Suggesting column types
- Registering custom SQL functions
- Quoting strings for use in SQL
- Reading rows from a file
- Setting the maximum CSV field size limit
- Detecting column types using TypeTracker
- SpatiaLite helpers
- Initialize SpatiaLite
- Finding SpatiaLite
- Adding geometry columns
- Creating a spatial index
🐣 Getting started
okay bestie, here's literally the cutest possible on-ramp: make a brand new SQLite database file with a chickens table in it, stuffed with four whole records. watch this 💅
from sqlite_utils import Database
db = Database("chickens.db")
db.table("chickens").insert_all([{
"name": "Azi",
"color": "blue",
}, {
"name": "Lila",
"color": "blue",
}, {
"name": "Suna",
"color": "gold",
}, {
"name": "Cardi",
"color": "black",
}])the inserted rows get saved to the database file immediately — no cap. methods like insert_all() commit their own changes, so you do not need to call commit() yourself. (want the whole tea on how that works? go look at Transactions and saving your changes.) 🫶
and you can loop through those rows like this:
for row in db.table("chickens").rows:
print(row)which prints out:
{'name': 'Azi', 'color': 'blue'}
{'name': 'Lila', 'color': 'blue'}
{'name': 'Suna', 'color': 'gold'}
{'name': 'Cardi', 'color': 'black'}when you wanna run a raw SQL query, that's db.query()'s moment:
for row in db.query("""
select color, count(*)
from chickens group by color
order by count(*) desc
"""):
print(row)which gives you back:
{'color': 'blue', 'count(*)': 2}
{'color': 'gold', 'count(*)': 1}
{'color': 'black', 'count(*)': 1}🔌 Connecting to or creating a database
you make a Database object by handing it either a path to a file on disk, or an SQLite3 database connection you already have. so cozy:
from sqlite_utils import Database
db = Database("my_database.db")this will create my_database.db if it's not already there. iconic behaviour honestly ✨
now — if what you actually want is to rebuild that database from absolute scratch (like, yeeting the existing file off disk first if it exists), pass recreate=True:
db = Database("my_database.db", recreate=True)instead of a file path you can also just drop in an existing SQLite connection:
import sqlite3
db = Database(sqlite3.connect("my_database.db"))🚨 ngl this one has rules: the connection must use Python's default transaction handling. connections made with the Python 3.12+sqlite3.connect(..., autocommit=True)orautocommit=Falseoptions get rejected with asqlite_utils.db.TransactionError— see Supported connection modes.
want an in-memory database? easy, bestie:
db = Database(memory=True)you can also make a named in-memory database. unlike regular memory databases, these can be reached by multiple threads — as long as at least one reference to the database still exists. and del db will clear that database right out of memory 🧚
db = Database(memory_name="my_shared_database")once you've made a Database, db.memory and db.memory_name will tell you whether it's backed by an in-memory database, and let you read its shared cache name. db.memory is True for any in-memory database, and db.memory_name holds the name you passed to memory_name=, otherwise None.
db = Database(memory_name="shared")
db.memory # True
db.memory_name # "shared"connections use PRAGMA recursive_triggers=on by default. if you do not want recursive triggers (https://www.sqlite.org/pragma.html#pragma_recursive_triggers) you can turn them off with:
db = Database(memory=True, recursive_triggers=False)by default, any sqlite-utils plugins (#plugins) that implement the prepare_connection(conn) (#plugins-hooks-prepare-connection) hook will run against the connection the moment you create the Database object. if you'd rather not run plugins at all, opt out with execute_plugins=False, like so:
db = Database(memory=True, execute_plugins=False)and you can pass strict=True to switch on SQLite STRICT mode (https://www.sqlite.org/stricttables.html) for every table created through this database object 👑
db = Database("my_database.db", strict=True)🚪 Closing a database
Database objects keep a connection to the underlying SQLite database. you can explicitly close that connection with the .close() method:
db = Database("my_database.db")
# ... use the database ...
db.close()the Database object also works as a context manager, which closes the connection automatically when the with block exits:
with Database("my_database.db") as db:
db["my_table"].insert({"name": "Example"})
# Connection is automatically closed hereexiting the block is the same as calling db.close(): the connection gets closed, and any transaction still open at that point gets rolled back. that matches SQLite's own behaviour when a connection closes, fr fr.
honestly though, this rarely matters in practice. everything that writes to the database — including raw db.execute() statements — commits automatically, so a transaction can only still be open here if you explicitly started one with db.begin() and haven't committed it yet. and in that case, the decision to commit is yours: committing automatically on exit could silently persist half-finished work — like if your code returned early out of the block. call db.commit() when the work is actually complete. periodt.
💀 note that this is not the same as the sqlite3.Connection context manager in the standard library, which commits on success but does not close the connection. see Transactions and saving your changes for the full transaction model.
🔗 Attaching additional databases
SQLite supports cross-database SQL queries, which can join data from tables living in more than one database file. the way that works is genuinely so good.
you attach an extra database using the .attach() method — give it an alias for that database plus the path to the SQLite file on disk.
db = Database("first.db")
db.attach("second", "second.db")
# Now you can run queries like this one:
print(db.query("""
select * from table_in_first
union all
select * from second.table_in_second
"""))you reference tables in the attached database by using the alias value you passed to db.attach(alias, filepath) as a prefix — like that second.table_in_second reference up there in the SQL query. 🌈
Note: in the CLI:sqlite-utils --attach(#cli-query-attach)
👀 Tracing queries
you can use the tracer mechanism to see the SQL queries SQLite is executing. a tracer is just a function you provide, and it gets called with sql and params arguments every single time SQL is executed. so nosy, so useful:
def tracer(sql, params):
print("SQL: {} - params: {}".format(sql, params))you pass that function to the Database() constructor like so:
db = Database(memory=True, tracer=tracer)you can also switch a tracer function on temporarily for a block of code using the with db.tracer(...) context manager:
db = Database(memory=True)
# ... later
with db.tracer(print):
db.table("dogs").insert({"name": "Cleo"})that example prints queries only for the duration of the with block. cute, contained, zero mess 🎀
🗣️ Executing queries
the Database class has several methods for directly executing SQL queries. main character behaviour.
db.query(sql, params)
the db.query(sql) function executes a SQL query and returns an iterator over Python dictionaries representing the resulting rows:
db = Database(memory=True)
db.table("dogs").insert_all([{"name": "Cleo"}, {"name": "Pancakes"}])
for row in db.query("select * from dogs"):
print(row)
# Outputs:
# {'name': 'Cleo'}
# {'name': 'Pancakes'}okay so bestie, the second you call db.query(), that SQL runs. immediately. 💅 no waiting around. the rows get fetched lazily as you iterate though, so a huge result set is not gonna just yeet itself into memory all at once 🎀
and because execution is instant, if your SQL is broken it raises an exception right there and then 😭 also — a statement like INSERT ... RETURNING? it takes effect, and it gets committed (unless you've got a transaction open) even if you never iterate over its results. periodt.
db.query() is only for SQL that returns rows, okay? if you hand it something that returns no rows — think an INSERT or UPDATE with no RETURNING clause — it's gonna raise a ValueError. and the rejected statement gets rolled back, so your database is left completely untouched 🫶 for those statements, use db.execute() instead.
ok so there's exactly one exception to that rolled-back guarantee: aPRAGMAstatement that returns no rows, likePRAGMA user_version = 5. it still raises aValueErrorbut it has already taken effect. some PRAGMAs straight up refuse to run inside a transaction, which is why PRAGMAs are executed outside the savepoint that rolls back the other rejected statements. so for PRAGMAs that don't return rows, usedb.execute()💖
now if a query returns more than one column with the same name — think a join between two tables that share column names — later occurrences get renamed with a numeric suffix, so every single value still shows up in your dictionary 🔥
row = next(db.query("select 1 as id, 2 as id, 3 as id"))
print(row)
# Outputs:
# {'id': 1, 'id_2': 2, 'id_3': 3}and if a suffix would collide with another column in the query, it gets skipped — select 1 as id, 2 as id, 3 as id_2 gives you {'id': 1, 'id_3': 2, 'id_2': 3}. the same renaming is applied by table.rows_where() and table.search() ✨
Note: in the CLI it's sqlite-utils query.
💫 db.execute(sql, params) 💫
okay so db.execute() and db.executescript() are basically little wrappers around .execute() and .executescript() on the underlying SQLite connection. if you've registered a tracer function, these wrappers log to it 🌸
db.execute(sql) returns a sqlite3.Cursor that was used to execute the SQL.
db = Database(memory=True)
db.table("dogs").insert({"name": "Cleo"})
cursor = db.execute("update dogs set name = 'Cleopaws'")
print(cursor.rowcount)
# Outputs the number of rows affected by the update
# In this case 2other cursor methods like .fetchone() and .fetchall() are also available, see the standard library documentation 📚
Note: write statements executed this way are committed automatically — unless a transaction's already open, in which case they become part of it. see Raw SQL writes with db.execute() 👀
🍓 Passing parameters 🍓
both db.query() and db.execute() take an optional second argument for parameters to pass to your SQL query 💖
it can be either a tuple/list or a dictionary, depending on which type of parameters you're using in the query. values passed this way get correctly quoted and escaped, which helps you dodge SQL injection vulnerabilities 🛡️
? parameters in the SQL query can be filled using a list:
db.execute("update dogs set name = ?", ["Cleopaws"])
# This will rename ALL dogs to be called "Cleopaws"named parameters using :name can be filled using a dictionary:
dog = next(db.query(
"select rowid, name from dogs where name = :name",
{"name": "Cleopaws"}
))
# dog is now {'rowid': 1, 'name': 'Cleopaws'}in this example next() is used to grab the first result from the iterator that db.query() returns ✨
🎀 Transactions and saving your changes 🎀
every single method in this library that writes to the database — insert(), upsert(), update(), delete(), delete_where(), transform(), create_table(), create_index(), enable_fts() and the rest — runs inside its own transaction and commits it before returning 😌 your changes hit disk as soon as the method call finishes:
db = Database("data.db")
db.table("news").insert({"headline": "Dog wins award"})
# The new row is already saved - no commit() requiredsame deal with raw SQL run through db.execute() — a write statement is committed as soon as it's run 💅
another way to think about it: every sqlite-utils method call is its own little unit of work 🧚 if several method calls all need to either succeed together or fail together, use db.atomic() to turn them into one single unit of work.
you never need to call commit(), and you do not need to close the database to persist your changes. there are exactly two situations where you actually need to think about transactions:
- you want to group several write ops together so they either all succeed or all fail — use
db.atomic(). - you're managing a transaction yourself with
db.begin(), in which case nothing gets committed until you commit — the library will never commit a transaction you opened.
with Database(...) as db: is not a transaction block, bestie 😭 it manages the lifetime of the database connection and closes it on exit. for an actual transaction, use with db.atomic():.
🔥 Grouping changes with db.atomic() 🔥
use db.atomic() to group multiple operations into a single transaction:
with db.atomic():
db.table("dogs").insert({"id": 1, "name": "Cleo"}, pk="id")
db.table("dogs").insert({"id": 2, "name": "Pancakes"})the transaction commits when the block exits ✨ if an exception is raised, changes made inside the block get rolled back.
this matters when several operations represent a single logical change. without db.atomic(), an earlier method call stays committed even if a later one fails:
# These are two separate transactions
db.table("accounts").update(1, {"balance": 90})
db.table("accounts").update(2, {"balance": 110})
# These updates either both succeed or both fail
with db.atomic():
db.table("accounts").update(1, {"balance": 90})
db.table("accounts").update(2, {"balance": 110})transactions can also make things faster 🏃♀️ calling insert() over and over outside db.atomic() creates and commits a whole separate transaction for every single call. for bulk inserts, prefer insert_all(). and if you need to call several different methods in a loop, wrap the whole loop in db.atomic():
with db.atomic():
for row in rows:
db.table("events").insert(row)db.atomic() can also be nested 💅 nested blocks use SQLite savepoints, so an exception in an inner block can roll back to that savepoint without rolling back the entire outer transaction:
with db.atomic():
db.table("dogs").insert({"id": 1, "name": "Cleo"}, pk="id")
try:
with db.atomic():
db.table("dogs").insert({"id": 2, "name": "Pancakes"})
raise ValueError("skip this one")
except ValueError:
pass
db.table("dogs").insert({"id": 3, "name": "Marnie"})the transaction opens with a deferred BEGIN — SQLite takes the locks it needs when the first statement inside the block runs 👑
🌈 Raw SQL writes with db.execute() 🌈
write statements run with db.execute() follow the same rule as literally everything else: they get committed automatically the moment they've run 🫶
db.execute("insert into news (headline) values (?)", ["Dog wins award"])
# Already committeddb.execute() plays nice with sqlite-utils transaction handling. calling db.conn.execute() directly bypasses that policy and leaves transaction handling up to Python's underlying sqlite3.Connection. prefer db.execute() unless you deliberately need that lower-level API 😌
if a transaction is open — either because the call happens inside a db.atomic() block, or after db.begin() — the statement becomes part of that transaction instead, and commits when the transaction commits:
with db.atomic():
db.execute("insert into news (headline) values (?)", ["Dog wins award"])
db.execute("insert into news (headline) values (?)", ["Cat unimpressed"])
# Both rows committed togetherone corner case, bestie: a row-returning write like INSERT ... RETURNING executed through db.execute() cannot be auto-committed, because its rows haven't been read yet 😭 call db.commit() after fetching them, or use db.query() for those statements — that executes the write and commits it right away 💖
👑 Managing transactions yourself 👑
you can take full manual control using the db.begin(), db.commit() and db.rollback() methods:
db.begin()
db.table("news").insert({"headline": "Dog wins award"})
if all_looks_good:
db.commit()
else:
db.rollback()db.begin() raises sqlite3.OperationalError if a transaction's already open. db.commit() and db.rollback() do nothing if there's no open transaction 🙈
the library will never commit a transaction you opened. if you call write methods like insert() — or use db.atomic() — while your transaction is open, they participate in it using SQLite savepoints instead of committing: exiting an atomic() block releases its savepoint, but nothing is saved to disk until you commit the outer transaction yourself. and if you roll back, their changes get rolled back too 💅
prefer db.atomic() or db.begin(), db.commit() and db.rollback() over mixing sqlite-utils transaction methods with calls to db.conn.commit(), db.conn.rollback() or raw transaction-control SQL. mixing the two layers makes it way harder to tell which layer owns the current transaction 😵💫
some related safeguards to be aware of:
db.enable_wal()anddb.disable_wal()raise asqlite_utils.db.TransactionErrorif called while a transaction is open — because changing the journal mode would commit it as a side effect 💀table.transform()raises asqlite_utils.db.TransactionErrorif called while a transaction is open withPRAGMA foreign_keysenabled and the table is referenced by foreign keys with destructiveON DELETEactions — because the pragma can't be turned off mid-transaction to protect those referencing rows. see Foreign keys and transactions.- closing the database — explicitly with
db.close(), or by exiting awith Database(...) as db:block — rolls back any transaction that's still open. see Closing a database.
🔌 Supported connection modes 🫶
Okay so before we go any further, bestie — db.atomic() and the automatic per-method transactions are lowkey picky about which kind of connection they'll accept. 💅
They currently require a connection that's using Python's legacy transaction control mode — that's sqlite3.LEGACY_TRANSACTION_CONTROL on Python 3.12 and later. So if you hand Database() a connection you built with the Python 3.12+ sqlite3.connect(..., autocommit=True) or autocommit=False options, it will not vibe with you. You get a sqlite_utils.db.TransactionError thrown straight at your face. 😭
Why tho? Let me explain, because it's actually so valid:
- Connections using
autocommit=Falseare not supported because Python keeps a transaction open continuously. sqlite-utils usesConnection.in_transactionto tell its own transactions apart from transactions opened by its caller, and that distinction is simply not available in this mode. It's giving "can't tell whose transaction is whose" 💀 - Connections using
autocommit=Trueare also currently rejected, because sqlite-utils has not formally exposed that as a supported configuration.
So yeah — legacy transaction control mode or nothing, no cap. 🦋
📂 Accessing tables
Tables get accessed with the db.table() method, like this:
table = db.table("my_table")Using this factory function is what lets you set Table configuration options. Any additional keyword arguments you pass to db.table() will be used if a further method call causes the table to actually get created. ✨
Now listen — the db.table() method will always return a sqlite_utils.db.Table <#reference-db-table> instance, or it raises a sqlite_utils.db.NoTable exception if that table name is actually a SQL view. (Rude, but fair.)
You can also access tables or views using dictionary-style syntax, like this:
table_or_view = db["my_table_or_view_name"]And here's the tea: if a table you access with either of these methods doesn't exist yet, it will be created the first time you try to insert or upsert data into it. Iconic "build it as we go" energy. 👑
👀 Accessing views
SQL views can be accessed using the db.view() method, like so:
view = db.view("my_view")This returns a sqlite_utils.db.View <#reference-db-view> instance, or it raises a sqlite_utils.db.NoView exception if the view doesn't exist. Simple as that. 🌸
📜 Listing tables
You can list the names of tables in a database using the .table_names() method:
>>> db.table_names()
['dogs']Only want the FTS4 tables? Use .table_names(fts4=True). For FTS5, use .table_names(fts5=True). ✨
You can also iterate through the table objects themselves using the .tables property:
>>> db.tables
[<Table dogs>]🗒️ Note: In the CLI: sqlite-utils tables <#cli-tables>
🪞 Listing views
.view_names() shows you a list of views in the database:
>>> db.view_names()
['good_dogs']And you can iterate through view objects using the .views property:
>>> db.views
[<View good_dogs>]View objects are basically Table objects' chill cousin, except that any attempts to insert or update data will throw an error. 🚫 So here's the full list of methods and properties you get on a view object:
columnscolumns_dictcountschemarowsrows_where(where, where_args, order_by, select)drop()
🗒️ Note: In the CLI: sqlite-utils views <#cli-views>
🐾 Listing rows
To iterate through dictionaries for each of the rows in a table, use .rows:
>>> db = sqlite_utils.Database("dogs.db")
>>> for row in db.table("dogs").rows:
... print(row)
{'id': 1, 'age': 4, 'name': 'Cleo'}
{'id': 2, 'age': 2, 'name': 'Pancakes'}You can filter rows by a WHERE clause using .rows_where(where, where_args):
>>> db = sqlite_utils.Database("dogs.db")
>>> for row in db.table("dogs").rows_where("age > ?", [3]):
... print(row)
{'id': 1, 'age': 4, 'name': 'Cleo'}So the first argument is a fragment of SQL. The second, optional argument is values to be passed to that fragment — you can use ? placeholders and pass an array, or you can use :named parameters and pass a dictionary, like this:
>>> for row in db.table("dogs").rows_where("age > :age", {"age": 3}):
... print(row)
{'id': 1, 'age': 4, 'name': 'Cleo'}Wanna return custom columns instead of the default that uses select *? Just pass select="column1, column2":
>>> db = sqlite_utils.Database("dogs.db")
>>> for row in db.table("dogs").rows_where(select='name, age'):
... print(row)
{'name': 'Cleo', 'age': 4}To specify an order, use the order_by= argument:
>>> for row in db.table("dogs").rows_where("age > 1", order_by="age"):
... print(row)
{'id': 2, 'age': 2, 'name': 'Pancakes'}
{'id': 1, 'age': 4, 'name': 'Cleo'}And yes bestie, you can use order_by="age desc" for descending order. 🔽
You can even order all the records in the table by just excluding the where argument entirely:
>>> for row in db.table("dogs").rows_where(order_by="age desc"):
... print(row)
{'id': 1, 'age': 4, 'name': 'Cleo'}
{'id': 2, 'age': 2, 'name': 'Pancakes'}This method also accepts offset= and limit= arguments, for specifying an OFFSET and a LIMIT for the SQL query:
>>> for row in db.table("dogs").rows_where(order_by="age desc", limit=1):
... print(row)
{'id': 1, 'age': 4, 'name': 'Cleo'}🗒️ Note: In the CLI: sqlite-utils rows <#cli-rows>
🔢 Counting rows
To count the number of rows that would be returned by a where filter, use .count_where(where, where_args):
>>> db.table("dogs").count_where("age > ?", [1])
2🔑 Listing rows with their primary keys
Sometimes it's genuinely so useful to retrieve the primary key alongside each row, so you can pass that key (or primary key tuple) straight into the .get() or .update() methods. 🎀
The .pks_and_rows_where() method takes the same signature as .rows_where() (with the exception of the select= parameter) but returns a generator that yields pairs of (primary key, row dictionary).
The primary key value will usually be a single value, but it can also be a tuple if the table has a compound primary key.
And if the table is a rowid table (aka it has no explicit primary key column), then that ID will be returned instead.
>>> db = sqlite_utils.Database(memory=True)
>>> db.table("dogs").insert({"name": "Cleo"})
>>> for pk, row in db.table("dogs").pks_and_rows_where():
... print(pk, row)
1 {'rowid': 1, 'name': 'Cleo'}
>>> db.table("dogs_with_pk").insert({"id": 5, "name": "Cleo"}, pk="id")
>>> for pk, row in db.table("dogs_with_pk").pks_and_rows_where():
... print(pk, row)
5 {'id': 5, 'name': 'Cleo'}
>>> db.table("dogs_with_compound_pk").insert(
... {"species": "dog", "id": 3, "name": "Cleo"},
... pk=("species", "id")
... )
>>> for pk, row in db.table("dogs_with_compound_pk").pks_and_rows_where():
... print(pk, row)
('dog', 3) {'species': 'dog', 'id': 3, 'name': 'Cleo'}🎯 Retrieving a specific record
You can retrieve a record by its primary key using table.get():
>>> db = sqlite_utils.Database("dogs.db")
>>> print(db.table("dogs").get(1))
{'id': 1, 'age': 4, 'name': 'Cleo'}If the table has a compound primary key, you can pass in the primary key values as a tuple:
>>> db.table("compound_dogs").get(("mixed", 3))And if the record does not exist, a NotFoundError will be raised:
from sqlite_utils.db import NotFoundError
try:
row = db.table("dogs").get(5)
except NotFoundError:
print("Dog not found")🧾 Showing the schema
The db.schema property returns the full SQL schema for the database as a string:
>>> db = sqlite_utils.Database("dogs.db")
>>> print(db.schema)
CREATE TABLE "dogs" (
"id" INTEGER PRIMARY KEY,
"name" TEXT
);🗒️ Note: In the CLI: sqlite-utils schema <#cli-schema>
🛠️ Creating tables
The easiest way to create a new table? Just insert a record into it. Fr fr. 💖
from sqlite_utils import Database
import sqlite3
db = Database("dogs.db")
dogs = db.table("dogs")
dogs.insert({
"name": "Cleo",
"twitter": "cleopaws",
"age": 3,
"is_good_dog": True,
})This will automatically create a new table called "dogs" with the following schema:
CREATE TABLE dogs (
name TEXT,
twitter TEXT,
age INTEGER,
is_good_dog INTEGER
)You can also specify a primary key by passing the pk= parameter to the .insert() call. Heads up though — this will only be obeyed if the record being inserted is what causes the table to be created:
dogs.insert({
"id": 1,
"name": "Cleo",
"twitter": "cleopaws",
"age": 3,
"is_good_dog": True,
}, pk="id")After inserting a row like this, the dogs.last_rowid property will return the SQLite rowid assigned to the most recently inserted record. ✨
And the dogs.last_pk property will return the last inserted primary key value, if you specified one. This can be super duper useful when you're writing code that creates foreign keys or many-to-many relationships. 🔗
🎨 Custom column order and column types
The order of the columns in your table is derived from the order of the keys in the dictionary — provided you're on Python 3.6 or later. 🐍
Wanna explicitly set the order of the columns yourself? Do it with the column_order= parameter:
db.table("dogs").insert({
"id": 1,
"name": "Cleo",
"twitter": "cleopaws",
"age": 3,
"is_good_dog": True,
}, pk="id", column_order=("id", "twitter", "name"))You don't need to pass all the columns to the column_order parameter, bestie. If you only pass a subset of the columns, the remaining columns will be ordered based on the key order of the dictionary. 🌈
Now, column types get detected based on the example data you provide. But sometimes you'll find you need to over-ride those detected types — to create an integer column for data that was provided as a string, for example, or to make sure that a table where the first example was None gets created as an INTEGER rather than a TEXT column. You can do alllll of that using the columns= parameter:
db.table("dogs").insert({
"id": 1,
"name": "Cleo",
"age": "5",
}, pk="id", columns={"age": int, "weight": float})This will create a table with the following schema:
CREATE TABLE "dogs" (
"id" INTEGER PRIMARY KEY,
"name" TEXT,
"age" INTEGER,
"weight": REAL
)🏗️ Explicitly creating a table
You can directly create a new table without inserting any data into it at all, using the .create() method:
db.table("cats").create({
"id": int,
"name": str,
"weight": float,
}, pk="id")The first argument here is a dictionary specifying the columns you'd like to create. Each column is paired with a Python type indicating the type of column. And if you want the full details on exactly how these types work, go check out Adding columns. 💅
This method takes optional arguments pk=, column_order=, foreign_keys=, not_null=set() and defaults=dict() — and bestie, I'm gonna explain all of them below, don't even stress 💖
A sqlite_utils.utils.sqlite3.OperationalError will be raised if a table of that name already exists. Rude of it? Maybe. But it's just looking out for you.
You can pass ignore=True to ignore that error. You can also use if_not_exists=True to use the SQL CREATE TABLE IF NOT EXISTS pattern to achieve the same effect:
db.table("cats").create({
"id": int,
"name": str,
}, pk="id", if_not_exists=True)To drop and replace any existing table of that name, pass replace=True. This is a dangerous operation that will result in loss of existing data in the table. Like, actual data, gone, poof 💀 be so careful.
You can also pass transform=True to have any existing tables transformed to match your new table specification. This is a dangerous operation as it will drop columns that are no longer listed in your call to .create(), so be careful when running this. Not me warning you twice, it's just giving "we love you, please don't delete your columns" energy 🙈
db.table("cats").create({
"id": int,
"name": str,
"weight": float,
}, pk="id", transform=True)The transform=True option will update the table schema if any of the following have changed:
- The specified columns or their types
- The specified primary key
- The order of the columns, defined using
column_order= - The
not_null=ordefaults=arguments
Changes to foreign_keys= are not currently detected and applied by transform=True. Lowkey a known gap, no cap 👑
You can pass strict=True to create a table in STRICT mode:
db.table("cats").create({
"id": int,
"name": str,
}, strict=True)SQLite STRICT tables can use the ANY column type for values that should retain their exact SQLite storage class without coercion. Use the sqlite_utils.ANY marker type:
import sqlite_utils
db.table("events").create({
"id": int,
"payload": sqlite_utils.ANY,
}, pk="id", strict=True)An ANY column can store integers, floating point values, text, binary data or None. In a STRICT table a text value such as "000123" remains text with its leading zeroes intact — iconic, the zeroes are safe ✨ SQLite also accepts ANY columns in ordinary non-STRICT tables, but those columns apply numeric affinity and would store that same value as the integer 123. So like, same value, different vibe entirely.
Note: In the CLI: sqlite-utils create-table <#cli-create-table>
💫 Compound primary keys 💫
If you want to create a table with a compound primary key that spans multiple columns, you can do so by passing a tuple of column names to any of the methods that accept a pk= parameter. For example:
db.table("cats").create({
"id": int,
"breed": str,
"name": str,
"weight": float,
}, pk=("breed", "id"))This also works for the .insert(), .insert_all(), .upsert() and .upsert_all() methods. Equal opportunity primary keys 🫶
🌸 Specifying foreign keys 🌸
Any operation that can create a table (.create(), .insert(), .insert_all(), .upsert() and .upsert_all()) accepts an optional foreign_keys= argument which can be used to set up foreign key constraints for the table that is being created.
If you are using your database with [Datasette](https://datasette.io/), Datasette will detect these constraints and use them to generate hyperlinks to associated records. The way that it just... links things together? Obsessed.
The foreign_keys argument takes a list that indicates which foreign keys should be created. The list can take several forms. The simplest is a list of columns:
foreign_keys=["author_id"]The library will guess which tables you wish to reference based on the column names using the rules described in Adding foreign key constraints.
You can also be more explicit, by passing in a list of tuples:
foreign_keys=[
("author_id", "authors", "id")
]This means that the author_id column should be a foreign key that references the id column in the authors table.
You can leave off the third item in the tuple to have the referenced column automatically set to the primary key of that table. A full example:
db.table("authors").insert_all([
{"id": 1, "name": "Sally"},
{"id": 2, "name": "Asheesh"}
], pk="id")
db.table("books").insert_all([
{"title": "Hedgehogs of the world", "author_id": 1},
{"title": "How to train your wolf", "author_id": 2},
], foreign_keys=[
("author_id", "authors")
])🦋 Compound foreign keys 🦋
To create a compound (multi-column) foreign key, use tuples of column names in place of the single column names:
db.table("courses").create({
"course_code": str,
"campus_name": str,
"dept_code": str,
}, pk="course_code", foreign_keys=[
(("campus_name", "dept_code"), "departments", ("campus_name", "dept_code"))
])This creates a table-level constraint:
CREATE TABLE "courses" (
"course_code" TEXT PRIMARY KEY,
"campus_name" TEXT,
"dept_code" TEXT,
FOREIGN KEY ("campus_name", "dept_code") REFERENCES "departments"("campus_name", "dept_code")
)As with single columns, you can leave off the tuple of other columns to reference the compound primary key of the other table:
foreign_keys=[
(("campus_name", "dept_code"), "departments")
]To specify ON DELETE or ON UPDATE actions, pass ForeignKey objects instead:
from sqlite_utils.db import ForeignKey
db.table("books").create({
"id": int,
"author_id": int,
}, pk="id", foreign_keys=[
ForeignKey(
table="books", column="author_id",
other_table="authors", other_column="id",
on_delete="CASCADE",
)
])Foreign key actions are preserved by table.transform() — prior to sqlite-utils 4.0 they were silently dropped when a table was transformed. Silently?? Dropped?? The audacity of the old days 😭 gone but not forgotten.
🎀 Table configuration options 🎀
The .insert(), .upsert(), .insert_all() and .upsert_all() methods each take a number of keyword arguments, some of which influence what happens should they cause a table to be created and some of which affect the behavior of those methods.
You can set default values for these methods by accessing the table through the db.table(...) method (instead of using db.table("table_name")), like so:
table = db.table(
"authors",
pk="id",
not_null={"name", "score"},
column_order=("id", "name", "score", "url")
)
# Now you can call .insert() like so:
table.insert({"id": 1, "name": "Tracy", "score": 5})The configuration options that can be specified in this way are pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, hash_id_columns, alter, ignore, replace, extracts, conversions, columns, strict. These are all documented below. Yes, all fifteen of them, we're not skipping a single one 💅
🌈 Setting defaults and not null constraints 🌈
Each of the methods that can cause a table to be created take optional arguments not_null=set() and defaults=dict(). The methods that take these optional arguments are:
db.create_table(...)table.create(...)table.insert(...)table.insert_all(...)table.upsert(...)table.upsert_all(...)
You can use not_null= to pass a set of column names that should have a NOT NULL constraint set on them when they are created.
You can use defaults= to pass a dictionary mapping columns to the default value that should be specified in the CREATE TABLE statement.
Here's an example that uses these features:
db.table("authors").insert_all(
[{"id": 1, "name": "Sally", "score": 2}],
pk="id",
not_null={"name", "score"},
defaults={"score": 1},
)
db.table("authors").insert({"name": "Dharma"})
list(db.table("authors").rows)
# Outputs:
# [{'id': 1, 'name': 'Sally', 'score': 2},
# {'id': 3, 'name': 'Dharma', 'score': 1}]
print(db.table("authors").schema)
# Outputs:
# CREATE TABLE "authors" (
# "id" INTEGER PRIMARY KEY,
# "name" TEXT NOT NULL,
# "score" INTEGER NOT NULL DEFAULT 1
# )Note: In the CLI:sqlite-utils insert --not-nulland--default<#cli-defaults-not-null>
⭐ Renaming a table ⭐
The db.rename_table(old_name, new_name) method can be used to rename a table:
db.rename_table("my_table", "new_name_for_my_table")This executes the following SQL:
ALTER TABLE [my_table] RENAME TO [new_name_for_my_table]Note: In the CLI: sqlite-utils rename-table <#cli-renaming-tables>
💜 Duplicating tables 💜
The table.duplicate() method creates a copy of the table, copying both the table schema and all of the rows in that table:
db.table("authors").duplicate("authors_copy")The new authors_copy table will now contain a duplicate copy of the data from authors. It's giving twin energy, fr fr.
This method raises sqlite_utils.db.NoTable if the table does not exist. Like, it's not gonna make something out of nothing babe.
Note: In the CLI: sqlite-utils duplicate <#cli-duplicate-table>
🔥 Bulk inserts 🔥
If you have more than one record to insert, the insert_all() method is a much more efficient way of inserting them. Just like insert() it will automatically detect the columns that should be created, but it will inspect the first batch of 100 items to help decide what those column types should be.
Use it like this:
db.table("dogs").insert_all([{
"id": 1,
"name": "Cleo",
"twitter": "cleopaws",
"age": 3,
"is_good_dog": True,
}, {
"id": 2,
"name": "Marnie",
"twitter": "MarnieTheDog",
"age": 16,
"is_good_dog": True,
}], pk="id", column_order=("id", "twitter", "name"))The column types used in the CREATE TABLE statement are automatically derived from the types of data in that first batch of rows. Any additional columns in subsequent batches will cause a sqlite3.OperationalError exception to be raised unless the alter=True argument is supplied, in which case the new columns will be created. Surprise columns? Not on this method's watch 🚫
The function can accept an iterator or generator of rows and will commit them according to the batch size. The default batch size is 100, but you can specify a different size using the batch_size parameter:
db.table("big_table").insert_all(({
"id": 1,
"name": "Name {}".format(i),
} for i in range(10000)), batch_size=1000)You can skip inserting any records that have a primary key that already exists using ignore=True. This works with both .insert({...}, ignore=True) and .insert_all([...], ignore=True).
You can delete all the existing rows in the table before inserting the new records using truncate=True. This is useful if you want to replace the data in the table. Out with the old, in with the new, main character energy 💅
Pass analyze=True and SQLite runs ANALYZE against the table right after those new records land. Chef's kiss for query planning. 💅
💡 Note: over in CLI-land that's sqlite-utils insert <#cli-inserting-data>
📋 Inserting data from a list or tuple iterator
So dicts are the vibe, but bestie, you can also feed it an iterator of lists or tuples instead. The tea here: the first item your iterator yields has to be a list or tuple of string column names, and everything after that is lists or tuples of actual values:
db["creatures"].insert_all([
["name", "species"],
["Cleo", "dog"],
["Lila", "chicken"],
["Bants", "chicken"],
])Generators? Also slay:
def creatures():
yield "id", "name", "city"
yield 1, "Cleo", "San Francisco"
yield 2, "Lila", "Los Angeles"
db["creatures"].insert_all(creatures())Tuples and lists are both supported. No discrimination here, we love them equally. 🫶
🔁 Insert-replacing data
Okay so here's the drama: if you try to insert data with a primary key that already exists, .insert() or .insert_all() will throw a sqlite3.IntegrityError exception at you. Rude, but fair.
Here's a little example that catches that exception like a pro:
from sqlite_utils.utils import sqlite3
try:
db.table("dogs").insert({"id": 1, "name": "Cleo"}, pk="id")
except sqlite3.IntegrityError:
print("Record already exists with that primary key")Lowkey important: importing from sqlite_utils.utils.sqlite3 keeps your code working even if you're using the pysqlite3 library instead of the Python standard library sqlite3 module. Future-proofing, we love to see it. ✨
Want to just ignore that error instead? Pass ignore=True:
# This fails silently if a record with id=1 already exists
db.table("dogs").insert({"id": 1, "name": "Cleo"}, pk="id", ignore=True)And if you want to replace any existing records that have a matching primary key, hit it with replace=True on .insert() or .insert_all():
db.table("dogs").insert_all([{
"id": 1,
"name": "Cleo",
"twitter": "cleopaws",
"age": 3,
"is_good_dog": True,
}, {
"id": 2,
"name": "Marnie",
"twitter": "MarnieTheDog",
"age": 16,
"is_good_dog": True,
}], pk="id", replace=True)💀 Note: back before sqlite-utils 2.0, the.upsert()and.upsert_all()methods worked exactly the same way as.insert(replace=True)does today. Check out Upserting data for the new behaviour those methods got in 2.0.
💡 Note: in the CLI that's sqlite-utils insert --replace <#cli-insert-replace>
✏️ Updating a specific record
You can update a record by its primary key using table.update():
>>> db = sqlite_utils.Database("dogs.db")
>>> print(db.table("dogs").get(1))
{'id': 1, 'age': 4, 'name': 'Cleo'}
>>> db.table("dogs").update(1, {"age": 5})
>>> print(db.table("dogs").get(1))
{'id': 1, 'age': 5, 'name': 'Cleo'}The first argument to update() is the primary key. Could be a single value, or a tuple if that table has a compound primary key:
>>> db.table("compound_dogs").update((5, 3), {"name": "Updated"})The second argument is a dictionary of the columns that should be updated, along with their brand new values.
And if you want any missing columns to just magically add themselves, there's alter=True:
>>> db.table("dogs").update(1, {"breed": "Mutt"}, alter=True)🗑️ Deleting a specific record
You can delete a record using table.delete():
>>> db = sqlite_utils.Database("dogs.db")
>>> db.table("dogs").delete(1)The delete() method takes the primary key of the record. And yes, that can be a tuple of values if the row has a compound primary key:
>>> db.table("compound_dogs").delete((5, 3))🗑️✨ Deleting multiple records
Feeling destructive? You can delete all the records in a table that match a specific WHERE statement using table.delete_where():
>>> db = sqlite_utils.Database("dogs.db")
>>> # Delete every dog with age less than 3
>>> db.table("dogs").delete_where("age < ?", [3])Call table.delete_where() with no other arguments and it will delete every single row in the table. Gone. Poof. 😭
Pass analyze=True to run ANALYZE against the table after deleting the rows.
🔄 Upserting data
Upserting is that "insert if they don't exist, update them if they DO exist" energy, matched against their primary key. Main character behaviour honestly.
For example, given the dogs database you could upsert the record for Cleo like so:
db.table("dogs").upsert({
"id": 1,
"name": "Cleo",
"twitter": "cleopaws",
"age": 4,
"is_good_dog": True,
}, pk="id", column_order=("id", "twitter", "name"))If a record already exists with id=1, it gets updated to match those fields. If it does not exist, it gets created. Simple as that. ✨
Any existing columns that are not referenced in the dictionary passed to .upsert() will stay unchanged. So if you want to replace a record entirely, use .insert(doc, replace=True) instead.
Real talk: the pk and column_order parameters here are optional if you're certain the table has already been created. But you should pass them if the table might not exist yet at the time the first upsert is performed.
There's also an upsert_all() method, which behaves like insert_all() but performs upserts instead.
Important: every record you pass to upsert() or upsert_all() must include a value for each primary key column. A record without one could never match an existing row, so instead of quietly inserting a new row, a sqlite_utils.db.PrimaryKeyRequired exception gets raised. No sneaky business. 🚫
💀 Note:.upsert()and.upsert_all()in sqlite-utils 1.x worked like.insert(..., replace=True)and.insert_all(..., replace=True)do in 2.x. See issue #66https://github.com/simonw/sqlite-utils/issues/66for details of this change.
💡 Note: in the CLI that's sqlite-utils upsert <#cli-upsert>
🕰️ Alternative upserts using INSERT OR IGNORE
Upserts use INSERT INTO ... ON CONFLICT SET. But prior to sqlite-utils 4.0, these used a sequence of INSERT OR IGNORE followed by an UPDATE. That older method is still used for SQLite 3.23.1 and earlier. If you want to force the older implementation, pass use_old_upsert=True to the Database() constructor. Retro vibes, on purpose. 🧚
🎀 Converting data in columns
The table.convert(...) method can be used to apply a conversion function to the values in a column — either to update that column or to populate new ones. It's the Python library equivalent of the sqlite-utils convert <#cli-convert> command.
How does the magic work? It registers a custom SQLite function that applies a Python transformation, then runs a SQL query equivalent to UPDATE table SET column = convert_value(column); 🪄
To transform a specific column to uppercase, you'd do:
db.table("dogs").convert("name", lambda value: value.upper())You can pass a list of columns too, in which case the transformation gets applied to each one:
db.table("dogs").convert(["name", "twitter"], lambda value: value.upper())Want to save the output of the transformation to a different column? Use the output= parameter:
db.table("dogs").convert("name", lambda value: value.upper(), output="name_upper")That will add the new column if it does not already exist. You can pass output_type=int or some other type to control the type of the new column — otherwise it defaults to text.
And if you want to drop the original column after saving the results in a separate output column, pass drop=True.
You can even create multiple new columns from a single input column by passing multi=True plus a conversion function that returns a Python dictionary. This example creates new upper and lower columns populated from the single title column:
table.convert(
"title", lambda v: {"upper": v.upper(), "lower": v.lower()}, multi=True
)The .convert() method also accepts optional where= and where_args= parameters, which let you apply the conversion to just a subset of rows specified by a where clause. Here's how to apply the conversion only to rows with an id that's higher than 20:
table.convert("title", lambda v: v.upper(), where="id > :id", where_args={"id": 20})These behave the same as the corresponding parameters to the .rows_where() method, so you can use ? placeholders and a list of values instead of :named placeholders with a dictionary. Flexible era. 💅
🔍 Working with lookup tables
Okay so this is a genuinely useful pattern: when you're populating large tables, break common values out into lookup tables. Picture a table of Trees, where every tree has a species. Ideally those species get split out into a separate Species table, with each one assigned an integer primary key that can be referenced from the Trees table's species_id column. Normalisation, but make it cute. 🌸
🌴 Creating lookup tables explicitly
Calling db.table("Species").lookup({"name": "Palm"}) creates a table called Species (if one doesn't already exist) with two columns: id and name. It sets up a unique constraint on the name column so it's guaranteed not to contain duplicate rows. Then it inserts a new row with the name set to Palm and returns that new integer primary key value. Iconic.
If the Species table already exists, it'll insert the new row and return the primary key. And if a row with that name already exists? It just returns the corresponding primary key value directly. Efficient. ✨
If you call .lookup() against an existing table without the unique constraint, it will attempt to add the constraint, raising an IntegrityError if the constraint can't be created.
Pass in a dictionary with multiple values and both values get used to insert or retrieve the corresponding ID — and any unique constraint that gets created will cover all of those columns, for example:
db.table("Trees").insert({
"latitude": 49.1265976,
"longitude": 2.5496218,
"species": db.table("Species").lookup({
"common_name": "Common Juniper",
"latin_name": "Juniperus communis"
})
})The .lookup() method has an optional second argument which can be used to populate other columns in the table — but only if the row does not exist yet. Heads up: these columns will not be included in the unique index.
So to create a species record with a note on when it was first seen, you can use this:
db.table("Species").lookup({"name": "Palm"}, {"first_seen": "2021-03-04"})The first time this is called, the record gets created for name="Palm". Any subsequent calls with that same name will ignore the second argument, even if it includes different values. One and done. 🙈
None values are matched correctly too: calling .lookup() a second time with the same values will return the primary key of the existing row even if some of those values are None. No cap.
.lookup() also accepts keyword arguments, which get passed through to the insert() method and can be used to influence the shape of the created table. Supported parameters are:
pk— which defaults toidforeign_keyscolumn_ordernot_nulldefaultsextractsconversionscolumnsstrict
🦋 Populating lookup tables automatically during insert/upsert 🦋
okay so bestie, the big brain way to work with lookup tables is to define them using the extracts= parameter. It's accepted by .insert(), .upsert(), .insert_all(), .upsert_all() and by the .table(...) factory function. slay. 💅
extracts= tells it which columns should be "extracted" out into their own separate lookup table while the data is getting inserted.
And it can be either of these two things:
- a list of column names — in which case the extracted table names will match the column names exactly
- a dictionary mapping column names to the desired name of the extracted table
So if you wanna extract that species column out to a separate Species table, you do this:
# Using the table factory
trees = db.table("Trees", extracts={"species": "Species"})
trees.insert({
"latitude": 49.1265976,
"longitude": 2.5496218,
"species": "Common Juniper"
})
# If you want the table to be called 'species', you can do this:
trees = db.table("Trees", extracts=["species"])
# Using .insert() directly
db.table("Trees").insert({
"latitude": 49.1265976,
"longitude": 2.5496218,
"species": "Common Juniper"
}, extracts={"species": "Species"})Now listen closely because this part is important: None values are not extracted. No record gets created for them in the lookup table, and the column value just stays null. the way it should be. 🙈
👑 Working with many-to-many relationships 👑
sqlite-utils has a little shortcut for creating records with many-to-many relationships and it's called table.m2m(...). iconic behaviour honestly.
Here's how you create two brand new records and connect them through a many-to-many table in a single line of code:
db.table("dogs").insert({"id": 1, "name": "Cleo"}, pk="id").m2m(
"humans", {"id": 1, "name": "Natalie"}, pk="id"
)Running this example actually creates three tables: dogs, humans and a many-to-many dogs_humans table. It inserts one record into each of them. multitasking queen. ✨
The .m2m() method runs against the last record that was affected by .insert() or .update() — the record identified by the table.last_pk property. So if you want .m2m() to run against one specific record, select it first by passing its primary key to .update():
db.table("dogs").update(1).m2m(
"humans", {"id": 2, "name": "Simon"}, pk="id"
)The first argument to .m2m() can be either the name of a table as a string, or the table object itself. either one, we don't gatekeep. 🎀
The second argument can be a single dictionary record, or a list of dictionaries. Those dictionaries get passed straight to .upsert() against the table you specified.
Here's the alternative code that creates the dog record and adds two whole people to it:
db = Database(memory=True)
dogs = db.table("dogs", pk="id")
humans = db.table("humans", pk="id")
dogs.insert({"id": 1, "name": "Cleo"}).m2m(
humans, [
{"id": 1, "name": "Natalie"},
{"id": 2, "name": "Simon"}
]
)So the method goes looking for an existing many-to-many table by hunting for a table that has foreign key relationships against both of the tables in the relationship. detective mode. 🔍
If it can't find such a table, it creates a new one using the names of the two tables — dogs_humans in this example. You can customize the name of this table with the m2m_table= argument to .m2m().
⚠️ But if it finds multiple candidate tables with foreign keys to both of the specified tables, it will raise asqlite_utils.db.NoObviousTableexception. You can dodge that error by specifying the correct table usingm2m_table=.
The .m2m() method also takes an optional pk= argument to specify the primary key that should be used if the table gets created, plus an optional alter=True argument to say that any missing columns of an existing table should be added if they're needed. so considered. 💖
💜 Using m2m and lookup tables together 💜
You can work with (or create!) lookup tables as part of a call to .m2m() using the lookup= parameter. It accepts the same argument as table.lookup() does — a dictionary of values that should be used to look up or create a row in the lookup table.
This example creates a dogs table, populates it, creates a characteristics table, populates that, and sets up a many-to-many relationship between the two. Check it — it chains .m2m() twice to create two associated characteristics:
db = Database(memory=True)
dogs = db.table("dogs", pk="id")
dogs.insert({"id": 1, "name": "Cleo"}).m2m(
"characteristics", lookup={
"name": "Playful"
}
).m2m(
"characteristics", lookup={
"name": "Opinionated"
}
)And you can inspect the database to see the results like this:
>>> db.table_names()
['dogs', 'characteristics', 'characteristics_dogs']
>>> list(db.table("dogs").rows)
[{'id': 1, 'name': 'Cleo'}]
>>> list(db.table("characteristics").rows)
[{'id': 1, 'name': 'Playful'}, {'id': 2, 'name': 'Opinionated'}]
>>> list(db.table("characteristics_dogs").rows)
[{'characteristics_id': 1, 'dogs_id': 1}, {'characteristics_id': 2, 'dogs_id': 1}]
>>> print(db.table("characteristics_dogs").schema)
CREATE TABLE "characteristics_dogs" (
"characteristics_id" INTEGER REFERENCES "characteristics"("id"),
"dogs_id" INTEGER REFERENCES "dogs"("id"),
PRIMARY KEY ("characteristics_id", "dogs_id")
)look at that schema. gorgeous. 😭
🎀 Analyzing a column 🎀
The table.analyze_column(column) method is what the analyze-tables CLI command uses under the hood.
It takes these arguments and options:
column— required. The name of the column to analyze.common_limit— the number of most common values to return. Defaults to10.value_truncate— if set to an integer, values longer than this get truncated to this length. Defaults toNone.most_common— if set toFalse, themost_commonfield of the returnedColumnDetailswill be set toNone. Defaults toTrue.least_common— if set toFalse, theleast_commonfield of the returnedColumnDetailswill be set toNone. Defaults toTrue.
And it returns a ColumnDetails named tuple with these fields:
table— the name of the tablecolumn— the name of the columntotal_rows— the total number of rows in the tablenum_null— the number of rows for which this column is nullnum_blank— the number of rows for which this column is blank (the empty string)num_distinct— the number of distinct values in this columnmost_common— the N most common values as a list of(value, count)tuples, orNoneif the table consists entirely of distinct valuesleast_common— the N least common values as a list of(value, count)tuples, orNoneif the table is entirely distinct or if the number of distinct values is less than N (since they will already have been returned inmost_common)
🌸 Adding columns 🌸
You can add a new column to a table with the .add_column(col_name, col_type) method:
db.table("dogs").add_column("instagram", str)
db.table("dogs").add_column("weight", float)
db.table("dogs").add_column("dob", datetime.date)
db.table("dogs").add_column("image", "BLOB")
db.table("dogs").add_column("website") # str by defaultYou can specify that col_type argument either by using a SQLite type as a string, or by directly passing a Python type like str or float. either way bestie, it's giving flexible. ✨
The col_type is optional — if you omit it, the type of TEXT gets used.
SQLite types you can specify are "TEXT", "INTEGER", "FLOAT", "REAL", "BLOB" or "ANY". And you can use the sqlite_utils.ANY marker instead of the "ANY" string for extra style points.
If you pass a Python type, it gets mapped to SQLite types as shown here:
float: "REAL"
int: "INTEGER"
bool: "INTEGER"
str: "TEXT"
bytes: "BLOB"
datetime.datetime: "TEXT"
datetime.date: "TEXT"
datetime.time: "TEXT"
datetime.timedelta: "TEXT"
sqlite_utils.ANY: "ANY"
# If numpy is installed
np.int8: "INTEGER"
np.int16: "INTEGER"
np.int32: "INTEGER"
np.int64: "INTEGER"
np.uint8: "INTEGER"
np.uint16: "INTEGER"
np.uint32: "INTEGER"
np.uint64: "INTEGER"
np.float16: "REAL"
np.float32: "REAL"
np.float64: "REAL"You can also add a column that is a foreign key reference to another table using the fk parameter:
db.table("dogs").add_column("species_id", fk="species")This automatically detects the name of the primary key on the species table and uses that (plus its type) for the new column. it just knows. 🔮
And you can explicitly specify the column you wish to reference using fk_col:
db.table("dogs").add_column("species_id", fk="species", fk_col="ref")You can set a NOT NULL DEFAULT 'x' constraint on the new column using not_null_default:
db.table("dogs").add_column("friends_count", int, not_null_default=0)Note: in the CLI there's sqlite-utils add-column for this.
✨ Adding columns automatically on insert/update ✨
You can insert or update data that includes new columns and have the table automatically altered to fit the new schema using the alter=True argument. You can pass it to all four of .insert(), .upsert(), .insert_all() and .upsert_all(), or you can pass it to db.table(table_name, alter=True) to enable it by default for all method calls against that table instance.
db.table("new_table").insert({"name": "Gareth"})
# This will throw an exception:
db.table("new_table").insert({"name": "Gareth", "age": 32})
# This will succeed and add a new "age" integer column:
db.table("new_table").insert({"name": "Gareth", "age": 32}, alter=True)
# You can see confirm the new column like so:
print(db.table("new_table").columns_dict)
# Outputs this:
# {'name': <class 'str'>, 'age': <class 'int'>}
# This works too:
new_table = db.table("new_table", alter=True)
new_table.insert({"name": "Gareth", "age": 32, "shoe_size": 11})no migration pain, no drama. that's the vibe. 🫶
Note: in the CLI this is sqlite-utils insert --alter.
💅 Adding foreign key constraints 💅
real talk: the SQLite ALTER TABLE statement simply doesn't have the ability to add foreign key references to an existing column. it's not a skill issue, the feature just isn't there. 💀
The add_foreign_key() method here is a convenient wrapper around table.transform().
It's also totally possible to add foreign keys by going straight in and updating the sqlite_master table yourself. That's exactly what the sqlite-utils-fast-fks plugin does — it implements this pattern using code that shipped with sqlite-utils before version 3.35. 💅
Here's the mechanism in action, bestie:
db.table("authors").insert_all([
{"id": 1, "name": "Sally"},
{"id": 2, "name": "Asheesh"}
], pk="id")
db.table("books").insert_all([
{"title": "Hedgehogs of the world", "author_id": 1},
{"title": "How to train your wolf", "author_id": 2},
])
db.table("books").add_foreign_key("author_id", "authors", "id")The table.add_foreign_key(column, other_table, other_column) method takes the name of the column, the table being referenced, and the key column inside that other table. Skip the other_column argument and it'll just use the primary key from that other table automatically. Skip other_table and it goes full detective mode, guessing based on some simple rules:
- If the column is formatted like
author_id, it looks for tables calledauthororauthors👀 - If the column does not end in
_id, it tries looking for a table with the exact name of the column, or that name with an addeds
Before it does anything, this method checks that the foreign key you specified references tables and columns that actually exist, and that it doesn't clash with an existing foreign key. If those checks fail it raises a sqlite_utils.db.AlterError exception. Rude, but fair. 🙈
Want to just ignore the case where the key already exists? Use ignore=True:
db.table("books").add_foreign_key("author_id", "authors", "id", ignore=True)For a compound foreign key, pass tuples of columns:
db.table("courses").add_foreign_key(
("campus_name", "dept_code"), "departments", ("campus_name", "dept_code")
)Same energy as single columns — omit the other columns and it'll use the compound primary key of the other table. But heads up: with a compound foreign key, other_table must always be specified. No guessing allowed. 🎀
Use on_delete= and on_update= to specify ON DELETE and ON UPDATE actions for the foreign key:
db.table("books").add_foreign_key(
"author_id", "authors", "id", on_delete="CASCADE"
)That right there creates a foreign key with an ON DELETE CASCADE clause, so deleting an author nukes their books too (as long as foreign key enforcement is switched on with PRAGMA foreign_keys = ON). Valid actions are "SET NULL", "SET DEFAULT", "CASCADE", "RESTRICT" and the default, "NO ACTION".
Note: In the CLI: sqlite-utils add-foreign-key <#cli-add-foreign-key>
✨ Adding multiple foreign key constraints at once ✨
You can use db.add_foreign_keys(...) to add multiple foreign keys in one single go, no cap. This method takes a list of four-tuples, and each one specifies a table, column, other_table and other_column.
Here's an example adding two foreign keys at once:
db.add_foreign_keys([
("dogs", "breed_id", "breeds", "id"),
("dogs", "home_town_id", "towns", "id")
])It runs the same checks as .add_foreign_keys() and will raise sqlite_utils.db.AlterError if those checks fail.
Foreign keys that already exist are silently skipped, so repeated calls are idempotent — but only if they match exactly. Requesting a foreign key that exists with different ON DELETE/ON UPDATE actions raises AlterError: use table.transform() to change the actions of an existing foreign key.
Note: In the CLI: sqlite-utils add-foreign-keys <#cli-add-foreign-keys>
🌈 Adding indexes for all foreign keys 🌈
If you want to make sure that every single foreign key column in your database has a corresponding index, you can do that like this:
db.index_foreign_keys()Compound foreign keys get a single composite index across their columns. Iconic behavior, honestly.
Note: In the CLI: sqlite-utils index-foreign-keys <#cli-index-foreign-keys>
🦋 Dropping a table or view 🦋
You can drop a table or view with the .drop() method:
db.table("my_table").drop()
# Or for a view:
db.view("my_view").drop()
# Or for either:
db["table_or_view_name"].drop()Pass ignore=True if you want to ignore the error you'd get from the table or view not existing:
db.table("my_table").drop(ignore=True)Note: In the CLI:sqlite-utils drop-table <#cli-drop-table>andsqlite-utils drop-view <#cli-drop-view>
💅 Transforming a table 💅
The SQLite ALTER TABLE statement is... limited, bestie. It can add and drop columns, and rename tables, but it literally cannot change column types, change NOT NULL status, or change the primary key for a table. It's giving minimal effort. 💀
The table.transform() method can do all of those things, by implementing a multi-step pattern described in the SQLite documentation (https://www.sqlite.org/lang_altertable.html#otheralter):
- Start a transaction
CREATE TABLE tablename_new_x123with the required changes- Copy the old data into the new table using
INSERT INTO tablename_new_x123 SELECT * FROM tablename; DROP TABLE tablename;ALTER TABLE tablename_new_x123 RENAME TO tablename;- Commit the transaction
The .transform() method takes a number of parameters, and every single one of them is optional.
As a little bonus, calling .transform() will reformat the schema for that table stored in SQLite to make it more readable. It does this even if you call it without any arguments at all. Free glow-up. ✨
To keep the original table around instead of dropping it, pass the keep_table= option and specify the name you'd like it renamed to:
table.transform(types={"age": int}, keep_table="original_table")This method raises a sqlite_utils.db.TransformError exception if the table can't be transformed — usually because there are existing constraints or indexes that are incompatible with the modifications you're making to the columns.
Note: In the CLI: sqlite-utils transform <#cli-transform-table>
🧚 Altering column types 🧚
To alter the type of a column, use the types= argument:
# Convert the 'age' column to an integer, and 'weight' to a float
table.transform(types={"age": int, "weight": float})When a TEXT column is changed to INTEGER, FLOAT or REAL, exact empty-string values are stored as NULL. Other values, including whitespace-only strings, get copied normally.
See Adding columns for a list of available types.
🌸 Changing strict mode 🌸
The optional strict= parameter can change whether a table uses SQLite STRICT mode (https://www.sqlite.org/stricttables.html). Pass strict=True to convert a regular table to a strict table:
table.transform(strict=True)Pass strict=False to convert a strict table back into a regular, non-strict table:
table.transform(strict=False)If the table has ANY columns, converting it to non-strict mode can coerce text values that look numeric. For example, SQLite converts "000123" to the integer 123 when copying it into an ordinary ANY column. That's SQLite's documented distinction between STRICT and ordinary ANY columns (https://www.sqlite.org/stricttables.html#the_any_datatype).
The default is strict=None, which preserves whatever strict mode the table already has.
Passing strict=True raises sqlite_utils.db.TransformError if the available SQLite version doesn't support strict tables. No strict mode for you. 😭
Converting to a strict table validates all existing rows as they're copied into the replacement table. If a value is incompatible with its declared column type, SQLite raises sqlite3.IntegrityError and the transformation is rolled back, leaving the original table and its data completely unchanged.
👑 Renaming columns 👑
The rename= parameter can rename columns:
# Rename 'age' to 'initial_age':
table.transform(rename={"age": "initial_age"})🍓 Dropping columns 🍓
To drop columns, pass them in the drop= set:
# Drop the 'age' column:
table.transform(drop={"age"})⭐ Changing primary keys ⭐
To change the primary key for a table, use pk=. You can hand it a single column for a regular primary key, or a tuple of columns to create a compound primary key. Passing pk=None will remove the primary key entirely and convert the table into a rowid table.
# Make `user_id` the new primary key
table.transform(pk="user_id")💜 Changing not null status 💜
You can change the NOT NULL status of columns using not_null=. Pass it a set of columns to make those columns NOT NULL:
# Make the 'age' and 'weight' columns NOT NULL
table.transform(not_null={"age", "weight"})If you want to take existing NOT NULL columns and change them so they allow null values, pass a dictionary of true/false values instead:
# 'age' is NOT NULL but we want to allow NULL:
table.transform(not_null={"age": False})
# Make age allow NULL and switch weight to being NOT NULL:
table.transform(not_null={"age": False, "weight": True})🎀 Altering column defaults 🎀
The defaults= parameter can set or change the defaults for different columns:
# Set default age to 1:
table.transform(defaults={"age": 1})
# Now remove the default from that column:
table.transform(defaults={"age": None})🔥 Changing column order 🔥
The column_order= parameter can change the order of the columns. If you pass the names of a subset of the columns, those ones go first, and any columns you left out appear in their existing order after them.
# Change column order
table.transform(column_order=("name", "age", "id")🫶 Adding foreign key constraints 🫶
You can add one or more foreign key constraints to a table using the add_foreign_keys= parameter:
db.table("places").transform(
add_foreign_keys=(
("country", "country", "id"),
("continent", "continent", "id")
)
)This accepts the same arguments described in specifying foreign keys — so you can specify them as a full tuple of (column, other_table, other_column), or you can take a shortcut and pass just the name of the column, as long as the table can be automatically derived from the column name:
db.table("places").transform(
add_foreign_keys=(("country", "continent"))
)✨ Replacing foreign key constraints ✨
The foreign_keys= parameter is similar to add_foreign_keys= but can be used to replace all foreign key constraints on a table, dropping any that aren't explicitly mentioned:
db.table("places").transform(
foreign_keys=(
("continent", "continent", "id"),
)
)💖 Dropping foreign key constraints 💖
You can use .transform() to remove foreign key constraints from a table.
This example drops two foreign keys — the one from places.country to country.id and the one from places.continent to continent.id:
db.table("places").transform(
drop_foreign_keys=("country", "continent")
)A bare column name drops any foreign key that column participates in, including compound foreign keys. So if you wanna snipe one specific compound foreign key instead of nuking everything that touches that column, hand over a tuple of its columns:
db.table("courses").transform(
drop_foreign_keys=[("campus_name", "dept_code")]
)Renaming a column with rename= updates every foreign key that uses it 🫶 — and dropping a column with drop= also drops any foreign keys it's part of, which for a compound foreign key means the whole constraint goes poof, not just part of it. Bestie be aware 💅
✨ CHECK constraints ✨
.transform() is a loyal queen and preserves both column-level and table-level CHECK constraints. And if a column gets renamed, the references to that column inside the check expression get renamed right along with it. Slay ✨
A column-level check gets removed if its owning column is dropped. But — and this is the important bit — if you try to drop a column that's referenced by any remaining check, it raises TransformError instead of creating a schema that's invalid or, like, sneakily weaker than you thought. No silent downgrades here, we don't do that 🙅♀️
Comments sitting immediately before or after a column definition are preserved too 💖. They move with that column if it gets renamed or reordered, and they're removed if the column is dropped. A comment wedged between two column definitions counts as belonging to the following column, just so you know the vibes check is deterministic.
🌸 Tables referenced by views 🌸
Tables that are referenced by views can be transformed totally safely — the view definitions are left byte-for-byte unchanged, and the views keep reading from the live table even when you use keep_table= to keep a copy of the original hanging around. No cap.
A view that references a column the transform renamed or dropped will still exist, but it will raise a no such column error the next time it's queried 😭. That's just how SQLite views are built — the SQL is stored as text, so if you rename or drop columns a view depends on, you gotta go update that view definition yourself. It's giving "you broke it, you fix it" 🎀
To make all that work, the SQL produced by transform_sql() turns on PRAGMA legacy_alter_table for its ALTER TABLE ... RENAME TO statements, then restores the pragma back to whatever value it had when the SQL was generated. Why tho? Because without it, SQLite would try to rewrite references to the renamed table in every view definition — and that straight-up fails when a view references the table that was just dropped. 💀
🦋 Custom transformations with .transform_sql() 🦋
The .transform() method handles most cases bestie, but it does not automatically upgrade indexes, views or triggers associated with the table you're transforming. She's not doing all that.
So if you want to go more advanced, call table.transform_sql(...) with the same arguments you would've passed to table.transform(...). Same energy, different front door.
This method returns a list of SQL statements that should be executed to implement the change. Then you can tweak that SQL — or bolt on extra SQL statements of your own — before executing it yourself. Main character energy 🔥
💜 Foreign keys and transactions 💜
Okay this part matters, pay attention.
Because .transform() drops the old table, running it with PRAGMA foreign_keys enabled could fire ON DELETE actions on any tables that reference it — and an inbound ON DELETE CASCADE foreign key would silently delete those referencing rows 😱. Not okay. To prevent that, .transform() turns PRAGMA foreign_keys off for the duration of the operation and restores it afterwards, running PRAGMA foreign_key_check before committing.
Here's the catch though: PRAGMA foreign_keys cannot be changed inside a transaction. So this protection is literally impossible if you call .transform() while a transaction is already open — like inside a with db.atomic(): block or after db.begin(). If PRAGMA foreign_keys is on and another table references the table being transformed with a destructive ON DELETE action — CASCADE, SET NULL or SET DEFAULT — the method will refuse to run and raise a sqlite_utils.db.TransactionError:
from sqlite_utils.db import TransactionError
try:
with db.atomic():
db["authors"].transform(types={"id": str})
except TransactionError as ex:
print("Could not transform in transaction:", ex)To transform a table like that, either call .transform() outside the transaction, or execute PRAGMA foreign_keys = off before opening it:
db.execute("PRAGMA foreign_keys = off")
with db.atomic():
db["authors"].transform(types={"id": str})
db.execute("PRAGMA foreign_keys = on")Tables referenced by foreign keys without a destructive action (the default NO ACTION, or RESTRICT) can still be transformed inside a transaction — in that case sqlite-utils uses PRAGMA defer_foreign_keys to postpone the foreign key checks until the transaction commits. Deferred, not cancelled 🧚
👑 Extracting columns into a separate table 👑
The table.extract() method can be used to extract specified columns into a separate table. Normalize that data, bestie 📊
Imagine a Trees table that looks like this:
┌────┬──────────────┬─────────┐
│ id │ TreeAddress │ Species │
├────┼──────────────┼─────────┤
│ 1 │ 52 Vine St │ Palm │
├────┼──────────────┼─────────┤
│ 2 │ 12 Draft St │ Oak │
├────┼──────────────┼─────────┤
│ 3 │ 51 Dark Ave │ Palm │
├────┼──────────────┼─────────┤
│ 4 │ 1252 Left St │ Palm │
└────┴──────────────┴─────────┘The Species column is full of duplicate values (Palm Palm Palm... we see you 🌴). This database could be improved by extracting that column out into a separate Species table and pointing to it using a foreign key column.
The schema of the above table is:
CREATE TABLE "Trees" (
"id" INTEGER PRIMARY KEY,
"TreeAddress" TEXT,
"Species" TEXT
)Here's how to extract the Species column using .extract():
db.table("Trees").extract("Species")After running this code the table schema now looks like this:
CREATE TABLE "Trees" (
"id" INTEGER PRIMARY KEY,
"TreeAddress" TEXT,
"Species_id" INTEGER,
FOREIGN KEY(Species_id) REFERENCES Species(id)
)A new Species table will have been created with the following schema:
CREATE TABLE "Species" (
"id" INTEGER PRIMARY KEY,
"Species" TEXT
)The .extract() method defaults to creating a table with the same name as the column that was extracted, and adding a foreign key column called tablename_id. Cute defaults, honestly ✨
You can specify a custom table name using table=, and a custom foreign key name using fk_column=. This example creates a table called tree_species and a foreign key column called tree_species_id:
db.table("Trees").extract("Species", table="tree_species", fk_column="tree_species_id")The resulting schema looks like this:
CREATE TABLE "Trees" (
"id" INTEGER PRIMARY KEY,
"TreeAddress" TEXT,
"tree_species_id" INTEGER,
FOREIGN KEY(tree_species_id) REFERENCES tree_species(id)
)
CREATE TABLE "tree_species" (
"id" INTEGER PRIMARY KEY,
"Species" TEXT
)You can also extract multiple columns into the same external table. Say for example you have a table like this:
┌────┬──────────────┬────────────┬───────────┐
│ id │ TreeAddress │ CommonName │ LatinName │
├────┼──────────────┼────────────┼───────────┤
│ 1 │ 52 Vine St │ Palm │ Arecaceae │
├────┼──────────────┼────────────┼───────────┤
│ 2 │ 12 Draft St │ Oak │ Quercus │
├────┼──────────────┼────────────┼───────────┤
│ 3 │ 51 Dark Ave │ Palm │ Arecaceae │
├────┼──────────────┼────────────┼───────────┤
│ 4 │ 1252 Left St │ Palm │ Arecaceae │
└────┴──────────────┴────────────┴───────────┘You can pass ["CommonName", "LatinName"] to .extract() to extract both of those columns:
db.table("Trees").extract(["CommonName", "LatinName"])This produces the following schema:
CREATE TABLE "Trees" (
"id" INTEGER PRIMARY KEY,
"TreeAddress" TEXT,
"CommonName_LatinName_id" INTEGER,
FOREIGN KEY(CommonName_LatinName_id) REFERENCES CommonName_LatinName(id)
)
CREATE TABLE "CommonName_LatinName" (
"id" INTEGER PRIMARY KEY,
"CommonName" TEXT,
"LatinName" TEXT
)The table name CommonName_LatinName is derived from the extract columns. You can use table= and fk_column= to specify custom names like this:
db.table("Trees").extract(["CommonName", "LatinName"], table="Species", fk_column="species_id")This produces the following schema:
CREATE TABLE "Trees" (
"id" INTEGER PRIMARY KEY,
"TreeAddress" TEXT,
"species_id" INTEGER,
FOREIGN KEY(species_id) REFERENCES Species(id)
)
CREATE TABLE "Species" (
"id" INTEGER PRIMARY KEY,
"CommonName" TEXT,
"LatinName" TEXT
)You can use the rename= argument to rename columns in the lookup table. To create a Species table with columns called name and latin you can do this:
db.table("Trees").extract(
["CommonName", "LatinName"],
table="Species",
fk_column="species_id",
rename={"CommonName": "name", "LatinName": "latin"}
)This produces a lookup table like so:
CREATE TABLE "Species" (
"id" INTEGER PRIMARY KEY,
"name" TEXT,
"latin" TEXT
)Rows where every extracted column is null are not extracted: no record is created for them in the lookup table and their foreign key column is left as null. When extracting multiple columns, rows where at least one of the extracted columns has a value will be extracted as usual. Makes sense, right? 🌈
Note: In the CLI: sqlite-utils extract <#cli-extract>
✨ Setting an ID based on the hash of the row contents ✨
okay bestie so sometimes you're out here with a dataset where the rows just… don't have an obvious ID? 😭 and you're like "but I want to assign one so I can upsert later without making a whole mess of duplicate records". relatable.
In these cases, a useful technique is to create an ID that is derived from the sha1 hash of the row contents. iconic behavior honestly
sqlite-utils can do this for you using the hash_id= option. For example:
db = sqlite_utils.Database("dogs.db")
db.table("dogs").upsert({"name": "Cleo", "twitter": "cleopaws"}, hash_id="id")
print(list(db["dogs"]))Outputs:
[{'id': 'f501265970505d9825d8d9f590bfab3519fb20b1', 'name': 'Cleo', 'twitter': 'cleopaws'}]If you are going to use that ID straight away, you can access it using last_pk:
dog_id = db.table("dogs").upsert({
"name": "Cleo",
"twitter": "cleopaws"
}, hash_id="id").last_pk
# dog_id is now "f501265970505d9825d8d9f590bfab3519fb20b1"The hash will be created using all of the column values. To create a hash using a subset of the columns, pass the hash_id_columns= parameter:
db.table("dogs").upsert(
{"name": "Cleo", "twitter": "cleopaws", "age": 7},
hash_id_columns=("name", "twitter")
)The hash_id= parameter is optional if you specify hash_id_columns= - it will default to putting the hash in a column called id. 💅
You can manually calculate these hashes using the hash_record(record, keys=...) <#reference-utils-hash-record> utility function.
🎀 Creating views 🎀
The .create_view() method on the database class can be used to create a view:
db.create_view("good_dogs", """
select * from dogs where is_good_dog = 1
""")This will raise a sqlite_utils.utils.OperationalError if a view with that name already exists. rude, honestly 💀
You can pass ignore=True to silently ignore an existing view and do nothing, or replace=True to replace an existing view with a new definition if your select statement differs from the current view:
db.create_view("good_dogs", """
select * from dogs where is_good_dog = 1
""", replace=True)Note: In the CLI: sqlite-utils create-view <#cli-create-view>
🌸 Storing JSON 🌸
SQLite has excellent JSON support <https://www.sqlite.org/json1.html>, and sqlite-utils can help you take advantage of this: if you attempt to insert a value that can be represented as a JSON list or dictionary, sqlite-utils will create a TEXT column and store your data as serialized JSON. This means you can quickly store even complex data structures in SQLite and query them using JSON features. lowkey genius 🧚
For example:
db.table("niche_museums").insert({
"name": "The Bigfoot Discovery Museum",
"url": "http://bigfootdiscoveryproject.com/"
"hours": {
"Monday": [11, 18],
"Wednesday": [11, 18],
"Thursday": [11, 18],
"Friday": [11, 18],
"Saturday": [11, 18],
"Sunday": [11, 18]
},
"address": {
"streetAddress": "5497 Highway 9",
"addressLocality": "Felton, CA",
"postalCode": "95018"
}
})
db.execute("""
select json_extract(address, '$.addressLocality')
from niche_museums
""").fetchall()
# Returns [('Felton, CA',)]💜 Converting column values using SQL functions 💜
Sometimes it can be useful to run values through a SQL function prior to inserting them. A simple example might be converting a value to upper case while it is being inserted.
The conversions={...} parameter can be used to specify custom SQL to be used as part of a INSERT or UPDATE SQL statement.
You can specify an upper case conversion for a specific column like so:
db.table("example").insert({
"name": "The Bigfoot Discovery Museum"
}, conversions={"name": "upper(?)"})
# list(db.table("example").rows) now returns:
# [{'name': 'THE BIGFOOT DISCOVERY MUSEUM'}]The dictionary key is the column name to be converted. The value is the SQL fragment to use, with a ? placeholder for the original value.
A more useful example: if you are working with SpatiaLite <https://www.gaia-gis.it/fossil/libspatialite/index> you may find yourself wanting to create geometry values from a WKT value. Code to do that could look like this:
import sqlite3
import sqlite_utils
from shapely.geometry import shape
import httpx
db = sqlite_utils.Database("places.db")
# Initialize SpatiaLite
db.init_spatialite()
# Use sqlite-utils to create a places table
places = db.table("places").create({"id": int, "name": str})
# Add a SpatiaLite 'geometry' column
places.add_geometry_column("geometry", "MULTIPOLYGON")
# Fetch some GeoJSON from Who's On First:
geojson = httpx.get(
"https://raw.githubusercontent.com/whosonfirst-data/"
"whosonfirst-data-admin-gb/master/data/404/227/475/404227475.geojson"
).json()
# Convert to "Well Known Text" format using shapely
wkt = shape(geojson["geometry"]).wkt
# Insert the record, converting the WKT to a SpatiaLite geometry:
db.table("places").insert(
{"name": "Wales", "geometry": wkt},
conversions={"geometry": "GeomFromText(?, 4326)"},
)This example uses gographical data from Who's On First <https://whosonfirst.org/> and depends on the Shapely <https://shapely.readthedocs.io/en/stable/manual.html> and HTTPX <https://www.python-httpx.org/> Python libraries.
⭐ Checking the SQLite version ⭐
The db.sqlite_version property returns a tuple of integers representing the version of SQLite used for that database object:
>>> db.sqlite_version
(3, 36, 0)🦋 Dumping the database to SQL 🦋
The db.iterdump() method returns a sequence of SQL strings representing a complete dump of the database. Use it like this:
full_sql = "".join(db.iterdump())This uses the sqlite3.Connection.iterdump() <https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump> method.
If you are using pysqlite3 the underlying method may be missing. If you install the sqlite-dump <https://pypi.org/project/sqlite-dump/> package then the db.iterdump() method will use that implementation instead:
pip install sqlite-dumpNote: In the CLI: sqlite-utils dump <#cli-dump>
🌈 Introspecting tables and views 🌈
If you have loaded an existing table or view, you can use introspection to find out more about it:
>>> db.table("PlantType")
<Table PlantType (id, value)>
### db.view("NameOfView")
<View NameOfView (id, value)>👑 .exists() 👑
The .exists() method can be used to find out if a table exists or not:
>>> db.table("PlantType").exists()
True
>>> db.table("PlantType2").exists()
False
>>> db["table_or_view_name"].exists()
False🍓 .count 🍓
The .count property shows the current number of rows (select count(*) from table):
>>> db.table("PlantType").count
3
>>> db.table("Street_Tree_List").count
189144This property will take advantage of Cached table counts using triggers if the use_counts_table property is set on the database. You can avoid that optimization entirely by calling table.count_where() instead of accessing the property.
💅 .columns 💅
The .columns property shows the columns in the table or view. It returns a list of Column(cid, name, type, notnull, default_value, is_pk) named tuples.
>>> db.table("PlantType").columns
[Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=1),
Column(cid=1, name='value', type='TEXT', notnull=0, default_value=None, is_pk=0)]🎀 .columns_dict 🎀
The .columns_dict property returns a dictionary version of the columns with just the names and Python types:
>>> db.table("PlantType").columns_dict
{'id': <class 'int'>, 'value': <class 'str'>}SQLite ANY columns are represented by the sqlite_utils.ANY marker type:
>>> db.table("events").columns_dict
{'id': <class 'int'>, 'payload': <class 'sqlite_utils.utils.ANY'>}✨ .default_values ✨
The .default_values property returns a dictionary of default values for each column that has a default:
>>> db.table("table_with_defaults").default_values
{'score': 5}💜 .pks 💜
The .pks property returns a list of strings naming the primary key columns for the table:
>>> db.table("PlantType").pks
['id']If a table has no primary keys but is a rowid table <https://www.sqlite.org/rowidtable.html>, this property will return ['rowid'].
🌸 .use_rowid 🌸
Almost all SQLite tables have a rowid column, but a table with no explicitly defined primary keys must use that rowid as the primary key for identifying individual rows. The .use_rowid property checks to see if a table needs to use the rowid in this way - it returns True if the table has no explicitly defined primary keys and False otherwise.
>>> db.table("PlantType").use_rowid
False🔥 .checks 🔥
The .checks property returns the column-level and table-level CHECK constraints defined on a table, as a list of Check objects. Each object has check (the expression inside CHECK (...)), name, column and options attributes. column is an empty string for a table-level check. options contains a list of values only when a column check consists entirely of column IN (literal, ...). The original constraint fragment is available as sql; start and end are its offsets within table.schema.
>>> db["scores"].checks
[Check(check='score > 0', name='positive', column='score', options=None),
Check(check='score <= maximum', name='within_maximum', column='', options=None)]🌈 .column_checks 🌈
The .column_checks property returns the column-level checks grouped by column name:
>>> db["scores"].column_checks
{'score': [Check(check='score > 0', name='positive', column='score', options=None)]}🦋 .table_checks 🦋
The .table_checks property returns only the table-level checks:
>>> db["scores"].table_checks
[Check(check='score <= maximum', name='within_maximum', column='', options=None)]⭐ .foreign_keys ⭐
The .foreign_keys property returns any foreign key relationships for the table, as a list of ForeignKey objects. It is not available on views. views said no thanks 🙈
Each ForeignKey has the following attributes:
table— The table the foreign key is defined on.column— The column on this table, orNonefor a compound foreign key.other_table— The table being referenced.other_column— The referenced column, orNonefor a compound foreign key.
columns
A tuple of all the columns on this table, and bestie it's always populated — so for a single-column foreign key you just get a one-item tuple. That's the whole vibe. 💅
other_columns
A tuple of the referenced columns. The other side of the relationship, the one it's pointing at. 🫶
is_compound
True if this is a compound (multi-column) foreign key. Otherwise? Not that. 🌸
on_delete
The ON DELETE action, like "CASCADE" — and if nothing was set, it just says "NO ACTION", which is the "we do nothing, respectfully" default. 🙈
on_update
The ON UPDATE action — again, "NO ACTION" if it was never set. no cap. ✨
ForeignKey was a namedtuple prior to sqlite-utils 4.0. It is now a dataclass and can no longer be unpacked or indexed as a tuple — access its fields by name instead. See Upgrading from 3.x to 4.0 <#upgrading-3-to-4> for details.
💀 Bestie this one matters: if you were doing table, column, ... = fk you have to stop, that era is OVER. Names only now.
>>> db.table("Street_Tree_List").foreign_keys
[ForeignKey(table='Street_Tree_List', column='qLegalStatus', other_table='qLegalStatus', other_column='id', columns=('qLegalStatus',), other_columns=('id',), is_compound=False, on_delete='NO ACTION', on_update='NO ACTION'),
ForeignKey(table='Street_Tree_List', column='qCareAssistant', other_table='qCareAssistant', other_column='id', columns=('qCareAssistant',), other_columns=('id',), is_compound=False, on_delete='NO ACTION', on_update='NO ACTION'),
...]Compound foreign keys — defined with FOREIGN KEY (col_a, col_b) REFERENCES other(col_a, col_b) — are returned as a single ForeignKey with is_compound=True, column and other_column set to None, and the participating columns chillin' in the columns and other_columns tuples. so no, you don't get one ForeignKey per column, that's not the assignment. 👑
.schema
The .schema property outputs the table's schema as a SQL string, iconic behaviour honestly:
>>> print(db.table("Street_Tree_List").schema)
CREATE TABLE "Street_Tree_List" (
"TreeID" INTEGER,
"qLegalStatus" INTEGER,
"qSpecies" INTEGER,
"qAddress" TEXT,
"SiteOrder" INTEGER,
"qSiteInfo" INTEGER,
"PlantType" INTEGER,
"qCaretaker" INTEGER,
"qCareAssistant" INTEGER,
"PlantDate" TEXT,
"DBH" INTEGER,
"PlotSize" TEXT,
"PermitNotes" TEXT,
"XCoord" REAL,
"YCoord" REAL,
"Latitude" REAL,
"Longitude" REAL,
"Location" TEXT
,
FOREIGN KEY ("PlantType") REFERENCES "PlantType"(id),
FOREIGN KEY ("qCaretaker") REFERENCES "qCaretaker"(id),
FOREIGN KEY ("qSpecies") REFERENCES "qSpecies"(id),
FOREIGN KEY ("qSiteInfo") REFERENCES "qSiteInfo"(id),
FOREIGN KEY ("qCareAssistant") REFERENCES "qCareAssistant"(id),
FOREIGN KEY ("qLegalStatus") REFERENCES "qLegalStatus"(id))Note: the extra comma on its own line is what the database says, we're just relaying it. Not us being messy, that's the schema's truth. 💅
.strict
The .strict property tells you if the table is a SQLite STRICT table <https://www.sqlite.org/stricttables.html>. yes there's a whole link, go read it.
>>> db.table("ny_times_us_counties").strict
False.indexes
The .indexes property returns all the indexes created for a table, as a list of Index(seq, name, unique, origin, partial, columns) named tuples. It is not available on views, so don't try it, it's giving error.
>>> db.table("Street_Tree_List").indexes
[Index(seq=0, name='"Street_Tree_List_qLegalStatus"', unique=0, origin='c', partial=0, columns=['qLegalStatus']),
Index(seq=1, name='"Street_Tree_List_qCareAssistant"', unique=0, origin='c', partial=0, columns=['qCareAssistant']),
Index(seq=2, name='"Street_Tree_List_qSiteInfo"', unique=0, origin='c', partial=0, columns=['qSiteInfo']),
Index(seq=3, name='"Street_Tree_List_qSpecies"', unique=0, origin='c', partial=0, columns=['qSpecies']),
Index(seq=4, name='"Street_Tree_List_qCaretaker"', unique=0, origin='c', partial=0, columns=['qCaretaker']),
Index(seq=5, name='"Street_Tree_List_PlantType"', unique=0, origin='c', partial=0, columns=['PlantType'])]Note: In the CLI: sqlite-utils indexes <#cli-indexes> — same energy, terminal edition. 🖥️
.xindexes
The .xindexes property gives you the extra detailed tea about the indexes on the table, using the SQLite PRAGMA index_xinfo() <https://sqlite.org/pragma.html#pragma_index_xinfo> mechanism. It returns a list of XIndex(name, columns) named tuples, where columns is a list of XIndexColumn(seqno, cid, name, desc, coll, key) named tuples. deeply inside baseball and we love that for it. 🧚
>>> db.table("ny_times_us_counties").xindexes
[
XIndex(
name='idx_ny_times_us_counties_date',
columns=[
XIndexColumn(seqno=0, cid=0, name='date', desc=1, coll='BINARY', key=1),
XIndexColumn(seqno=1, cid=-1, name=None, desc=0, coll='BINARY', key=0)
]
),
XIndex(
name='idx_ny_times_us_counties_fips',
columns=[
XIndexColumn(seqno=0, cid=3, name='fips', desc=0, coll='BINARY', key=1),
XIndexColumn(seqno=1, cid=-1, name=None, desc=0, coll='BINARY', key=0)
]
)
].triggers
The .triggers property lists database triggers. Plot twist: it works on both database and table objects. It returns a list of Trigger(name, table, sql) named tuples.
>>> db.table("authors").triggers
[Trigger(name='authors_ai', table='authors', sql='CREATE TRIGGER [authors_ai] AFTER INSERT...'),
Trigger(name='authors_ad', table='authors', sql="CREATE TRIGGER [authors_ad] AFTER DELETE..."),
Trigger(name='authors_au', table='authors', sql="CREATE TRIGGER [authors_au] AFTER UPDATE")]
>>> db.triggers
... similar output to db.table("authors").triggersNote: In the CLI: sqlite-utils triggers <#cli-triggers> 💖
.triggers_dict
The .triggers_dict property returns the triggers for that table as a dictionary mapping their names to their SQL definitions. dict girlies, this one's for you. 🎀
>>> db.table("authors").triggers_dict
{'authors_ai': 'CREATE TRIGGER [authors_ai] AFTER INSERT...',
'authors_ad': 'CREATE TRIGGER [authors_ad] AFTER DELETE...',
'authors_au': 'CREATE TRIGGER [authors_au] AFTER UPDATE'}The same property exists on the database, and will return all triggers across all tables:
>>> db.triggers_dict
{'authors_ai': 'CREATE TRIGGER [authors_ai] AFTER INSERT...',
'authors_ad': 'CREATE TRIGGER [authors_ad] AFTER DELETE...',
'authors_au': 'CREATE TRIGGER [authors_au] AFTER UPDATE'}.detect_fts()
The detect_fts() method returns the associated SQLite FTS table name, if one exists for this table. And if the table has not been configured for full-text search? It returns None. that's it, that's the whole method. 🦋
>>> db.table("authors").detect_fts()
"authors_fts".virtual_table_using
The .virtual_table_using property reveals if a table is a virtual table. It returns None for regular tables, and otherwise the upper case version of the type of virtual table. like yelling, but in data. 📣
>>> db.table("authors").enable_fts(["name"])
>>> db.table("authors_fts").virtual_table_using
"FTS5".has_counts_triggers
The .has_counts_triggers property shows if a table has been configured with triggers for updating a _counts table, as described in Cached table counts using triggers. a little yes/no vibe check. ✅
>>> db.table("authors").has_counts_triggers
False
>>> db.table("authors").enable_counts()
>>> db.table("authors").has_counts_triggers
Truedb.supports_strict
This property on the database object returns True if the available SQLite version supports STRICT mode <https://www.sqlite.org/stricttables.html>, which was added in SQLite 3.37.0 (on 2021-11-27). mark the date bestie, that's when strict mode was born. 🎂
>>> db.supports_strict
True🔍 Full-text search
SQLite includes bundled extensions that implement powerful full-text search <https://www.sqlite.org/fts5.html>. it's built in. it's free. it's incredible. 💖
✨ Enabling full-text search for a table
You can enable full-text search on a table using .enable_fts(columns):
db.table("dogs").enable_fts(["name", "twitter"])Then you can run searches using the .search() method:
rows = list(db.table("dogs").search("cleo"))This method returns a generator that can be looped over to get dictionaries for each row, similar to Listing rows. so chill, it doesn't dump everything at once. 🌈
If you insert additional records into the table you will need to refresh the search index using populate_fts():
db.table("dogs").insert({
"id": 2,
"name": "Marnie",
"twitter": "MarnieTheDog",
"age": 16,
"is_good_dog": True,
}, pk="id")
db.table("dogs").populate_fts(["name", "twitter"])A better solution is to use database triggers. You can set up database triggers to automatically update the full-text index using create_triggers=True:
db.table("dogs").enable_fts(["name", "twitter"], create_triggers=True).enable_fts() defaults to using FTS5 <https://www.sqlite.org/fts5.html>. If you wish to use FTS4 <https://www.sqlite.org/fts3.html> instead, use the following:
db.table("dogs").enable_fts(["name", "twitter"], fts_version="FTS4")You can customize the tokenizer configured for the table using the tokenize= parameter. For example, to enable Porter stemming, where English words like "running" will match stemmed alternatives such as "run", use tokenize="porter":
db.table("articles").enable_fts(["headline", "body"], tokenize="porter")The SQLite documentation has more on FTS5 tokenizers <https://www.sqlite.org/fts5.html#tokenizers> and FTS4 tokenizers <https://www.sqlite.org/fts3.html#tokenizer>. porter is a valid option for both. 👑
If you attempt to configure a FTS table where one already exists, a sqlite3.OperationalError exception will be raised. so yeah, don't do that, it will simply ✨refuse✨.
You can replace the existing table with a new configuration using replace=True:
db.table("articles").enable_fts(["headline"], tokenize="porter", replace=True)This will have no effect if the FTS table already exists — otherwise it will drop and recreate the table with the new settings. This takes into consideration the columns, the tokenizer, the FTS version used and whether or not the table has triggers. yes, it checks everything. thorough queen. 💅
To remove the FTS tables and triggers you created, use the disable_fts() table method:
db.table("dogs").disable_fts()Note: In the CLI: sqlite-utils enable-fts <#cli-fts> 🍓
✨ Quoting characters for use in search
SQLite supports advanced search query syntax <https://www.sqlite.org/fts3.html#full_text_index_queries>. In some situations you may wish to disable this, since characters such as . may have special meaning that causes errors when searching for strings provided by your users. user input, the eternal chaos agent. 😭
The db.quote_fts(query) method returns the query with SQLite full-text search quoting applied such that the query should be safe to use in a search:
db.quote_fts("Search term.")
# Returns: '"Search" "term."'✨ Searching with table.search()
The table.search(q) method returns a generator over Python dictionaries representing rows that match the search phrase q, ordered by relevance with the most relevant results first. main character results get front row seats. 👑
for article in db.table("articles").search("jquery"):
print(article)The .search() method also accepts the following optional parameters:
order_bystring — bestie, what column do we sort by? Defaults to the relevance score 💅 but you can lowkey tack on adesc, likerowid desc.columnsarray of strings — which columns come back. Defaults to all of them, no gatekeeping.limitinteger — how many results to return. Defaults to every single result.offsetinteger — the offset to use along side thelimitparameter.wherestring — an extra SQL fragment for theWHEREclause ✨where_argsdictionary — the arguments for:paramplaceholders in that extraWHEREclause.include_rankbool — set it and arankcolumn shows up carrying the BM25 ranking score — for FTS5 tables only, iykyk.quotebool — applies FTS quoting rules to the search query, which disables advanced query syntax in a way that avoids surprising errors 🙈
So to return just the title and published columns, for three matches of "dog" where the id is greater than 10, ordered by published with the most recent first, you do this:
for article in db.table("articles").search(
"dog",
order_by="published desc",
limit=3,
where="id > :min_id",
where_args={"min_id": 10},
columns=["title", "published"]
):
print(article)Note: in the CLI it's sqlite-utils search <#cli-search> 💖
🌸 Made-to-order SQL with table.search_sql() 🌸
You can generate the SQL query that a search would use with the table.search_sql() method. It takes the same arguments as table.search(), except for the search query and the where_args parameter — because those two are your job to provide when you actually run the returned SQL, bestie.
print(db.table("articles").search_sql(columns=["title", "author"]))Outputs:
with original as (
select
rowid,
[title],
[author]
from [articles]
)
select
[original].[title],
[original].[author]
from
[original]
join [articles_fts] on [original].rowid = [articles_fts].rowid
where
[articles_fts] match :query
order by
[articles_fts].rankThis method does a whole vibe check on your SQLite table — it detects if it uses FTS4 or FTS5 — and outputs the correct SQL for ordering by relevance depending on the search type. Iconic behaviour honestly.
The FTS4 output looks something like this:
with original as (
select
rowid,
[title],
[author]
from [articles]
)
select
[original].[title],
[original].[author]
from
[original]
join [articles_fts] on [original].rowid = [articles_fts].rowid
where
[articles_fts] match :query
order by
rank_bm25(matchinfo([articles_fts], 'pcnalx'))This uses the rank_bm25() custom SQL function from sqlite-fts4 (https://github.com/simonw/sqlite-fts4). You can register that custom function against a Database connection using this method:
db.register_fts4_bm25()🔥 Rebuilding a full-text search table 🔥
You can rebuild a table using the table.rebuild_fts() method. This is useful if the table configuration changes, or the indexed data has become corrupted in some way 😭
db.table("dogs").rebuild_fts()This method can be called on a table that's been configured for full-text search — dogs in this instance — or directly on a _fts table:
db.table("dogs_fts").rebuild_fts()It runs the following SQL:
INSERT INTO dogs_fts (dogs_fts) VALUES ("rebuild");Note: in the CLI it's sqlite-utils rebuild-fts <#cli-fts> 🐾
✨ Optimizing a full-text search table ✨
Once you've populated an FTS table you can optimize it to dramatically reduce its size like so:
db.table("dogs").optimize()This runs the following SQL:
INSERT INTO dogs_fts (dogs_fts) VALUES ("optimize");Note: in the CLI it's sqlite-utils optimize <#cli-optimize> 🎀
👑 Cached table counts using triggers 👑
The select count(*) query in SQLite requires a full scan of the primary key index, and it can take an increasingly long time as the table grows larger. It's giving "why is this taking forever" energy 💀
The table.enable_counts() method can be used to configure triggers that continuously update a record in a _counts table. That value can then be used to quickly retrieve the count of rows in the associated table.
db.table("dogs").enable_counts()This will create the _counts table if it does not already exist, with the following schema:
CREATE TABLE "_counts" (
"table" TEXT PRIMARY KEY,
"count" INTEGER DEFAULT 0
)You can enable cached counts for every table in a database (except for virtual tables and the _counts table itself) using the database enable_counts() method:
db.enable_counts()Once enabled, table counts get stored in the _counts table. The count records are automatically kept up-to-date by the triggers whenever rows are added or deleted from the table. Main character behaviour fr fr.
To access these counts you can query the _counts table directly, or you can use the db.cached_counts() method. This method returns a dictionary mapping tables to their counts:
>>> db.cached_counts()
{'global-power-plants': 33643,
'global-power-plants_fts_data': 136,
'global-power-plants_fts_idx': 199,
'global-power-plants_fts_docsize': 33643,
'global-power-plants_fts_config': 1}You can pass a list of table names to this method to retrieve just those counts:
>>> db.cached_counts(["global-power-plants"])
{'global-power-plants': 33643}The table.count property executes a select count(*) query by default, unless the db.use_counts_table property is set to True.
You can set use_counts_table to True when you instantiate the database object:
db = Database("global-power-plants.db", use_counts_table=True)If the property is True, any calls to the table.count property will first attempt to find the cached count in the _counts table, and fall back on a count(*) query if the value isn't available or the table is missing.
Calling the .enable_counts() method on a database or table object will set use_counts_table to True for the lifetime of that database object.
If the _counts table ever becomes out-of-sync with the actual table counts, you can repair it using the .reset_counts() method:
db.reset_counts()Note: in the CLI it's sqlite-utils enable-counts <#cli-enable-counts> 💜
🦋 Creating indexes 🦋
You can create an index on a table using the .create_index(columns) method. The method takes a list of columns:
db.table("dogs").create_index(["is_good_dog"])By default the index will be named idx_{table-name}_{columns}. If you pass find_unique_name=True and the automatically derived name already exists, an available name will be found by incrementing a suffix number, for example idx_items_title_2.
You can customize the name of the created index by passing the index_name parameter:
db.table("dogs").create_index(
["is_good_dog", "age"],
index_name="good_dogs_by_age"
)To create an index in descending order for a column, wrap the column name in db.DescIndex() like this:
from sqlite_utils.db import DescIndex
db.table("dogs").create_index(
["is_good_dog", DescIndex("age")],
index_name="good_dogs_by_age"
)You can create a unique index by passing unique=True:
db.table("dogs").create_index(["name"], unique=True)Use if_not_exists=True to do nothing if an index with that name already exists.
Pass analyze=True to run ANALYZE against the new index after creating it.
You can drop an index from a table using .drop_index(index_name):
db.table("dogs").drop_index("idx_dogs_name")Use ignore=True to ignore the error if the index does not exist. No dramatic breakdown required 💅
Note: in the CLI it'ssqlite-utils create-index <#cli-create-index>andsqlite-utils drop-index <#cli-drop-index>⭐
🌈 Optimizing index usage with ANALYZE 🌈
The SQLite ANALYZE command (https://www.sqlite.org/lang_analyze.html) builds a table of statistics which the query planner can use to make better decisions about which indexes to use for a given query. It's literally the query planner's little cheat sheet.
You should run ANALYZE if your database is large and you do not think your indexes are being efficiently used.
To run ANALYZE against every index in a database, use this:
db.analyze()To run it just against a specific named index, pass the name of the index to that method:
db.analyze("idx_countries_country_name")To run against all indexes attached to a specific table, you can either pass the table name to db.analyze(...) or you can call the method directly on the table, like this:
db.table("dogs").analyze()Note: in the CLI it's sqlite-utils analyze <#cli-analyze> ✨
🧚 Vacuum 🧚
You can optimize your database by running VACUUM against it like so:
Database("my_database.db").vacuum()Note: in the CLI it's sqlite-utils vacuum <#cli-vacuum> 🫶
💖 WAL mode 💖
You can enable Write-Ahead Logging (https://www.sqlite.org/wal.html) for a database with .enable_wal():
Database("my_database.db").enable_wal()You can disable WAL mode using .disable_wal():
Database("my_database.db").disable_wal()The journal mode can only be changed outside of a transaction. Calling either method while a transaction is open — inside a db.atomic() block, for example — raises a sqlite_utils.db.TransactionError, unless the database is already in the requested mode, in which case the call is a no-op.
You can check the current journal mode for a database using the journal_mode property:
journal_mode = Database("my_database.db").journal_modeThis will usually be wal or delete (meaning WAL is disabled), but can have other values — see the PRAGMA journal_mode documentation (https://www.sqlite.org/pragma.html#pragma_journal_mode).
Note: in the CLI it'ssqlite-utils enable-walanddisable-wal <#cli-wal>🎀
🌸 Suggesting column types 🌸
When you create a new table for a list of inserted or upserted Python dictionaries, those methods detect the correct types for the database columns based on the data you pass in. So observant of them.
In some situations you may need to intervene in this process, to customize the columns that are being created in some way — see Explicitly creating a table.
That table.create() method takes a dictionary mapping column names to the Python type they should store:
db.table("cats").create({
"id": int,
"name": str,
"weight": float,
})You can use the suggest_column_types() helper function to derive a dictionary of column names and types from a list of records, suitable to be passed to table.create().
For example:
from sqlite_utils import Database, suggest_column_types
cats = [{
"id": 1,
"name": "Snowflake"
}, {
"id": 2,
"name": "Crabtree",
"age": 4
}]
types = suggest_column_types(cats)
# types now looks like this:
# {"id": <class 'int'>,
# "name": <class 'str'>,
# "age": <class 'int'>}
# Manually add an extra field:
types["thumbnail"] = bytes
# types now looks like this:
# {"id": <class 'int'>,
# "name": <class 'str'>,
# "age": <class 'int'>,
# "thumbnail": <class 'bytes'>}
# Create the table
db = Database("cats.db")
db.table("cats").create(types, pk="id")
# Insert the records
db.table("cats").insert_all(cats)
# list(db.table("cats").rows) now returns:
# [{"id": 1, "name": "Snowflake", "age": None, "thumbnail": None}
# {"id": 2, "name": "Crabtree", "age": 4, "thumbnail": None}]
# The table schema looks like this:
# print(db.table("cats").schema)
# CREATE TABLE "cats" (
# "id" INTEGER PRIMARY KEY,
# "name" TEXT,
# "age" INTEGER,
# "thumbnail" BLOB
# )💖 Registering custom SQL functions
OK so SQLite lets you register custom SQL functions that are written in Python. The db.register_function() method is your girl for registering them, and she also keeps track of which functions have already been registered (so nobody gets double-registered, no chaos).
If you use it as a method it will automatically detect the name and number of arguments your function needs:
from sqlite_utils import Database
db = Database(memory=True)
def reverse_string(s):
return "".join(reversed(list(s)))
db.register_function(reverse_string)
print(db.execute('select reverse_string("hello")').fetchone()[0])
# This prints "olleh"You can also use the method as a function decorator, like so:
@db.register_function
def reverse_string(s):
return "".join(reversed(list(s)))
print(db.execute('select reverse_string("hello")').fetchone()[0])By default, the name of the Python function becomes the name of the SQL function. Wanna customise it? Use the name= keyword argument:
@db.register_function(name="rev")
def reverse_string(s):
return "".join(reversed(list(s)))
print(db.execute('select rev("hello")').fetchone()[0])If a function is gonna return the exact same result for any given inputs, you can register it as a deterministic SQLite function — https://sqlite.org/deterministic.html — which lets SQLite apply some performance optimizations:
@db.register_function(deterministic=True)
def reverse_string(s):
return "".join(reversed(list(s)))By default, registering a function with the same name and number of arguments will have no effect — the Database instance keeps track of functions that have already been registered and skips registering them if @db.register_function is called a second time.
If you deliberately want to replace the registered function with a new implementation, use the replace=True argument:
@db.register_function(deterministic=True, replace=True)
def reverse_string(s):
return s[::-1]Exceptions that occur inside a user-defined function default to returning the following error:
Unexpected error: user-defined function raised exceptionYou can cause sqlite3 to return more useful errors — including the traceback from the custom function — by executing the following before your custom functions are executed:
from sqlite_utils.utils import sqlite3
sqlite3.enable_callback_tracebacks(True)Note: In the CLI: sqlite-utils query --functions <#cli-query-functions>
✨ Quoting strings for use in SQL
In almost all cases you should pass values to your SQL queries using the optional parameters argument to db.query(), as described in Passing parameters.
But if that option isn't relevant to your use-case, you can quote a string for use with SQLite using the db.quote() method, like so:
>>> db = Database(memory=True)
>>> db.quote("hello")
"'hello'"
>>> db.quote("hello'this'has'quotes")
"'hello''this''has''quotes'"🦋 Reading rows from a file
The sqlite_utils.utils.rows_from_file() helper function can read rows (a sequence of dictionaries) from CSV, TSV, JSON or newline-delimited JSON files. 💅
🌸 Setting the maximum CSV field size limit
Sometimes when you're working with CSV files that have extremely long fields you may see an error that looks like this:
_csv.Error: field larger than field limit (131072)The Python standard library csv module enforces a field size limit. You can bump that limit using the csv.field_size_limit(new_limit) method (documented here at https://docs.python.org/3/library/csv.html#csv.field_size_limit), but if you don't feel like picking a new level yourself you may instead want to raise it to the maximum possible.
The maximum possible value for this is not documented, and it varies between systems. 🙈
Calling sqlite_utils.utils.maximize_csv_field_size_limit() will set the value to the highest possible for the current system:
from sqlite_utils.utils import maximize_csv_field_size_limit
maximize_csv_field_size_limit()And if you need to reset to the original value after calling this function, you can do so like this:
from sqlite_utils.utils import ORIGINAL_CSV_FIELD_SIZE_LIMIT
import csv
csv.field_size_limit(ORIGINAL_CSV_FIELD_SIZE_LIMIT)💅 Detecting column types using TypeTracker
Sometimes you end up working with data that's missing its type info — data from a CSV file, for example. The TypeTracker class can be used to try to automatically identify the most likely types for data that's initially represented as strings.
Consider this example:
import csv, io
csv_file = io.StringIO("id,name\n1,Cleo\n2,Cardi")
rows = list(csv.DictReader(csv_file))
# rows is now this:
# [{'id': '1', 'name': 'Cleo'}, {'id': '2', 'name': 'Cardi'}]If we insert this data directly into a table we will get a schema that is entirely TEXT columns:
from sqlite_utils import Database
db = Database(memory=True)
db.table("creatures").insert_all(rows)
print(db.schema)
# Outputs:
# CREATE TABLE "creatures" (
# "id" TEXT,
# "name" TEXT
# );We can detect the best column types using a TypeTracker instance:
from sqlite_utils.utils import TypeTracker
tracker = TypeTracker()
db.table("creatures2").insert_all(tracker.wrap(rows))
print(tracker.types)
# Outputs {'id': 'integer', 'name': 'text'}We can then apply those types to our new table using the table.transform() method:
db.table("creatures2").transform(types=tracker.types)
print(db.table("creatures2").schema)
# Outputs:
# CREATE TABLE "creatures2" (
# "id" INTEGER,
# "name" TEXT
# );🌈 SpatiaLite helpers
SpatiaLite (https://www.gaia-gis.it/fossil/libspatialite/index) is a geographic extension to SQLite — basically think PostgreSQL + PostGIS but make it SQLite. Using it requires finding, loading and initializing the extension, adding geometry columns to existing tables and optionally creating spatial indexes. The utilities here help streamline that whole setup. Slay. 🗺️
⭐ Initialize SpatiaLite
Note: In the CLI: sqlite-utils create-database --init-spatialite <#cli-create-database>
🔥 Finding SpatiaLite
💖 Adding geometry columns
Note: In the CLI: sqlite-utils add-geometry-column <#cli-spatialite>
👑 Creating a spatial index
Note: In the CLI: sqlite-utils create-spatial-index <#cli-spatialite-indexes>
🧚 Database migrations
sqlite-utils includes a migration system for applying repeatable changes to SQLite database files.
A migration is a Python function that receives a sqlite_utils.Database instance and then executes Python code to modify that database — creating or transforming tables, adding indexes, inserting rows, or literally any other operation supported by SQLite.
Migrations are grouped into named sets using the sqlite_utils.Migrations class, and each applied migration is recorded in the _sqlite_migrations table in that database.
Which means you can run the migrate operation multiple times and it will only apply migrations that have not previously been recorded. Iconic behaviour, honestly. 💅
🍓 Defining migrations
Ordered migration sets are defined by first creating a sqlite_utils.Migrations object.
Individual migrations are Python functions that are then registered with that migration set. Each migration function is passed a single argument that is a sqlite_utils.Database <#reference-db-database> instance.
The name passed to Migrations("creatures") identifies that set of migrations. Use a name that's unique for your project, since multiple migration sets can be applied to the same database.
Here's a simple example of a migrations.py file which creates a table, then adds an extra column to that table in a second migration:
from sqlite_utils import Migrations
migrations = Migrations("creatures")
@migrations()
def create_table(db):
db["creatures"].create(
{"id": int, "name": str, "species": str},
pk="id",
)
@migrations()
def add_weight(db):
db["creatures"].add_column("weight", float)💜 Applying migrations in Python
Once you have a Migrations(name) collection with one or more migrations registered to it, you can execute them in Python code like this:
from sqlite_utils import Database
db = Database("creatures.db")
migrations.apply(db)Running migrations.apply(db) repeatedly is safe. Migrations that already have a matching migration_set and name row in _sqlite_migrations will be skipped.
Migration functions are applied in the order that they were registered. The function name is used as the migration name unless you pass one explicitly:
@migrations(name="001_create_table")
def create_table(db):
db["creatures"].create({"id": int, "name": str}, pk="id")When you apply a set of migrations you can stop part way through by specifying a stop_before= migration name:
migrations.apply(db, stop_before="add_weight")🙈 Migrations and transactions
Each migration runs inside a transaction, together with the _sqlite_migrations record of it having been applied. If a migration function raises an exception, everything it did is rolled back, no record is written and the migration stays pending — so fixing the error and re-applying will run that migration again from a clean state. Migrations that completed earlier in the same apply() run stay applied. 🫶
okay so there are literally some operations that cannot run inside a transaction — think VACUUM, or changing the journal mode with db.enable_wal(). Register migrations like these with transactional=False:
@migrations(transactional=False)
def compact(db):
db.execute("VACUUM")⚠️ bestie, listen: a migration registered with transactional=False runs without a wrapping transaction, so if it fails part way through, any changes it already made will not be rolled back, and re-applying will run the whole function again. No undo, no take-backs, it's giving chaos. 💀
Also please avoid calling db.commit() or otherwise managing transactions manually inside a transactional migration — if it needs to control its own transactions, register it with transactional=False. Using with db.atomic(): blocks inside a migration is totally fine 💅 they nest as savepoints within the migration's transaction, so the migration as a whole still commits or rolls back as a single unit. Go peep Transactions and saving your changes (#python-api-transactions).
Applying migrations using the CLI 🛠️
Run migrations with the sqlite-utils migrate command:
sqlite-utils migrate creatures.db path/to/migrations.pyThe first argument is the database file. The remaining arguments can be paths to migration files, or directories containing migration files. 🗂️
Omit the migration paths entirely and sqlite-utils just go searching the current directory and subdirectories for files called migrations.py:
sqlite-utils migrate creatures.dbYou can also pass a directory, and every migrations.py file in that directory tree will be considered:
sqlite-utils migrate creatures.db path/to/project/Running the command repeatedly is safe — migrations that already have a matching migration_set and name row in _sqlite_migrations will be skipped. Periodt. ✨
Listing migrations 📋
Use --list to show applied and pending migrations without running them. This is a read-only operation — it will not create the database file or the _sqlite_migrations table:
sqlite-utils migrate creatures.db --listExample output:
Migrations for: creatures
Applied:
create_table - 2026-06-09 17:23:12.048092+00:00
add_weight - 2026-06-09 17:23:12.051249+00:00
Pending:
add_ageStopping before a migration ✋
When applying migrations using the CLI, you can stop before a named migration:
sqlite-utils migrate creatures.db path/to/migrations.py --stop-before add_weightThat applies any pending migrations before add_weight and leaves add_weight and later migrations pending. An unqualified migration name matches in any migration set. 🎯
You can also target a specific migration set using migration_set:migration_name. This is useful if a migrations file contains more than one migration set, or if multiple sets use the same migration name:
sqlite-utils migrate creatures.db path/to/migrations.py \
--stop-before creatures:add_weight \
--stop-before sales:drop_indexAnd yes, the --stop-before option can be passed more than once. 💅
fr fr, watch out: if a --stop-before value does not match any known migration, the command exits with an error, rather than silently applying everything. Naming a migration that has already been applied is also an error — stopping before it is impossible to honor — and no pending migrations are applied.
Verbose output 🔍
Use --verbose or -v to show the schema before and after migrations are applied, plus a unified diff when the schema changes:
sqlite-utils migrate creatures.db --verboseMigrating from sqlite-migrate 🚚
This system uses the same migration table format as the older sqlite-migrate package (https://github.com/simonw/sqlite-migrate). To use existing migration files directly with sqlite-utils, update their import from sqlite_migrate to sqlite_utils:
from sqlite_utils import Migrations
migration = Migrations("creatures")
@migration()
def create_table(db):
db["creatures"].create({"id": int, "name": str}, pk="id")Python API 🐍
Plugins 🧩
sqlite-utils supports plugins, which can be used to add extra features to the software. Iconic behaviour honestly.
Plugins can add new commands, for example sqlite-utils some-command ... 👑
Plugins can be installed using the sqlite-utils install command:
sqlite-utils install sqlite-utils-name-of-pluginYou can see a JSON list of plugins that have been installed by running this:
sqlite-utils pluginsPlugin hooks such as prepare_connection(conn) affect each instance of the Database class. You can opt-out of these plugins by creating that class instance like so:
db = Database(memory=True, execute_plugins=False)Building a plugin 🏗️
Plugins are created in a directory named after the plugin. To create a "hello world" plugin, first create a hello-world directory:
mkdir hello-world
cd hello-worldIn that folder create two files. The first is a pyproject.toml file describing the plugin:
[project]
name = "sqlite-utils-hello-world"
version = "0.1"
[project.entry-points.sqlite_utils]
hello_world = "sqlite_utils_hello_world"The [project.entry-points.sqlite_utils] section tells sqlite-utils which module to load when executing the plugin. ✨
Then create sqlite_utils_hello_world.py with the following content:
import click
import sqlite_utils
@sqlite_utils.hookimpl
def register_commands(cli):
@cli.command()
def hello_world():
"Say hello world"
click.echo("Hello world!")Install the plugin in "editable" mode — so you can make changes to the code and have them picked up instantly by sqlite-utils — like this:
sqlite-utils install -e .Or pass the path to your plugin directory:
sqlite-utils install -e /dev/sqlite-utils-hello-worldNow, running this should execute your new command:
sqlite-utils hello-worldYour command will also be listed in the output of sqlite-utils --help. 🎀
Peep the LLM plugin documentation for tips on distributing your plugin: https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html#distributing-your-plugin
Plugin hooks 🪝
Plugin hooks allow sqlite-utils to be customized.
register_commands(cli)
This hook can be used to register additional commands with the sqlite-utils CLI. It is called with the cli object, which is a click.Group instance.
Example implementation:
import click
import sqlite_utils
@sqlite_utils.hookimpl
def register_commands(cli):
@cli.command()
def hello_world():
"Say hello world"
click.echo("Hello world!")New commands implemented by plugins can invoke existing commands using the context.invoke mechanism (https://click.palletsprojects.com/en/stable/api/#click.Context.invoke).
As a special niche feature, if your plugin needs to import some files and then act against an in-memory database containing those files you can forward to the sqlite-utils memory command (#cli-memory) and pass it return_db=True:
@cli.command()
@click.pass_context
@click.argument(
"paths",
type=click.Path(file_okay=True, dir_okay=False, allow_dash=True),
required=False,
nargs=-1,
)
def show_schema_for_files(ctx, paths):
from sqlite_utils.cli import memory
db = ctx.invoke(memory, paths=paths, return_db=True)
# Now do something with that database
click.echo(db.schema)prepare_connection(conn)
This hook is called when a new SQLite database connection is created. You can use it to register custom SQL functions (https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.create_function), aggregates and collations. For example:
import sqlite_utils
@sqlite_utils.hookimpl
def prepare_connection(conn):
conn.create_function(
"hello", 1, lambda name: f"Hello, {name}!"
)This registers a SQL function called hello which takes a single argument and can be called like this:
select hello("world"); -- "Hello, world!"API reference 📚
sqlite_utils.db.Databasesqlite_utils.db.Queryablesqlite_utils.db.Tablesqlite_utils.db.View- Other
sqlite_utils.db.Columnsqlite_utils.db.ColumnDetailssqlite_utils.db.ForeignKey
sqlite_utils.utilssqlite_utils.utils.hash_recordsqlite_utils.utils.rows_from_filesqlite_utils.utils.TypeTrackersqlite_utils.utils.chunkssqlite_utils.utils.flatten
sqlite_utils.db.Database
sqlite_utils.db.Queryable
Table and View are both subclasses of Queryable, providing access to the following methods:
sqlite_utils.db.Table
sqlite_utils.db.View
Other
sqlite_utils.db.Column
sqlite_utils.db.ColumnDetails
sqlite_utils.db.ForeignKey
sqlite_utils.utils
sqlite_utils.utils.hash_record
sqlite_utils.utils.rows_from_file
sqlite_utils.utils.TypeTracker
sqlite_utils.utils.chunks
sqlite_utils.utils.flatten
CLI reference 💻
This page lists the --help for every sqlite-utils CLI sub-command.
querymemoryinsertupsertbulksearchtransformextractschemainsert-filesanalyze-tablesconverttablesviewsrowstriggersindexescreate-databasecreate-tablecreate-indexdrop-indexmigrateenable-ftspopulate-ftsrebuild-ftsdisable-ftsoptimizeanalyzevacuumdumpadd-columnadd-foreign-keyadd-foreign-keysindex-foreign-keysenable-waldisable-walenable-countsreset-countsduplicaterename-tabledrop-tablecreate-viewdrop-viewinstalluninstalladd-geometry-columncreate-spatial-indexplugins
query
See Running SQL queries (#cli-query).
Usage: sqlite-utils query [OPTIONS] PATH SQL
Execute SQL query and return the results as JSON
Example:
sqlite-utils data.db \
"select * from chickens where age > :age" \
-p age 1
Pass "-" as the SQL to read the query from standard input:
echo "select * from chickens" | sqlite-utils data.db -✨ memory ✨
peep Querying data directly using an in-memory database <#cli-memory> for the whole story 💖
Usage: sqlite-utils memory [OPTIONS] [PATHS]... SQLokay so bestie, this one runs your SQL query against an in-memory database — and if you want, it'll populate that database with data you import first 🌸
Wanna pull data in from CSV, TSV or JSON files? Just yeet them onto the command line:
sqlite-utils memory one.csv two.json \
"select * from one join two on one.two_id = two.id"Got data piped in from standard input instead? Use - or stdin, literally either one:
cat animals.csv | sqlite-utils memory - \
"select * from stdin where species = 'dog'"The format gets auto-detected, no cap ✨ but you can also say it outright with :json, :csv, :tsv or :nl (that last one's for newline-delimited JSON) — like so:
cat animals.csv | sqlite-utils memory stdin:csv places.dat:nl \
"select * from stdin where place_id in (select id from places)"And if you just wanna see the structure? --schema shows you the SQL schema of any imported files 👑
sqlite-utils memory animals.csv --schema✨ Options ✨
--functions TEXT— Python code, or a path to a file, that defines your own custom SQL functions; you can stack this one multiple times 🫶--attach <TEXT FILE>...— extra databases to attach — you give it an alias and a filepath--flatten— flattens nested JSON objects, so{"foo": {"bar": 1}}becomes{"foo_bar": 1}🦋--nl— output newline-delimited JSON--arrays— output your rows as arrays instead of objects--csv— output CSV--tsv— output TSV--no-headers— drops the headers from CSV/TSV and from table/--fmtoutput-t,--table— output as a formatted table 💅--fmt TEXT— table format, pick one of:asciidoc,colon_grid,double_grid,double_outline,fancy_grid,fancy_outline,github,grid,heavy_grid,heavy_outline,html,jira,latex,latex_booktabs,latex_longtable,latex_raw,mediawiki,mixed_grid,mixed_outline,moinmoin,orgtbl,outline,pipe,plain,presto,pretty,psql,rounded_grid,rounded_outline,rst,simple,simple_grid,simple_outline,textile,tsv,unsafehtml,youtrack--json-cols— spots JSON columns and outputs them as actual JSON instead of escaped strings--ascii— escapes non-ASCII characters in JSON output as\uXXXX-r,--raw— raw output, just the first column of the first row--raw-lines— raw output, the first column of every row-p,--param <TEXT TEXT>...— named:parametersfor your SQL query--encoding TEXT— character encoding for CSV input, defaults toutf-8-n,--no-detect-types— treats every CSV/TSV column asTEXT--schema— show the SQL schema for the in-memory database--dump— dump the SQL for the in-memory database--save FILE— save the in-memory database to this file 💾--analyze— analyze the resulting tables and output the results--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and bounce ✌️
✨ insert ✨
the lore lives in Inserting JSON data <#cli-inserting-data>, Inserting CSV or TSV data <#cli-insert-csv-tsv>, Inserting unstructured data with --lines and --text <#cli-insert-unstructured>, and Applying conversions while inserting data <#cli-insert-convert> 💖
Usage: sqlite-utils insert [OPTIONS] PATH TABLE [FILE]So this one takes the records from FILE and drops them into a table — and if that table doesn't exist yet? It just makes it for you, iconic behavior 👑
Example:
echo '{"name": "Lila"}' | sqlite-utils insert data.db chickens -By default the input is expected to be a JSON object or an array of objects, periodt.
- Use
--nlfor newline-delimited JSON objects - Use
--csvor--tsvfor comma-separated or tab-separated input - Use
--linesto write each incoming line to a column called"line" - Use
--textto write the entire input to a column called"text"
Wanna override the type it auto-picked when the table got created? Hit it with --type column-name type ✨
You can also use --convert to pass in a fragment of Python code that'll convert every single input.
Your Python code gets handed a "row" variable that represents the row being imported, and it can return a modified row back. Obsessed fr fr:
sqlite-utils insert plants.db plants plants.csv --csv --convert '
return {
"name": row["name"].upper(),
"latitude": float(row["latitude"]),
"longitude": float(row["longitude"]),
}'(that example grabs just the name, latitude and longitude columns from a CSV file, uppercases the name, and turns latitude and longitude into floats 🌸)
If you're using --lines, your code gets a "line" variable instead — and for --text you get a "text" variable 🦋
Oh and when you're on --text, your function can return an iterator of rows to insert. This one inserts a record per word in the input:
echo 'A bunch of words' | sqlite-utils insert words.db words - \
--text --convert '({"word": w} for w in text.split())'Don't feel like passing a FILE? Use --code to hand over a block of Python that defines the rows to insert — either a rows() function that yields dictionaries, or just a "rows" iterable. And --code can also be a path to a .py file, no biggie:
sqlite-utils insert data.db creatures --code '
def rows():
yield {"id": 1, "name": "Cleo"}
yield {"id": 2, "name": "Suna"}
' --pk id✨ Options ✨
--pk TEXT— the column(s) to use as the primary key, e.g.id--code TEXT— Python code defining arows()function or an iterable of rows to insert--flatten— flattens nested JSON objects, so{"a": {"b": 1}}becomes{"a_b": 1}--nl— expect newline-delimited JSON-c,--csv— expect CSV input--tsv— expect TSV input--empty-null— treats empty strings asNULL--lines— treats each line as a single value called'line'--text— treats the input as a single value called'text'--convert TEXT— Python code to convert each item 🧚--import TEXT— Python modules to import--delimiter TEXT— the delimiter to use for CSV files--quotechar TEXT— the quote character to use for CSV/TSV--sniff— detect the delimiter and quote character--no-headers— the CSV file has no header row--encoding TEXT— character encoding for input, defaults toutf-8--batch-size INTEGER— commit every X records--stop-after INTEGER— stop after X records--alter— alter the existing table to add any missing columns--not-null TEXT— columns that should be created asNOT NULL--default <TEXT TEXT>...— a default value that should be set for a column--type <TEXT CHOICE>...— the column types to use when creating the table--no-detect-types— treats every CSV/TSV column asTEXT--analyze— runANALYZEat the end of this operation--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint--silent— don't show the progress bar 🙈--strict— appliesSTRICTmode to the created table--ignore— ignores records if the pk already exists--replace— replaces records if the pk already exists--truncate— truncates the table before inserting records, if the table already exists ⚠️ (yes it wipes it, be so for real)-h,--help— show this message and dip
✨ upsert ✨
more on this in Upserting data <#cli-upsert> 💖
Usage: sqlite-utils upsert [OPTIONS] PATH TABLE [FILE]Upsert records based on their primary key, bestie. It works just like 'insert' — except if an incoming record's primary key matches a record that already exists, the existing one gets updated instead 🔥
And if the table already exists and has a primary key, you can straight up omit --pk 👑
oh and one more thing before you go — --type takes a column-name type pair and overrides the type that would've been auto-picked when the table gets created. you're the one in charge now 💅
Example, bestie:
echo '[
{"id": 1, "name": "Lila"},
{"id": 2, "name": "Suna"}
]' | sqlite-utils upsert data.db chickens - --pk id✨ Options ✨
--pk TEXT— the column(s) to use as the primary key, likeid--code TEXT— Python code defining arows()function or an iterable of rows to insert--flatten— flatten nested JSON objects, so{"a": {"b": 1}}becomes{"a_b": 1}--nl— expect newline-delimited JSON-c,--csv— expect CSV input--tsv— expect TSV input--empty-null— treat empty strings asNULL--lines— treat each line as a single value calledline--text— treat input as a single value calledtext--convert TEXT— Python code to convert each item--import TEXT— Python modules to import--delimiter TEXT— delimiter to use for CSV files--quotechar TEXT— quote character to use for CSV/TSV--sniff— detect delimiter and quote character--no-headers— CSV file has no header row--encoding TEXT— character encoding for input, defaults toutf-8--batch-size INTEGER— commit every X records--stop-after INTEGER— stop after X records--alter— alter an existing table to add any missing columns--not-null TEXT— columns that should be created asNOT NULL--default <TEXT TEXT>...— default value that should be set for a column--type <TEXT CHOICE>...— column types to use when creating the table--no-detect-types— treat all CSV/TSV columns asTEXT--analyze— runANALYZEat the end of this operation--load-extension TEXT— path to SQLite extension, with optional:entrypoint--silent— do not show progress bar--strict— apply STRICT mode to created table-h,--help— show this message and exit.
🌈 bulk
peep Executing SQL in bulk <#cli-bulk> for the full tea 🌸
Usage: sqlite-utils bulk [OPTIONS] PATH SQL FILERun parameterized SQL against the list of documents you hand it. yes king, batch that SQL 🔥
Example:
echo '[
{"id": 1, "name": "Lila2"},
{"id": 2, "name": "Suna2"}
]' | sqlite-utils bulk data.db '
update chickens set name = :name where id = :id
' -Options
--batch-size INTEGER— commit every X records--functions TEXT— Python code or a file path defining custom SQL functions; can be used multiple times--flatten— flatten nested JSON objects, so{"a": {"b": 1}}becomes{"a_b": 1}--nl— expect newline-delimited JSON-c,--csv— expect CSV input--tsv— expect TSV input--empty-null— treat empty strings asNULL--lines— treat each line as a single value calledline--text— treat input as a single value calledtext--convert TEXT— Python code to convert each item--import TEXT— Python modules to import--delimiter TEXT— delimiter to use for CSV files--quotechar TEXT— quote character to use for CSV/TSV--sniff— detect delimiter and quote character--no-headers— CSV file has no header row--encoding TEXT— character encoding for input, defaults toutf-8--load-extension TEXT— path to SQLite extension, with optional:entrypoint-h,--help— show this message and exit.
🔍 search
full rundown lives at Executing searches <#cli-search> 💖
Usage: sqlite-utils search [OPTIONS] PATH DBTABLE QRun a full-text search against this table. go find your people 🦋
Example:
sqlite-utils search data.db chickens lilaOptions
-o,--order TEXT— order by ('column'or'column desc')-c,--column TEXT— columns to return--limit INTEGER— number of rows to return — defaults to everything--sql— show SQL query that would be run--quote— apply FTS quoting rules to search term--nl— output newline-delimited JSON--arrays— output rows as arrays instead of objects--csv— output CSV--tsv— output TSV--no-headers— omit headers from CSV/TSV and table/--fmtoutput-t,--table— output as a formatted table--fmt TEXT— table format, and it's giving ✨options✨: one ofasciidoc,colon_grid,double_grid,double_outline,fancy_grid,fancy_outline,github,grid,heavy_grid,heavy_outline,html,jira,latex,latex_booktabs,latex_longtable,latex_raw,mediawiki,mixed_grid,mixed_outline,moinmoin,orgtbl,outline,pipe,plain,presto,pretty,psql,rounded_grid,rounded_outline,rst,simple,simple_grid,simple_outline,textile,tsv,unsafehtml,youtrack--json-cols— detect JSON cols and output them as JSON, not escaped strings--ascii— escape non-ASCII characters in JSON output as\uXXXX--load-extension TEXT— path to SQLite extension, with optional:entrypoint-h,--help— show this message and exit.
🦋 transform
details at Transforming tables <#cli-transform-table> 🎀
Usage: sqlite-utils transform [OPTIONS] PATH TABLETransform a table beyond the capabilities of ALTER TABLE. like, ALTER TABLE could never 💀
Example:
sqlite-utils transform mydb.db mytable \
--drop column1 \
--rename column2 column_renamedOptions
--type <TEXT CHOICE>...— change column type toINTEGER,TEXT,FLOAT,REAL,BLOBorANY--drop TEXT— drop this column--rename <TEXT TEXT>...— rename this column to X-o,--column-order TEXT— reorder columns--not-null TEXT— set this column toNOT NULL--not-null-false TEXT— removeNOT NULLfrom this column--pk TEXT— make this column the primary key--pk-none— remove primary key (convert to rowid table)--default <TEXT TEXT>...— set default value for this column--default-none TEXT— remove default from this column--add-foreign-key <TEXT TEXT TEXT>...— add a foreign key constraint from a column to another table with another column--drop-foreign-key TEXT— drop foreign key constraint for this column--strict/--no-strict— enable or disable STRICT mode (default: preserve current mode)--sql— output SQL without executing it--load-extension TEXT— path to SQLite extension, with optional:entrypoint-h,--help— show this message and exit.
🧚 extract
the whole story is at Extracting columns into a separate table <#cli-extract> 💜
Usage: sqlite-utils extract [OPTIONS] PATH TABLE COLUMNS...Pull one or more columns out into a separate table. decluttering is self care, periodt ✨
Example:
sqlite-utils extract trees.db Street_Trees speciesOptions
--table TEXT— name of the other table to extract columns to--fk-column TEXT— name of the foreign key column to add to the table--rename <TEXT TEXT>...— rename this column in extracted table--load-extension TEXT— path to SQLite extension, with optional:entrypoint-h,--help— show this message and exit.
📜 schema
Showing the schema <#cli-schema> has the receipts 🫶
Usage: sqlite-utils schema [OPTIONS] PATH [TABLES]...Show the full schema for this database, or just for the tables you name.
Example:
sqlite-utils schema trees.dbOptions
--load-extension TEXT— path to SQLite extension, with optional:entrypoint-h,--help— show this message and exit.
🍓 insert-files
Inserting data from files <#cli-insert-files> for the full breakdown 🌸
Usage: sqlite-utils insert-files [OPTIONS] PATH TABLE FILE_OR_DIR...Insert one or more files using BLOB columns in the specified table. your files, but make it database 💅
Example:
sqlite-utils insert-files pics.db images *.gif \
-c name:name \
-c content:content \
-c content_hash:sha256 \
-c created:ctime_iso \
-c modified:mtime_iso \
-c size:size \
--pk nameOptions
-c,--column TEXT— column definitions for the table--pk TEXT— column to use as primary key--alter— alter table to add missing columns--replace— replace files with matching primary key--upsert— upsert files with matching primary key--name TEXT— file name to use--text— store file content asTEXT, notBLOB--encoding TEXT— character encoding for input, defaults toutf-8-s,--silent— don't show a progress bar--load-extension TEXT— path to SQLite extension, with optional:entrypoint-h,--help— show this message and exit.
👑 analyze-tables
Analyzing tables <#cli-analyze-tables>, iykyk 💖
Usage: sqlite-utils analyze-tables [OPTIONS] PATH [TABLES]...Analyze the columns in one or more tables. get to know your data fr fr 🔥
Example:
sqlite-utils analyze-tables data.db treesOptions
-c,--column TEXT— specific columns to analyze--save— save results to_analyze_tablestable--common-limit INTEGER— how many common values--no-most— skip most common values--no-least— skip least common values--load-extension TEXT— path to SQLite extension, with optional:entrypoint-h,--help— show this message and exit.
🎀 convert
Converting data in columns <#cli-convert> is where it's at ⭐
Usage: sqlite-utils convert [OPTIONS] DB_PATH TABLE COLUMNS... CODEConvert columns using Python code you supply. For example:
sqlite-utils convert my.db mytable mycolumn \
'"\n".join(textwrap.wrap(value, 10))' \
--import=textwrap"value" is a variable holding the column value you're converting, bestie 💖 that's the whole vibe.
And CODE can also just be a reference to a callable that takes the value, like:
sqlite-utils convert my.db mytable date r.parsedate
sqlite-utils convert my.db mytable data json.loads --import jsonOh and pro tip: use "-" for CODE to read Python code straight from standard input 🫶 stdin girlies rise up.
🧁 The recipe functions
These common operations are available as recipe functions — lowkey iconic, they come pre-packaged:
r.jsonsplit(value: 'str', delimiter: 'str' = ',', type: 'Callable[[str], object]' = <class 'str'>) -> 'str'
Takes a string like a,b,c and converts it into a JSON array ["a", "b", "c"] ✨ slay.
r.parsedate(value: 'str', dayfirst: 'bool' = False, yearfirst: 'bool' = False, errors: 'object | None' = None) -> 'str | None'
Parses a date and converts it to ISO date format: yyyy-mm-dd
dayfirst=True: treat xx as the day in xx/yy/zzyearfirst=True: treat xx as the year in xx/yy/zzerrors=r.IGNOREto ignore values that cannot be parsederrors=r.SET_NULLto set values that cannot be parsed to null
r.parsedatetime(value: 'str', dayfirst: 'bool' = False, yearfirst: 'bool' = False, errors: 'object | None' = None) -> 'str | None'
Parses a datetime and converts it to ISO datetime format: yyyy-mm-ddTHH:MM:SS
dayfirst=True: treat xx as the day in xx/yy/zzyearfirst=True: treat xx as the year in xx/yy/zzerrors=r.IGNOREto ignore values that cannot be parsederrors=r.SET_NULLto set values that cannot be parsed to null
You can use these recipes like so 🍓:
sqlite-utils convert my.db mytable mycolumn \
'r.jsonsplit(value, delimiter=":")'🌸 Options 🌸
--import TEXT— Python modules to import 📦--dry-run— Show results of running this against first 10 rows (a lil sneak peek, no cap 👀)--multi— Populate columns for keys in returned dictionary--where TEXT— Optional where clause-p,--param <TEXT TEXT>...— Named:parametersfor where clause--output TEXT— Optional separate column to populate with the output--output-type [integer|float|blob|text]— Column type to use for the output column--drop— Drop original column afterwards 💀 (yes it's gone gone)-s,--silent— Don't show a progress bar--pdb— Open pdb debugger on first error 🙈-h,--help— Show this message and exit.
👑 tables
See Listing tables <#cli-tables>.
Usage: sqlite-utils tables [OPTIONS] PATH
List the tables in the database
Example:
sqlite-utils tables trees.db✨ Options ✨
--fts4— Just show FTS4 enabled tables--fts5— Just show FTS5 enabled tables--counts— Include row counts per table--nl— Output newline-delimited JSON--arrays— Output rows as arrays instead of objects--csv— Output CSV--tsv— Output TSV--no-headers— Omit headers from CSV/TSV and table/--fmtoutput-t,--table— Output as a formatted table--fmt TEXT— Table format - one of asciidoc, colon_grid, double_grid, double_outline, fancy_grid, fancy_outline, github, grid, heavy_grid, heavy_outline, html, jira, latex, latex_booktabs, latex_longtable, latex_raw, mediawiki, mixed_grid, mixed_outline, moinmoin, orgtbl, outline, pipe, plain, presto, pretty, psql, rounded_grid, rounded_outline, rst, simple, simple_grid, simple_outline, textile, tsv, unsafehtml, youtrack--json-cols— Detect JSON cols and output them as JSON, not escaped strings--ascii— Escape non-ASCII characters in JSON output as\uXXXX--columns— Include list of columns for each table--schema— Include schema for each table--load-extension TEXT— Path to SQLite extension, with optional:entrypoint-h,--help— Show this message and exit.
🦋 views
See Listing views <#cli-views>.
Usage: sqlite-utils views [OPTIONS] PATH
List the views in the database
Example:
sqlite-utils views trees.db✨ Options ✨
--counts— Include row counts per view--nl— Output newline-delimited JSON--arrays— Output rows as arrays instead of objects--csv— Output CSV--tsv— Output TSV--no-headers— Omit headers from CSV/TSV and table/--fmtoutput-t,--table— Output as a formatted table--fmt TEXT— Table format - one of asciidoc, colon_grid, double_grid, double_outline, fancy_grid, fancy_outline, github, grid, heavy_grid, heavy_outline, html, jira, latex, latex_booktabs, latex_longtable, latex_raw, mediawiki, mixed_grid, mixed_outline, moinmoin, orgtbl, outline, pipe, plain, presto, pretty, psql, rounded_grid, rounded_outline, rst, simple, simple_grid, simple_outline, textile, tsv, unsafehtml, youtrack--json-cols— Detect JSON cols and output them as JSON, not escaped strings--ascii— Escape non-ASCII characters in JSON output as\uXXXX--columns— Include list of columns for each view--schema— Include schema for each view--load-extension TEXT— Path to SQLite extension, with optional:entrypoint-h,--help— Show this message and exit.
🌈 rows
See Returning all rows in a table <#cli-rows>.
Usage: sqlite-utils rows [OPTIONS] PATH DBTABLE
Output all rows in the specified table
Example:
sqlite-utils rows trees.db Trees✨ Options ✨
-c,--column TEXT— Columns to return--where TEXT— Optional where clause-o,--order TEXT— Order by ('column'or'column desc')-p,--param <TEXT TEXT>...— Named:parametersfor where clause--limit INTEGER— Number of rows to return - defaults to everything--offset INTEGER— SQL offset to use--nl— Output newline-delimited JSON--arrays— Output rows as arrays instead of objects--csv— Output CSV--tsv— Output TSV--no-headers— Omit headers from CSV/TSV and table/--fmtoutput-t,--table— Output as a formatted table--fmt TEXT— Table format - one of asciidoc, colon_grid, double_grid, double_outline, fancy_grid, fancy_outline, github, grid, heavy_grid, heavy_outline, html, jira, latex, latex_booktabs, latex_longtable, latex_raw, mediawiki, mixed_grid, mixed_outline, moinmoin, orgtbl, outline, pipe, plain, presto, pretty, psql, rounded_grid, rounded_outline, rst, simple, simple_grid, simple_outline, textile, tsv, unsafehtml, youtrack--json-cols— Detect JSON cols and output them as JSON, not escaped strings--ascii— Escape non-ASCII characters in JSON output as\uXXXX--load-extension TEXT— Path to SQLite extension, with optional:entrypoint-h,--help— Show this message and exit.
💜 triggers
See Listing triggers <#cli-triggers>.
Usage: sqlite-utils triggers [OPTIONS] PATH [TABLES]...
Show triggers configured in this database
Example:
sqlite-utils triggers trees.db✨ Options ✨
--nl— Output newline-delimited JSON--arrays— Output rows as arrays instead of objects--csv— Output CSV--tsv— Output TSV--no-headers— Omit headers from CSV/TSV and table/--fmtoutput-t,--table— Output as a formatted table--fmt TEXT— Table format - one of asciidoc, colon_grid, double_grid, double_outline, fancy_grid, fancy_outline, github, grid, heavy_grid, heavy_outline, html, jira, latex, latex_booktabs, latex_longtable, latex_raw, mediawiki, mixed_grid, mixed_outline, moinmoin, orgtbl, outline, pipe, plain, presto, pretty, psql, rounded_grid, rounded_outline, rst, simple, simple_grid, simple_outline, textile, tsv, unsafehtml, youtrack--json-cols— Detect JSON cols and output them as JSON, not escaped strings--ascii— Escape non-ASCII characters in JSON output as\uXXXX--load-extension TEXT— Path to SQLite extension, with optional:entrypoint-h,--help— Show this message and exit.
🎀 indexes
See Listing indexes <#cli-indexes>.
Usage: sqlite-utils indexes [OPTIONS] PATH [TABLES]...
Show indexes for the whole database or specific tables
Example:
sqlite-utils indexes trees.db Trees--aux— also grab the auxiliary columns 👀--nl— output newline-delimited JSON, one lil object per line fr--arrays— output rows as arrays instead of objects, so no keys, just vibes--csv— output CSV 📄--tsv— output TSV 📄--no-headers— omit headers from CSV/TSV and from table/--fmtoutput. yes, both. she's versatile 💅-t,--table— output as a formatted table, pretty pretty ✨--fmt TEXT— the table format!! pick one of:asciidoc,colon_grid,double_grid,double_outline,fancy_grid,fancy_outline,github,grid,heavy_grid,heavy_outline,html,jira,latex,latex_booktabs,latex_longtable,latex_raw,mediawiki,mixed_grid,mixed_outline,moinmoin,orgtbl,outline,pipe,plain,presto,pretty,psql,rounded_grid,rounded_outline,rst,simple,simple_grid,simple_outline,textile,tsv,unsafehtml,youtrack— that's the whole menu, bestie, no substitutions 🍓--json-cols— detects JSON cols and outputs them as actual JSON, not as escaped strings. it's giving ✨foresight✨--ascii— escape any non-ASCII characters in JSON output as\uXXXX--load-extension TEXT— path to a SQLite extension, with an optional:entrypointtacked on 🧚-h,--help— show this message and exit, always there for u 🫶
✨ create-database ✨
See Creating an empty database <#cli-create-database>.
Usage: sqlite-utils create-database [OPTIONS] PATH
Create a new empty database file
Example:
sqlite-utils create-database trees.dbLike. a whole new database. empty. a blank canvas. main character energy fr 🌱
--enable-wal— turn on WAL mode for the database u just created--init-spatialite— enable SpatiaLite on the newly created database 🗺️--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
🎀 create-table 🎀
See Creating tables <#cli-create-table>.
Usage: sqlite-utils create-table [OPTIONS] PATH TABLE COLUMNS...
Add a table with the specified columns. Columns should be specified using
name, type pairs, for example:
sqlite-utils create-table my.db people \
id integer \
name text \
height real \
photo blob --pk id
Valid column types are text, integer, real, float, blob and any.so like, u feed it name, type pairs and it builds u a table. valid column types are text, integer, real, float, blob and any — memorize those, they're the only ones invited to the party 💖
--pk TEXT— the column that gets to be the primary key 👑--not-null TEXT— columns that should be created asNOT NULL--default <TEXT TEXT>...— default value that should be set for a column--fk <TEXT TEXT TEXT>...— column, other table, other column — sets that trio up as a foreign key 🔗--ignore— if the table already exists, do nothing. chill.--replace— if the table already exists, replace it. not chill (affectionate)--transform— if the table already exists, try to transform the schema ✨--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint--strict— apply STRICT mode to the created table. no chaos allowed 💅-h,--help— show this message and exit
💫 create-index 💫
See Creating indexes <#cli-create-index>.
Usage: sqlite-utils create-index [OPTIONS] PATH TABLE COLUMN...
Add an index to the specified table for the specified columns
Example:
sqlite-utils create-index chickens.db chickens name
To create an index in descending order:
sqlite-utils create-index chickens.db chickens -- -namethat -- before -name is the whole trick btw, it's how u get descending order without the CLI thinking -name is a flag 🙈
--name TEXT— an explicit name for the new index--unique— make this a unique index 🌈--if-not-exists,--ignore— ignore it if the index already exists--analyze— runANALYZEafter creating the index--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
🦋 drop-index 🦋
See Dropping indexes <#cli-drop-index>.
Usage: sqlite-utils drop-index [OPTIONS] PATH TABLE INDEX
Drop an index by index name from the specified table
Example:
sqlite-utils drop-index chickens.db chickens idx_chickens_nameindex name, not column name — read the fine print bestie, it matters 😭
--ignore— ignore it if the index does not exist--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
🌸 migrate 🌸
See Running migrations <#cli-migrate>.
Usage: sqlite-utils migrate [OPTIONS] DB_PATH [MIGRATIONS]...
Apply pending database migrations.
Usage:
sqlite-utils migrate database.db
This will find the migrations.py file in the current directory or
subdirectories and apply any pending migrations.
Or pass paths to one or more migrations.py files directly:
sqlite-utils migrate database.db path/to/migrations.py
Pass --list to see a list of applied and pending migrations without applying
them.
Use --stop-before migration_set:name to stop before a migration. This option
can be used multiple times.soooo — it hunts down migrations.py in the current directory or subdirectories, applies anything pending, and u can also just hand it one or more migrations.py paths directly 📂
--stop-before TEXT— stop before applying this migration. useset:nameto target a migration set. (and yes u can stack this option multiple times)--list— list migrations without running them-v,--verbose— show verbose output, for the nosy girlies-h,--help— show this message and exit
🔥 enable-fts 🔥
See Configuring full-text search <#cli-fts>.
Usage: sqlite-utils enable-fts [OPTIONS] PATH TABLE COLUMN...
Enable full-text search for specific table and columns
Example:
sqlite-utils enable-fts chickens.db chickens namefull-text search?? on?? ur?? table?? iconic. 🔍
--fts4— use FTS4--fts5— use FTS5--tokenize TEXT— tokenizer to use, e.g.porter--create-triggers— create triggers to update the FTS tables when the parent table changes. she keeps herself in sync, queen 👑--replace— replace the existing FTS configuration if it exists--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
💜 populate-fts 💜
Usage: sqlite-utils populate-fts [OPTIONS] PATH TABLE COLUMN...
Re-populate full-text search for specific table and columns
Example:
sqlite-utils populate-fts chickens.db chickens namesame energy as enable-fts but for when the search index needs a lil refresher 🧖
--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
⭐ rebuild-fts ⭐
Usage: sqlite-utils rebuild-fts [OPTIONS] PATH [TABLES]...
Rebuild all or specific full-text search tables
Example:
sqlite-utils rebuild-fts chickens.db chickensgive it no tables and it rebuilds all of them, give it tables and it's surgical about it 💅
--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
🫶 disable-fts 🫶
Usage: sqlite-utils disable-fts [OPTIONS] PATH TABLE
Disable full-text search for specific table
Example:
sqlite-utils disable-fts chickens.db chickensbreaking up with full-text search, respectfully ✌️
--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
✨ optimize ✨
See Optimize <#cli-optimize>.
Usage: sqlite-utils optimize [OPTIONS] PATH [TABLES]...
Optimize all full-text search tables and then run VACUUM - should shrink the
database file
Example:
sqlite-utils optimize chickens.dboptimizes every FTS table then runs VACUUM, and the whole point is that the database file should get smaller 🪄
--no-vacuum— don't runVACUUM--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
💖 analyze 💖
See Optimizing index usage with ANALYZE <#cli-analyze>.
Usage: sqlite-utils analyze [OPTIONS] PATH [NAMES]...
Run ANALYZE against the whole database, or against specific named indexes and
tables
Example:
sqlite-utils analyze chickens.dbno names → the WHOLE database. names → just those indexes and tables. ur choice bestie 📊
-h,--help— show this message and exit
🌈 vacuum 🌈
See Vacuum <#cli-vacuum>.
Usage: sqlite-utils vacuum [OPTIONS] PATH
Run VACUUM against the database
Example:
sqlite-utils vacuum chickens.dbspring cleaning for ur database file 🧹
-h,--help— show this message and exit
🎀 dump 🎀
See Dumping the database to SQL <#cli-dump>.
Usage: sqlite-utils dump [OPTIONS] PATH
Output a SQL dump of the schema and full contents of the database
Example:
sqlite-utils dump chickens.dbschema AND full contents, it's not holding back 😤
--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
🦋 add-column 🦋
See Adding columns <#cli-add-column>.
Usage: sqlite-utils add-column [OPTIONS] PATH TABLE COL_NAME
[integer|int|float|real|text|str|blob|bytes|any]
Add a column to the specified table
Example:
sqlite-utils add-column chickens.db chickens weight floatthe type arg is optional and it's a whole lil enum: integer, int, float, real, text, str, blob, bytes, any 🌸
--fk TEXT— table to reference as a foreign key--fk-col TEXT— referenced column on that foreign key table — if u omit it, it'll automatically use the primary key 👑--not-null-default TEXT— add aNOT NULL DEFAULT 'TEXT'constraint--ignore— if the column already exists, do nothing--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
🔗 add-foreign-key 🔗
See Adding foreign key constraints <#cli-add-foreign-key>.
Usage: sqlite-utils add-foreign-key [OPTIONS] PATH TABLE COLUMN [OTHER_TABLE]
[OTHER_COLUMN]
Add a new foreign key constraint to an existing table
Example:
sqlite-utils add-foreign-key my.db books author_id authors id--ignore— if the foreign key already exists, do nothing--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
💫 add-foreign-keys 💫
See Adding multiple foreign keys at once <#cli-add-foreign-keys>.
Usage: sqlite-utils add-foreign-keys [OPTIONS] PATH [FOREIGN_KEY]...
Add multiple new foreign key constraints to a database
Example:
sqlite-utils add-foreign-keys my.db \
books author_id authors id \
authors country_id countries idbatch mode. plural. many keys, one command, efficiency queen 💅
--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit
👑 index-foreign-keys
okay so bestie, this one pairs with Adding indexes for all foreign keys (#cli-index-foreign-keys) — go read that for the whole lore.
Usage: sqlite-utils index-foreign-keys [OPTIONS] PATHWhat it does: makes sure every single foreign key column has an index on it. Every. Single. One. No column left un-indexed 💅
Example:
sqlite-utils index-foreign-keys chickens.dbOptions:
--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
✨ enable-wal
Full tea over in WAL mode (#cli-wal).
Usage: sqlite-utils enable-wal [OPTIONS] PATH...It turns WAL ON for your database files. That's it, that's the job 🦋
Example:
sqlite-utils enable-wal chickens.dbOptions:
--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
🌸 disable-wal
Usage: sqlite-utils disable-wal [OPTIONS] PATH...The reverse girlie: turns WAL OFF for your database files 💜
Example:
sqlite-utils disable-wal chickens.dbOptions:
--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
🎀 enable-counts
More context in Enabling cached counts (#cli-enable-counts).
Usage: sqlite-utils enable-counts [OPTIONS] PATH [TABLES]...It sets up triggers so a _counts table gets updated with row counts 🤩
Example:
sqlite-utils enable-counts chickens.dbOptions:
--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
💫 reset-counts
Usage: sqlite-utils reset-counts [OPTIONS] PATHRe-does the calculated counts living in the _counts table. Fresh numbers, no cap ✨
Example:
sqlite-utils reset-counts chickens.dbOptions:
--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
🫶 duplicate
See Duplicating tables (#cli-duplicate-table).
Usage: sqlite-utils duplicate [OPTIONS] PATH TABLE NEW_TABLECreates a duplicate of this table, copying across the schema and all the row data. Schema plus data, no crumbs left behind 😌
Options:
--ignore— if the table does not exist, do nothing--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
🦋 rename-table
See Renaming a table (#cli-renaming-tables).
Usage: sqlite-utils rename-table [OPTIONS] PATH TABLE NEW_NAMERename this table. Glow-up unlocked 💅
Options:
--ignore— if the table does not exist, do nothing--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
💀 drop-table
See Dropping tables (#cli-drop-table).
Usage: sqlite-utils drop-table [OPTIONS] PATH TABLEDrops the specified table. Gone. Bye. Periodt.
Example:
sqlite-utils drop-table chickens.db chickensOptions:
--ignore— if the table does not exist, do nothing--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
ngl, dropping a table is permanent. Be sure, bestie 🙈
🌈 create-view
See Creating views (#cli-create-view).
Usage: sqlite-utils create-view [OPTIONS] PATH VIEW SELECTMakes a view for whatever SELECT query you hand it 🎀
Example:
sqlite-utils create-view chickens.db heavy_chickens \
'select * from chickens where weight > 3'Options:
--ignore— if the view already exists, do nothing--replace— if the view already exists, replace it--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
🍓 drop-view
See Dropping views (#cli-drop-view).
Usage: sqlite-utils drop-view [OPTIONS] PATH VIEWDrops the specified view ✨
Example:
sqlite-utils drop-view chickens.db heavy_chickensOptions:
--ignore— if the view does not exist, do nothing--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
🧚 install
See Installing packages (#cli-install).
Usage: sqlite-utils install [OPTIONS] [PACKAGES]...Installs packages from PyPI straight into the same environment as sqlite-utils 👑
Options:
-U,--upgrade— upgrade packages to latest version-e,--editable TEXT— install a project in editable mode from this path-h,--help— show this message and exit.
💜 uninstall
See Uninstalling packages (#cli-uninstall).
Usage: sqlite-utils uninstall [OPTIONS] PACKAGES...Uninstalls Python packages from the sqlite-utils environment. It's giving spring cleaning 🧹
Options:
-y,--yes— don't ask for confirmation-h,--help— show this message and exit.
🌟 add-geometry-column
See SpatiaLite helpers (#cli-spatialite).
Usage: sqlite-utils add-geometry-column [OPTIONS] DB_PATH TABLE COLUMN_NAMEAdds a SpatiaLite geometry column to a table that already exists. Requires the SpatiaLite extension, so make sure that's present.
By default this command tries to load the SpatiaLite extension from the usual paths. Wanna load it from a specific path instead? Use --load-extension 🦋
Options:
-t,--type [point|linestring|polygon|multipoint|multilinestring|multipolygon|geometrycollection|geometry]— specify a geometry type for this column. [default: GEOMETRY]--srid INTEGER— Spatial Reference ID. See https://spatialreference.org for details on specific projections. [default: 4326]--dimensions TEXT— coordinate dimensions. UseXYZfor three-dimensional geometries.--not-null— add aNOT NULLconstraint.--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
🔥 create-spatial-index
See Adding spatial indexes (#cli-spatialite-indexes).
Usage: sqlite-utils create-spatial-index [OPTIONS] DB_PATH TABLE COLUMN_NAMECreates a spatial index on a SpatiaLite geometry column. Heads up though — the table and the geometry column have to already exist before you try to add a spatial index, fr fr.
By default this command tries to load the SpatiaLite extension from the usual paths. Need a specific path? That's what --load-extension is for 💖
Options:
--load-extension TEXT— path to a SQLite extension, with an optional:entrypoint-h,--help— show this message and exit.
🎀 plugins
Usage: sqlite-utils plugins [OPTIONS]Lists your installed plugins. Little roster check 📋
Options:
-h,--help— show this message and exit.
🚀 Upgrading
This page describes the changes you may need to make to your own code or scripts when upgrading between major versions of sqlite-utils.
For the full list of changes in every release, go see the Changelog (#changelog).
🌈 Upgrading from 3.x to 4.0
✨ Requirements
- Python 3.10 or higher is required.
- The
clickdependency must be version 8.3.1 or later.
💅 Command-line changes
- Type detection is now the default for CSV and TSV imports.
sqlite-utils insertandsqlite-utils upsertnow detect column types when importing CSV or TSV data — previously every column was created asTEXTunless you passed--detect-types. To get the old behavior back, pass the new--no-detect-typesflag:
sqlite-utils insert data.db rows data.csv --csv --no-detect-types- Two related things have been removed:
- The
SQLITE_UTILS_DETECT_TYPESenvironment variable. - The old
-d/--detect-typesflag itself. Since detection is now the default, that flag did nothing — remove it from any scripts that used it.
- The
- The
convertcommand no longer skips falsey values.sqlite-utils convertpreviously skipped values that evaluated toFalse(empty strings,0) unless you passed--no-skip-false. All values are now converted and the--no-skip-falseflag has been removed. drop-tableanddrop-viewcheck the object type.sqlite-utils drop-tablenow refuses to drop a view, anddrop-viewrefuses to drop a table. Previously each one would silently drop the wrong type of object if the name matched. If you relied on that (unlikely), use the matching command instead.sqlite-utils tuihas moved to a plugin. The optional terminal interface is now provided by thesqlite-utils-tuiplugin (https://github.com/simonw/sqlite-utils-tui):
sqlite-utils install sqlite-utils-tui👑 Python API changes
db.query()now rejects SQL that does not return rows. This is likely the most common change you'll need to make to existing code.db.query()used to accept any SQL statement — passing one that returns no rows, such as anINSERTorUPDATEwithout aRETURNINGclause or aCREATE TABLE, did nothing at all, silently. Those statements now raise aValueError, and are rolled back so they have no effect on the database. Transaction control statements (BEGIN,COMMIT,END,ROLLBACK,SAVEPOINT,RELEASE) plusVACUUM,ATTACHandDETACHare also rejected with aValueError, without being executed at all. Usedb.execute()for statements that do not return rows:
# 3.x accepted this but silently did nothing:
db.query("update dogs set name = 'Cleopaws'")
# In 4.0 use execute() for SQL that does not return rows:
db.execute("update dogs set name = 'Cleopaws'")db.query()executes immediately.db.query(sql)previously returned a generator that did not execute the SQL until you started iterating over it. The SQL now runs as soon as the method is called — rows are still fetched lazily, but errors in your SQL raise at thedb.query()call site rather than on first iteration, and a write with aRETURNINGclause takes effect even if you never iterate over its results.db.table()no longer returns views.db.table(name)now raises asqlite_utils.db.NoTableexception ifnameis a SQL view. Use the newdb.view(name)method for views:
table = db.table("my_table")
view = db.view("my_view")db["name"]still returns either aTableor aViewdepending on what exists in the database.- Upserts use
INSERT ... ON CONFLICT. Upsert operations now use SQLite'sINSERT ... ON CONFLICT SETsyntax rather than the previousINSERT OR IGNOREfollowed byUPDATE. If your code depends on the old behavior, passuse_old_upsert=Trueto theDatabase()constructor — see Alternative upserts using INSERT OR IGNORE (#python-api-old-upsert). - Upsert records must include their primary keys, bestie 💖
table.upsert()andtable.upsert_all()now raisesqlite_utils.db.PrimaryKeyRequiredif a record is missing a value for any primary key column (or hasNonefor one of them). Before this, those records just got quietly inserted as new rows. Sneaky! 🙈 Relatedly,pk=is now optional when the table already exists with a primary key — it gets detected automatically. 😌 - Floating point columns are now
REAL✨ Auto-detected floating point columns are created with the correct SQLite typeREALinstead ofFLOAT. So if your code inspects column types, expectREAL, periodt. - Generated schemas use double quotes — tables created by this library now wrap table and column names in
"double-quotes"where they previously used[square-braces]. If you comparetable.schemastrings against expected values you will need to update them. No cap. table.convert()no longer skips falsey values 🫶 Matching the CLI change above,table.convert()now converts every value. Theskip_falseparameter has been removed — it previously defaulted toTrue, skipping empty strings and other falsey values.- Null values are no longer extracted into lookup tables 💅
table.extract()and thesqlite-utils extractcommand leave rows alone if every extracted column is null — the new foreign key column is left as null instead of pointing at an all-null record in the lookup table. Theextracts=insert option similarly keepsNonevalues as null. Relatedly,table.lookup()now compares values usingISso that looking up a value containingNonereturns the existing matching row — previously it inserted a duplicate row on every single call. not that 😭 ensure_autocommit_off()is nowensure_autocommit_on()🦋 Thedb.ensure_autocommit_off()context manager has been renamed todb.ensure_autocommit_on(). The old name described the literal opposite of what the method did: it temporarily puts the connection into driver-level autocommit mode (by settingisolation_level = None), so that statements such asPRAGMA journal_mode=walcan run outside of an implicit transaction. The behavior is unchanged — just update any calls to use the new name.View.enable_fts()has been removed 💀 TheViewclass previously had anenable_fts()method that existed only to raiseNotImplementedError— full-text search is not supported for views, that's the whole tea. Calling it now raisesAttributeErrorlike any other missing method.ForeignKeyis now a dataclass, not a namedtuple 👑 TheForeignKeyobjects returned bytable.foreign_keysgained new fields —columns,other_columns,is_compound,on_deleteandon_update— so that compound (multi-column) foreign keys and foreign key actions can be represented. To make room for those fields cleanly,ForeignKeyis now a dataclass rather than a namedtuple, so it can no longer be unpacked or indexed as a tuple. Access its fields by name instead:
# 3.x - tuple unpacking, no longer works:
for table, column, other_table, other_column in db["courses"].foreign_keys:
...
# 4.0 - access fields by name:
for fk in db["courses"].foreign_keys:
fk.table, fk.column, fk.other_table, fk.other_columnAttempting the old unpacking or fk[0] indexing now raises TypeError, so any code using those patterns will fail loudly rather than silently misbehave — which, honestly, iconic behavior. Like the old namedtuple, ForeignKey instances are immutable and hashable — they can be collected into sets and used as dictionary keys. Note that equality now includes the on_delete and on_update actions: a ForeignKey with ON DELETE CASCADE is not equal to one without. ✨
- Compound foreign keys — previously returned as one
ForeignKeyper column, misleadingly suggesting several independent single-column keys — are now returned as a singleForeignKeywithis_compound=True. For these, the scalarcolumnandother_columnfields areNone; use thecolumnsandother_columnstuples instead. Single-column foreign keys are unaffected apart from the class change:column/other_columnbehave as before andcolumns/other_columnsare one-item tuples. 🎀 - Two related behavior changes to
table.transform()🌈 compound foreign keys now survive a transform (previously they were split into separate single-column keys), andON DELETE/ON UPDATEactions such asON DELETE CASCADEare now preserved (previously they were silently stripped from the schema). Gone and never coming back, the silent stripping. - Validation errors raise
ValueError🔥 Invalid arguments to Python API methods — for examplecreate_table()with no columns, orignore=Truetogether withreplace=True— now raiseValueError. They previously raisedAssertionErrorfrom bareassertstatements, which were silently skipped underpython -O. Wild. - Transaction behavior is now well-defined ⭐ 4.0 introduces the
db.atomic()context manager (#python-api-atomic) and uses it consistently for every write operation — the full model is described in Transactions and saving your changes (#python-api-transactions). Changes you may notice:- Write statements executed with raw
db.execute()calls now commit automatically, unless a transaction is already open, in which case they join it. Previously they opened an implicit transaction that nothing committed — so if your code useddb.execute()for writes and relied ondb.conn.rollback()to undo them, open an explicit transaction with the newdb.begin()method first. 🫡 - Multi-step operations such as
table.transform()no longer commit an existing transaction you have open — they use savepoints inside it instead. db.enable_wal()anddb.disable_wal()raise asqlite_utils.db.TransactionErrorif called while a transaction is open, instead of silently committing it.- Using
Databaseas a context manager (with Database(path) as db:) closes the connection on exit without committing — a transaction you explicitly opened withdb.begin()and did not commit is rolled back. Database()rejects connections created with the Python 3.12+sqlite3.connect(..., autocommit=True)orautocommit=Falseoptions, raisingsqlite_utils.db.TransactionError. On those connections every write the library made was silently discarded when the connection closed. yeah, all of it. poof. 💀
- Write statements executed with raw
🎁 Packaging changes
sqlite-utilsnow usespyproject.tomlin place ofsetup.py.pipis now a runtime dependency, used by thesqlite-utils installanduninstallcommands.
🌟 New features to be aware of
Not breaking changes, but new in 4.0 and worth knowing about when you upgrade, bestie:
- A database migrations system (
#migrations), incorporating the functionality of thesqlite-migrateplugin. If you used that plugin, the built-in system reads the same_sqlite_migrationstable — your applied migrations will not run again. Update your migration files to usefrom sqlite_utils import Migrations. 🧚 db.atomic()(#python-api-atomic) for nested transaction support. Nested!! 💜table.insert_all()andtable.upsert_all()accept an iterator of lists or tuples as an alternative to dictionaries — see Inserting data from a list or tuple iterator (#python-api-insert-lists).
🌸 Upgrading from 2.x to 3.0
The 3.0 release redesigned search. The breaking changes were minor:
table.search()returns a generator of dictionaries, sorted by relevance. It previously returned a list of tuples sorted by rowid.- The
-cshortcut for--csvand the-fshortcut for--fmtwere removed from the CLI — use the full option names.
🍓 Upgrading from 1.x to 2.0
The 2.0 release changed the meaning of upsert. In 1.x, table.upsert() and table.upsert_all() actually performed INSERT OR REPLACE operations — entirely replacing the existing row. Since 2.0 an upsert updates only the columns you provide, leaving other columns untouched. glow-up fr fr ✨
If you want the 1.x behavior, use table.insert(..., replace=True) or table.insert_all(..., replace=True) instead.
💖 Contributing
Development of sqlite-utils takes place in the sqlite-utils GitHub repository (https://github.com/simonw/sqlite-utils).
All improvements to the software should start with an issue. Read How I build a feature (https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/) for a detailed description of the recommended process for building bug fixes or enhancements. 📝
🦋 Obtaining the code
To work on this library locally, first checkout the code:
git clone git@github.com:simonw/sqlite-utils
cd sqlite-utilsUse uv run to run the development version of the tool:
uv run sqlite-utils --help🧪 Running the tests
Use uv run to run the tests:
uv run pytest📚 Building the documentation
To build the documentation run this command:
uv run make livehtml --directory docsThis will start a server on port 8000 that will serve the documentation and live-reload any time you make an edit to a .rst file. obsessed 😍
The cog tool (https://github.com/nedbat/cog) is used to maintain portions of the documentation. You can run it like so:
uv run cog -r docs/*.rst💅 Linting and formatting
sqlite-utils uses Black (https://black.readthedocs.io/) for code formatting, and flake8 (https://flake8.pycqa.org/) and mypy (https://mypy.readthedocs.io/) for linting and type checking:
uv run black .Linting tools can be run like this:
uv run flake8
uv run mypy sqlite_utilsAll three of these tools are run by our CI mechanism against every commit and pull request. every single one. 👑
🌈 Using Just
If you install Just (https://github.com/casey/just) you can use it to manage your local development environment.
To run all of the tests and linters:
justTo run tests, or run a specific test module or test by name:
just test # All tests
just test tests/test_cli_memory.py # Just this module
just test -k test_memory_no_detect_types # Just this testTo run just the linters:
just lintTo apply Black to your code:
just blackTo update documentation using Cog:
just cogTo run the live documentation server (this will run Cog first):
just docsAnd to list all available commands:
just -l🚀 Release process
Releases are performed using tags. When a new release is published on GitHub, a GitHub Actions workflow (https://github.com/simonw/sqlite-utils/blob/main/.github/workflows/publish.yml) will perform the following:
- Run the unit tests against all supported Python versions. If the tests pass...
- Build a wheel bundle of the underlying Python source code
- Push that new wheel up to PyPI: (
https://pypi.org/project/sqlite-utils/)
To deploy new releases you will need to have push access to the GitHub repository.
sqlite-utils follows Semantic Versioning (https://semver.org/):
major.minor.patchWe increment major for backwards-incompatible releases.
We increment minor for new features.
We increment patch for bugfix releass. 🎀
To drop a new version you start by making a commit that bumps the version number in pyproject.toml and updates the changelog (linked over at #changelog) with the highlights of the new version. There's an example commit you can peek at right here: https://github.com/simonw/sqlite-utils/commit/b491f22d817836829965516983a3f4c3c72c05fc — it looks a little something like this:
# Update changelog
git commit -m " Release 3.29
Refs #423, #458, #467, #469, #470, #471, #472, #475" -a
git pushReferencing the issues that are part of the release in the commit message is a whole thing, bestie — it's what makes the name of the release show up on those issue pages, like this one here: https://github.com/simonw/sqlite-utils/issues/458#ref-commit-b491f22 ✨
You can generate the list of issue references for a specific release by copying and pasting text from the release notes or the GitHub changes-since-last-release view into this Extract issue numbers from pasted text tool: https://observablehq.com/@simonw/extract-issue-numbers-from-pasted-text 🎀
To create the tag for the release, make a new release on GitHub matching the new version number: https://github.com/simonw/sqlite-utils/releases/new. You can convert the release notes to Markdown by copying and pasting the rendered HTML into this Paste to Markdown tool: https://euangoddard.github.io/clipboard2markdown/ 💅
🌈 Changelog 🌈
quick key bestie: every#NNNbelow points athttps://github.com/simonw/sqlite-utils/issues/NNN, unless it's a pull request, in which case it'shttps://github.com/simonw/sqlite-utils/pull/NNN. I'll tell you which one each time so nothing gets lost 💖
4.2.1 (2026-08-13)
- Fix for a
No module named 'typing_extensions'crashing bug that got accidentally shipped in version 4.2. (issue#842)
4.2 (2026-08-13)
- Brand new
table.checks,table.column_checksandtable.table_checksintrospection properties that expose column-level and table-level CHECK constraints. No cap, this is so useful. (issue#834) - New
sqlite_utils.ANYmarker type for creating and introspecting SQLite ANY columns. The Python API and the CLI can create, add and transform these columns, andtable.transform()andtable.extract()now preserve ANY columns and their values in STRICT tables. (issue#790) table.default_valuesnow unescapes doubled single quotes in string defaults, so a default such as'O''Brien'comes back as"O'Brien". Thanks, ikatyal2110! (pull#811)table.default_valuesnow decodes unquotedTRUE,FALSEandNULLdefault literals asTrue,FalseandNonerespectively. (issue#836)table.enable_fts(..., tokenize=...)andsqlite-utils enable-fts --tokenizenow safely quote the tokenizer argument, which stops a crafted value from injecting additional SQL. Thanks, Bunlong Heng! (pull#828)rows_where(),pks_and_rows_where(),search()andsearch_sql()now supportoffset=without requiringlimit=. Thesqlite-utils rows --offsetoption now works without--limittoo. Thanks, ethanhawkes-gif! (issue#816, pull#821)- Empty or whitespace-only input passed to
rows_from_file()is now handled as an empty CSV file instead of raisingcsv.Error. Thanks, Rami Abdelrazzaq! (issue#808, pull#837) sqlite-utils convert --dry-runnow works for table and column names that contain closing square brackets. (issue#829)table.indexesandtable.xindexesnow work for table, index and column names that contain double quotes. This also fixestable.transform()for tables with those identifiers. Thanks, nyxst4ck! (issue#824, pull#825)- Improved type annotations throughout the package, plus Pyright regression checks added to CI. (issue
#833) - Changing a TEXT column to INTEGER, FLOAT or REAL using
table.transform()orsqlite-utils transformnow converts exact empty strings to NULL. Before this, they stayed as empty strings sitting in a numeric column, which was lowkey cursed. Thanks, ikatyal2110! (issue#488, pull#805)
And table.transform() can handle so many more edge-cases now, it's giving main character energy:
table.transform()now preserves column-level and composite UNIQUE constraints, including constraint names, collations, sort order and ON CONFLICT behavior. Renaming columns updates those constraints, and dropping any constituent column removes the entire constraint. (issue#762)table.transform()now preserves AUTOINCREMENT primary keys and their sequence high-water marks. Before, a transform would strip out AUTOINCREMENT and could reuse deleted row IDs — absolute nightmare fuel. (issue#602)table.transform()now preserves CHECK constraints, including comments inside their expressions. Renaming a column rewrites identifier references in checks without touching string literals or function names. Dropping a column drops a check owned by that column, and raisesTransformErrorif a remaining check depends on it. (issue#762)table.transform()now preserves comments sitting immediately before or after column definitions. Those comments move with the column when it's renamed or reordered, and they get removed if the column is dropped. (issue#762)table.transform(rename=...)now preserves explicit indexes on renamed columns by dropping and recreating those indexes against the new column names. Before this, it just raised aTransformError. (issue#822)table.transform()now works for tables that are referenced by views. Previously theALTER TABLE ... RENAME TOstep raisedno such tableif a view referenced the table being transformed. View definitions are left unchanged — see Tables referenced by views (#python-api-transform-views). This also fixes a bug wheretransform(keep_table=...)silently rewrote dependent views to point at the frozen backup table instead of the live one. (issue#831)
3.39.1 (2026-07-25)
- Fixed a bug where
table.delete_where()left the connection in an open transaction, which meant deleted rows would get silently restored when the connection was closed. The audacity of that bug, fr fr. (issue#815)
4.1.1 (2026-07-12)
table.transform()now raises aTransactionErrorif it's called while a transaction is open withPRAGMA foreign_keysenabled and the table is referenced by foreign keys with destructive ON DELETE actions — CASCADE, SET NULL or SET DEFAULT. The pragma can't be changed inside a transaction, so previously dropping the old table as part of the transform could fire those actions and silently delete or modify referencing rows. See Foreign keys and transactions (#python-api-transform-foreign-keys-transactions) for details and workarounds. (issue#794)- The CLI (
#cli) and Python API (#python-api) docs now cross-reference each other: CLI sections link to the equivalent Python API functionality, and Python API sections link back to the corresponding CLI command. Obsessed with the mutual linking, honestly. (issue#791)
4.1 (2026-07-11)
sqlite-utils insertandsqlite-utils upsertnow accept a--codeoption for providing a block of Python code (#cli-insert-code) — or a path to a.pyfile — that defines arows()function or arowsiterable of rows to insert, as an alternative to importing from a file. (issue#684)sqlite-utils insertandsqlite-utils upsertnow accept--type column-name typeto override the type that gets automatically chosen when the table is created (#cli-insert-csv-tsv-column-types). This is chef's kiss for CSV or TSV columns such as ZIP codes that look like integers but should be stored as TEXT to preserve leading zeros. (issue#131)- New
table.drop_index(name)method andsqlite-utils drop-indexcommand for dropping an index by name. Both acceptignore=True/--ignoreto ignore a missing index, because sometimes you just don't want drama. (issue#626) sqlite-utils querycan now read the SQL query from standard input by passing-in place of the query, for example:
echo "select * from dogs" | sqlite-utils query dogs.db -(issue #765)
sqlite-utils upsertcan now infer the primary key of an existing table, so--pkcan be omitted when upserting into a table that already has a primary key. So convenient of it.table.transform()andtable.transform_sql()now acceptstrict=Trueorstrict=Falseto change a table's SQLite strict mode (https://www.sqlite.org/stricttables.html). Omitting the option preserves the existing mode. (issue#787)- The
sqlite-utils transformcommand now accepts--strictand--no-strictto change a table's strict mode. (issue#787)
4.0 (2026-07-07)
The 4.0 release includes some minor backwards-incompatible fixes (which is why the major version number got bumped, iykyk) and introduces three major new features:
- Database migrations (
#migrations), providing a structured mechanism for evolving a project's schema over time. (issue#752) - Nested transaction support (
#python-api-atomic) viadb.atomic(), plus numerous improvements to how transactions work across the library. (issue#755) - Support for compound foreign keys (
#python-api-compound-foreign-keys), including creation, transformation and introspection throughtable.foreign_keys(#python-api-introspection-foreign-keys). (issue#594)
Other notable changes include:
- Upserts now use SQLite's
INSERT ... ON CONFLICT ... DO UPDATE SETsyntax, detect existing table primary keys automatically and reject records that are missing required primary key values. (issue#652) db.query()now runs immediately, and it straight-up rejects statements that don't return rows — no cap. Wanna do writes or DDL? That'sdb.execute()'s job, bestie. 💅- CSV and TSV imports now detect column types by default ✨, but when you're inserting into a table that already exists, those existing column types are preserved. (#679 —
https://github.com/simonw/sqlite-utils/issues/679) - Foreign key handling now preserves
ON DELETE/ON UPDATEactions during transforms, and resolves referenced primary keys way more accurately. (#530 —https://github.com/simonw/sqlite-utils/issues/530) - Column names you pass to Python API methods are now matched case-insensitively, mirroring SQLite's own identifier behavior — iconic, honestly. (#760 —
https://github.com/simonw/sqlite-utils/issues/760) - The command-line tool now emits UTF-8 JSON output by default 🌈, with
--asciiavailable to restore the old escaped output if you need it. (#625 —https://github.com/simonw/sqlite-utils/issues/625) table.extract()andextracts=no longer create lookup table records for all-null values. (#186 —https://github.com/simonw/sqlite-utils/issues/186)
See Upgrading from 3.x to 4.0 (#upgrading-3-to-4) for details on the backwards-incompatible changes — that's the part you actually need to read before you upgrade, fr fr. 🫶
The detailed release notes for the features and fixes shipped during the 4.0 pre-release cycle are available in 4.0a0, 4.0a1, 4.0rc1, 4.0rc2, 4.0rc3 and 4.0rc4.
🐛 Bug fixes since 4.0rc4 🐛
- Fixed the 4.0 regressions in insert/upsert against tables that use SQLite's implicit
rowidprimary key. Passingpk="rowid",pk="_rowid_"orpk="oid"now works again for rowid tables, andlast_pkgets set correctly. (#781 —https://github.com/simonw/sqlite-utils/issues/781) - Fixed
insert(..., ignore=True)andinsert_all(..., ignore=True)so an ignored insert that conflicts with an existing primary key row now reports that existing row inlast_rowidandlast_pkwhere possible. This also works for compound primary keys and list-mode inserts. (#783 —https://github.com/simonw/sqlite-utils/issues/783)
🌸 4.0rc4 (2026-07-06) 🌸
- Breaking change, so listen up:
table.extract()— and thesqlite-utils extractcommand — no longer extract rows where every extracted column is null. Those rows now keep a null value in the new foreign key column instead of pointing at an all-null record in the lookup table. When you're extracting multiple columns, rows are still extracted if at least one of the columns has a value. (#186 —https://github.com/simonw/sqlite-utils/issues/186) - The
extracts=option totable.insert()and friends no longer creates a lookup table record forNonevalues — the column value just stays null. Before this, every batch of inserted rows containing aNonevalue would add a duplicate null record to the lookup table. 😭 - Fixed a bug where
table.lookup()inserted a duplicate row on every single call if any of the lookup values wereNone. Lookup values are now compared usingISso thatNonevalues match existing rows correctly. 💀 - JSON output from the command-line tool no longer escapes non-ASCII characters, so
sqlite-utils data.db "select '日本語' as text"now outputs[{"text": "日本語"}]. This matches how values were already stored by insert, and how CSV/TSV output already behaved. A new--asciioption restores the previous behavior of escaping non-ASCII characters, for output destinations that cannot handle UTF-8 — seeUnicode characters in JSON(#cli-query-json-ascii). The option is available on thequery,rows,search,tables,views,triggers,indexesandmemorycommands. Theconvert --multi --dry-runpreview and plugins output also no longer escape non-ASCII characters. (#625 —https://github.com/simonw/sqlite-utils/issues/625) --no-headersnow omits the header row from--fmtand--tableoutput, not just CSV and TSV output. (#566 —https://github.com/simonw/sqlite-utils/issues/566)table.insert_all(..., pk=...)now raisesInvalidColumnsifpk=names columns that don't exist in an existing table. Before, this was all over the place — single-row inserts raised aKeyErrorwhile other row counts succeeded. So messy. (#732 —https://github.com/simonw/sqlite-utils/issues/732)- Fixed an
IndexErrorfromtable.insert(..., pk=..., ignore=True)when an ignored insert followed writes to another table on the same connection.last_pkis now populated from the explicit primary key value instead of looking up a stalelastrowid. (#554 —https://github.com/simonw/sqlite-utils/issues/554) - Fixed a bug where a failed write statement executed with
db.execute()left the driver's implicit transaction open. Every subsequent write then joined that phantom transaction, which nothing ever committed, so their work was silently rolled back when the connection was closed. The implicit transaction opened by a failed statement is now rolled back before the exception is raised. A failed write inside a transaction opened withdb.begin()ordb.atomic()leaves that transaction open and untouched, same as before. - Fixed a bug where transaction-control statements prefixed with an empty statement, like
db.query("; COMMIT"), or with a UTF-8 byte order mark, slipped past the check that rejects them — committing the caller's open transaction before raising a confusingOperationalError. The keyword scanner used bydb.query()anddb.execute()now skips leading;and byte order marks, matching what thesqlite3driver tolerates before the first token, so these statements are rejected with aValueErrorwithout being executed. The same fix meansdb.execute("; BEGIN")no longer auto-commits the transaction it just opened. 🎀 - Documented a limitation of
db.query(): aPRAGMAstatement that returns no rows raises aValueErrorbut still takes effect, becausePRAGMAstatements run outside the savepoint guard used to roll back other rejected statements. Usedb.execute()for row-lessPRAGMAstatements. - Fixed exception masking when a statement destroys the enclosing transaction. An error such as a
RAISE(ROLLBACK)trigger or anINSERT OR ROLLBACKconflict rolls back the whole transaction, destroying every savepoint — the cleanup indb.atomic()anddb.query()then failed withOperationalError: no such savepoint(orcannot rollback - no transaction is active), hiding the originalIntegrityErrorfrom code that tried to catch it. Cleanup now checks whether a transaction is still open first, so the original exception propagates. 🙈 sqlite-utils migrate --listis now read-only even when the migrations file uses the legacysqlite_migrate.Migrationsclass, whose listing methods create the_sqlite_migrationstable as a side effect. The listing now runs inside a transaction that is rolled back. ✨sqlite-utils insert ... --pk <missing column>andsqlite-utils extract <missing column>now show a cleanError:message instead of a raw Python traceback. Theextractcommand also shows a clean error when pointed at a view.- Fixed a bug where running
table.extract()more than once against the same lookup table inserted duplicate rows for values containing null — SQLite unique indexes treatNULLvalues as distinct, soINSERT OR IGNOREalone could not dedupe them. Each repeat extract added another copy that nothing referenced. The insert now uses anIS-basedNOT EXISTSguard so null-containing rows match existing lookup rows. db.add_foreign_keys()no longer silently ignores requestedON DELETE/ON UPDATEactions when a foreign key with the same columns already exists — it raisesAlterErrorsuggestingtable.transform(), since the actions of an existing foreign key cannot be changed in place. Exact duplicates, including actions, are still skipped so repeated calls stay idempotent. The method also now validates that compound foreign keys have the same number of columns on both sides, instead of silently discarding the extra columns. 💯db.ensure_autocommit_on()now raisesTransactionErrorif called while a transaction is open. Assigningisolation_levelcommits any pending transaction as a side effect, so entering the block silently committed the caller's open transaction and made a laterrollback()a total no-op. 😤sqlite-utils migrate --stop-beforenow exits with an error if the named migration has already been applied. Previously the name passed validation but was only checked against pending migrations, so every migration after it was silently applied — the exact outcome--stop-beforeexists to prevent.Migrations.apply(db, stop_before=...)raisesValueErrorin the same situation, before applying anything.- Fixed a regression where
table.insert(..., pk=..., alter=True)raisedInvalidColumnsif the primary key column did not exist in the table yet. Withalter=Truethe check now waits until the record keys are known, so a pk column supplied by the records is added by the alter as it was in 3.x. A pk column found in neither the table nor the records still raisesInvalidColumns. - Fixed a bug where inserting CSV or TSV data into an existing table rewrote that table's column types to match the incoming file. Type detection is the default in 4.0, so
sqlite-utils insert data.db places places.csv --csvagainst a table with aTEXTzip code column would convert the column toINTEGERand corrupt values with leading zeros —"01234"became1234. Detected types are now only applied when the insert or upsert command creates the table. 🔥 - Fixed
pks_and_rows_where()raisingAttributeErrorwhen called on a view, and it no longer double-quotes the synthesizedrowidcolumn in its generated SQL — SQLite turns a double-quoted identifier that does not resolve into a string literal, which on a view produced a confusingKeyErrorinstead of theOperationalErrorraised in 3.x. Compound primary keys returned by this method now followPRIMARY KEYdeclaration order. - The
foreign_keys=argument tocreate()andinsert()accepts a mixed list ofForeignKeyobjects, tuples and column name strings again. In 4.0 pre-releases, mixingForeignKeyobjects with tuples raised aValueError— a regression from 3.x, whereForeignKeywas anamedtupleand passed the tuple checks. 🌈 ForeignKeyobjects are hashable again. The 4.0 change fromnamedtupletodataclassaccidentally made them unhashable, breaking patterns likeset(table.foreign_keys)that worked in 3.x.ForeignKeyis now a frozen dataclass — immutable and hashable, like thenamedtuplewas. 👑- Fixed a bug where compound primary key columns came back in table column order instead of PRIMARY KEY declaration order 💀 For a table declared as
CREATE TABLE other (b TEXT, a TEXT, PRIMARY KEY (a, b))an implicitFOREIGN KEY (x, y) REFERENCES otherwas being introspected as referencing(b, a)when SQLite actually resolves it as(a, b)— and then runningtransform()on a table like that rewrote the schema with the inverted column order, silently reversing the meaning of the constraint and causing foreign key errors on perfectly valid data 😭table.pks, compound foreign key guessing andtransform()now all use the primary key declaration order, andtransform()no longer reorders a compoundPRIMARY KEY (b, a)into table column order. slay.
4.0rc3 (2026-07-05) ✨
💅 Breaking changes 💅
table.foreign_keysnow returnsForeignKeyobjects that are dataclasses instead ofnamedtupleinstances, so bestie you can no longer unpack or index them as(table, column, other_table, other_column)tuples — access the fields by name instead, ok? 👑 Compound (multi-column) foreign keys are now represented as a singleForeignKeywithis_compound=Trueand populatedcolumns/other_columnstuples, wherecolumnandother_columnareNone. Before, they came back as oneForeignKeyper column, which was lowkey misleading and made it look like you had several independent foreign keys 🙈 See Upgrading from 3.x to 4.0 for the details. (#594 —https://github.com/simonw/sqlite-utils/issues/594)- Removed support for using
sqlean.pyas a drop-in replacement for the Python standard librarysqlite3module.sqlite-utilswill now usepysqlite3if it's installed, otherwise it usessqlite3from the standard library. periodt. - The
db.ensure_autocommit_off()context manager has been renamed todb.ensure_autocommit_on(), because the old name literally described the opposite of what it did 😭 not me naming my function wrong. The method temporarily puts the connection into driver-level autocommit mode — by settingisolation_level = None— so that statements such asPRAGMA journal_mode=walcan run outside of an implicit transaction. (#705 —https://github.com/simonw/sqlite-utils/issues/705)
🦋 Compound foreign key support 🦋
- Tables can now be created with compound foreign keys, by passing tuples of column names in
foreign_keys=:foreign_keys=[(("campus_name", "dept_code"), "departments")]. The referenced columns default to the compound primary key of the other table. Compound keys are rendered as table-levelFOREIGN KEYconstraints in the generated schema. iconic. table.transform()now preserves compound foreign keys, applying any column renames to them. Dropping a column that is part of a compound foreign key drops the whole constraint, matching the existing single-column behavior.drop_foreign_keys=accepts a bare column name — which drops any foreign key that column participates in — or a tuple of columns to target a compound key precisely 💖table.add_foreign_key()anddb.add_foreign_keys()accept tuples of column names to add a compound foreign key to an existing table.db.index_foreign_keys()creates a single composite index for a compound foreign key. one index, one vibe ✨
🔥 Other foreign key improvements 🔥
ForeignKeynow exposeson_deleteandon_updatefields reflecting the foreign key'sON DELETE/ON UPDATEactions, andtable.transform()preserves those actions. Previously a transform silently stripped clauses such asON DELETE CASCADEfrom the table schema 💀table.add_foreign_key()accepts newon_delete=andon_update=parameters for creating foreign keys with actions, e.g.table.add_foreign_key("author_id", "authors", "id", on_delete="CASCADE"). (#530 —https://github.com/simonw/sqlite-utils/issues/530)- Foreign keys declared as
REFERENCES other_tablewith no explicit column are now resolved to the other table's primary key bytable.foreign_keys, instead of reportingother_column=None. we love an auto-resolve 🫶 - Fixed a
TypeErrorwhen sortingForeignKeyobjects where some were compound.
🌸 Case-insensitive column matching 🌸
Column names passed to Python API methods are now matched against the table schema case-insensitively, mirroring how SQLite itself treats identifiers. Previously many methods accepted mixed-case identifiers in the SQL they generated but then failed — or silently did nothing — when performing Python-side comparisons against the schema. (#760 — https://github.com/simonw/sqlite-utils/issues/760) Fixes include:
table.insert()andtable.upsert()now populatetable.last_pkcorrectly when thepk=argument uses different casing to the table schema or the record keys — previously this raised aKeyErrorafter the row had already been written 😭- Upserts no longer raise or misbehave when the casing of
pk=differs from the casing of the record keys. The primary key columns are correctly excluded from the generatedDO UPDATE SETclause. table.transform()argumentstypes=,rename=,drop=,pk=,not_null=,defaults=,column_order=anddrop_foreign_keys=all resolve column names case-insensitively. Previously options likerename={"name": "title"}against a column calledNamewere silently ignored 🫠db.create_table(..., transform=True)now recognizes existing columns that differ only by case, instead of attempting to add them again and failing with duplicate column name. The casing used in the existing schema is preserved 💜table.lookup()returns the primary key value even ifpk=casing differs from the schema, and recognizes existing unique indexes case-insensitively instead of creating redundant ones.table.extract()andtable.convert()— includingmulti=Trueandoutput=— accept column names in any casing.- Foreign key columns are validated and recorded using the casing of the actual schema columns, in
foreign_keys=when creating tables,db.add_foreign_keys(),table.add_foreign_key()andtable.add_column(fk_col=...). Duplicate foreign key detection is also case-insensitive ✨ table.create()withpk=,not_null=,defaults=orcolumn_order=referencing columns using different casing no longer creates an unwanted extra primary key column or raises aValueError.
🎀 Everything else 🎀
- Fixed a bug where
table.transform()could convertDEFAULT TRUE,DEFAULT FALSEandDEFAULT NULLcolumn defaults into quoted string defaults when rebuilding a table. Thanks, Vincent Gao (https://github.com/gaoflow). (#764 —https://github.com/simonw/sqlite-utils/pull/764)
4.0rc2 (2026-07-04) ⭐
💅 Breaking changes: 💅
- Write statements executed with
db.execute()are now committed automatically, unless a transaction is already open in which case they join it. Previously they opened an implicit transaction that stayed open until something committed it — writes appeared to work when read on the same connection but were silently rolled back when the connection closed 💀 Code that relied on rolling back uncommitteddb.execute()writes should use the newdb.begin()method to open an explicit transaction first. The transaction model is documented in full at Transactions and saving your changes. db.query()now executes its SQL as soon as it is called, rather than waiting until the returned generator is first iterated. Rows are still fetched lazily during iteration. SQL errors are now raised at the call site, statements such asINSERT ... RETURNINGare executed and committed immediately without needing to iterate over their results, and passing a statement that returns no rows — previously a silent no-op — now raises aValueErrorrecommendingdb.execute()instead. A statement rejected this way is rolled back before the error is raised, so it has no effect on the database. no cap.- Python API validation errors now raise
ValueErrorinstead ofAssertionError. Previously invalid arguments — such ascreate_table()with no columns,transform()on a table that does not exist, or passing bothignore=Trueandreplace=True— were rejected using bareassertstatements, which are silently skipped when Python runs with the-Oflag 😳 Code that caughtAssertionErrorfor these cases should catchValueErrorinstead. table.upsert()andtable.upsert_all()now raisePrimaryKeyRequiredif a record is missing a value for any primary key column, or has a value ofNonefor one. Previously such records — which can never match an existing row — were quietly inserted as brand new rows, or triggered a confusingKeyErrorafter the insert had already taken place 🙈db.enable_wal()anddb.disable_wal()now raise asqlite_utils.db.TransactionErrorif called while a transaction is open. Previously they would silently commit the open transaction as a side effect of changing the journal mode, breaking the rollback guarantee ofdb.atomic()and of user-managed transactions.- The
Viewclass no longer has anenable_fts()method. It existed only to raiseNotImplementedError, since full-text search is not supported for views — calling it now raisesAttributeErrorinstead, and the method no longer appears in the API reference. Thesqlite-utils enable-ftscommand shows a clean error when pointed at a view. goodbye pointless method 💅 - The no-op
-d/--detect-typesflag has been removed from theinsertandupsertcommands. Type detection has been the default for CSV/TSV data since 4.0a1, so the flag did nothing — invocations using it should simply drop it.--no-detect-typesremains available to disable detection. Database()now raises asqlite_utils.db.TransactionErrorif passed a connection created with the Python 3.12+sqlite3.connect(..., autocommit=True)orautocommit=Falseoptions.commit()androllback()behave differently on those connections, which previously caused every write made by the library to be silently discarded when the connection closed 😭
🎀 Everything else: 🎀
- Fixed a bug where
table.delete_where(),table.optimize()andtable.rebuild_fts()did not commit their changes, leaving the connection inside an open transaction. Their work — and any subsequent writes — could then be silently rolled back when the connection was closed. All three now usedb.atomic(), consistent with the other write methods. 🫶 - The
sqlite-utils drop-tablecommand now refuses to drop a view, anddrop-viewrefuses to drop a table. Previously each would silently drop the wrong type of object if the name matched. Both now exit with an error suggesting the correct command to use. - Migrations from the new migrations system <#migrations> now run inside a transaction, together with the record of the migration having been applied. So if a migration throws an exception? Its changes get rolled back, it stays pending, and you can safely re-apply it once you've fixed the error. ✨ Migrations that can't run inside a transaction — like ones doing
VACUUM— can opt out with@migrations(transactional=False). See <#migrations-transactions> Migrations and transactions for the deets. table.upsert()andtable.upsert_all()now sniff out the primary key or compound primary key of an existing table all by themselves, so thepk=argument is no longer required when you're upserting into a table that already has a primary key. she's independent now 💅db.table(table_name).insert({})can now insert a row made entirely of default values into an existing table, usingINSERT INTO ... DEFAULT VALUES. (issue#759— https://github.com/simonw/sqlite-utils/issues/759)- Glow-up for the
sqlite-utils migratecommand:--stop-beforevalues that don't match any known migration are now an error instead of being silently ignored,--stop-beforenow plays nice with migration files that still use the oldersqlite_migrate.Migrationsclass, and--listis now a read-only operation that no longer creates the database file or the migrations tracking table. Alsomigrations.applied()now returns migrations in the order they were applied. 🔥 - Brand new
db.begin(),db.commit()anddb.rollback()methods for taking manual control of transactions, as an alternative to thedb.atomic()context manager. - Fresh docs alert: <#python-api-transactions> Transactions and saving your changes explains how transactions work and when changes get committed, and there's a new <#upgrading> Upgrading page detailing the changes you need to move between major versions. 🎀
🌈 4.0rc1 (2026-06-21)
- New migrations system <#migrations>, absorbing all the functionality that used to live in the separate
sqlite-migrateplugin (https://github.com/simonw/sqlite-migrate). You define migration sets with the newsqlite_utils.Migrationsclass and apply them with thesqlite-utils migratecommand or the migrations Python API <#migrations-python>. (issue#752— https://github.com/simonw/sqlite-utils/issues/752) - New
db.atomic()context manager giving you nested transaction support <#python-api-atomic> using SQLite transactions and savepoints. Internal multi-step operations liketable.transform()now use this mechanism so they don't unexpectedly commit a transaction you already had open. (issue#755— https://github.com/simonw/sqlite-utils/issues/755) - Database objects work as context managers now <#python-api-close>, auto-closing the connection when the
withblock exits. The CLI also closes database and file handles way more reliably, which fixed a whole bunch ofResourceWarningwarnings. (issue#692— https://github.com/simonw/sqlite-utils/issues/692) - The
sqlite-utils convertcommand can now take a direct callable reference liker.parsedateorjson.loads --import jsonas the conversion code, instead of you having to call it explicitly liker.parsedate(value). (issue#686— https://github.com/simonw/sqlite-utils/issues/686) - Fixed a bug where CSV or TSV files with only a header row could crash
sqlite-utils insertandsqlite-utils memorywhen type detection was enabled. Thanks, Rami Abdelrazzaq (https://github.com/RamiNoodle733). (issue#702— https://github.com/simonw/sqlite-utils/issues/702, pull#707— https://github.com/simonw/sqlite-utils/pull/707) - Fixed a bug where installed plugins could get loaded while running the test suite, even though the test-mode safeguard is supposed to disable plugin loading. Thanks, Rami Abdelrazzaq (https://github.com/RamiNoodle733). (issue
#713— https://github.com/simonw/sqlite-utils/issues/713, pull#719— https://github.com/simonw/sqlite-utils/pull/719) table.detect_fts()now recognizes legacy FTS virtual tables that quote thecontent=table name using square brackets, sotable.enable_fts(..., replace=True)can replace them properly. (issue#694— https://github.com/simonw/sqlite-utils/issues/694)- Now depends on Click 8.3.1 or later, which let us yeet the compatibility workarounds for Click's Sentinel default values. (issue
#666— https://github.com/simonw/sqlite-utils/issues/666) - Improved type annotations all over the package, with
tynow running in CI. (issue#697— https://github.com/simonw/sqlite-utils/issues/697) - Dev tooling now uses
uvdependency groups, with separatedevanddocsgroups. (issue#691— https://github.com/simonw/sqlite-utils/issues/691) - The test suite now runs against Python
3.15-dev. (issue#738— https://github.com/simonw/sqlite-utils/issues/738)
🦋 3.39 (2025-11-24)
- Fixed a bug with
sqlite-utils installwhen the tool had been installed usinguv. (issue#687— https://github.com/simonw/sqlite-utils/issues/687) - The
--functionsargument now optionally accepts a path to a Python file as an alternative to a string full of code, and can be specified multiple times — see <#cli-query-functions> Defining custom SQL functions. (issue#659— https://github.com/simonw/sqlite-utils/issues/659) sqlite-utilsnow requires Python 3.10 or higher.
💜 4.0a1 (2025-11-23)
- Breaking change: The
db.table(table_name)method now only works with tables. To access a SQL view, usedb.view(view_name)instead. (issue#657— https://github.com/simonw/sqlite-utils/issues/657) - The
table.insert_all()andtable.upsert_all()methods can now accept an iterator of lists or tuples as an alternative to dictionaries. The first item should be a list/tuple of column names. See <#python-api-insert-lists> Inserting data from a list or tuple iterator for the full rundown. (issue#672— https://github.com/simonw/sqlite-utils/issues/672) - Breaking change: The default floating point column type has changed from
FLOATtoREAL, which is the correct SQLite type for floating point values. This affects auto-detected columns when inserting data. (issue#645— https://github.com/simonw/sqlite-utils/issues/645) - Now uses
pyproject.tomlinstead ofsetup.pyfor packaging. (issue#675— https://github.com/simonw/sqlite-utils/issues/675) - Tables in the Python API now do a much better job of remembering the primary key and other schema details from when they were first created. (issue
#655— https://github.com/simonw/sqlite-utils/issues/655) - Breaking change: The
table.convert()andsqlite-utils convertmechanisms no longer skip values that evaluate toFalse. Previously you needed the--skip-falseoption — it's been removed. (issue#542— https://github.com/simonw/sqlite-utils/issues/542) - Breaking change: Tables created by this library now wrap table and column names in
"double-quotes"in the schema. Previously they used[square-braces]. (issue#677— https://github.com/simonw/sqlite-utils/issues/677) - The
--functionsCLI argument now accepts a path to a Python file in addition to a string full of Python code. It can also be specified multiple times now. (issue#659— https://github.com/simonw/sqlite-utils/issues/659) - Breaking change: Type detection is now the default behavior for the
insertandupsertCLI commands when importing CSV or TSV data. Previously every column was treated asTEXTunless you passed--detect-types. Use the new--no-detect-typesflag to get the old behavior back. TheSQLITE_UTILS_DETECT_TYPESenvironment variable has been removed. (issue#679— https://github.com/simonw/sqlite-utils/issues/679)
🌸 4.0a0 (2025-05-08)
- Upsert operations now use SQLite's
INSERT ... ON CONFLICT SETsyntax on all SQLite versions later than3.23.1. Heads up: this is a very slight breaking change for apps that depend on the oldINSERT OR IGNOREfollowed byUPDATEbehavior. (issue#652— https://github.com/simonw/sqlite-utils/issues/652) - Python library girlies can opt in to the previous implementation by passing
use_old_upsert=Trueto theDatabase()constructor — see <#python-api-old-upsert> Alternative upserts using INSERT OR IGNORE. - Dropped support for Python 3.8, added support for Python 3.13. (issue
#646— https://github.com/simonw/sqlite-utils/issues/646) sqlite-utils tuiis now provided by thesqlite-utils-tuiplugin (https://github.com/simonw/sqlite-utils-tui). (issue#648— https://github.com/simonw/sqlite-utils/issues/648)- The test suite now also runs against SQLite
3.23.1, the last version (from 2018-04-10) before the newINSERT ... ON CONFLICT SETsyntax was added. (issue#654— https://github.com/simonw/sqlite-utils/issues/654)
🍓 3.38 (2024-11-23)
- Plugins can now reuse the implementation of the
sqlite-utils memoryCLI command with the newreturn_db=Trueparameter. (issue#643— https://github.com/simonw/sqlite-utils/issues/643) table.transform()now recreates indexes after transforming a table. A newsqlite_utils.db.TransformErrorexception is raised if those indexes can't be recreated because of conflicting changes to the table, like a column rename. Thanks, Mat Miller (https://github.com/matdmiller). (issue#633— https://github.com/simonw/sqlite-utils/issues/633)table.search()now accepts aninclude_rank=Trueparameter, which makes the resulting rows carry arankcolumn showing the calculated relevance score. Thanks, liunux4odoo (https://github.com/liunux4odoo). (pull#628— https://github.com/simonw/sqlite-utils/pull/628)- Fixed an error that happened when creating a strict table with at least one floating point column. Those
FLOATcolumns are now correctly created asREALtoo — but only for strict tables. (issue#644— https://github.com/simonw/sqlite-utils/issues/644)
🧚 3.37 (2024-07-18)
- The
create-tableandinsert-filescommands all now accept multiple--pkoptions for compound primary keys. (issue#620— https://github.com/simonw/sqlite-utils/issues/620) - Now tested against Python 3.13 pre-release. (pull
#619— https://github.com/simonw/sqlite-utils/pull/619) - Fixed a crash that can happen in environments with a broken
numpyinstallation, which produced amodule 'numpy' has no attribute 'int8'.(issue#632— https://github.com/simonw/sqlite-utils/issues/632)
👑 3.36 (2023-12-07)
- Support for creating tables in SQLite
STRICTmode (https://www.sqlite.org/stricttables.html). Thanks, Taj Khattra (https://github.com/tkhattra). (issue#344— https://github.com/simonw/sqlite-utils/issues/344)- The CLI commands
create-table,insertandupsertall now accept a--strictoption.
- The CLI commands
- Python methods that can build a table, aka
table.create(), plusinsert/upsert/insert_all/upsert_all, now all take an optionalstrict=Trueparameter. 💅 - The
transformcommand and thetable.transform()method keep strict mode intact while they transform a table. no data lost, bestie. - The
sqlite-utils create-tablecommand now acceptsstr,intandbytesas aliases fortext,integerandblobrespectively. (#606 <https://github.com/simonw/sqlite-utils/issues/606>)
🌈 3.35.2 (2023-11-03)
- The
--load-extension=spatialiteoption and thefind_spatialite()utility function (<#python-api-gis-find-spatialite>) both now work correctly on arm64 Linux. Thanks, Mike Coats <https://github.com/MikeCoats>. (#599 <https://github.com/simonw/sqlite-utils/issues/599>) 🦋 - Fixed the bug where
sqlite-utils insertcould make your terminal cursor just... vanish. gone. so rude. Thanks, Luke Plant <https://github.com/spookylukey>. (#433 <https://github.com/simonw/sqlite-utils/issues/433>) 😭 datetime.timedeltavalues are now stored asTEXTcolumns. Thanks, Harald Nezbeda <https://github.com/nezhar>. (#522 <https://github.com/simonw/sqlite-utils/issues/522>)- The test suite now also runs against Python 3.12. ✨
💜 3.35.1 (2023-09-08)
- Fixed a bug where
table.transform()(<#python-api-transform>) would sometimes re-assign therowidvalues for a table instead of keeping them consistent across the whole operation. (#592 <https://github.com/simonw/sqlite-utils/issues/592>)
👑 3.35 (2023-08-17)
Adding foreign keys to a table no longer uses PRAGMA writable_schema = 1 to go in and directly manipulate the sqlite_master table. That approach was throwing errors in some Python installations where the SQLite library was compiled in a way that blocked it from working — macOS especially. Foreign keys are now added using the table transformation (<#python-api-transform>) mechanism instead. (#577 <https://github.com/simonw/sqlite-utils/issues/577>)
Lowkey important warning though: this new mechanism creates a full copy of the table, so for big tables it is likely to be significantly slower. The tradeoff is that it will no longer trigger table sqlite_master may not be modified errors on platforms that don't support PRAGMA writable_schema = 1. 🫠
A new plugin, sqlite-utils-fast-fks <https://github.com/simonw/sqlite-utils-fast-fks>, is now available for developers who still want to use that faster but riskier implementation. iykyk.
Other changes:
- The
table.transform()method (<#python-api-transform>) has two new parameters:foreign_keys=lets you replace the foreign key constraints defined on a table, andadd_foreign_keys=lets you specify new foreign keys to add. These sit alongside the existingdrop_foreign_keys=parameter. (#577 <https://github.com/simonw/sqlite-utils/issues/577>) - The
sqlite-utils transform(<#cli-transform-table>) command has a new--add-foreign-keyoption, which can be called multiple times to add foreign keys to a table that is being transformed. (#585 <https://github.com/simonw/sqlite-utils/issues/585>) 🎀 sqlite-utils convert(<#cli-convert>) now has a--pdboption for opening a debugger on the first error encountered in your conversion script. (#581 <https://github.com/simonw/sqlite-utils/issues/581>)- Fixed a bug where the
sqlite-utils install -e '.[test]'option did not work correctly.
🧚 3.34 (2023-07-22)
This release introduces a whole new plugin system (<#plugins>). Read more about it in sqlite-utils now supports plugins <https://simonwillison.net/2023/Jul/24/sqlite-utils-plugins/>. (#567 <https://github.com/simonw/sqlite-utils/issues/567>) not me being emotional about plugins, okay maybe a little. 🥹
- Documentation describing how to build a plugin (<#plugins-building>).
- Plugin hook:
register_commands(cli)(<#plugins-hooks-register-commands>), so plugins can add extra commands tosqlite-utils. (#569 <https://github.com/simonw/sqlite-utils/issues/569>) - Plugin hook:
prepare_connection(conn)(<#plugins-hooks-prepare-connection>). Plugins can use this to help prepare the SQLite connection to do things like registering custom SQL functions. Thanks, Alex Garcia <https://github.com/asg017>. (#574 <https://github.com/simonw/sqlite-utils/issues/574>) sqlite_utils.Database(..., execute_plugins=False)option for disabling plugin execution. (#575 <https://github.com/simonw/sqlite-utils/issues/575>)sqlite-utils install -e path-to-directoryoption for installing editable code. This option is useful during the development of a plugin. (#570 <https://github.com/simonw/sqlite-utils/issues/570>)- The
table.create(...)method now acceptsreplace=Trueto drop and replace an existing table with the same name, orignore=Trueto silently do nothing if a table with the same name already exists. (#568 <https://github.com/simonw/sqlite-utils/issues/568>) 💅 sqlite-utils insert ... --stop-after 10option for stopping the insert after a specified number of records. Works for theupsertcommand as well. (#561 <https://github.com/simonw/sqlite-utils/issues/561>)- The
--csvand--tsvmodes forinsertnow accept a--empty-nulloption, which causes empty strings in the CSV file to be stored as null in the database. (#563 <https://github.com/simonw/sqlite-utils/issues/563>) - New
db.rename_table(table_name, new_name)method for renaming tables. (#565 <https://github.com/simonw/sqlite-utils/issues/565>) sqlite-utils rename-table my.db table_name new_namecommand for renaming tables. (#565 <https://github.com/simonw/sqlite-utils/issues/565>) 🌸- The
table.transform(...)method now takes an optionalkeep_table=new_table_nameparameter, which will cause the original table to be renamed tonew_table_namerather than being dropped at the end of the transformation. (#571 <https://github.com/simonw/sqlite-utils/issues/571>) - Documentation now notes that calling
table.transform()with no arguments at all will reformat the SQL schema stored by SQLite to be more aesthetically pleasing. (#564 <https://github.com/simonw/sqlite-utils/issues/564>) glow-up for your schema, honestly.
🍓 3.33 (2023-06-25)
sqlite-utilswill now usesqlean.py<https://github.com/nalgeon/sqlean.py> in place ofsqlite3if it is installed in the same virtual environment. This is useful for Python environments with either an outdated version of SQLite or with restrictions on SQLite such as disabled extension loading, or restrictions resulting in thesqlite3.OperationalError: table sqlite_master may not be modifiederror. (#559 <https://github.com/simonw/sqlite-utils/issues/559>)- New
with db.ensure_autocommit_off()context manager, which ensures that the database is in autocommit mode for the duration of a block of code. This is used bydb.enable_wal()anddb.disable_wal()to ensure they work correctly withpysqlite3andsqlean.py. - New
db.iterdump()method, providing an iterator over SQL strings representing a dump of the database. This usessqlite-dumpif it is available, otherwise falling back on theconn.iterdump()method fromsqlite3. Bothpysqlite3andsqlean.pyomit support foriterdump()— this method helps paper over that difference. 🩹
🌟 3.32.1 (2023-05-21)
- Examples in the CLI documentation (<#cli>) can now all be copied and pasted without needing to remove a leading
$. (#551 <https://github.com/simonw/sqlite-utils/issues/551>) copy-paste slay. - Documentation now covers Setting up shell completion (<#installation-completion>) for
bashandzsh. (#552 <https://github.com/simonw/sqlite-utils/issues/552>)
💖 3.32 (2023-05-21)
- New experimental
sqlite-utils tuiinterface for interactively building command-line invocations, powered by Trogon <https://github.com/Textualize/trogon>. This requires an optional dependency, installed usingsqlite-utils install trogon. (#545 <https://github.com/simonw/sqlite-utils/issues/545>) it's giving ✨main character CLI✨. - The
sqlite-utils analyze-tablescommand (documentation <#cli-analyze-tables>) now has a--common-limit 20option for changing the number of common/least-common values shown for each column. (#544 <https://github.com/simonw/sqlite-utils/issues/544>) sqlite-utils analyze-tables --no-mostand--no-leastoptions for disabling calculation of most-common and least-common values.- If a column contains only null values,
analyze-tableswill no longer attempt to calculate the most common and least common values for that column. (#547 <https://github.com/simonw/sqlite-utils/issues/547>) - Calling
sqlite-utils analyze-tableswith non-existent columns in the-c/--columnoption now results in an error message. (#548 <https://github.com/simonw/sqlite-utils/issues/548>) - The
table.analyze_column()method (documented here <#python-api-analyze-column>) now acceptsmost_common=Falseandleast_common=Falseoptions for disabling calculation of those values.
🦋 3.31 (2023-05-08)
- Dropped support for Python 3.6. Tests now ensure compatibility with Python 3.11. (#517 <https://github.com/simonw/sqlite-utils/issues/517>)
- Automatically locates the SpatiaLite extension on Apple Silicon. Thanks, Chris Amico. (#536 <https://github.com/simonw/sqlite-utils/pull/536>)
- New
--raw-linesoption for thesqlite-utils queryandsqlite-utils memorycommands, which outputs just the raw value of the first column of every row. (#539 <https://github.com/simonw/sqlite-utils/issues/539>) - Fixed a bug where
table.upsert_all()failed if thenot_null=option was passed. (#538 <https://github.com/simonw/sqlite-utils/issues/538>) - Fixed a
ResourceWarningwhen usingsqlite-utils insert. (#534 <https://github.com/simonw/sqlite-utils/issues/534>) - Now shows a more detailed error message when
sqlite-utils insertis called with invalid JSON. (#532 <https://github.com/simonw/sqlite-utils/issues/532>) table.convert(..., skip_false=False)and thesqlite-utils convert --no-skip-falseoptions, for avoiding a misfeature where theconvert()(<#python-api-convert>) mechanism skips rows in the database with a falsey value for the specified column. Fixing this by default would be a backwards-incompatible change and is under consideration for a4.0release in the future. (#527 <https://github.com/simonw/sqlite-utils/issues/527>) 🚨 yes that's a real "we can't just fix it yet" moment.- Tables can now be created with self-referential foreign keys. Thanks, Scott Perry. (#537 <https://github.com/simonw/sqlite-utils/pull/537>) iconic.
sqlite-utils transformno longer breaks if a table defines default values for columns. Thanks, Kenny Song. (#509 <https://github.com/simonw/sqlite-utils/issues/509>)- Fixed a bug where repeated calls to
table.transform()did not work correctly. Thanks, Martin Carpenter. (#525 <https://github.com/simonw/sqlite-utils/issues/525>) 💜 - So much nicer error message when
rows_from_file()gets handed a file-like object that isn't in binary mode. No more cryptic nonsense. (#520https://github.com/simonw/sqlite-utils/issues/520)
🦋 3.30 (2022-10-25)
- Now tested against Python 3.11, periodt. (#502
https://github.com/simonw/sqlite-utils/issues/502) - New
table.search_sql(include_rank=True)option, which slaps arankcolumn into the generated SQL. Thanks, Jacob Chapman. (#480https://github.com/simonw/sqlite-utils/pull/480) - Progress bars now show up for newline-delimited JSON files when you use the
--nloption. Thanks, Mischa Untaga. (#485https://github.com/simonw/sqlite-utils/issues/485) - New
db.close()method — say bye to your database on purpose. (#504https://github.com/simonw/sqlite-utils/issues/504) - Conversion functions you pass to
table.convert(...)(#python-api-convert) can now return lists or dictionaries, and those get inserted into the database as JSON strings. Iconic behavior honestly. (#495https://github.com/simonw/sqlite-utils/issues/495) - New
sqlite-utils installandsqlite-utils uninstallcommands for installing packages into the same virtual environment assqlite-utils, described here (#cli-install). (#483https://github.com/simonw/sqlite-utils/issues/483) - New
sqlite_utils.utils.flatten()(#reference-utils-flatten) utility function. (#500https://github.com/simonw/sqlite-utils/issues/500) - Documentation on using Just (
#contributing-just) to run tests, linters and build documentation. - Documentation now covers the Release process (
#release-process) for this package.
🌸 3.29 (2022-08-27)
- The
sqlite-utils query,memoryandbulkcommands now all accept a new--functionsoption. You hand it a string of Python code, and any callable objects defined in that code become available to SQL queries as custom SQL functions. See Defining custom SQL functions (#cli-query-functions) for the details. (#471https://github.com/simonw/sqlite-utils/issues/471) - The
db[table].create(...)method now accepts a newtransform=Trueparameter. If the table already exists it will be transformed (#python-api-transform) to match the schema configuration options you passed in. Which could mean columns getting added or dropped, column types changing, column order updating, or not null and default values for columns being set. (#467https://github.com/simonw/sqlite-utils/issues/467) - Related to the above: the
sqlite-utils create-tablecommand now accepts a--transformoption. - New introspection property:
table.default_valuesreturns a dictionary mapping each column name that has a default value over to the configured default value. (#475https://github.com/simonw/sqlite-utils/issues/475) - The
--load-extensionoption can now be given a path to a compiled SQLite extension module plus the name of an entrypoint, separated by a colon — for example--load-extension ./lines0:sqlite3_lines0_noread_init. This feature is modelled on code first contributed to Datasette (https://github.com/simonw/datasette/pull/1789) by Alex Garcia. (#470https://github.com/simonw/sqlite-utils/issues/470) - Functions registered with the
db.register_function()(#python-api-register-function) method can now have a custom name specified using the newdb.register_function(fn, name=...)parameter. (#458https://github.com/simonw/sqlite-utils/issues/458) sqlite-utils rows(#cli-rows) has a new--orderoption for specifying the sort order of the rows you get back. (#469https://github.com/simonw/sqlite-utils/issues/469)- All of the CLI options that take Python code blocks can now be used to define functions that access modules imported in that same block of code — no
globalkeyword required. life-changing (#472https://github.com/simonw/sqlite-utils/issues/472) - Fixed a bug where
table.extract()would behave wrong for columns containing null values. Thanks, Forest Gregg. (#423https://github.com/simonw/sqlite-utils/issues/423) - New tutorial: Cleaning data with sqlite-utils and Datasette (
https://datasette.io/tutorials/clean-data) shows you how to usesqlite-utilsto import and clean an example CSV file. 🧚 - Datasette and sqlite-utils now have a Discord community. Join the Discord here (
https://discord.gg/Ass7bCAMDw).
🎀 3.28 (2022-07-15)
- New
table.duplicate(new_name)(#python-api-duplicate) method for making a copy of a table with a matching schema and row contents. Thanks, David (https://github.com/davidleejy). (#449https://github.com/simonw/sqlite-utils/issues/449) - New
sqlite-utils duplicate data.db table_name new_nameCLI command for Duplicating tables (#cli-duplicate-table). (#454https://github.com/simonw/sqlite-utils/issues/454) sqlite_utils.utils.rows_from_file()is now a documented API (#reference-utils-rows-from-file). You can use it to read a sequence of dictionaries from a file-like object containing CSV, TSV, JSON or newline-delimited JSON. You can pass it an explicit format, or let it try to detect the format automatically. (#443https://github.com/simonw/sqlite-utils/issues/443)sqlite_utils.utils.TypeTrackeris now a documented API for detecting the likely column types for a sequence of string rows — see Detecting column types using TypeTracker (#python-api-typetracker). (#445https://github.com/simonw/sqlite-utils/issues/445)sqlite_utils.utils.chunks()is now a documented API for splitting an iterator into chunks (#reference-utils-chunks). (#451https://github.com/simonw/sqlite-utils/issues/451)sqlite-utils enable-ftsnow has a--replaceoption for replacing the existing FTS configuration for a table. (#450https://github.com/simonw/sqlite-utils/issues/450)- The
create-index,add-columnandduplicatecommands all now take a--ignoreoption for ignoring errors should the database not be in the right state for them to operate. (#450https://github.com/simonw/sqlite-utils/issues/450)
✨ 3.27 (2022-06-14)
See also the annotated release notes (https://simonwillison.net/2022/Jun/19/weeknotes/#sqlite-utils-3-27) for this release.
- Documentation now uses the Furo (
https://github.com/pradyunsg/furo) Sphinx theme. (#435https://github.com/simonw/sqlite-utils/issues/435) - Code examples in documentation now have a "copy to clipboard" button. (#436
https://github.com/simonw/sqlite-utils/issues/436) sqlite_utils.utils.utils.rows_from_file()is now a documented API, see Reading rows from a file (#python-api-rows-from-file). (#443https://github.com/simonw/sqlite-utils/issues/443)rows_from_file()has two new parameters to help handle CSV files that have rows with more values than the CSV file's headings list out:ignore_extras=Trueandextras_key="name-of-key". (#440https://github.com/simonw/sqlite-utils/issues/440)sqlite_utils.utils.maximize_csv_field_size_limit()helper function for increasing the field size limit for reading CSV files to its maximum, see Setting the maximum CSV field size limit (#python-api-maximize-csv-field-size-limit). (#442https://github.com/simonw/sqlite-utils/issues/442)table.search(where=, where_args=)parameters for adding extra WHERE clauses to a search query. Thewhere=parameter is available ontable.search_sql(...)as well. See Searching withtable.search()(#python-api-fts-search). (#441https://github.com/simonw/sqlite-utils/issues/441)- Fixed a bug where
table.detect_fts()and other search-related functions could fail if two FTS-enabled tables had names that were prefixes of each other. 😭 (#434https://github.com/simonw/sqlite-utils/issues/434)
💜 3.26.1 (2022-05-02)
- Now depends on
click-default-group-wheel(https://github.com/simonw/click-default-group-wheel), a pure Python wheel package. Which means you can install and use this package with Pyodide (https://pyodide.org/), the thing that runs Python entirely in your browser using WebAssembly. (#429https://github.com/simonw/sqlite-utils/pull/429)
Try that out using the Pyodide REPL (https://pyodide.org/en/stable/console.html):
>>> import micropip
>>> await micropip.install("sqlite-utils")
>>> import sqlite_utils
>>> db = sqlite_utils.Database(memory=True)
>>> list(db.query("select 3 * 5"))
[{'3 * 5': 15}]🍓 3.26 (2022-04-13)
- New
errors=r.IGNORE/r.SET_NULLparameter for ther.parsedatetime()andr.parsedate()convert recipes (#cli-convert-recipes). (#416https://github.com/simonw/sqlite-utils/issues/416) - Fixed a bug where
--multicould not be used together with--dry-runfor theconvert(#cli-convert) command. (#415https://github.com/simonw/sqlite-utils/issues/415) - New documentation: Defining a
convert()function (#cli-convert-complex). (#420https://github.com/simonw/sqlite-utils/issues/420) - More robust detection for whether or not
deterministic=Trueis supported. (#425https://github.com/simonw/sqlite-utils/issues/425)
⭐ 3.25.1 (2022-03-11)
- Improved display of type information and parameters in the API reference documentation (
#reference). (#413https://github.com/simonw/sqlite-utils/issues/413)
🌈 3.25 (2022-03-01)
- New
hash_id_columns=parameter for creating a primary key that's a hash of the content of specific columns — see Setting an ID based on the hash of the row contents (#python-api-hash) for the tea. (#343https://github.com/simonw/sqlite-utils/issues/343) - New
db.sqlite_version(#python-api-sqlite-version) property, returning a tuple of integers representing the version of SQLite, for example(3, 38, 0). - Fixed a bug where
register_function(deterministic=True)(#python-api-register-function) caused errors on versions of SQLite before 3.8.3. (#408https://github.com/simonw/sqlite-utils/issues/408) - New documented
hash_record(record, keys=...)(#reference-utils-hash-record) function.
💅 3.24 (2022-02-15)
- SpatiaLite helpers for the
sqlite-utilscommand-line tool — thanks, Chris Amico. (#398https://github.com/simonw/sqlite-utils/issues/398)sqlite-utils create-database(#cli-create-database)--init-spatialiteoption for initializing SpatiaLite on a newly created database.sqlite-utils add-geometry-column(#cli-spatialite) command for adding geometry columns.sqlite-utils create-spatial-index(#cli-spatialite-indexes) command for adding spatial indexes.
db[table].create(..., if_not_exists=True)— a new option so you can create a table only if it doesn't already exist yet (peep the docs at#python-api-explicit-create). Issue #397 →https://github.com/simonw/sqlite-utils/issues/397✨Database(memory_name="my_shared_database")— a parameter for spinning up a named in-memory database (see#python-api-connect) that multiple connections can share between themselves. Issue #405 →https://github.com/simonw/sqlite-utils/issues/405. lowkey iconic 💅- The docs now spill the tea on how to add a primary key to a rowid table (that's the
#cli-transform-table-add-primary-key-to-rowidsection) usingsqlite-utils transform. Issue #403 →https://github.com/simonw/sqlite-utils/issues/403📖
💜 3.23 (2022-02-03)
k so this release drops four brand new utility methods for working with SpatiaLite — https://www.gaia-gis.it/fossil/libspatialite/index. Thank you Chris Amico, you ate that! PR #385 → https://github.com/simonw/sqlite-utils/pull/385 🗺️
sqlite_utils.utils.find_spatialite()— sniffs out where the SpatiaLite module lives on disk (see#python-api-gis-find-spatialite). 🔍db.init_spatialite()— gets SpatiaLite initialized for whatever database you hand it (see#python-api-gis-init-spatialite). 🚀table.add_geometry_column(...)— adds a geometry column onto a table that already exists (see#python-api-gis-add-geometry-column). 📐table.create_spatial_index(...)— builds a spatial index for a column (see#python-api-gis-create-spatial-index). 🧭sqlite-utils batchnow takes a--batch-sizeoption. Issue #392 →https://github.com/simonw/sqlite-utils/issues/392⚡
🦋 3.22.1 (2022-01-25)
- Every single command now shows example usage inside its
--help— go look at the CLI reference at#cli-reference. Issue #384 →https://github.com/simonw/sqlite-utils/issues/384🫶 - The Python library docs got themselves a whole new Getting started section (that's
#python-api-getting-started). Issue #387 →https://github.com/simonw/sqlite-utils/issues/387🌱 - Docs are on Plausible analytics now —
https://plausible.io/. Issue #389 →https://github.com/simonw/sqlite-utils/issues/389📊
🌸 3.22 (2022-01-11)
- A whole new CLI reference page (see
#cli-reference) that literally lists the output of--helpfor every single one of the CLI commands. Issue #383 →https://github.com/simonw/sqlite-utils/issues/383📚 sqlite-utils rowspicked up--limitand--offsetoptions so you can paginate through your data. Issue #381 →https://github.com/simonw/sqlite-utils/issues/381📄sqlite-utils rowsalso got--whereand-poptions for filtering a table with a WHERE query — check "Returning all rows in a table" at#cli-rows. Issue #382 →https://github.com/simonw/sqlite-utils/issues/382🔎
👑 3.21 (2022-01-10)
CLI and Python library glow-ups that help you run ANALYZE — https://www.sqlite.org/lang_analyze.html — after you create indexes or insert rows, so the SQLite query planner can get better performance when it runs against those indexes. ✨
Three new CLI commands, bestie: create-database, analyze and bulk.
More details and examples live in the annotated release notes → https://simonwillison.net/2022/Jan/11/sqlite-utils/ 🎀
- New
sqlite-utils create-databasecommand for creating fresh empty database files. Issue #348 →https://github.com/simonw/sqlite-utils/issues/348🗃️ - New Python methods for running ANALYZE against a database, a table or an index:
db.analyze()andtable.analyze()— see "Optimizing index usage with ANALYZE" at#python-api-analyze. Issue #366 →https://github.com/simonw/sqlite-utils/issues/366🧠 - New
sqlite-utils analyzecommand (see#cli-analyze) for running ANALYZE straight from the CLI. Issue #379 →https://github.com/simonw/sqlite-utils/issues/379💻 - The
create-index,insertandupsertcommands all got a new--analyzeoption that runs ANALYZE once the command finishes. Issue #379 →https://github.com/simonw/sqlite-utils/issues/379🔥 - New
sqlite-utils bulkcommand (see#cli-bulk) that imports records exactly likesqlite-utils insertdoes (from JSON, CSV or TSV) and uses them to bulk execute a parametrized SQL query. Issue #375 →https://github.com/simonw/sqlite-utils/issues/375🚚 - The CLI tool can now be run with
python -m sqlite_utilstoo. Issue #368 →https://github.com/simonw/sqlite-utils/issues/368🐍 - Using
--fmtnow implies--table, so you don't have to pass both options anymore. Issue #374 →https://github.com/simonw/sqlite-utils/issues/374💫 - The
--convertfunction applied to rows can now modify the row in place. Issue #371 →https://github.com/simonw/sqlite-utils/issues/371✏️ - The
insert-filescommand (see#cli-insert-files) supports two new columns:stemandsuffix. Issue #372 →https://github.com/simonw/sqlite-utils/issues/372🌿 - The
--nlimport option now ignores blank lines in the input. Issue #376 →https://github.com/simonw/sqlite-utils/issues/376🙈 - Fixed a bug where streaming input into the
insertcommand with--batch-size 1looked like it only committed after several rows had been ingested — caused by unnecessary input buffering. Issue #364 →https://github.com/simonw/sqlite-utils/issues/364🐛
🍓 3.20 (2022-01-05)
sqlite-utils insert ... --linesinserts the lines from a file into a table with a singlelinecolumn — see "Inserting unstructured data with --lines and --text" at#cli-insert-unstructured. 📝sqlite-utils insert ... --textinserts the contents of the file into a table with a singletextcolumn and a single row. 📃sqlite-utils insert ... --convertlets you hand over a Python function that gets used to convert every row as it's being inserted into the database. See "Applying conversions while inserting data" at#cli-insert-convertfor the details — including the special behavior when you pair it with--linesand--text. Issue #356 →https://github.com/simonw/sqlite-utils/issues/356🔄sqlite-utils convertnow accepts a code value of-to read code from standard input. Issue #353 →https://github.com/simonw/sqlite-utils/issues/353⌨️sqlite-utils convertalso now accepts code that defines a namedconvert(value)function — see "Converting data in columns" at#cli-convert. 🧩- New
db.supports_strictproperty showing whether the database connection supports SQLite strict tables →https://www.sqlite.org/stricttables.html🧱 - New
table.strictproperty (see.strictat#python-api-introspection-strict) telling you whether the table uses strict mode. Issue #344 →https://github.com/simonw/sqlite-utils/issues/344💀 - Fixed a bug where
sqlite-utils upsert ... --detect-typeswas ignoring the--detect-typesoption. Issue #362 →https://github.com/simonw/sqlite-utils/issues/362🐞
🧚 3.19 (2021-11-20)
- The
table.lookup()method (see#python-api-lookup-tables) now accepts keyword arguments that match the ones on the underlyingtable.insert()method:foreign_keys=,column_order=,not_null=,defaults=,extracts=,conversions=andcolumns=. You can also passpk=now to specify a different column name to use for the primary key. Issue #342 →https://github.com/simonw/sqlite-utils/issues/342🔑
🌈 3.18 (2021-11-14)
- The
table.lookup()method now takes an optional second argument that can be used to populate columns only the first time a record is created — see "Working with lookup tables" at#python-api-lookup-tables. Issue #339 →https://github.com/simonw/sqlite-utils/issues/339🎯 sqlite-utils memorynow has a--flattenoption for flattening nested JSON objects (see#cli-inserting-data-flatten) into separate columns, consistent with whatsqlite-utils insertdoes. Issue #332 →https://github.com/simonw/sqlite-utils/issues/332🪄- New
table.create_index(..., find_unique_name=True)parameter that finds an available name for the index it's creating even if the default name is already taken. Which meansindex-foreign-keyswill work even if one of the indexes it tries to create clashes with an existing index name. Issue #335 →https://github.com/simonw/sqlite-utils/issues/335🏷️ - Added
py.typedto the module, so mypy —http://mypy-lang.org/— should now correctly pick up the type annotations. Thanks, Andreas Longo. Issue #331 →https://github.com/simonw/sqlite-utils/issues/331✅ - Now depends on
python-dateutilinstead of depending ondateutils. Thanks, Denys Pavlov. Issue #324 →https://github.com/simonw/sqlite-utils/issues/324📅 table.create()(see "Explicitly creating a table" at#python-api-explicit-create) now handlesdict,listandtupletypes, mapping them toTEXTcolumns in SQLite so they can be stored encoded as JSON. Issue #338 →https://github.com/simonw/sqlite-utils/issues/338🧺- Inserted data with square braces in the column names — like a CSV file containing a
item[price]column — now gets those braces converted to underscores:item_price_. Previously columns like that just got rejected with an error. Issue #329 →https://github.com/simonw/sqlite-utils/issues/329🔧 - Now also tested against Python 3.10. PR #330 →
https://github.com/simonw/sqlite-utils/pull/330🥳
💖 3.17.1 (2021-09-22)
sqlite-utils memory(see#cli-memory) now works even if the files you pass to it share the same file name. Issue #325 →https://github.com/simonw/sqlite-utils/issues/325📂sqlite-utils query(see#cli-query) now returns[]in JSON mode if no rows come back. Issue #328 →https://github.com/simonw/sqlite-utils/issues/328🫙
🌟 3.17 (2021-08-24)
- The
sqlite-utils memorycommand (see#cli-memory) has a new--analyzeoption that runs the equivalent of theanalyze-tablescommand (see#cli-analyze-tables) directly against the in-memory database created from the incoming CSV or JSON data. Issue #320 →https://github.com/simonw/sqlite-utils/issues/320🧪 sqlite-utils insert-files(see#cli-insert-files) can now insert file contents intoTEXTcolumns in addition to the defaultBLOB. Pass the--textoption, or usecontent_textas a column specifier. Issue #319 →https://github.com/simonw/sqlite-utils/issues/319📥
🎀 3.16 (2021-08-18)
- Type signatures added to more methods, including
table.resolve_foreign_keys(),db.create_table_sql(),db.create_table()andtable.create(). Issue #314 →https://github.com/simonw/sqlite-utils/issues/314🖊️ - New
db.quote_fts(value)method — see "Quoting characters for use in search" at#python-api-quote-fts. Thanks, Mark Neumann. Issue #246 →https://github.com/simonw/sqlite-utils/issues/246💬 table.search()now accepts an optionalquote=Trueparameter. Issue #296 →https://github.com/simonw/sqlite-utils/issues/296🔍- The CLI command
sqlite-utils searchnow accepts a--quoteoption. Issue #296 →https://github.com/simonw/sqlite-utils/issues/296✅ - Fixed a bug where
--no-headersand--tsvoptions tosqlite-utils insert(see#cli-insert-csv-tsv) couldn't be used together. Issue #295 →https://github.com/simonw/sqlite-utils/issues/295🐛 - Various small improvements to the API reference (see
#reference) documentation. 📝
3.15.1 (2021-08-10)
- The Python library now has type annotations on almost all of the methods, plus detailed docstrings describing each one, bestie. (
#311·https://github.com/simonw/sqlite-utils/issues/311) ✨ - New API reference documentation page (
#reference) — powered by those very docstrings. Iconic. - Fixed a bug where
.add_foreign_keys()was out here NOT raising an error when called against a View. (#313·https://github.com/simonw/sqlite-utils/issues/313) 💀 - Fixed a bug where
.delete_where()returned a[]instead of returningselfif you called it against a table that doesn't exist. (#315·https://github.com/simonw/sqlite-utils/issues/315)
3.15 (2021-08-09)
- New
sqlite-utils insert --flattenoption for flattening nested JSON objects (#cli-inserting-data-flatten) so you get tables with column names liketopkey_nestedkey. Obsessed. (#310·https://github.com/simonw/sqlite-utils/issues/310) - Fixed several spelling mistakes in the documentation, spotted using codespell (
https://til.simonwillison.net/python/codespell). 🎀 - Errors that happen while you're using the
sqlite-utilsCLI tool now show the responsible SQL and query parameters, if possible. (#309·https://github.com/simonw/sqlite-utils/issues/309)
3.14 (2021-08-02)
This release introduces the new sqlite-utils convert command (#cli-convert, #251 · https://github.com/simonw/sqlite-utils/issues/251) and the corresponding table.convert(...) Python method (#python-api-convert, #302 · https://github.com/simonw/sqlite-utils/issues/302). These tools can be used to apply a Python conversion function to one or more columns of a table — either updating the column in place, or using transformed data from that column to populate one or more other columns.
This command-line example uses the Python standard library textwrap module (https://docs.python.org/3/library/textwrap.html) to wrap the content of the content column in the articles table to 100 characters:
$ sqlite-utils convert content.db articles content \
'"\n".join(textwrap.wrap(value, 100))' \
--import=textwrapThe same operation in Python code looks like this:
import sqlite_utils, textwrap
db = sqlite_utils.Database("content.db")
db["articles"].convert("content", lambda v: "\n".join(textwrap.wrap(v, 100)))See the full documentation for the sqlite-utils convert command (#cli-convert) and the table.convert(...) Python method (#python-api-convert) for more details.
Also in this release:
- The new
table.count_where(...)method, for counting rows in a table that match a specific SQLWHEREclause. (#305·https://github.com/simonw/sqlite-utils/issues/305) 🔥 - New
--silentoption for thesqlite-utils insert-filescommand (#cli-insert-files) to hide the terminal progress bar, consistent with the--silentoption forsqlite-utils convert. (#301·https://github.com/simonw/sqlite-utils/issues/301)
3.13 (2021-07-24)
- The
sqlite-utils schema my.db table1 table2command now accepts optional table names. (#299·https://github.com/simonw/sqlite-utils/issues/299) sqlite-utils memory --helpnow describes the--schemaoption. 📝
3.12 (2021-06-25)
- New
db.query(sql, params)method (#python-api-query), which executes a SQL query and returns the results as an iterator over Python dictionaries. (#290·https://github.com/simonw/sqlite-utils/issues/290) 🦋 - This project now uses
flake8and has started to usemypy. (#291·https://github.com/simonw/sqlite-utils/issues/291) - New documentation on contributing (
#contributing) to this project. (#292·https://github.com/simonw/sqlite-utils/issues/292)
3.11 (2021-06-20)
- New
sqlite-utils memory data.csv --schemaoption, for outputting the schema of the in-memory database generated from one or more files. See--schema,--analyze,--dumpand--save(#cli-memory-schema-dump-save). (#288·https://github.com/simonw/sqlite-utils/issues/288) ✨ - Added installation instructions (
#installation). (#286·https://github.com/simonw/sqlite-utils/issues/286)
3.10 (2021-06-19)
This release introduces the sqlite-utils memory command, which can be used to load CSV or JSON data into a temporary in-memory database and run SQL queries (including joins across multiple files) directly against that data. Main character command fr fr.
Also new: sqlite-utils insert --detect-types, sqlite-utils dump, table.use_rowid plus some smaller fixes.
sqlite-utils memory
This example of sqlite-utils memory retrieves information about all of the repositories in the Dogsheep (https://github.com/dogsheep) organization on GitHub using this JSON API (https://api.github.com/users/dogsheep/repos), sorts them by their number of stars and outputs a table of the top five (using -t):
$ curl -s 'https://api.github.com/users/dogsheep/repos' \
| sqlite-utils memory - '
select full_name, forks_count, stargazers_count
from stdin order by stargazers_count desc limit 5
' -t
full_name forks_count stargazers_count
--------------------------------- ------------- ------------------
dogsheep/twitter-to-sqlite 12 225
dogsheep/github-to-sqlite 14 139
dogsheep/dogsheep-photos 5 116
dogsheep/dogsheep.github.io 7 90
dogsheep/healthkit-to-sqlite 4 85The tool works against files on disk as well. This example joins data from two CSV files:
$ cat creatures.csv
species_id,name
1,Cleo
2,Bants
2,Dori
2,Azi
$ cat species.csv
id,species_name
1,Dog
2,Chicken
$ sqlite-utils memory species.csv creatures.csv '
select * from creatures join species on creatures.species_id = species.id
'
[{"species_id": 1, "name": "Cleo", "id": 1, "species_name": "Dog"},
{"species_id": 2, "name": "Bants", "id": 2, "species_name": "Chicken"},
{"species_id": 2, "name": "Dori", "id": 2, "species_name": "Chicken"},
{"species_id": 2, "name": "Azi", "id": 2, "species_name": "Chicken"}]Here the species.csv file becomes the species table, the creatures.csv file becomes the creatures table and the output is JSON, the default output format. 🐣
You can also use the --attach option to attach existing SQLite database files to the in-memory database, in order to join data from CSV or JSON directly against your existing tables.
Full documentation of this new feature is available in Querying data directly using an in-memory database (#cli-memory). (#272 · https://github.com/simonw/sqlite-utils/issues/272)
sqlite-utils insert --detect-types
The sqlite-utils insert command (#cli-inserting-data) can be used to insert data from JSON, CSV or TSV files into a SQLite database file. The new --detect-types option (shortcut -d), when used in conjunction with a CSV or TSV import, will automatically detect if columns in the file are integers or floating point numbers as opposed to treating everything as a text column and create the new table with the corresponding schema. See Inserting CSV or TSV data (#cli-insert-csv-tsv) for details. (#282 · https://github.com/simonw/sqlite-utils/issues/282) 💅
Other changes
- Bug fix:
table.transform(), when run against a table without explicit primary keys, would incorrectly create a new version of the table with an explicit primary key column calledrowid. (#284·https://github.com/simonw/sqlite-utils/issues/284) 😭 - New
table.use_rowidintrospection property, see.use_rowid(#python-api-introspection-use-rowid). (#285·https://github.com/simonw/sqlite-utils/issues/285) - The new
sqlite-utils dump file.dbcommand outputs a SQL dump that can be used to recreate a database. (#274·https://github.com/simonw/sqlite-utils/issues/274) -hnow works as a shortcut for--help, thanks Loren McIntyre. (#276·https://github.com/simonw/sqlite-utils/issues/276) 🫶- Now using pytest-cov (
https://pytest-cov.readthedocs.io/) and Codecov (https://about.codecov.io/) to track test coverage — currently at 96%. (#275·https://github.com/simonw/sqlite-utils/issues/275) 👑 - SQL errors that occur when using
sqlite-utils queryare now displayed as CLI errors.
3.9.1 (2021-06-12)
- Fixed bug when using
table.upsert_all()to create a table with only a single column that is treated as the primary key. (#271·https://github.com/simonw/sqlite-utils/issues/271)
3.9 (2021-06-11)
- New
sqlite-utils schemacommand showing the full SQL schema for a database, see Showing the schema (CLI) (#cli-schema). (#268·https://github.com/simonw/sqlite-utils/issues/268) 🌸 db.schemaintrospection property exposing the same feature to the Python library, see Showing the schema (Python library) (#python-api-schema).
3.8 (2021-06-02)
- New
sqlite-utils indexescommand to list indexes in a database, see Listing indexes (#cli-indexes). (#263·https://github.com/simonw/sqlite-utils/issues/263) 🎀 table.xindexesintrospection property returning more details about that table's indexes, see.xindexes(#python-api-introspection-xindexes). (#261·https://github.com/simonw/sqlite-utils/issues/261)
3.7 (2021-05-28)
- New
table.pks_and_rows_where()method returning(primary_key, row_dictionary)tuples — see Listing rows with their primary keys (#python-api-pks-and-rows-where). (#240·https://github.com/simonw/sqlite-utils/issues/240) ✨ - Fixed bug with
table.add_foreign_key()against columns containing spaces. (#238·https://github.com/simonw/sqlite-utils/issues/238) table_or_view.drop(ignore=True)option for avoiding errors if the table or view does not exist. (#237·https://github.com/simonw/sqlite-utils/issues/237)sqlite-utils drop-view --ignoreandsqlite-utils drop-table --ignoreoptions. (#237·https://github.com/simonw/sqlite-utils/issues/237)- Fixed a bug with inserts of nested JSON containing non-ascii strings — thanks, Dylan Wu. (
#257·https://github.com/simonw/sqlite-utils/issues/257) 🙈 - Suggest
--alterif an error occurs caused by a missing column. (#259·https://github.com/simonw/sqlite-utils/issues/259) - Support creating indexes with columns in descending order, see API documentation (
#python-api-create-index) and CLI documentation (#cli-create-index). (#260·https://github.com/simonw/sqlite-utils/issues/260) 🔥 - Correctly handle CSV files that start with a UTF-8 BOM. (
#250·https://github.com/simonw/sqlite-utils/issues/250)
3.6 (2021-02-18) 🌈
okay bestie, this one is a big deal — it adds the ability to run queries that join data from more than one database file, which is basically the same energy as the cross database querying feature that showed up in Datasette 0.55 💅 (<https://docs.datasette.io/en/stable/changelog.html#v0-55>).
db.attach(alias, filepath)Python method — lets you attach extra databases onto the same connection, seedb.attach()in the Python API documentation (<#python-api-attach>). (<https://github.com/simonw/sqlite-utils/issues/113>)--attachoption — attaches extra aliased databases so you can run SQL queries against them straight from the command-line, see attaching additional databases in the CLI documentation (<#cli-query-attach>). (<https://github.com/simonw/sqlite-utils/issues/236>)
3.5 (2021-02-14) 💖
sqlite-utils insert --sniffoption for detecting the delimiter and quote character used by a CSV file, see Alternative delimiters and quote characters (<#cli-insert-csv-tsv-delimiter>). (<https://github.com/simonw/sqlite-utils/issues/230>)- The
table.rows_where(),table.search()andtable.search_sql()methods all now take optionaloffset=andlimit=arguments. (<https://github.com/simonw/sqlite-utils/issues/231>) - New
--no-headersoption forsqlite-utils insert --csvto handle CSV files that are missing the header row, see CSV files without a header row (<#cli-insert-csv-tsv-no-header>). (<https://github.com/simonw/sqlite-utils/issues/228>) - Fixed bug where inserting data with extra columns in subsequent chunks would throw an error. Thanks @nieuwenhoven (<https://github.com/nieuwenhoven>) for the fix. (<https://github.com/simonw/sqlite-utils/issues/234>)
- Fixed bug importing CSV files with columns containing more than 128KB of data. (<https://github.com/simonw/sqlite-utils/issues/229>)
- Test suite now runs in CI against Ubuntu, macOS and Windows. Thanks @nieuwenhoven (<https://github.com/nieuwenhoven>) for the Windows test fixes. (<https://github.com/simonw/sqlite-utils/issues/232>)
3.4.1 (2021-02-05) 🩹
- Fixed a code import bug that slipped in to 3.4. (<https://github.com/simonw/sqlite-utils/issues/226>)
3.4 (2021-02-05) ✨
sqlite-utils insert --csvnow accepts optional--delimiterand--quotecharoptions. See Alternative delimiters and quote characters (<#cli-insert-csv-tsv-delimiter>). (<https://github.com/simonw/sqlite-utils/issues/223>)
3.3 (2021-01-17) 🦋
- The
table.m2m()method now accepts an optionalalter=Trueargument to specify that any missing columns should be added to the referenced table. See Working with many-to-many relationships (<#python-api-m2m>). (<https://github.com/simonw/sqlite-utils/issues/222>)
3.2.1 (2021-01-12) 🙈
- Fixed a bug where
.add_missing_columns()failed to take case insensitive column names into account. (<https://github.com/simonw/sqlite-utils/issues/221>)
3.2 (2021-01-03) 👑
This release introduces a whole new mechanism for speeding up count(*) queries using cached table counts, stored in a _counts table and updated by triggers. This mechanism is described in Cached table counts using triggers (<#python-api-cached-table-counts>), and you can turn it on using Python API methods or the new enable-counts CLI command. (<https://github.com/simonw/sqlite-utils/issues/212>)
table.enable_counts()method for enabling these triggers on a specific table.db.enable_counts()method for enabling triggers on every table in the database. (<https://github.com/simonw/sqlite-utils/issues/213>)- New
sqlite-utils enable-counts my.dbcommand for enabling counts on all or specific tables, see Enabling cached counts (<#cli-enable-counts>). (<https://github.com/simonw/sqlite-utils/issues/214>) - New
sqlite-utils triggerscommand for listing the triggers defined for a database or specific tables, see Listing triggers (<#cli-triggers>). (<https://github.com/simonw/sqlite-utils/issues/218>) - New
db.use_counts_tableproperty which, ifTrue, causestable.countto read from the_countstable. (<https://github.com/simonw/sqlite-utils/issues/215>) table.has_counts_triggersproperty revealing if a table has been configured with the new_countsdatabase triggers.db.reset_counts()method andsqlite-utils reset-countscommand for resetting the values in the_countstable. (<https://github.com/simonw/sqlite-utils/issues/219>)- The previously undocumented
db.escape()method has been renamed todb.quote()and is now covered by the documentation: Quoting strings for use in SQL (<#python-api-quote>). (<https://github.com/simonw/sqlite-utils/issues/217>) - New
table.triggers_dictanddb.triggers_dictintrospection properties. (<https://github.com/simonw/sqlite-utils/issues/211>, <https://github.com/simonw/sqlite-utils/issues/216>) sqlite-utils insertnow shows a more useful error message for invalid JSON. (<https://github.com/simonw/sqlite-utils/issues/206>)
3.1.1 (2021-01-01) 🎀
- Fixed failing test caused by
optimizesometimes creating larger database files. (<https://github.com/simonw/sqlite-utils/issues/209>) - Documentation now lives on <https://sqlite-utils.datasette.io/>
- README now includes
brew install sqlite-utilsinstallation method.
3.1 (2020-12-12) ⭐
- New command:
sqlite-utils analyze-tables my.dboutputs useful information about the table columns in the database, such as the number of distinct values and how many rows are null. See Analyzing tables (<#cli-analyze-tables>) for documentation. (<https://github.com/simonw/sqlite-utils/issues/207>) - New
table.analyze_column(column)Python method used by theanalyze-tablescommand — see Analyzing a column (<#python-api-analyze-column>). - The
table.update()method now correctly handles values that should be stored as JSON. Thanks, Andreas Madsack. (<https://github.com/simonw/sqlite-utils/pull/204>)
3.0 (2020-11-08) 🔥
This release introduces a new sqlite-utils search command for searching tables, see Executing searches (<#cli-search>). (<https://github.com/simonw/sqlite-utils/issues/192>)
The table.search() method has been redesigned, see Searching with table.search() (<#python-api-fts-search>). (<https://github.com/simonw/sqlite-utils/issues/197>)
The release includes minor backwards-incompatible changes, hence the version bump to 3.0. Those changes, which should not affect most users, are:
- The
-cshortcut option for outputting CSV is no longer available. The full--csvoption is required instead. - The
-fshortcut for--fmthas also been removed - use--fmt. - The
table.search()method now defaults to sorting by relevance, not sorting by rowid. (<https://github.com/simonw/sqlite-utils/issues/198>) - The
table.search()method now returns a generator over a list of Python dictionaries. It previously returned a list of tuples.
Also in this release:
- The
query,tables,rowsandsearchCLI commands now accept a new--tsvoption which outputs the results in TSV. (<https://github.com/simonw/sqlite-utils/issues/193>) - A new
table.virtual_table_usingproperty reveals if a table is a virtual table, and returns the upper case type of virtual table (e.g.FTS4orFTS5) if it is. It returnsNoneif the table is not a virtual table. (<https://github.com/simonw/sqlite-utils/issues/196>) - The new
table.search_sql()method returns the SQL for searching a table, see Building SQL queries withtable.search_sql()(<#python-api-fts-search-sql>). sqlite-utils rowsnow accepts multiple optional-cparameters specifying the columns to return. (<https://github.com/simonw/sqlite-utils/issues/200>)
Changes since the 3.0a0 alpha release:
- The
sqlite-utils searchcommand now defaults to returning every result, unless you add a--limit 20option. - The
sqlite-utils search -candtable.search(columns=[])options are now fully respected. (<https://github.com/simonw/sqlite-utils/issues/201>)
2.23 (2020-10-28) 🌸
table.m2m(other_table, records)method now takes any iterable, not just a list or tuple. Thanks, Adam Wolf. (<https://github.com/simonw/sqlite-utils/pull/189>)sqlite-utils insertnow displays a progress bar for CSV or TSV imports. (<https://github.com/simonw/sqlite-utils/issues/173>)- New
@db.register_function(deterministic=True)option for registering deterministic SQLite functions in Python 3.8 or higher. (<https://github.com/simonw/sqlite-utils/issues/191>)
2.22 (2020-10-16) 💜
- New
--encodingoption for processing CSV and TSV files that use a non-utf-8 encoding, for both theinsertandupdatecommands. (<https://github.com/simonw/sqlite-utils/issues/182>) - The
--load-extensionoption is now available to many more commands. (<https://github.com/simonw/sqlite-utils/issues/137>) --load-extension=spatialitecan be used to load SpatiaLite from common installation locations, if it is available. (<https://github.com/simonw/sqlite-utils/issues/136>)- Tests now also run against Python 3.9. (<https://github.com/simonw/sqlite-utils/issues/184>)
- Passing
pk=["id"]now has the same effect as passingpk="id". (<https://github.com/simonw/sqlite-utils/issues/181>)
2.21 (2020-09-24) 🧚
table.extract()andsqlite-utils extractnow apply much, much faster - one example operation reduced from twelve minutes to just four seconds! (<https://github.com/simonw/sqlite-utils/issues/172>)sqlite-utils extractno longer shows a progress bar, because it's fast enough not to need one.- New
column_order=option fortable.transform()which can be used to alter the order of columns in a table. (<https://github.com/simonw/sqlite-utils/issues/175>) sqlite-utils transform --column-order=option (with a-oshortcut) for changing column order. (<https://github.com/simonw/sqlite-utils/issues/176>)- The
table.transform(drop_foreign_keys=)parameter and thesqlite-utils transform --drop-foreign-keyoption have changed. They now accept just the name of the column rather than requiring all three of the column, other table and other column. This is technically a backwards-incompatible change but I chose not to bump the major version number because the transform feature is so new. (<https://github.com/simonw/sqlite-utils/issues/177>) - The table
.disable_fts(),.rebuild_fts(),.delete(),.delete_where()and.add_missing_columns()methods all now returnself, which means they can be chained together with other table operations.
2.20 (2020-09-22) 🦋
okay so this release dropped TWO whole new superpowers and honestly? iconic. say hi to transform (#114 — https://github.com/simonw/sqlite-utils/issues/114) and extract (#42 — https://github.com/simonw/sqlite-utils/issues/42). 💅
Transform ✨
so SQLite's ALTER TABLE has several documented limitations — the receipts are at https://sqlite.org/lang_altertable.html — and bestie, the table.transform() Python method plus the sqlite-utils transform CLI command just... work around all of them. the pattern is: a brand new table gets created with the exact structure you want, the data gets copied over, and then the old table gets dropped and replaced by the new one. clean, surgical, no mess. 💖
you can use these tools to:
- change column types
- rename columns
- drop columns
- add and remove
NOT NULLand defaults - remove foreign key constraints
- and more
full deets live in transforming tables (CLI) at #cli-transform-table and transforming tables (Python library) at #python-api-transform. go read them, i'll wait. 🎀
Extract 🌸
sometimes a table — especially one you imported straight from a CSV — is just out here duplicating itself. like, a Trees table might have a Species column with only a few dozen unique values while the table itself is holding thousands of rows. wasteful!! 😭
the table.extract() method and the sqlite-utils extract commands can pull a column — or multiple columns — out into its own separate lookup table and set up a foreign key relationship from the original table back to it. it's giving database normalization, periodt.
the Python library extract() documentation at #python-api-extract explains how extraction works in detail, and Extracting columns into a separate table at #cli-extract in the CLI docs has a whole detailed example. 🫶
Other changes ⭐
- The
@db.register_functiondecorator can be used to quickly register Python functions as custom SQL functions — see Registering custom SQL functions at#python-api-register-function. (#162—https://github.com/simonw/sqlite-utils/issues/162) - The
table.rows_where()method now accepts an optionalselect=argument for specifying which columns should be selected — see Listing rows at#python-api-rows.
2.19 (2020-09-20) 💜
- New
sqlite-utils add-foreign-keyscommand for Adding multiple foreign keys at once at#cli-add-foreign-keys. (#157—https://github.com/simonw/sqlite-utils/issues/157) - New
table.enable_fts(..., replace=True)argument for replacing an existing FTS table with a new configuration. (#160—https://github.com/simonw/sqlite-utils/issues/160) - New
table.add_foreign_key(..., ignore=True)argument for ignoring a foreign key if it already exists. (#112—https://github.com/simonw/sqlite-utils/issues/112)
2.18 (2020-09-08) 🧚
table.rebuild_fts()method for rebuilding an FTS index — see Rebuilding a full-text search table at#python-api-fts-rebuild. (#155—https://github.com/simonw/sqlite-utils/issues/155)sqlite-utils rebuild-fts data.dbcommand for rebuilding FTS indexes across all tables, or just specific tables. (#155—https://github.com/simonw/sqlite-utils/issues/155)table.optimize()method no longer deletes junk rows from the*_fts_docsizetable. this was added in2.17but it turns out runningtable.rebuild_fts()is a better solution to this problem. ngl, growth. 🌈- Fixed a bug where rows with additional columns that are inserted after the first batch of records could cause an error due to breaking SQLite's maximum number of parameters. Thanks, Simon Wiles. (
#145—https://github.com/simonw/sqlite-utils/issues/145)
2.17 (2020-09-07) 🎀
this release handles a bug where replacing rows in FTS tables could result in growing numbers of unnecessary rows in the associated *_fts_docsize table. (#149 — https://github.com/simonw/sqlite-utils/issues/149)
PRAGMA recursive_triggers=onby default for all connections. you can turn it off withDatabase(recursive_triggers=False). (#152—https://github.com/simonw/sqlite-utils/issues/152)table.optimize()method now deletes unnecessary rows from the*_fts_docsizetable. (#153—https://github.com/simonw/sqlite-utils/issues/153)- New
tracermethod for tracking underlying SQL queries — see Tracing queries at#python-api-tracing. (#150—https://github.com/simonw/sqlite-utils/issues/150) - Neater indentation for schema SQL. (
#148—https://github.com/simonw/sqlite-utils/issues/148) - Documentation for the
sqlite_utils.AlterErrorexception thrown byadd_foreign_keys().
2.16.1 (2020-08-28) 💖
insert_all(..., alter=True)now works for columns introduced after the first 100 records. Thanks, Simon Wiles! (#139—https://github.com/simonw/sqlite-utils/issues/139)- Continuous Integration is now powered by GitHub Actions. (
#143—https://github.com/simonw/sqlite-utils/issues/143)
2.16 (2020-08-21) 🔥
--load-extensionoption forsqlite-utils queryfor loading SQLite extensions. (#134—https://github.com/simonw/sqlite-utils/issues/134)- New
sqlite_utils.utils.find_spatialite()function for finding SpatiaLite in common locations. (#135—https://github.com/simonw/sqlite-utils/issues/135)
2.15.1 (2020-08-12) 🍓
- Now available as a sdist package on PyPI in addition to a wheel. (
#133—https://github.com/simonw/sqlite-utils/issues/133)
2.15 (2020-08-10) 👑
- New
db.enable_wal()anddb.disable_wal()methods for enabling and disabling Write-Ahead Logging (https://www.sqlite.org/wal.html) for a database file — see WAL mode at#python-api-walin the Python API documentation. - Also
sqlite-utils enable-wal file.dbandsqlite-utils disable-wal file.dbcommands for doing the same thing on the command-line — see WAL mode (CLI) at#cli-wal. (#132—https://github.com/simonw/sqlite-utils/issues/132)
2.14.1 (2020-08-05) ⭐
- Documentation improvements.
2.14 (2020-08-01) 🌸
- The
insert-filescommand at#cli-insert-filescan now read from standard input:
cat dog.jpg | sqlite-utils insert-files dogs.db pics - --name=dog.jpg(#127 — https://github.com/simonw/sqlite-utils/issues/127)
- You can now specify a full-text search tokenizer using the new
tokenize=parameter toenable_fts()at#python-api-fts. this means you can enable Porter stemming on a table by runningdb["articles"].enable_fts(["headline", "body"], tokenize="porter"). (#130—https://github.com/simonw/sqlite-utils/issues/130) - You can also set a custom tokenizer using the
sqlite-utils enable-ftscommand at#cli-fts, via the new--tokenizeoption.
2.13 (2020-07-29) 🦋
memoryviewanduuid.UUIDobjects are now supported.memoryviewobjects will be stored usingBLOBanduuid.UUIDobjects will be stored usingTEXT. (#128—https://github.com/simonw/sqlite-utils/issues/128)
2.12 (2020-07-27) 💀
the theme of this release is better tools for working with binary data. the new insert-files command can be used to insert binary files directly into a database table, and other commands have been improved with better support for BLOB columns. obsessed.
sqlite-utils insert-files my.db gifs *.gifcan now insert the contents of files into a specified table. the columns in the table can be customized to include different pieces of metadata derived from the files. see Inserting data from files at#cli-insert-files. (#122—https://github.com/simonw/sqlite-utils/issues/122)--rawoption tosqlite-utils query— for outputting just a single raw column value — see Returning raw data, such as binary content at#cli-query-raw. (#123—https://github.com/simonw/sqlite-utils/issues/123)- JSON output now encodes
BLOBvalues as special base64 objects — see Returning JSON at#cli-query-json. (#125—https://github.com/simonw/sqlite-utils/issues/125) - The same format of JSON base64 objects can now be used to insert binary data — see Inserting JSON data at
#cli-inserting-data. (#126—https://github.com/simonw/sqlite-utils/issues/126) - The
sqlite-utils querycommand can now accept named parameters, e.g.sqlite-utils :memory: "select :num * :num2" -p num 5 -p num2 6— see Returning JSON at#cli-query-json. (#124—https://github.com/simonw/sqlite-utils/issues/124)
2.11 (2020-07-08) 🎀
- New
--truncateoption tosqlite-utils insert, andtruncate=Trueargument to.insert_all(). Thanks, Thomas Sibley. (#118—https://github.com/simonw/sqlite-utils/pull/118) - The
sqlite-utils querycommand now runs updates in a transaction. Thanks, Thomas Sibley. (#120—https://github.com/simonw/sqlite-utils/pull/120)
2.10.1 (2020-06-23) 💜
- Added documentation for the
table.pksintrospection property. (#116—https://github.com/simonw/sqlite-utils/issues/116)
2.10 (2020-06-12) 🔥
- The
sqlite-utilscommand now supportsUPDATE/INSERT/DELETEin addition toSELECT. (#115—https://github.com/simonw/sqlite-utils/issues/115)
2.9.1 (2020-05-11) ✨
- Added custom project links to the PyPI listing at
https://pypi.org/project/sqlite-utils/.
2.9 (2020-05-10) 🌈
- New
sqlite-utils drop-tablecommand — see Dropping tables at#cli-drop-table. (#111—https://github.com/simonw/sqlite-utils/issues/111) - New
sqlite-utils drop-viewcommand — see Dropping views at#cli-drop-view. - Python
decimal.Decimalobjects are now stored asFLOAT. (#110—https://github.com/simonw/sqlite-utils/issues/110)
2.8 (2020-05-03) 🍓
- New
sqlite-utils create-tablecommand — see Creating tables at#cli-create-table. (#27—https://github.com/simonw/sqlite-utils/issues/27) - New
sqlite-utils create-viewcommand — see Creating views at#cli-create-view. (#107—https://github.com/simonw/sqlite-utils/issues/107)
2.7.2 (2020-05-02) 🦋
db.create_view(...)now has additional parametersignore=Trueorreplace=True— see Creating views at#python-api-create-view. (#106—https://github.com/simonw/sqlite-utils/issues/106)
2.7.1 (2020-05-01) 🧚
- New
sqlite-utils views my.dbcommand for listing views in a database — see Listing views at#cli-views. (#105—https://github.com/simonw/sqlite-utils/issues/105) sqlite-utils tables(andviews) has a new--schemaoption which outputs the table/view schema — see Listing tables at#cli-tables. (#104—https://github.com/simonw/sqlite-utils/issues/104)- Nested structures containing invalid JSON values (e.g. Python bytestrings) are now serialized using
repr()instead of throwing an error. (#102—https://github.com/simonw/sqlite-utils/issues/102)
2.7 (2020-04-17) 👑
- New
columns=argument for the.insert(),.insert_all(),.upsert()and.upsert_all()methods, for over-riding the auto-detected types for columns and specifying additional columns that should be added when the table is created. see Custom column order and column types at#python-api-custom-columns. (#100—https://github.com/simonw/sqlite-utils/issues/100)
2.6 (2020-04-15) 🌸
- Fresh little argument on
table.rows_where(..., order_by="age desc"), bestie — peep Listing rows for the whole story. (https://github.com/simonw/sqlite-utils/issues/76)
2.5 (2020-04-12) ✨
- Panda's
Timestampis now stored as a SQLiteTEXTcolumn. Thank you so much,b0b5h4rp13! (https://github.com/simonw/sqlite-utils/issues/96) table.last_pkis now only available for inserts or upserts of a single record. No group-chat scenarios, single record only. (https://github.com/simonw/sqlite-utils/issues/98)- New
Database(filepath, recreate=True)parameter, for when you need to delete the database and recreate it from scratch. (https://github.com/simonw/sqlite-utils/issues/97)
2.4.4 (2020-03-23) 🦋
- Fixed a bug where columns with literally only null values were not being correctly created. (
https://github.com/simonw/sqlite-utils/issues/95)
2.4.3 (2020-03-23) 🎀
- The column type suggestion code is no longer confused by null values. She's unbothered now. (
https://github.com/simonw/sqlite-utils/issues/94)
2.4.2 (2020-03-14) 💖
table.column_dictsnow works with all column types — before this it would throw errors on any type other thanTEXT,BLOB,INTEGERorFLOAT. (https://github.com/simonw/sqlite-utils/issues/92)- Documentation for the
NotFoundErrorthrown bytable.get(pk)— go read Retrieving a specific record.
2.4.1 (2020-03-01) 🌺
table.enable_fts()now works with columns that contain spaces. Yes, spaces. We been knew. (https://github.com/simonw/sqlite-utils/issues/90)
2.4 (2020-02-26) 🔥
table.disable_fts()can now be used to yeet FTS tables and triggers that were created usingtable.enable_fts(...). (https://github.com/simonw/sqlite-utils/issues/88)- The
sqlite-utils disable-ftscommand can be used to remove FTS tables and triggers from the command-line. Same energy, terminal edition. (https://github.com/simonw/sqlite-utils/issues/88) - Trying to create table columns with square braces (
[or]) in the name now raises an error. Nope nope nope. (https://github.com/simonw/sqlite-utils/issues/86) - Subclasses of
dict,listandtupleare now detected as needing a JSON column. (https://github.com/simonw/sqlite-utils/issues/87)
2.3.1 (2020-02-10) 🫶
table.create_index()now works for columns that contain spaces. (https://github.com/simonw/sqlite-utils/issues/85)
2.3 (2020-02-08) 💅
table.exists()is now a method, not a property. This was not a documented part of the API before, so honestly? I'm calling this a non-breaking change. Periodt. (https://github.com/simonw/sqlite-utils/issues/83)
2.2.1 (2020-02-06) 😭
- Fixed a bug where
.upsert(..., hash_id="pk")threw an error. (https://github.com/simonw/sqlite-utils/issues/84)
2.2 (2020-02-01) 🌈
- New feature!!
sqlite_utils.suggest_column_types([records])returns the suggested column types for a list of records. Go see Suggesting column types. (https://github.com/simonw/sqlite-utils/issues/81) - This replaces the undocumented
table.detect_column_types()method. Out with the old, in with the iconic.
2.1 (2020-01-30) ⭐
- New feature:
conversions={...}can be passed to the.insert()family of functions, so you can specify SQL conversions that should be applied to values being inserted or updated. See Converting column values using SQL functions. (https://github.com/simonw/sqlite-utils/issues/73)
2.0.1 (2020-01-05) 🔔
- The
.upsert()and.upsert_all()methods now raise asqlite_utils.db.PrimaryKeyRequiredexception if you call them without specifying the primary key column usingpk=. (https://github.com/simonw/sqlite-utils/issues/73)
2.0 (2019-12-29) 💀
This release changes the behaviour of upsert. It's a breaking change, hence the big shiny 2.0.
The upsert command-line utility and the .upsert() and .upsert_all() Python API methods have had their behaviour altered. They used to completely replace the affected records: now, they update the specified values on existing records but leave other columns unaffected.
See Upserting data using the Python API and Upserting data using the CLI for the full details.
If you want the old behaviour — the one where records got completely replaced — you can use $ sqlite-utils insert ... --replace on the command-line, and .insert(..., replace=True) and .insert_all(..., replace=True) in the Python API. See Insert-replacing data using the Python API and Insert-replacing data using the CLI for more.
For the full background on this change, see issue #66 (https://github.com/simonw/sqlite-utils/issues/66).
1.12.1 (2019-11-06) 🧚
- Fixed the error thrown when
.insert_all()and.upsert_all()were called with empty lists. (https://github.com/simonw/sqlite-utils/issues/52)
1.12 (2019-11-04) 💜
Python library utilities for deleting records. (https://github.com/simonw/sqlite-utils/issues/62)
db["tablename"].delete(4)to delete by primary key — see Deleting a specific record.db["tablename"].delete_where("id > ?", [3])to delete by a where clause — see Deleting multiple records.
1.11 (2019-09-02) 🍓
Option to create triggers that automatically keep FTS tables up-to-date with newly inserted, updated and deleted records. Thank you, Amjith Ramanujam! (https://github.com/simonw/sqlite-utils/pull/57)
sqlite-utils enable-fts ... --create-triggers— see Configuring full-text search using the CLI.db["tablename"].enable_fts(..., create_triggers=True)— see Configuring full-text search using the Python library.- Support for introspecting triggers for a database or table — see Introspecting tables and views. (
https://github.com/simonw/sqlite-utils/issues/59)
1.10 (2019-08-23) 🌸
Ability to introspect and run queries against views. (https://github.com/simonw/sqlite-utils/issues/54)
db.view_names()method anddb.viewsproperty.- Separate
ViewandTableclasses, both subclassing the newQueryableclass. view.drop()method.
See Listing views.
1.9 (2019-08-04) 🦋
table.m2m(...)method for creating many-to-many relationships: Working with many-to-many relationships. (https://github.com/simonw/sqlite-utils/issues/23)
1.8 (2019-07-28) 🎀
table.update(pk, values)method: Updating a specific record. (https://github.com/simonw/sqlite-utils/issues/35)
1.7.1 (2019-07-28) 💖
- Fixed a bug where inserting records with 11 columns in a batch of 100 triggered a "too many SQL variables" error. (
https://github.com/simonw/sqlite-utils/issues/50) - Documentation and tests for the
table.drop()method: Dropping a table or view.
1.7 (2019-07-24) 🌺
Support for lookup tables.
- New
table.lookup({...})utility method for building and querying lookup tables — see Working with lookup tables. (https://github.com/simonw/sqlite-utils/issues/44) - New
extracts=table configuration option — see Populating lookup tables automatically during insert/upsert. (https://github.com/simonw/sqlite-utils/issues/46) - Use
pysqlite3if it is available, otherwise usesqlite3from the standard library. - Table options can now be passed to the new
db.table(name, **options)factory function, in addition to being passed toinsert_all(records, **options)and friends — see Table configuration options. - In-memory databases can now be created using
db = Database(memory=True).
1.6 (2019-07-18) 🔥
sqlite-utils insertcan now accept TSV data via the new--tsvoption. (https://github.com/simonw/sqlite-utils/issues/41)
1.5 (2019-07-14) 💅
- Support for compound primary keys. (
https://github.com/simonw/sqlite-utils/issues/36)- Configure these using the CLI tool by passing
--pkmultiple times. - In Python, pass a tuple of columns to the
pk=(..., ...)argument: Compound primary keys.
- Configure these using the CLI tool by passing
- New
table.get()method for retrieving a record by its primary key: Retrieving a specific record. (https://github.com/simonw/sqlite-utils/issues/39)
1.4.1 (2019-07-14) 🫶
- Assorted minor documentation fixes: changes since 1.4 (
https://github.com/simonw/sqlite-utils/compare/1.4...1.4.1)
1.4 (2019-06-30) 😭
- Added the
sqlite-utils index-foreign-keyscommand (docs) and thedb.index_foreign_keys()method (docs). (https://github.com/simonw/sqlite-utils/issues/33)
1.3 (2019-06-28) 🌈
- New mechanism for adding multiple foreign key constraints at once:
db.add_foreign_keys()documentation. (https://github.com/simonw/sqlite-utils/issues/31)
1.2.2 (2019-06-25) ⭐
- Fixed a bug where
datetime.timewas not being handled correctly.
1.2.1 (2019-06-20) 🔔
- Check that the column exists before attempting to add a foreign key. (
https://github.com/simonw/sqlite-utils/issues/29)
1.2 (2019-06-12) 💀
- Improved foreign key definitions: you no longer need to specify the
column,other_tableANDother_columnto define a foreign key — if you omit theother_tableorother_column, the script will attempt to guess the correct values by introspecting the database. See Adding foreign key constraints for details. (https://github.com/simonw/sqlite-utils/issues/25) - Ability to set
NOT NULLconstraints andDEFAULTvalues when creating tables. (https://github.com/simonw/sqlite-utils/issues/24) Documentation: Setting defaults and not null constraints (Python API), Setting defaults and not null constraints (CLI). - Support for
not_null_default=X/--not-null-defaultfor setting aNOT NULL DEFAULT 'x'when adding a new column. Documentation: Adding columns (Python API), Adding columns (CLI).
1.1 (2019-05-28) 🧚
- Support for
ignore=True/--ignore, for ignoring inserted records if the primary key already exists. (https://github.com/simonw/sqlite-utils/issues/21) Documentation: Inserting data (Python API), Inserting data (CLI). - Ability to add a column that is a foreign key reference using
fk=.../--fk. (https://github.com/simonw/sqlite-utils/issues/16) Documentation: Adding columns (Python API), Adding columns (CLI).
1.0.1 (2019-05-27) 💜
sqlite-utils rows data.db table --json-cols— fixed a bug where--json-colswas not being obeyed.
🌟 1.0 (2019-05-24)
bestie we made it to 1.0 🎉 big day for the sqlite-utils cinematic universe:
- Option to automatically add new columns if you attempt to insert or upsert data with extra fields:
sqlite-utils insert ... --alter— see Adding columns automatically with the sqlite-utils CLI (#cli-add-column-alter)db["tablename"].insert(record, alter=True)— see Adding columns automatically using the Python API (#python-api-add-column-alter)
- New
--json-colsoption for outputting nested JSON, see Nested JSON values (#cli-json-values)
🌸 0.14 (2019-02-24)
- Ability to create unique indexes:
db["mytable"].create_index(["name"], unique=True) db["mytable"].create_index(["name"], if_not_exists=True)$ sqlite-utils create-index mydb.db mytable col1 [col2...], see Creating indexes (#cli-create-index)table.add_column(name, type)method, see Adding columns (#python-api-add-column)$ sqlite-utils add-column mydb.db mytable nameofcolumn, see Adding columns (#cli-add-column) (CLI)db["books"].add_foreign_key("author_id", "authors", "id"), see Adding foreign key constraints (#python-api-add-foreign-key)$ sqlite-utils add-foreign-key books.db books author_id authors id, see Adding foreign key constraints (#cli-add-foreign-key) (CLI)- Improved (but backwards-incompatible)
foreign_keys=argument to various methods, see Specifying foreign keys (#python-api-foreign-keys)
✨ 0.13 (2019-02-23)
- New
--tableand--fmtoptions can be used to output query results in a variety of visual table formats, see Table-formatted output (#cli-query-table) - New
hash_id=argument can now be used for Setting an ID based on the hash of the row contents (#python-api-hash) - Can now derive correct column types for numpy int, uint and float values
table.last_idhas been renamed totable.last_rowidtable.last_pknow contains the last inserted primary key, ifpk=was specified- Prettier indentation in the
CREATE TABLEgenerated schemas
🦋 0.12 (2019-02-22)
- Added
db[table].rowsiterator — see Listing rows (#python-api-rows) - Replaced
sqlite-utils jsonandsqlite-utils csvwith a new default subcommand calledsqlite-utils querywhich defaults to JSON and takes formatting options--nl,--csvand--no-headers— see Returning JSON (#cli-query-json) and Returning CSV or TSV (#cli-query-csv) - New
sqlite-utils rows data.db name-of-tablecommand, see Returning all rows in a table (#cli-rows) sqlite-utils tablecommand now takes options--countsand--columnsplus the standard output format options, see Listing tables (#cli-tables)
💜 0.11 (2019-02-07)
New commands for enabling FTS against a table and columns, like:
sqlite-utils enable-fts db.db mytable col1 col2See Configuring full-text search (#cli-fts).
🍓 0.10 (2019-02-06)
Handle datetime.date and datetime.time values. 🕰️
New option for efficiently inserting rows from a CSV:
sqlite-utils insert db.db foo - --csv🎀 0.9 (2019-01-27)
Improved support for newline-delimited JSON.
sqlite-utils insert has two new command-line options:
--nlmeans "expect newline-delimited JSON". This is an extremely efficient way of loading in large amounts of data, especially if you pipe it into standard input.--batch-size=1000lets you increase the batch size (default is100). A commit will be issued every X records. This also controls how many initial records are considered when detecting the desired SQL table schema for the data.
In the Python API, the table.insert_all(...) method can now accept a generator as well as a list of objects. This will be efficiently used to populate the table no matter how many records are produced by the generator.
The Database() constructor can now accept a pathlib.Path object in addition to a string or an existing SQLite connection object.
🌈 0.8 (2019-01-25)
Two new commands: sqlite-utils csv and sqlite-utils json
These commands execute a SQL query and return the results as CSV or JSON. See Returning CSV or TSV (#cli-query-csv) and Returning JSON (#cli-query-json) for more details, iykyk.
$ sqlite-utils json --help
Usage: sqlite-utils json [OPTIONS] PATH SQL
Execute SQL query and return the results as JSON
Options:
--nl Output newline-delimited JSON
--arrays Output rows as arrays instead of objects
--help Show this message and exit.$ sqlite-utils csv --help
Usage: sqlite-utils csv [OPTIONS] PATH SQL
Execute SQL query and return the results as CSV
Options:
--no-headers Exclude headers from CSV output
--help Show this message and exit.👑 0.7 (2019-01-24)
This release implements the sqlite-utils command-line tool with a number of useful subcommands. main character energy fr fr 💅
sqlite-utils tables demo.dblists the tables in the databasesqlite-utils tables demo.db --fts4shows just the FTS4 tablessqlite-utils tables demo.db --fts5shows just the FTS5 tablessqlite-utils vacuum demo.dbrunsVACUUMagainst the databasesqlite-utils optimize demo.dbrunsOPTIMIZEagainst all FTS tables, thenVACUUMsqlite-utils optimize demo.db --no-vacuumrunsOPTIMIZEbut skipsVACUUM
The two most useful subcommands are upsert and insert, which allow you to ingest JSON files with one or more records in them, creating the corresponding table with the correct columns if it does not already exist. See Inserting JSON data (#cli-inserting-data) for more details.
sqlite-utils insert demo.db dogs dogs.json --pk=idinserts new records fromdogs.jsoninto thedogstablesqlite-utils upsert demo.db dogs dogs.json --pk=idupserts records, replacing any records with duplicate primary keys
One backwards incompatible change, sorry bestie: the db["table"].table_names property is now a method:
db["table"].table_names()returns a list of table namesdb["table"].table_names(fts4=True)returns a list of just the FTS4 tablesdb["table"].table_names(fts5=True)returns a list of just the FTS5 tables
A few other changes:
- Plenty of updated documentation, including full coverage of the new command-line tool
- Allow column names to be reserved words (use correct SQL escaping)
- Added automatic column support for
bytesanddatetime.datetime
🧚 0.6 (2018-08-12)
.enable_fts()now takes optional argumentfts_version, defaults to FTS5. Use FTS4 if the version of SQLite bundled with your Python does not support FTS5- New optional
column_order=argument to.insert()and friends for providing a partial or full desired order of the columns when a database table is created - New documentation (
#python-api) for.insert_all()and.upsert()and.upsert_all()
💖 0.5 (2018-08-05)
db.tablesanddb.table_namesintrospection propertiesdb.indexesproperty for introspecting indexestable.create_index(columns, index_name)methoddb.create_view(name, sql)method- Table methods can now be chained, plus added
table.last_idfor accessing the last inserted row ID
⭐ 0.4 (2018-07-31)
enable_fts(),populate_fts()andsearch()table methods
🫶 0.3.1 (2018-07-31)
- Documented related projects
- Added badges to the documentation
🌱 0.3 (2018-07-31)
- New
Tableclass representing a table in the SQLite database
🎈 0.2 (2018-07-28)
- Initial release to PyPI
👑 Author
Simon Willison
©️ Copyright
2018-2022, Simon Willison
and that's a wrap on the whole page, bestie 💅✨ it's giving SQLITE-UTILS(1), source/version from Aug 15, 2026, and honestly? iconic. go forth and slay your databases 🦋💖
📦 where this page comes from
- ubuntu stonking: sqlite-utils 4.2.1-1 · Apache-2.0
- 💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on : a language model's rewrite of the original, under its terms