PYTHONP...ARKSUITE(1)

📦 version 1.14.0, in arch rolling

💖 NAME 💖

okay so bestie, welcome to pythonperformancebenchmarksuite — the Python Performance Benchmark Suite Documentation, aka the manual page stylized as PYTHONP...ARKSUITE(1). The pyperformance project is literally intended to be an authoritative source of benchmarks for all Python implementations. The focus is on real-world benchmarks, not synthetic ones, and it uses whole applications when possible. iconic behavior, honestly. 💅

pyperformance documentation: <http://pyperformance.readthedocs.io/>
pyperformance GitHub project: <https://github.com/python/pyperformance> (source code, issues)
Download pyperformance on PyPI: <https://pypi.python.org/pypi/pyperformance>

pyperformance is distributed under the MIT license.

📚 Documentation:

✨ USAGE ✨

🎀 Installation 🎀

The command to install pyperformance is:

python3 -m pip install pyperformance

The command installs a new pyperformance program.

If needed, pyperf and six dependencies are installed automatically.

pyperformance works on Python 3.6 and newer, but it may work on Python 3.4 and 3.5.

At runtime, Python development files (header files) may be needed to install some dependencies like dulwich_log or psutil, to build their C extension. Commands on Fedora to install dependencies:

Python 3: sudo dnf install python3-devel
PyPy: sudo dnf install pypy-devel

🪟 Windows notes 🪟

On Windows, to allow pyperformance to build dependencies from source like greenlet, dulwich or psutil, if you want to use a python.exe built from source, you should not use the python.exe directly. Instead, you must run the little-known command PC\layout to create a filesystem layout that resembles an installed Python:

.\python.bat -m PC.layout --preset-default --copy installed -v

(Use the --help flag for more info about PC\layout.)

Now you can use the "installed" Python executable:

installed\python.exe -m pip install pyperformance
installed\python.exe -m pyperformance run ...

Using an actually installed Python executable (e.g. via py) works fine too.

🏃‍♀️ Run benchmarks 🏃‍♀️

Commands to compare Python 3.6 and Python 3.7 performance:

pyperformance run --python=python3.6 -o py36.json
pyperformance run --python=python3.7 -o py37.json
pyperformance compare py36.json py37.json

Note: python3 -m pyperformance ... syntax works as well (ex: python3 -m pyperformance run -o py37.json), but requires to install pyperformance on each tested Python version.

JSON files are produced by the pyperf module and so can be analyzed using pyperf commands:

python3 -m pyperf show py36.json
python3 -m pyperf check py36.json
python3 -m pyperf metadata py36.json
python3 -m pyperf stats py36.json
python3 -m pyperf hist py36.json
python3 -m pyperf dump py36.json
(...)

It's also possible to use pyperf to compare results of two JSON files:

python3 -m pyperf compare_to py36.json py37.json --table

🌈 Basic commands 🌈

pyperformance actions:

run                 Run benchmarks on the running python
show                Display a benchmark file
compare             Compare two benchmark files
list                List benchmarks of the running Python
list_groups         List benchmark groups of the running Python
venv                Actions on the virtual environment

💅 Common options 💅

Options available to all commands:

-h, --help — show this help message and exit

🔥 run 🔥

Run benchmarks on the running python.

Usage:

pyperformance run [-h] [-r] [-f] [--debug-single-value] [-v] [-m]
                     [--affinity CPU_LIST] [-o FILENAME]
                     [--append FILENAME] [--manifest MANIFEST]
                     [--timeout TIMEOUT] [-b BM_LIST]
                     [--inherit-environ VAR_LIST] [-p PYTHON]
                     [--hook HOOK]

options:

-h, --help — show this help message and exit
-r, --rigorous — spend longer running tests to get more accurate results
-f, --fast — get rough answers quickly
--debug-single-value — debug: fastest mode, only compute a single value
-v, --verbose — print more output
-m, --track-memory — track memory usage. This only works on Linux.
--affinity CPU_LIST — specify CPU affinity for benchmark runs. This way, benchmarks can be forced to run on a given CPU to minimize run to run variation.
-o FILENAME, --output FILENAME — run the benchmarks on only one interpreter and write benchmark into FILENAME. Provide only baseline_python, not changed_python.
--append FILENAME — add runs to an existing file, or create it if it doesn't exist
--timeout TIMEOUT — specify a timeout in seconds for a single benchmark run (default: disabled)
--manifest MANIFEST — benchmark manifest file to use
-b BM_LIST, --benchmarks BM_LIST — comma-separated list of benchmarks to run. Can contain both positive and negative arguments: --benchmarks=run_this,also_this,-not_this. If there are no positive arguments, we'll run all benchmarks except the negative arguments. Otherwise we run only the positive arguments.
--inherit-environ VAR_LIST — comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses.
-p PYTHON, --python PYTHON — Python executable (default: use running Python)
--same-loops SAME_LOOPS — use the same number of loops as a previous run (i.e., don't recalibrate). Should be a path to a .json file from a previous run.
--hook HOOK — apply the given pyperf hook when running the benchmarks.

🦋 show 🦋

Display a benchmark file.

Usage:

show FILENAME

positional arguments:

FILENAME

🍓 compare 🍓

Compare two benchmark files.

Usage:

pyperformance compare [-h] [-v] [-O STYLE] [--csv CSV_FILE]
                                [--inherit-environ VAR_LIST] [-p PYTHON]
                                baseline_file.json changed_file.json

positional arguments:

baseline_file.json
changed_file.json

options:

-v, --verbose — print more output
-O STYLE, --output_style STYLE — what style the benchmark output should take. Valid options are 'normal' and 'table'. Default is normal.
--csv CSV_FILE — name of a file the results will be written to, as a three-column CSV file containing minimum runtimes for each benchmark.
--inherit-environ VAR_LIST — comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses.
-p PYTHON, --python PYTHON — Python executable (default: use running Python)

🌸 list 🌸

List benchmarks of the running Python.

Usage:

pyperformance list [-h] [--manifest MANIFEST] [-b BM_LIST]
                             [--inherit-environ VAR_LIST] [-p PYTHON]

options:

--manifest MANIFEST — benchmark manifest file to use
-b BM_LIST, --benchmarks BM_LIST — comma-separated list of benchmarks to run. Can contain both positive and negative arguments: --benchmarks=run_this,also_this,-not_this. If there are no positive arguments, we'll run all benchmarks except the negative arguments. Otherwise we run only the positive arguments.
--inherit-environ VAR_LIST — comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses.
-p PYTHON, --python PYTHON — Python executable (default: use running Python)

Use python3 -m pyperformance list -b all to list all benchmarks.

🧚 list_groups 🧚

List benchmark groups of the running Python.

Usage:

pyperformance list_groups [-h] [--manifest MANIFEST]
                                    [--inherit-environ VAR_LIST]
                                    [-p PYTHON]

options:

--manifest MANIFEST — benchmark manifest file to use
--inherit-environ VAR_LIST — comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses.
-p PYTHON, --python PYTHON — Python executable (default: use running Python)

💜 venv 💜

Actions on the virtual environment.

Actions:

show      Display the path to the virtual environment and its status
          (created or not)
create    Create the virtual environment
recreate  Force the recreation of the the virtual environment
remove    Remove the virtual environment

Common options:

--venv VENV — path to the virtual environment
--inherit-environ VAR_LIST — comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses.
-p PYTHON, --python PYTHON — Python executable (default: use running Python)

⭐ venv show ⭐

Display the path to the virtual environment and its status (created or not).

Usage:

pyperformance venv show [-h] [--venv VENV]
                                  [--inherit-environ VAR_LIST] [-p PYTHON]

🎀 venv create 🎀

Create the virtual environment.

Usage:

okay so continuing right where we left off — the venv create synopsis is literally this, and yes, all those flags are in the usage:

pyperformance venv create [-h] [--venv VENV]
                          [--manifest MANIFEST] [-b BM_LIST]
                          [--inherit-environ VAR_LIST]
                          [-p PYTHON]

options:

--manifest MANIFEST — benchmark manifest file to use
-b BM_LIST, --benchmarks BM_LIST — Comma-separated list of benchmarks to run. Can contain both positive and negative arguments: --benchmarks=run_this,also_this,-not_this. If there are no positive arguments, we'll run all benchmarks except the negative arguments. Otherwise we run only the positive arguments.

🎀 `venv recreate` 🎀

Force the recreation of the the virtual environment. (yes, the double “the” is canon, bestie 💅)

Usage:

pyperformance venv recreate [-h] [--venv VENV]
                            [--manifest MANIFEST] [-b BM_LIST]
                            [--inherit-environ VAR_LIST]
                            [-p PYTHON]

options:

--manifest MANIFEST — benchmark manifest file to use
-b BM_LIST, --benchmarks BM_LIST — Comma-separated list of benchmarks to run. Can contain both positive and negative arguments: --benchmarks=run_this,also_this,-not_this. If there are no positive arguments, we'll run all benchmarks except the negative arguments. Otherwise we run only the positive arguments.

💀 `venv remove` 💀

Remove the virtual environment. bye-bye venv, you served looks ✨

Usage:

pyperformance venv remove [-h] [--venv VENV]
                          [--inherit-environ VAR_LIST]
                          [-p PYTHON]

🔥 Compile Python to run benchmarks 🔥

okay so pyperformance has these actions:

pyperformance actions:

   compile             Compile and install CPython and run benchmarks on
                       installed Python
   compile_all         Compile and install CPython and run benchmarks on
                       installed Python on all branches and revisions of
                       CONFIG_FILE
   upload              Upload JSON results to a Codespeed website

All these commands require a configuration file.

Here's the simple config that works for compile but not for compile_all nor upload, and it lives at doc/benchmark.conf:

[config]
json_dir = ~/prog/python/bench_json

[scm]
repo_dir = ~/prog/python/master
update = True

[compile]
bench_dir = ~/prog/python/bench_tmpdir

[run_benchmark]
system_tune = True
affinity = 2,3

And here's the annotated sample, doc/benchmark.conf.sample, comments and all, because they are lowkey important:

[config]
# Directory where JSON files are written.
# - uploaded files are moved to json_dir/uploaded/
# - results of patched Python are written into json_dir/patch/
json_dir = ~/json

# If True, compile CPython in debug mode (LTO and PGO disabled),
# run benchmarks with --debug-single-sample, and disable upload.
#
# Use this option to quickly test a configuration.
debug = False

[scm]
# Directory of CPython source code (Git repository)
repo_dir = ~/cpython

# Update the Git repository (git fetch)?
update = True

# Name of the Git remote, used to create revision of
# the Git branch. For example, use revision 'remotes/origin/3.6'
# for the branch '3.6'.
git_remote = remotes/origin

[compile]
# Create files into bench_dir:
# - bench_dir/bench-xxx.log
# - bench_dir/prefix/: where Python is installed
# - bench_dir/venv/: Virtual environment used by pyperformance
bench_dir = ~/bench_tmpdir

# Link Time Optimization (LTO)?
lto = True

# Profiled Guided Optimization (PGO)?
pgo = True

# Build the experimental just-in-time (JIT) compiler?
# Possible values are:
# - no: (default) do not build the JIT or the micro-op interpreter.
#   The new PYTHON_JIT environment variable has no effect.
# - yes: build the JIT and enable it by default. PYTHON_JIT=0 can be used to
#   disable it at runtime.
# - yes-off: build the JIT, but do not enable it by default. PYTHON_JIT=1 can
#   be used to enable it at runtime.
# - interpreter: do not build the JIT, but do build and enable the micro-op
#   interpreter. This is useful for those of us who find ourselves developing
#   or debugging micro-ops (but don’t want to deal with the JIT).
#   PYTHON_JIT=0 can be used to disable the micro-op interpreter at runtime.
jit = no

# The space-separated list of libraries that are package-only,
# i.e., locally installed but not on header and library paths.
# For each such library, determine the install path and add an
# appropriate subpath to CFLAGS and LDFLAGS declarations passed
# to configure. As an exception, the prefix for openssl, if that
# library is present here, is passed via the --with-openssl
# option. Currently, this only works with Homebrew on macOS.
# If running on macOS with Homebrew, you probably want to use:
#     pkg_only = openssl readline sqlite3 xz zlib
# The version of zlib shipping with macOS probably works as well,
# as long as Apple's SDK headers are installed.
pkg_only =

# Install Python? If false, run Python from the build directory
#
# WARNING: Running Python from the build directory introduces subtle changes
# compared to running an installed Python. Moreover, creating a virtual
# environment using a Python run from the build directory fails in many cases,
# especially on Python older than 3.4. Only disable installation if you
# really understand what you are doing!
install = True

# Specify '-j' parameter in 'make' command
jobs = 8

[run_benchmark]
# Run "sudo python3 -m pyperf system tune" before running benchmarks?
system_tune = True

# --manifest option for 'pyperformance run'
manifest =

# --benchmarks option for 'pyperformance run'
benchmarks =

# --affinity option for 'pyperf system tune' and 'pyperformance run'
affinity =

# Upload generated JSON file?
#
# Upload is disabled on patched Python, in debug mode or if install is
# disabled.
upload = False

# --rigorous option for 'pyperformance run'
rigorous = False

# Configuration to upload results to a Codespeed website
[upload]
url =
environment =
executable =
project =

[compile_all]
# List of CPython Git branches
branches = default 3.6 3.5 2.7

# List of revisions to benchmark by compile_all
[compile_all_revisions]
# list of 'sha1=' (default branch: 'master') or 'sha1=branch'
# used by the "pyperformance compile_all" command
# e.g.:
11159d2c9d6616497ef4cc62953a5c3cc8454afb =

🛠️ `compile` 🛠️

Compile Python, install Python and run benchmarks on the installed Python. the whole build-and-bench glow-up ✨

Usage:

pyperformance compile [-h] [--patch PATCH] [-U] [-T]
                      [--inherit-environ VAR_LIST] [-p PYTHON]
                      config_file revision [branch]

positional arguments:

config_file — Configuration filename
revision — Python benchmarked revision
branch — Git branch

options:

--patch PATCH — Patch file
-U, --no-update — Don't update the Git repository
-T, --no-tune — Don't run pyperf system tune to tune the system for benchmarks
--inherit-environ VAR_LIST — Comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses.
-p PYTHON, --python PYTHON — Python executable (default: use running Python)

Notes:

Note: PGO is broken on Ubuntu 14.04 LTS with GCC 4.8.4-2ubuntu1~14.04: Modules/socketmodule.c:7743:1: internal compiler error: in edge_badness, at ipa-inline.c:895

🌈 `compile_all` 🌈

Compile all branches and revisions of CONFIG_FILE. yes, all of them, no cap 👑

Usage:

pyperformance compile_all [-h] [--inherit-environ VAR_LIST] [-p PYTHON]
                          config_file

positional arguments:

config_file — Configuration filename

options:

--inherit-environ VAR_LIST — Comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses.
-p PYTHON, --python PYTHON — Python executable (default: use running Python)

🦋 `upload` 🦋

Upload results from a JSON file to a Codespeed website.

Usage:

pyperformance upload [-h] [--inherit-environ VAR_LIST] [-p PYTHON]
                     config_file json_file

positional arguments:

config_file — Configuration filename
json_file — JSON filename

options:

--inherit-environ VAR_LIST — Comma-separated list of environment variable names that are inherited from the parent environment when running benchmarking subprocesses.
-p PYTHON, --python PYTHON — Python executable (default: use running Python)

✨ How to get stable benchmarks ✨

Run python3 -m pyperf system tune command
Compile Python using LTO (Link Time Optimization) and PGO (profile guided optimizations): use the pyperformance compile command with uses LTO and PGO by default
See advices of the pyperf documentation: How to get reproductible benchmark results http://pyperf.readthedocs.io/en/latest/run_benchmark.html#how-to-get-reproductible-benchmark-results.

🫶 pyperformance virtual environment 🫶

To run benchmarks, pyperformance first creates a virtual environment. It installs requirements with fixed versions to get a reproductible environment. The system Python has unknown module installed with unknown versions, and can have .pth files run at Python startup which can modify Python behaviour or at least slow down Python startup. so yeah, the venv is doing serious main character work 💅

👑 What is the goal of pyperformance 👑

A benchmark is always written for a specific purpose. Depending how the benchmark is written and how the benchmark is run, the result can be different and so have a different meaning.

The pyperformance benchmark suite has multiple goals:

help you spot a performance regression in a Python implementation 👀
prove that an optimization actually makes Python faster and doesn't bring performance regressions, or at least only minor ones
compare two implementations of Python against each other, like CPython vs PyPy, for example
show off Python performance in a way that ideally represents how apps behave in production

💅 Don't disable GC nor ASLR

So pyperf and the pyperformance benchmarks are literally built to give you reproducible results, but not at the cost of running your benchmarks in some special mode that nobody would ever use in production. Periodt. For that exact reason, the Python garbage collector, the Python randomized hash function, and system ASLR (Address Space Layout Randomization) are not disabled. Benchmarks also don't call gc.collect() — because CPython implements it with stop-the-world (https://en.wikipedia.org/wiki/Tracing_garbage_collection#Stop-the-world_vs._incremental_vs._concurrent), so real applications don't call it either, because that would absolutely kill their performance. 💀

📈 Include outliers and spikes

Also, bestie, even though the pyperf documentation does explain how to reduce random noise from the system and other applications, some benchmarks use the system — so their timing can change depending on system workload, on I/O performance, etc. Outliers and temporary spikes in results are not automatically removed 🚫 — the values get summarized by computing the average (arithmetic mean) and standard deviation, which "contains" those spikes, instead of using the median and median absolute deviation, which would ignore outliers. And that's a deliberate choice, because applications running in production do get hit by these temporary slowdowns caused by all sorts of things, like a garbage collection or a JIT compilation. ✨

🔥 Warmups and steady state

A borderline issue here is benchmark "warmups". The first values of each worker process are always slower: 10% slower in the best case, and it can be 1000% slower or more on PyPy 😭. Right now (2017-04-14), pyperformance ignores the first values, treating them as warmup, until a benchmark reaches its "steady state". That "steady state" can include temporary spikes every 5 values (like, say, caused by the garbage collector), and it can still imply further JIT compiler optimizations — but with a "low" impact on the pyperformance average.

To be clear: "warmup" and "steady state" are a work-in-progress and a very complex topic, especially on PyPy and its JIT compiler. 🫶

📝 Notes

A tool for comparing the performance of two Python implementations.

pyperformance will run Student's two-tailed T test on the benchmark results at the 95% confidence level to indicate whether the observed difference is statistically significant. 📊

If you omit the -b option, you get the default group of benchmarks being run — omitting -b is the same as specifying -b default. Slay.

To run every single benchmark pyperformance knows about, use -b all. To see a full list of all available benchmarks, use --help. 👑

Negative benchmark specifications are supported too: -b -2to3 will run every benchmark in the default group except for 2to3 (same thing as -b default,-2to3). And -b all,-django runs all benchmarks except the Django templates benchmark. Negative groups (like -b -default) are not supported 🙅. Positive benchmarks are parsed before the negative benchmarks are subtracted.

If --track_memory is passed, pyperformance will continuously sample the benchmark's memory usage. This currently only works on Linux 2.6.16 and higher, or Windows with PyWin32. And because --track_memory introduces performance jitter while collecting those memory measurements, only memory usage is reported in the final report. 🧠

🏆 BENCHMARKS 🏆

Also see Custom Benchmarks <> for how to create your own benchmarks or use someone else's. 🛠️

🌈 Available Groups

Just like individual benchmarks (see "Available benchmarks" below), benchmark groups are allowed after the -b option. Use python3 -m pyperformance list_groups to list groups and their benchmarks.

Available benchmark groups:

all: group including all benchmarks
apps: "High-level" applicative benchmarks (2to3, Chameleon, Tornado HTTP)
default: group of benchmarks run by default by the run command
math: float and integers
regex: collection of regular expression benchmarks
serialize: benchmarks on pickle and json modules
startup: collection of microbenchmarks focused on Python interpreter start-up time.
template: templating libraries

Use the python3 -m pyperformance list_groups command to list groups and their benchmarks. (Yes, same command, it's that useful 💅)

🔍 Available Benchmarks

In pyperformance 0.5.5, the following microbenchmarks have been removed because they're too short, not representative of real applications, and too unstable:

call_method_slots
call_method_unknown
call_method
call_simple
pybench

🧰 2to3

Run the 2to3 tool on the pyperformance/benchmarks/data/2to3/ directory: it's a copy of the django/core/*.py files of Django 1.1.4 — 9 files total.

Run the python -m lib2to3 -f all <files> command where python is sys.executable. So the test doesn't only measure the performance of Python itself, but also the performance of the lib2to3 module, which can totally change depending on the Python version. 🐍

Note: the files are called .py.txt instead of .py so that PEP 8 checks don't run on them, and more generally so that nothing modifies them.

🌳 async_tree

An async workload benchmark that calls asyncio.gather() on a tree (6 levels deep, 6 branches per level) with leaf nodes simulating some [potentially] async work (depending on the benchmark variant). Available variants:

async_tree: no actual async work at any leaf node.
async_tree_io: all leaf nodes simulate async IO workload (async sleep 50ms).
async_tree_memoization: all leaf nodes simulate async IO workload with 90% of the data memoized.
async_tree_cpu_io_mixed: half of the leaf nodes simulate a CPU-bound workload (math.factorial(500)) and the other half simulate the same workload as the async_tree_memoization variant.

These benchmarks also have an "eager" flavor that uses the asyncio eager task factory, if available. ⚡

🔡 base64

Benchmark the base64 module's encoding and decoding functions. Each algorithm has _small and _large variants that test both encode and decode in a single benchmark:

_small: balanced iterations across 20B, 127B, 3KiB, and 9KB data sizes, more likely to show the impact of overhead.
_large: large data focus with 100KiB and ~1MiB data sizes, likely to demonstrate implementation efficiency.

Available benchmarks:

base64_small, base64_large: standard Base64 encoding and decoding
urlsafe_base64_small: URL-safe Base64 (small only, since URLs shouldn't contain huge data)
base32_small, base32_large: Base32 encoding and decoding
base16_small, base16_large: Base16/hex encoding and decoding
ascii85_small, ascii85_large: Ascii85 encoding and decoding (includes the wrapcol=76 code path)
base85_small, base85_large: Base85 encoding and decoding

See the base64 module: https://docs.python.org/dev/library/base64.html.

🦎 chameleon

Render a template using the chameleon module to create an HTML table of 500 lignes and 10 columns.

See the chameleon.PageTemplate class.

🌀 chaos

Create chaosgame-like fractals. Command line options:

   --thickness THICKNESS
                         Thickness (default: 0.25)
   --width WIDTH         Image width (default: 256)
   --height HEIGHT       Image height (default: 256)
   --iterations ITERATIONS
                         Number of iterations (default: 5000)
   --filename FILENAME.PPM
                         Output filename of the PPM picture
   --rng-seed RNG_SEED   Random number generator seed (default: 1234)

When the --filename option is used, the timing includes the time it takes to create the PPM file. ⏱️

(There's a picture right here — a chaos game render from the bm_chaos benchmark — and another image too. 🎨)

Image generated by bm_chaos (took 3 sec on CPython 3.5) with the command:

python3 pyperformance/benchmarks/bm_chaos.py --worker -l1 -w0 -n1 --filename chaos.ppm --width=512 --height=512 --iterations 50000

🔐 crypto_pyaes

Benchmark a pure-Python implementation of the AES block-cipher in CTR mode using the pyaes module.

The benchmark is slower on CPython 3 compared to CPython 2.7, because CPython 3 has no more "small int" type (int). The CPython 3 int type now always has an arbitrary size, like the CPython 2.7 long type did. 💀

See pyaes (https://github.com/ricmoo/pyaes): a pure-Python implementation of the AES block cipher algorithm and the common modes of operation (CBC, CFB, CTR, ECB and OFB).

📋 deepcopy

Benchmark the Python copy.deepcopy method. The deepcopy method is performed on a nested dictionary and a dataclass. 🫶

💠 deltablue

DeltaBlue benchmark.

Ported for the PyPy project. Contributed by Daniel Lindsley.

This implementation of the DeltaBlue benchmark was directly ported from V8's source code (https://github.com/v8/v8/blob/master/benchmarks/deltablue.js), which was in turn derived from the Smalltalk implementation by John Maloney and Mario Wolczko. The original Javascript implementation was licensed under the GPL.

It's been updated in places to be more idiomatic to Python (for loops over collections, a couple magic methods, OrderedCollection being a list & things altering those collections changed to the builtin methods) but it largely retains the layout & logic from the original. (Ugh.) 😭

🎭 django_template

Use the Django template system to build a 150x150-cell HTML table.

Use the Context and Template classes of the django.template module.

🌿 dulwich_log

Iterate on commits of the asyncio Git repository using the Dulwich module. Uses the pyperformance/benchmarks/data/asyncio.git/ repository.

Pseudo-code of the benchmark:

repo = dulwich.repo.Repo(repo_path)
head = repo.head()
for entry in repo.get_walker(head):
    pass

See the Dulwich project: https://www.dulwich.io/.

📚 docutils

Use Docutils (https://docutils.sourceforge.io/) to convert Docutils' documentation to HTML. Representative of building a medium-sized documentation set. 🎀

🐣 fannkuch

The Computer Language Benchmarks Game: http://benchmarksgame.alioth.debian.org/

Contributed by Sokolov Yura, modified by Tupteq.

🧮 float

An artificial, floating point-heavy benchmark originally used by Factor.

Create 100,000 point objects which compute math.cos(), math.sin() and math.sqrt()

Changed in version 0.5.5: use __slots__ on the Point class to focus the benchmark on float rather than testing the performance of class attributes.

🧵 genshi

Render a template using Genshi (the genshi.template module):

genshi_text — render a HTML template using the NewTextTemplate class ✨
genshi_xml — render an XML template using the MarkupTemplate class 🌸

See the Genshi project: http://pythonhosted.org/Genshi/.

go 🔥

Artificial intelligence playing the Go board game. Use Zobrist hashing: https://en.wikipedia.org/wiki/Zobrist_hashing.

hexiom 🧩

Solver of Hexiom board game (level 25 by default). Command line option:

--level {2,10,20,25,30,36} — Hexiom board level (default: 25)

hg_startup 🧚

Get Mercurial's help screen.

Measure the performance of the python path/to/hg help command using pyperf.Runner.bench_command(), where python is sys.executable and path/to/hg is the Mercurial program installed in a virtual environment.

The bench_command() redirects stdout and stderr into /dev/null.

See the Mercurial project: https://www.mercurial-scm.org/.

html5lib 🌈

Parse the pyperformance/benchmarks/data/w3_tr_html5.html HTML file (132 KB) using html5lib. The file is the HTML 5 specification, but truncated to parse the file in less than 1 second (around 250 ms).

On CPython, after 3 warmups, the benchmarks enters a cycle of 5 values: every 5th value is 10% slower. Plot of 1 run of 50 values (the warmup is not rendered): [image: html5lib values] [image]

See the html5lib project: https://html5lib.readthedocs.io/.

json_dumps, json_loads 🍓

Benchmark dumps() and loads() functions of the json module.

bm_json_dumps.py command line option:

--cases CASES — Comma separated list of cases. Available cases: EMPTY, SIMPLE, NESTED, HUGE. By default, run all cases.

logging 💜

Benchmarks on the logging module:

logging_format — Benchmark logger.warn(fmt, str)
logging_simple — Benchmark logger.warn(msg)
logging_silent — Benchmark logger.debug(msg) when the log is ignored

Script command line option:

format
silent
simple

See the logging module: https://docs.python.org/dev/library/logging.html.

mako 🎀

Use the Mako template system to build a 150x150-cell HTML table. Includes:

two template inheritances
HTML escaping, XML escaping, URL escaping, whitespace trimming
function definitions and calls
for loops

See the Mako project: http://docs.makotemplates.org/.

mdp 🦋

Battle with damages and topological sorting of nodes in a graph.

See Topological sorting: https://en.wikipedia.org/wiki/Topological_sorting.

meteor_contest ⭐

Solver for Meteor Puzzle board.

Meteor Puzzle board: http://benchmarksgame.alioth.debian.org/u32/meteor-description.html#meteor

The Computer Language Benchmarks Game: http://benchmarksgame.alioth.debian.org/

Contributed by Daniel Nanz, 2008-08-21.

nbody 💫

N-body benchmark from the Computer Language Benchmarks Game. Microbenchmark on floating point operations.

This is intended to support Unladen Swallow's perf.py. Accordingly, it has been modified from the Shootout version:

Accept standard Unladen Swallow benchmark options.
Run report_energy()/advance() in a loop.
Reimplement itertools.combinations() to work with older Python versions.

Pulled from: http://benchmarksgame.alioth.debian.org/u64q/program.php?test=nbody&lang=python3&id=1

Contributed by Kevin Carson. Modified by Tupteq, Fredrik Johansson, and Daniel Nanz.

python_startup, python_startup_nosite 🚀

python_startup — Measure the Python startup time, run python -c pass where python is sys.executable
python_startup_nosite — Measure the Python startup time without importing the site module, run python -S -c pass where python is sys.executable

Run the benchmark with pyperf.Runner.bench_command().

nqueens 👑

Simple, brute-force N-Queens solver.

See Eight queens puzzle: https://en.wikipedia.org/wiki/Eight_queens_puzzle.

pathlib 📁

Test the performance of operations of the pathlib module of the standard library.

This benchmark stresses the creation of small objects, globbing, and system calls.

See the documentation of the pathlib module: https://docs.python.org/dev/library/pathlib.html.

pickle 🥒

pickle benchmarks (serialize):

pickle — use the cPickle module to pickle a variety of datasets.
pickle_dict — microbenchmark; use the cPickle module to pickle a lot of dicts.
pickle_list — microbenchmark; use the cPickle module to pickle a lot of lists.
pickle_pure_python — use the pure-Python pickle module to pickle a variety of datasets.

unpickle benchmarks (deserialize):

unpickle — use the cPickle module to unpickle a variety of datasets.
unpickle_list
unpickle_pure_python — use the pure-Python pickle module to unpickle a variety of datasets.

pidigits 🥧

Calculating 2,000 digits of π. This benchmark stresses big integer arithmetic.

Command line option:

--digits DIGITS — Number of computed pi digits (default: 2000)

Adapted from code on: http://benchmarksgame.alioth.debian.org/

pyflate 🗜️

Benchmark of a pure-Python bzip2 decompressor: decompress the pyperformance/benchmarks/data/interpreter.tar.bz2 file in memory.

You may use and distribute this code under any DFSG-compatible license (eg. BSD, GNU GPLv2).

Stand-alone pure-Python DEFLATE (gzip) and bzip2 decoder/decompressor. This is probably most useful for research purposes/index building; there is certainly some room for improvement in the Huffman bit-matcher.

With the as-written implementation, there was a known bug in BWT decoding to do with repeated strings. This has been worked around; see bwt_reverse(). Correct output is produced in all test cases but ideally the problem would be found...

raytrace 🌅

Simple raytracer.

Command line options:

--width WIDTH — Image width (default: 100)
--height HEIGHT — Image height (default: 100)
--filename FILENAME.PPM — Output filename of the PPM picture

This file contains definitions for a simple raytracer. Copyright Callum and Tony Garnock-Jones, 2008.

This file may be freely redistributed under the MIT license: http://www.opensource.org/licenses/mit-license.php

From https://leastfixedpoint.com/tonyg/kcbbs/lshift_archive/toy-raytracer-in-python-20081029.html [image: Pure Python raytracer] [image]

Image generated by the command (took 68.4 sec on CPython 3.5):

python3 pyperformance/benchmarks/bm_raytrace.py --worker --filename=raytrace.ppm  -l1 -w0 -n1 -v --width=800 --height=600

regex_compile 🧵

Stress the performance of Python's regex compiler, rather than the regex execution speed.

Benchmark how quickly Python's regex implementation can compile regexes.

We bring in all the regexes used by the other regex benchmarks, capture them by stubbing out the re module, then compile those regexes repeatedly. We muck with the re module's caching to force it to recompile every regex we give it.

regex_dna 🧬

regex DNA benchmark using "fasta" to generate the test case.

The Computer Language Benchmarks Game: http://benchmarksgame.alioth.debian.org/

regex-dna Python 3 #5 program: contributed by Dominique Wahli 2to3 modified by Justin Peel

fasta Python 3 #3 program: modified by Ian Osgood modified again by Heinrich Acker modified by Justin Peel Modified by Christopher Sean Forgeron

regex_effbot 🧪

Some of the original benchmarks used to tune mainline Python's current regex engine.

regex_v8 🏎️

Python port of V8's regex benchmark.

Automatically generated on 2009-01-30.

This benchmark is generated by loading 50 of the most popular pages on the web and logging all regexp operations performed. Each operation is given a weight that is calculated from an estimate of the popularity of the pages where it occurs and the number of times it is executed while loading each page. Finally the literal letters in the data are encoded using ROT13 in a way that does not affect how the regexps match their input.

Ported to Python for Unladen Swallow. The original JS version can be found at https://github.com/v8/v8/blob/master/benchmarks/regexp.js, r1243.

richards 🧠

The classic Python Richards benchmark.

Based on a Java version.

Based on original version written in BCPL by Dr Martin Richards in 1981 at Cambridge University Computer Laboratory, England and a C++ version derived from a Smalltalk version written by L Peter Deutsch.

scimark 📊

scimark_sor — Successive over-relaxation (SOR) https://en.wikipedia.org/wiki/Successive_over-relaxation benchmark
scimark_sparse_mat_mult — sparse matrix https://en.wikipedia.org/wiki/Sparse_matrix multiplication https://en.wikipedia.org/wiki/Matrix_multiplication_algorithm benchmark
scimark_monte_carlo — benchmark on the Monte Carlo algorithm https://en.wikipedia.org/wiki/Monte_Carlo_algorithm to compute the area of a disc
scimark_lu — LU decomposition https://en.wikipedia.org/wiki/LU_decomposition benchmark
scimark_fft — Fast Fourier transform (FFT) https://en.wikipedia.org/wiki/Fast_Fourier_transform benchmark

spectral_norm 🌌

MathWorld: "Hundred-Dollar, Hundred-Digit Challenge Problems", Challenge #3. http://mathworld.wolfram.com/Hundred-DollarHundred-DigitChallengeProblems.html

The Computer Language Benchmarks Game: http://benchmarksgame.alioth.debian.org/u64q/spectralnorm-description.html#spectralnorm

Contributed by Sebastien Loisel. Fixed by Isaac Gouy. Sped up by Josh Goldfoot. Dirtily sped up by Simon Descarpentries. Concurrency by Jason Stitt.

sqlalchemy_declarative, sqlalchemy_imperative 🗄️

sqlalchemy_declarative — SQLAlchemy Declarative benchmark using SQLite
sqlalchemy_imperative — SQLAlchemy Imperative benchmark using SQLite

See the SQLAlchemy project: https://www.sqlalchemy.org/.

sqlite_synth 🪶

Benchmark Python aggregate for SQLite.

The goal of the benchmark (written for PyPy) is to test CFFI performance and going back and forth between SQLite and Python a lot. Therefore the queries themselves are really simple.

See the SQLite project: https://www.sqlite.org/ and the Python sqlite3 module (stdlib): https://docs.python.org/dev/library/sqlite3.html.

sympy 🧮

Benchmark on the sympy module:

sympy_expand — Benchmark sympy.expand()
sympy_integrate — Benchmark sympy.integrate()
sympy_str — Benchmark str(sympy.expand())
sympy_sum — Benchmark sympy.summation()

On CPython, some sympy_sum values are 5%-10% slower:

$ python3 -m pyperf dump sympy_sum.json
Run 1: 1 warmup, 50 values, 1 loop
- warmup 1: 404 ms (+63%)
- value 1: 244 ms
- value 2: 245 ms
- value 3: 258 ms <----
- value 4: 245 ms
- value 5: 245 ms
- value 6: 279 ms (+12%) <----
- value 7: 246 ms
- value 8: 244 ms
- value 9: 245 ms
- value 10: 255 ms <----
- value 11: 245 ms
- value 12: 245 ms
- value 13: 256 ms <----
- value 14: 248 ms
- value 15: 245 ms
- value 16: 245 ms
...

Then there's a plot of 1 run of 50 values — and no, the warmup does not get rendered, it's not invited to the graph party 🎀 — literally tagged as [image: sympy_sum values] [image] in the docs.

See the sympy project over at http://www.sympy.org/ 🧮

💰 telco

The Telco Benchmark, aka the one that measures how fast your decimal math goes — it's giving "phone bill but make it a stress test" fr fr:

http://speleotrove.com/decimal/telco.html
http://speleotrove.com/decimal/telcoSpec.html
So first, a call type indicator, c, gets set from the bottom (least significant) bit of the duration — which is why c is always 0 or 1, no in-between bestie ✨
Then a rate, r, is determined from the call type. Calls with c=0 get the low rate r of 0.0013; the rest (the "distance calls") get the "premium" r of 0.00894. And roughly-ish those rates are in Euros or dollarates per second 💸
Next a price, p, for the call gets calculated with p=r*n. That gets rounded to exactly 2 fractional digits using round-half-even (that's Banker's rounding-to-nearest, iykyk) 🎯
A basic tax, b, is calculated: b=p*0.0675 (that's 6.75%). It's truncated to exactly 2 fractional digits (round-down), and then the total basic tax variable gets incremented: sumB=sumB+b 🧾
For distance calls specifically: a distance tax, d, is calculated as d=p*0.0341 (3.41%). Truncated to exactly 2 fractional digits (round-down again), and then the total distance tax variable gets incremented: sumD=sumD+d
The total price, t, is calculated as t=p+b — and if it's a distance call, then t=t+d too ✨
Then the total prices variable gets incremented: sumT=sumT+t
Finally the total price, t, gets converted to a string, s.

The Python version of this benchmark is implemented with the decimal module, no cap 🐍

See the Python decimal module (stdlib) at https://docs.python.org/dev/library/decimal.html

🌪️ tornado_http

Benchmark of the HTTP server from the tornado module. Yeah. That's the whole vibe.

See the Tornado project at http://www.tornadoweb.org/ 🌈

📦 unpack_sequence

A microbenchmark for unpacking lists and tuples — tiny but iconic 💅

Pseudo-code moment:

a, b, c, d, e, f, g, h, i, j = to_unpack

...where to_unpack is either tuple(range(10)) or list(range(10)). That's it, that's the whole benchmark, and honestly? Slay.

🌲 xml_etree

Benchmarking the ElementTree API from the xml.etree module 🧚

xml_etree_generate: Create an XML document
xml_etree_iterparse: Benchmark etree.iterparse()
xml_etree_parse: Benchmark etree.parse()
xml_etree_process: Process an XML document

See the Python xml.etree.ElementTree module (stdlib) at https://docs.python.org/dev/library/xml.etree.elementtree.html

🧪 CUSTOM BENCHMARKS 🧪

So pyperformance comes with its own whole set of benchmarks (go look at the Benchmarks section, it's right up there ⬆️). But plot twist: it also supports using custom benchmarks, because of course it does, it's a generous king 👑

🛠️ Using Custom Benchmarks

To actually use custom benchmarks, you're gonna need the --manifest CLI option, and you hand it the path to the manifest file that describes those benchmarks. That's the whole ritual 🕯️

📁 The pyperformance File Formats

pyperformance uses two file formats to identify benchmarks:

manifest — a set of benchmarks
metadata — a single benchmark

For each benchmark there are two required files and several optional ones. Those files are expected to live in a specific directory structure (unless you customize it in the metadata, because you're a main character like that 💅).

The structure (peek below) is built so it's easy to maintain a benchmark (or a whole set of them) on GitHub and distribute it on PyPI. It also simplifies publishing a Python project's benchmarks. The alternative is just pointing people at a repo, which is... a choice 🙈

Benchmarks can inherit metadata from other metadata files. This is super useful for keeping common metadata for a set of benchmarks (like "version") in one single file. Same energy: benchmarks for a Python project can inherit metadata from that project's pyproject.toml 🫶

And sometimes a benchmark will have one or more variants that run using the same script. Variants like this are supported by pyperformance without requiring much extra effort — you love to see it ✨

🗂️ Benchmark Directory Structure

Normally a benchmark is structured like this:

bm_NAME/
   data/  # if needed
   requirements.txt  # lock file, if any
   pyproject.toml
   run_benchmark.py

(Note the bm_ prefix on the directory name — don't forget it, it matters 👀)

pyproject.toml holds the metadata. run_benchmark.py holds the actual benchmark code. Both are necessary, no skipping.

pyperformance treats the metadata file as the fundamental source of information about a benchmark. A manifest for a set of benchmarks is effectively a mapping of names to metadata files. So a metadata file is essential — non-negotiable. It can be located anywhere on disk. However, if it isn't located in the structure described above, then the metadata has to identify where to find the other files.

Other than that, only a benchmark script (like run_benchmark.py up there) is required. All the other files are optional, they're just there for the vibes 🎀

When a benchmark has variants, each one gets its own metadata file right next to the normal pyproject.toml, named bm_NAME.toml. (Again — note the bm_ prefix.) The format of variant metadata files is exactly the same. pyperformance treats them the same, except that the sibling pyproject.toml is inherited by default. Inherited, like your taste in music 💀

📜 Manifest Files

A manifest file identifies a set of benchmarks, plus (optionally) how they should be grouped. pyperformance uses the manifest to figure out which benchmarks are available to run (and therefore which ones run by default).

A manifest normally looks like this:

[benchmarks]

name        metafile
bench1      somedir/bm_bench1/pyproject.toml
bench2      somedir/pyproject.toml
bench3      ../anotherdir

The benchmarks section is a table with rows of tab-separated-values. The name value is how pyperformance will identify the benchmark. The metafile value is where pyperformance will look for that benchmark's metadata. And if a metafile is a directory, it goes looking for pyproject.toml inside that directory 🔍

🏷️ Benchmark Groups

The other sections in the manifest file are all about grouping:

[benchmarks]

name        metafile
bench1      somedir/bm_bench1
bench2      somedir/bm_bench2
bench3      anotherdir/mybench.toml

[groups]
tag1
tag2

[group default]
bench2
bench3

[group tricky]
bench2

The groups section specifies the available groups that may be identified by benchmark tags (more on tags in the metadata section below 👇). Any other group sections in the manifest get automatically added to the list of available groups, no manual labour required 💅

If no default group is specified, then one gets added automatically with all the benchmarks from the benchmarks section in it. And if there's no groups section and no individual group sections (other than default), then the set of all tags of the known benchmarks is treated as groups. A group named all is also automatically added, and it has all known benchmarks in it. Iconic behaviour, honestly 👑

Benchmarks can be excluded from a group by using a - (minus) prefix. Any benchmark already in the list (at that point) that matches gets dropped from the list. And if the first entry in the section is an exclusion, then all known benchmarks get added to the list first before the exclusion is applied. Order matters, bestie 🫠

For example:

[benchmarks]

name        metafile
bench1      somedir/bm_bench1
bench2      somedir/bm_bench2
bench3      anotherdir/mybench.toml

[group default]
-bench1

Which means by default only bench2 and bench3 are run. bench1 said "not today" 💀

🔗 Merging Manifests

To combine manifests, use the [includes] section in the manifest:

[includes]
project1/benchmarks/MANIFEST
project2/benchmarks/MANIFEST
<default>

And note that <default> is the same as including the manifest file for the default pyperformance benchmarks. Free real estate ✨

🏡 A Local Benchmark Suite

Often a project has more than one benchmark and treats the whole thing as a suite. pyperformance handles this with zero extra work from you, king behaviour 🫶

In the directory holding the manifest file, put all the benchmarks. Then put <local> in the metafile column, like this:

[benchmarks]

name        metafile
bench1      <local>
bench2      <local>
bench3      <local>
bench4      <local>
bench5      <local>

It'll go looking for DIR/bm_NAME/pyproject.toml 🎯

And if there are variants too, identify the main benchmark in the metafile value, like this:

[benchmarks]

name        metafile
bench1      <local>
bench2      <local>
bench3      <local>
variant1    <local:bench3>
variant2    <local:bench3>

pyperformance will look for DIR/bm_BASE/bm_NAME.toml, where BASE is the part after local: — literally just chop it at the colon 💅

🐍 A Project's Benchmark Suite

A Python project can identify its benchmark suite by putting the path to the manifest file in the project's top-level pyproject.toml. And you can identify additional manifests too, because why not:

[tool.pyperformance]
manifest = "..."
manifests = ["...", "..."]

(Reminder bestie: that's the pyproject.toml, not the manifest file. Different girl, different file 🙈)

📝 Benchmark Metadata Files

A benchmark's metadata file (usually pyproject.toml) follows the format specified in PEP 621 at https://www.python.org/dev/peps/pep-0621 and PEP 518 at https://www.python.org/dev/peps/pep-0518. So there are two supported sections in the file: project and tool.pyperformance ☑️

A typical metadata file looks something like this:

[project]
version = "0.9.1"
dependencies = ["pyperf"]
dynamic = ["name"]

[tool.pyperformance]
name = "my_benchmark"

And a highly detailed one might look like this:

[project]
name = "pyperformance_bm_json_dumps"
version = "0.9.1"
description = "A benchmark for json.dumps()"
requires-python = ">=3.8"
dependencies = ["pyperf"]
urls = {repository = "https://github.com/python/pyperformance"}
dynamic = ["version"]

[tool.pyperformance]
name = "json_dumps"
tags = "serialize"
runscript = "bench.py"
datadir = ".data-files/extras"
extra_opts = ["--special"]

🧬 Inheritance

For one benchmark to inherit from another one (or from common metadata), the inherits field is available to you:

[project]
dependencies = ["pyperf"]
dynamic = ["name", "version"]

[tool.pyperformance]
name = "my_benchmark"
inherits = "../common.toml"

okay so, all values in either section of the inherited metadata are treated as defaults, on top of which the current metadata is applied. In the above example, for instance, a value for version in common.toml would be used here.

If the inherits value is a directory (even for ..) then base.toml in that directory will be inherited.

For variants, the base pyproject.toml is the default value for inherits.

✨ Inferred Values ✨

In some situations, omitted values will be inferred from other available data (even for required fields).

project.name <= tool.pyperformance.name
project.* <= inherited metadata (except for "name" and "dynamic")
tool.pyperformance.name <= metadata filename
tool.pyperformance.* <= inherited metadata (except for "name" and "inherits")

When the name is inferred from the filename for a regularly structured benchmark, the bm_ prefix is removed from the benchmark's directory. If it is a variant that prefix is removed from the metadata filename, as well as the .toml suffix.

🎀 The `[project]` Section 🎀

Here's the table translated into list form, bestie — same fields, same X marks, no ASCII-art squinting 🫶

Columns: field, type, R, T, B, D. Legend: R = required, T = inferred from the tool section, B = inferred from the inherited metadata, D = for default benchmarks, inferred from pyperformance. An X means checked; a blank means not marked.

project.name — type str; R: yes; T: yes; B: —; D: —
project.version — type ver; R: yes; T: —; B: yes; D: yes
project.dependencies — type [str]; R: —; T: —; B: yes; D: —
project.dynamic — type [str]; R: —; T: —; B: —; D: —

dynamic is required by PEP 621 for when a field will be filled in dynamically by the tool. This is especially important for required fields.

All other PEP 621 fields are optional (e.g. requires-python = ">=3.8", {repository = "https://github.com/..."}).

🦋 The `[tool.pyperformance]` Section 🦋

Columns: field, type, R, B, F. Legend: R = required, B = inferred from the inherited metadata, F = inferred from filename. An X means checked; a blank means not marked.

tool.name — type str; R: yes; B: —; F: yes
tool.tags — type [str]; R: —; B: yes; F: —
tool.extra_opts — type [str]; R: —; B: yes; F: —
tool.inherits — type file; R: —; B: —; F: —
tool.runscript — type file; R: —; B: yes; F: —
tool.datadir — type file; R: —; B: yes; F: —

Here's the extra deets:

tags: optional list of names to group benchmarks
extra_opts: optional list of args to pass to tool.runscript
runscript: the benchmark script to use instead of run_benchmark.py.

🌈 CPYTHON RESULTS, 2017 🌈

This page lists benchmarks which became faster in CPython.

🔥 Optimizations 🔥

💅 2016-12-14: speedup method calls 💅

Optimization: Speedup method calls 1.2x — https://bugs.python.org/issue26110, commit f2392133 — https://github.com/python/cpython/commit/f2392133eba777f05947a8996c507690b95379c3.

Benchmark table, list-ified ✨ Columns: Benchmark, 2016-12-01 (27580c1fb5e8), 2017-01-01 (67e1aa0b58be).

call_method — 2016-12-01: 14.1 ms; 2017-01-01: 11.2 ms: 1.26x faster (-21%)
call_method_slots — 2016-12-01: 13.9 ms; 2017-01-01: 11.1 ms: 1.25x faster (-20%)
call_method_unknown — 2016-12-01: 16.0 ms; 2017-01-01: 14.3 ms: 1.12x faster (-11%)

🌸 2016-04-22: pymalloc allocator 🌸

Optimization: PyMem_Malloc() now uses the fast pymalloc allocator — http://bugs.python.org/issue26249, commit f5c4b990 — https://github.com/python/cpython/commit/f5c4b99034fae12ac2b9498dd12b5b3f352b90c8.

unpickle_list — 10.4 us → 7.64 us: 1.36x faster (-27%) 🔥
json_dumps — 28.0 ms → 25.2 ms: 1.11x faster (-10%)
unpickle_pure_python — 741 us → 678 us: 1.09x faster (-9%)
unpickle — 33.9 us → 31.3 us: 1.08x faster (-8%)
meteor_contest — 197 ms → 183 ms: 1.08x faster (-7%)
mako — 36.9 ms → 34.3 ms: 1.07x faster (-7%)
pathlib — 41.0 ms → 38.4 ms: 1.07x faster (-6%)
call_method_slots — 14.8 ms → 13.9 ms: 1.07x faster (-6%)
telco — 19.5 ms → 18.3 ms: 1.07x faster (-6%)
scimark_lu — 413 ms → 388 ms: 1.07x faster (-6%)
nqueens — 221 ms → 207 ms: 1.07x faster (-6%)
fannkuch — 937 ms → 882 ms: 1.06x faster (-6%)
regex_compile — 319 ms → 301 ms: 1.06x faster (-6%)
raytrace — 1.16 sec → 1.09 sec: 1.06x faster (-5%)
pickle_pure_python — 1.11 ms → 1.05 ms: 1.05x faster (-5%)
genshi_text — 70.1 ms → 66.6 ms: 1.05x faster (-5%)

🕰️ 2015-12-07: Optimize ElementTree.iterparse(), xml_etree_iterparse

The glow-up in question, bestie:

Optimization: Issue #25638: Optimized ElementTree.iterparse(); it is now 2x faster <http://bugs.python.org/issue25638>, commit 9ec5e25f2 <https://github.com/python/cpython/commit/9ec5e25f26a490510bb5da5c26a276cd30a263a0>.

The receipts (columns are 2015-12-01 (df144092a340) and 2016-01-01 (71db90356390), iykyk):

xml_etree_iterparse — 423 ms → 206 ms: 2.05x faster (-51%) 😭👑

Like, it halved. half. the. time. iconic behavior only.

🌈 2015-09-19: PGO uses test suite, pidigits

Optimization: Issue #24915: Add Clang support to PGO builds and use the test suite for profile data <http://bugs.python.org/issue24915>, commit 7188a3ef <https://github.com/python/cpython/commit/7188a3efe07b9effdb760f3a96783f250214f0be>.

So yeah — Clang joined the PGO party 💅 and the test suite got drafted into being the profile data. Slay. 🎀

✨ Changes of at least 5% ✨

Okay so comparing 2015-09-18 (4b363e270108) against 2015-09-18_22-13 (7188a3efe07b) — literally so much got faster, look at this glow-up 💅

pickle — 2015-09-18 (4b363e270108): 33.7 us → 2015-09-18_22-13 (7188a3efe07b): 26.4 us, that's 1.28x faster (-22%) 🔥
pidigits — 2015-09-18 (4b363e270108): 332 ms → 2015-09-18_22-13 (7188a3efe07b): 286 ms, that's 1.16x faster (-14%)
pickle_list — 2015-09-18 (4b363e270108): 9.90 us → 2015-09-18_22-13 (7188a3efe07b): 8.84 us, that's 1.12x faster (-11%)
unpickle — 2015-09-18 (4b363e270108): 37.2 us → 2015-09-18_22-13 (7188a3efe07b): 33.3 us, that's 1.12x faster (-11%)
unpickle_list — 2015-09-18 (4b363e270108): 11.1 us → 2015-09-18_22-13 (7188a3efe07b): 9.95 us, that's 1.11x faster (-10%)
regex_dna — 2015-09-18 (4b363e270108): 330 ms → 2015-09-18_22-13 (7188a3efe07b): 297 ms, that's 1.11x faster (-10%)
regex_effbot — 2015-09-18 (4b363e270108): 6.43 ms → 2015-09-18_22-13 (7188a3efe07b): 5.80 ms, that's 1.11x faster (-10%)
pickle_dict — 2015-09-18 (4b363e270108): 69.3 us → 2015-09-18_22-13 (7188a3efe07b): 64.1 us, that's 1.08x faster (-8%)
mako — 2015-09-18 (4b363e270108): 39.1 ms → 2015-09-18_22-13 (7188a3efe07b): 36.2 ms, that's 1.08x faster (-7%)
call_simple — 2015-09-18 (4b363e270108): 12.2 ms → 2015-09-18_22-13 (7188a3efe07b): 11.6 ms, that's 1.05x faster (-5%)
genshi_xml — 2015-09-18 (4b363e270108): 175 ms → 2015-09-18_22-13 (7188a3efe07b): 166 ms, that's 1.05x faster (-5%)

😭 Changes of at least 5%, sadly two benchmarks also became slower

Not everything ate, bestie. Comparing 2015-09-18_14-32-master-4b363e270108 with 2015-09-18_22-13-master-7188a3efe07b:

unpickle_pure_python — 2015-09-18_14-32-master-4b363e270108: 776 us → 2015-09-18_22-13-master-7188a3efe07b: 821 us, that's 1.06x slower (+6%) 💀
regex_v8 — 2015-09-18_14-32-master-4b363e270108: 49.5 ms → 2015-09-18_22-13-master-7188a3efe07b: 52.6 ms, that's 1.06x slower (+6%) 💀

🎀 2015-05-30: C implementation of `collections.OrderedDict`, html5lib

Optimization: Issue #16991: Add a C implementation of collections.OrderedDict <http://bugs.python.org/issue16991>, commit 96c6af9b <https://github.com/python/cpython/commit/96c6af9b207c188c52ac53ce87bb7f2dea3f328b>.

Comparing 2015-05-02 (3b4d30a27bd6) with 2015-06-01 (41874c570cf3):

html5lib — 2015-05-02 (3b4d30a27bd6): 285 ms → 2015-06-01 (41874c570cf3): 233 ms, that's 1.23x faster (-19%) 🦋

🌸 2015-05-23: C implementation of `functools.lru_cache()`, sympy

Optimization: Issue #14373: Added C implementation of functools.lru_cache() <http://bugs.python.org/issue14373>, commit 1c858c35 <https://github.com/python/cpython/commit/1c858c352b8c11419f79f586334c49378726dba8>.

✨ Changes of at least 5% ✨

Comparing 2015-05-23_19-15-master-c70908558d8e with 2015-05-23_19-42-master-1c858c352b8c:

sympy_expand — 2015-05-23_19-15-master-c70908558d8e: 1.45 sec → 2015-05-23_19-42-master-1c858c352b8c: 1.14 sec, that's 1.27x faster (-21%) 🔥
sympy_sum — 2015-05-23_19-15-master-c70908558d8e: 308 ms → 2015-05-23_19-42-master-1c858c352b8c: 247 ms, that's 1.25x faster (-20%)
sympy_str — 2015-05-23_19-15-master-c70908558d8e: 621 ms → 2015-05-23_19-42-master-1c858c352b8c: 500 ms, that's 1.24x faster (-19%)
sympy_integrate — 2015-05-23_19-15-master-c70908558d8e: 54.2 ms → 2015-05-23_19-42-master-1c858c352b8c: 45.7 ms, that's 1.19x faster (-16%)
scimark_lu — 2015-05-23_19-15-master-c70908558d8e: 497 ms → 2015-05-23_19-42-master-1c858c352b8c: 471 ms, that's 1.06x faster (-5%)

and yes the table's technically over, bestie, but this footnote ate so we're keeping it:

💅 pickle_dict is seen as 1.06x slower, but since pickle doesn't use functools.lru_cache(), the change is ignored in the table. (yes that's the whole reason — no cap, it's just not fair to compare)

🔥 Slowdown

🐢 `2016-09-11`: `regex_compile`

Okay so this one is not a glow-up, it's the opposite, and we're being honest about it:

Slowdown: convert re flags to (much friendlier) IntFlag constants — issue #28082 (http://bugs.python.org/issue28082), commit f93395bc (https://github.com/python/cpython/commit/f93395bc5125c99539597bf134ca8bcf9707655b).

And here's the receipts, with all three timestamps: 2016-04-01 (6b6abd4cf10e), 2016-07-01 (355048970b2a), and 2016-10-01 (78a111c7d867) 💖

regex_compile
- 2016-04-01 (6b6abd4cf10e): 339 ms
- 2016-07-01 (355048970b2a): 309 ms: 1.10x faster (-9%)
- 2016-10-01 (78a111c7d867): 383 ms: 1.13x slower (+13%)

So yeah, it got slower and we said that out loud. Disrespectful but documented. 😭

🕰️ Timeline

🌸 April, 2016 -> May, 2016

This stretch of 2016-04-01 .. 2016-05-01 was giving main character energy — look at all these faster results. Columns are 2016-04-01 (dcfebb32e277) and 2016-05-01 (f1e2671fdf88) 🦋

nqueens — 255 ms → 207 ms: 1.23x faster (-19%)
raytrace — 1.31 sec → 1.09 sec: 1.19x faster (-16%)
float — 290 ms → 243 ms: 1.19x faster (-16%)
chaos — 273 ms → 235 ms: 1.16x faster (-14%)
hexiom — 21.0 ms → 18.6 ms: 1.13x faster (-11%)
deltablue — 16.4 ms → 14.6 ms: 1.12x faster (-11%)
go — 557 ms → 502 ms: 1.11x faster (-10%)
nbody — 254 ms → 232 ms: 1.10x faster (-9%)

Every single one of them went faster, like a full sweep. slay after slay after slay ✨

👑 `call_method`

Okay bestie, this is the long-haul glow-up: Timeline 2016-04-01 .. 2017-01-01, four checkpoints deep. The column snapshots are 2016-04-01 (6b6abd4cf10e), 2016-07-01 (355048970b2a), 2016-10-01 (78a111c7d867), and 2017-01-01 (67e1aa0b58be) 💜

call_method
- 2016-04-01 (6b6abd4cf10e): 15.8 ms
- 2016-07-01 (355048970b2a): 14.9 ms: 1.06x faster (-6%)
- 2016-10-01 (78a111c7d867): 14.1 ms: 1.13x faster (-11%)
- 2017-01-01 (67e1aa0b58be): 11.2 ms: 1.42x faster (-29%)
call_method_slots
- 2016-04-01 (6b6abd4cf10e): 15.7 ms
- 2016-07-01 (355048970b2a): 15.2 ms: 1.03x faster (-3%)
- 2016-10-01 (78a111c7d867): 14.0 ms: 1.13x faster (-11%)
- 2017-01-01 (67e1aa0b58be): 11.1 ms: 1.42x faster (-29%)
call_method_unknown
- 2016-04-01 (6b6abd4cf10e): 17.7 ms
- 2016-07-01 (355048970b2a): 15.9 ms: 1.11x faster (-10%)
- 2016-10-01 (78a111c7d867): 15.6 ms: 1.13x faster (-11%)
- 2017-01-01 (67e1aa0b58be): 14.3 ms: 1.23x faster (-19%)

call_method and call_method_slots dead-ass tied at the end there (11.2 ms vs 11.1 ms, both 1.42x faster (-29%)) and honestly? iconic. 🫶

(…and that's the table closed out, bestie — moving on 💅)

🦋 `crypto_pyaes`

so like, here's the glow-up arc for crypto_pyaes:

crypto_pyaes — 2016-04-01 (master) 226 ms → 2016-05-01 (master) 205 ms: 1.10x faster (-9%) ✨

2016-03-01 .. 2016-06-01:

crypto_pyaes — 2016-03-01 (13d09afff127) 231 ms → 2016-06-01 (d80ab7d94578) 199 ms: 1.16x faster (-14%) 🔥

🌸 `json_loads`

Progress on 21 months, 2015-01-01 .. 2016-10-01 — the way she just kept improving, we love to see it:

json_loads — 2015-01-01 (52074ac866eb) 64.0 us → 2016-10-01 (78a111c7d867) 56.6 us: 1.13x faster (-11%) 🎀

🧚 `logging_silent`

logging_silent — 2016-01-01 (899b72cee21c) 718 ns → 2016-07-01 (355048970b2a) 606 ns: 1.18x faster (-16%) 👑

🍓 `pickle`

pickle, 2016-08-02 .. 2016-09-08:

pickle — 2016-08-02 (133138a284be) 25.5 us → 2016-09-08 (10427f44852b) 21.4 us: 1.19x faster (-16%) 💖

pickle dict/list:

pickle_dict — 2016-04-01 (6b6abd4cf10e) 64.5 us → 2016-10-01 (78a111c7d867) 57.7 us: 1.12x faster (-11%) ✨
pickle_list — 2016-04-01 (6b6abd4cf10e) 9.06 us → 2016-10-01 (78a111c7d867) 7.79 us: 1.16x faster (-14%) ✨

unpickle:

unpickle — 2015-07-01 (d7982beca93c) 36.9 us → 2015-10-01 (30b7138fe12b) 32.8 us: 1.13x faster (-11%) 🫶

🎀 `python_startup`

2015-04-01 .. 2015-10-01 — okay so this one lowkey went the other way, rip:

python_startup — 2015-04-01 (4fd929b43121) 16.4 ms → 2015-10-01 (30b7138fe12b) 17.2 ms: 1.05x slower (+5%) 😭
python_startup_no_site — 2015-04-01 (4fd929b43121) 8.65 ms → 2015-10-01 (30b7138fe12b) 8.90 ms: 1.03x slower (+3%) 😭

2016-04-01 .. 2017-01-01 — and then she recovered, periodt:

python_startup — 2016-04-01 (6b6abd4cf10e) 17.3 ms → 2017-01-01 (67e1aa0b58be) 14.5 ms: 1.20x faster (-16%) 🔥
python_startup_no_site — 2016-04-01 (6b6abd4cf10e) 8.89 ms → 2017-01-01 (67e1aa0b58be) 8.39 ms: 1.06x faster (-6%) 🌈

🦋 `regex_compile`

three snapshots, and ngl bestie she zigzagged on this one:

regex_compile — 2016-04-01 (6b6abd4cf10e) 339 ms
- → 2016-07-01 (355048970b2a) 309 ms: 1.10x faster (-9%) ✨
- → 2016-10-01 (78a111c7d867) 383 ms: 1.13x slower (+13%) 💀

👑 `telco`

the long-haul icon herself, five checkpoints deep:

telco — 2016-01-01 (899b72cee21c) 19.6 ms
- → 2016-04-01 (6b6abd4cf10e) 19.2 ms: 1.02x faster (-2%)
- → 2016-07-01 (355048970b2a) 18.3 ms: 1.08x faster (-7%)
- → 2016-10-01 (78a111c7d867) 15.1 ms: 1.30x faster (-23%)
- → 2017-03-31 (cdcac039fb44) 13.9 ms: 1.41x faster (-29%) 🔥

🌸 scimark

The window is literally 2016-10-01 .. 2017-03-31 ✨ so here's the glow-up timeline:

scimark_lu
- 2016-10-01 (78a111c7d867) → 423 ms
- 2017-01-01 (67e1aa0b58be) → 378 ms: 1.12x faster (-11%)
- 2017-03-31 (cdcac039fb44) → 318 ms: 1.33x faster (-25%)
scimark_sor
- 2016-10-01 (78a111c7d867) → 426 ms
- 2017-01-01 (67e1aa0b58be) → 403 ms: 1.06x faster (-5%)
- 2017-03-31 (cdcac039fb44) → 375 ms: 1.14x faster (-12%)

🧚 sqlalchemy_declarative

Two snapshots, one icon:

sqlalchemy_declarative
- 2014-10-01 (5a789f7eaf81) → 345 ms
- 2015-10-01 (30b7138fe12b) → 301 ms: 1.15x faster (-13%)

She ate and left no crumbs 💅

🦋 sympy

Window: 2016-04-01 .. 2016-10-01, okay so three checkpoints bestie:

sympy_expand
- 2016-04-01 (6b6abd4cf10e) → 1.10 sec
- 2016-07-01 (355048970b2a) → 1.01 sec: 1.09x faster (-8%)
- 2016-10-01 (78a111c7d867) → 942 ms: 1.17x faster (-14%)
sympy_integrate
- 2016-04-01 (6b6abd4cf10e) → 46.6 ms
- 2016-07-01 (355048970b2a) → 42.9 ms: 1.09x faster (-8%)
- 2016-10-01 (78a111c7d867) → 41.2 ms: 1.13x faster (-11%)
sympy_sum
- 2016-04-01 (6b6abd4cf10e) → 247 ms
- 2016-07-01 (355048970b2a) → 233 ms: 1.06x faster (-6%)
- 2016-10-01 (78a111c7d867) → 199 ms: 1.24x faster (-19%)
sympy_str
- 2016-04-01 (6b6abd4cf10e) → 483 ms
- 2016-07-01 (355048970b2a) → 454 ms: 1.07x faster (-6%)
- 2016-10-01 (78a111c7d867) → 427 ms: 1.13x faster (-12%)

🎀 xml_etree_generate

Five checkpoints, no cap, she's been consistent the whole time:

xml_etree_generate
- 2015-04-01 (4fd929b43121) → 282 ms
- 2015-07-01 (d7982beca93c) → 267 ms: 1.06x faster (-5%)
- 2015-10-01 (30b7138fe12b) → 256 ms: 1.10x faster (-9%)
- 2016-01-01 (899b72cee21c) → 237 ms: 1.19x faster (-16%)
- 2016-07-01 (355048970b2a) → 212 ms: 1.33x faster (-25%)

👑 CHANGELOG 👑

The lore. The origin story. Read it and weep 🫶

⭐ Version 1.14.0

Bump pyperf to 2.10.0
Add base64 module benchmark (b64, b32, b16, a85, b85)
Add FastAPI HTTP benchmark
Add YAML parsing benchmark
Respect rigorous setting in benchmark configuration files

⭐ Version 1.13.0 (2025-10-27)

Re-enable xdsl benchmark
Remove code for Python 3.7 and previous versions
CI updates:
- fix workflow ubuntu-latest
- remove unused Azure Pipelines config
- update checkout version
- setup dependabot
- update setup-python version

⭐ Version 1.12.0 (2025-10-24)

Add xdsl benchmark (but disabled by default due to a regression)
Add a new quadtree nbody simulation using the Barnes Hut algorithm
Add argparse benchmark
Add NetworkX benchmarks
Add Sphinx benchmark
Add decimal module benchmark
Add BPE tokeniser benchmark
Skip dask benchmark on Windows for Python 3.13
Bump dask[distributed] to 2024.10.1 for Windows compatibility
Bump greenlet to 3.2.4 for compatibility with 3.13+
Bump Chameleon to 4.6.0
Bump tornado to 6.5.0
Bump pyperf to 2.9.0
Bump sqlglot to V2
Bump mypy to 1.18.2
Fix check for editable mode
Fix --same-loops handling
Fix: pass --warmup and --timeout flags to pyperf
Add -V/--version CLI argument
Add --hook CLI argument
Run ruff format and check under pre-commit and GitHub Actions
Add support for experimental JIT builds
Add Arm64 CI
Add Free-threading CI
Make Windows a non-experimental platform in CI
Add support Python 3.14 and drop Python 3.7, 3.8, 3.9

⭐ Version 1.11.0 (2024-03-09)

Add a --same-loops option to the run command to use the exact same number of loops as a previous run (without recalibrating).
Bump pyperf to 2.6.3
Fix the django_template benchmark for compatibilty with 3.13
Fix benchmark.conf.sample

⭐ Version 1.10.0 (2023-10-22)

Add benchmark for asyncio_webockets
Expose --min-time from pyperf to pyperformance CLI
Bump coverage to 7.3.2 for compatibilty with 3.13
Bump greenlet to 3.0.0rc3 for compatibilty with 3.13

⭐ Version 1.0.9 (2023-06-14)

Vendor lib2to3 for Python 3.13+
Add TaskGroups variants to async_tree benchmarks

⭐ Version 1.0.8 (2023-06-02)

Move the main requirements.txt file to pyperformance/requirements so that dependabot can only run on that one file
Update dependencies of benchmarks not to specify setuptools
On older versions of Python, skip benchmarks that use features introduced in newer Python versions
Support --inherit-environ when reusing a venv
Use tomllib/tomli over toml
Update MANIFEST.in to include cert files for asyncio_tcp_ssl benchmark
Fix undefined variable issue when raising VenvPipInstallFailedError
Add mypy config; run mypy in CI
Fix typo of str.partition from _pyproject_toml.py
Add version of Richards benchmark that uses super()
Add a benchmark for runtime-checkable protocols
Extend async tree benchmarks to cover eager task execution

⭐ Version 1.0.7 (2023-04-22)

Upgrade pyperf from 2.5.0 to 2.6.0
Clean unused imports and other small code details
Migrage to the pyproject.toml based project
Fix the django_template benchmark due to lack of distutils
Add benchmark for toml
Add benchmark for comprehensions
Add benchmark for asyncio_tcp_ssl
Add benchmark for asyncio_tcp
Add benchmark for Dask scheduler
Add the gc benchmarks to the MANIFEST file

⭐ Version 1.0.6 (2022-11-20)

Okay this one is a whole saga, sit down 🍓

Upgrade pyperf from 2.4.1 to 2.5.0
Add a benchmark to measure gc traversal
Add jobs field in compile section to specify make -j param
Add benchmark for Docutils
Add async_generators benchmark
Add benchmark for IPC
Fix Manifest Group
Fix installing dev build of pyperformance inside compile/compile_all
Always upload, even when some benchmarks fail
Add sqlglot benchmarks
Support reporting geometric mean by tags
Allow for specifying local wheels and sdists as dependencies
Add a benchmark based on python -m pprint
Add mdp back into the default group
Add coroutines benchmark
Reduce noise in generators benchmark
Add benchmark for deepcopy
Add coverage benchmark
Add generators benchmark
Add benchmark for async tree workloads
Support relative paths to manifest files
Add support for multiple benchmark groups in a manifest
Fix --inherit-environ issue
Use working Genshi 0.7.7

⭐ Version 1.0.4 (2022-01-25)

Re-release support for user-defined benchmark after fixing problem with virtual environments.

⭐ Version 1.0.3 (2021-12-20)

Support user-defined benchmark suites.

More parts incoming, don't you dare close the tab bestie 💖

🌈 Version 1.0.2 (2021-05-11) 🌈

The henshi benchmark is temporarily disabled, bestie, since is no longer compatible with Python 3.11. 😭 we'll miss her fr.
The html5lib benchmark is reenabled — html5lib 1.1 has been released, so she's back and thriving ✨
Requirements updated. 🫶
Travis CI? gone. GitHub Actions? iconic. 💅 replaced, no cap.
The development branch master got renamed to main. See https://sfconservancy.org/news/2020/jun/23/gitbranchname/ for the rationale — receipts, we love to see it 👑

💖 Version 1.0.1 (2020-03-26) 💖

No more six module usage since Python 2 is no longer supported — Python 2 specific code? removed, periodt. ✂️
Update dependencies, and yes we're listing all of them because that's the vibe check:
- django: 3.0 => 3.0.4
- dulwich: 0.19.14 => 0.19.15
- mako: 1.1.0 = > 1.1.2
- mercurial: 5.1.1 => 5.3.1
- psutil: 5.6.7 => 5.7.0
- pyperf: 1.7.0 => 2.0.0
- sqlalchemy: 1.3.12 => 1.3.15
- sympy: 1.5 => 1.5.1
- tornado: 6.0.3 => 6.0.4
Removed the six, html5lib and mercurial requirements. bye besties 👋
pip-tools (pip-compile) is now used to update dependencies. slay, automation 💅

🦋 Version 1.0.0 (2019-12-17) 🦋

pyflate benchmarks are enabled on Python 3. let's gooo 🔥
The spambayes benchmark is removed — it is not compatible with Python 3, so she had to go 😔
The 2n3:benchmark group is removed, gone, deleted. 🗑️
Python 2.7 support is dropped — because old Django and Tornado versions are not compatible with incoming Python 3.9. it was time, ngl 🫶
The html5lib benchmark is temporarily disabled, since it's no longer compatible with Python 3.9. 💀
Update requirements:
- Django: 1.11.22 => 3.0
- Mako: 1.0.14 => 1.1.0
- SQLAlchemy: 1.3.6 => 1.3.12
- certifi: 2019.6.16 => 2019.11.28
- docutils: 0.15.1 => 0.15.2
- dulwich: 0.19.11 => 0.19.14
- mercurial: 5.0.2 => 5.1.1
- psutil: 5.6. => 5.6.7
- pyperf: 1.6.1 => 1.7.0
- six: 1.12. => 1.13.0
- sympy: 1.4 => 1.5

🌸 Version 0.9.1 (2019-07-29) 🌸

hg_startup is enabled on Python 3 ✨
Compatibility with Python 3.8 beta 2 is fixed 🛠️
Update requirements:
- certifi: 2019.3.9 => 2019.6.16
- Chameleon: 3.6.1 => 3.6.2
- Django: 1.11.20 => 1.11.22
- docutils: 0.14 => 0.15.1.post1
- Mako: 1.0.10 => 1.0.14
- mercurial: 5.0 => 5.0.2
- pathlib2: 2.3.3 => 2.3.4
- psutil: 5.6.2 => 5.6.3
- SQLAlchemy: 1.3.4 => 1.3.6

🎀 Version 0.9.0 (2019-05-29) 🎀

The project got renamed from "performance" to "pyperformance" 👑 glow up era.
pyperf upgraded from version 1.6.0 to 1.6.1. The project itself has been renamed from "perf" to "pyperf", and the imports are updated to match. iykyk 💅
Issue #54: Genshi updated to 0.7.3 — it is now compatible with Python 3.8. 🎉
Update requirements:
- Mako: 1.0.9= > 1.0.10
- SQLAlchemy: 1.3.3 => 1.3.4

⭐ Version 0.8.0 (2019-05-10) ⭐

The compile command gets a "pkg_only" option in benchmark.conf, plus support for native libraries that are installed but not on path. Patch by Robert Grimm — ty king 🫶
Travis configuration updated: trusty image, pip cache. Patch by Inada Naoki 💖
tornado upgraded to 5.1.1. Patch by Inada Naoki ✨
The compile command is fixed on Mac OS: no program extension. Patch by Anthony Shaw 🛠️
Update requirements:
- Chameleon: 3.4 => 3.6.1
- Django: 1.11.16 => 1.11.20
- Genshi: 0.7.1 => 0.7.2
- Mako: 1.0.7 => 1.0.9
- MarkupSafe: 1.0 => 1.1.1
- SQLAlchemy: 1.2.12 => 1.3.3
- certifi: 2018.10.15 => 2019.3.9
- dulwich: 0.19.6 => 0.19.11
- mercurial: 4.7.2 => 5.0
- mpmath: 1.0.0 => 1.1.0
- pathlib2: 2.3.2 => 2.3.3
- perf: 1.5.1 => 1.6.0
- psutil: 5.4.7 => 5.6.2
- six: 1.11.0 => 1.12.0
- sympy: 1.3 => 1.4
- tornado: 4.5.3 => 5.1.1

🧚 Version 0.7.0 (2018-10-16) 🧚

python_startup gets a --exit option. cute ✨
Update requirements:
- certifi: 2017.11.5 => 2018.10.15
- Chameleon: 3.2 => 3.4
- Django: 1.11.9 => 1.11.16
- dulwich: 0.18.6 => 0.19.6
- Genshi: 0.7 => 0.7.1
- mercurial: 4.4.2 => 4.7.2
- pathlib2: 2.3.0 => 2.3.2
- psutil: 5.4.3 => 5.4.7
- SQLAlchemy: 1.2.0 => 1.2.12
- sympy: 1.1.1 => 1.3
Issue #40 fixed for pip 10 and newer: indirect dependencies are removed. Indirect dependencies were used to install cffi, but Mercurial 4.0 doesn't depend on cffi anymore. ✂️ no cap.

🍓 Version 0.6.1 (2018-01-11) 🍓

inherit-environ fixed: it now propagates to recursive invocations of performance in the compile and compile_all commands. 🫶
The --track-memory option is fixed, thanks to the update to perf 1.5. ✨
Update requirements
- certifi: 2017.4.17 => 2017.11.5
- Chameleon: 3.1 => 3.2
- Django: 1.11.3 => 1.11.9
- docutils: 0.13.1 => 0.14
- dulwich: 0.17.3 => 0.18.6
- html5lib: 0.999999999 => 1.0.1
- Mako: 1.0.6 => 1.0.7
- mercurial: 4.2.2 => 4.4.2
- mpmath: 0.19 => 1.0.0
- perf: 1.4 => 1.5.1 (fix --track-memory option)
- psutil: 5.2.2 => 5.4.3
- pyaes: 1.6.0 => 1.6.1
- six: 1.10.0 => 1.11.0
- SQLAlchemy: 1.1.11 => 1.2.0
- sympy: 1.0 => 1.1.1
- tornado: 4.5.1 => 4.5.3

💜 Version 0.6.0 (2017-07-06) 💜

warn changed to warning in bm_logging.py. Why? because in Python 3, Logger.warn() calls warnings.warn() to log a deprecation warning, so is slower than Logger.warning(). performance queen behavior 💅
The logging_silent microbenchmark suite is added again. she's back 🎉
compile command: the Git repository is updated before getting the revision. 🛠️
Update requirements
- perf: 1.3 => 1.4 (fix parse_cpu_list(): strip also NUL characters)
- Django: 1.11.1 => 1.11.3
- mercurial: 4.2 => 4.2.2
- pathlib2: 2.2.1 => 2.3.0
- SQLAlchemy: 1.1.10 => 1.1.11

🌺 Version 0.5.5 (2017-05-29) 🌺

On the 2.x branch on CPython, compile now passes --enable-unicode=ucs4 to the configure script on all platforms — except on Windows, which uses UTF-16 because of its 16-bit wchar_t. 💀 windows stays special.
The float benchmark now uses __slots__ on the Point class. optimizing, we love to see it ✨
The following microbenchmarks are removed. They have been moved to the pymicrobench project at https://github.com/vstinner/pymicrobench because they are too short, not representative of real applications and are too unstable. 😔 moving on besties:
- pybench microbenchmark suite
- call_simple
- call_method
- call_method_unknown
- call_method_slots
- logging_silent: values are faster than 1 ns on PyPy with 2^27 loops! (and around 0.7 us on CPython) 💀 literally faster than light.
Update requirements
- Django: 1.11 => 1.11.1
- SQLAlchemy: 1.1.9 => 1.1.10
- certifi: 2017.1.23 => 2017.4.17
- perf: 1.2 => 1.3
- mercurial: 4.1.2 => 4.2
- tornado: 4.4.3 => 4.5.1

🎀 Version 0.5.4 (2017-04-10) 🎀

New documentation created at http://pyperformance.readthedocs.io/ — go read it, bestie 📖
"CPython results, 2017" added to the doc: significant performance changes, significant optimizations, timeline, etc. the receipts are immaculate 👑
The show command doesn't need to create a virtual env anymore. less work, same slay 💅
New commands added:
- pyperformance compile: compile, install and benchmark
- pyperformance compile_all: benchmark multiple branches and revisions of Python
- pyperformance upload: upload a JSON file to a Codespeed
setup.py: dependencies added to the perf and six modules. 🫶
bm_xml_etree now uses "_pure_python" in benchmark names if the accelerator is explicitly disabled. ✨
Upgrade requirements:
- Django: 1.10.6 -> 1.11
- SQLAlchemy: 1.1.6 -> 1.1.9
- mercurial: 4.1.1 -> 4.1.2
- perf: 1.1 => 1.2
- psutil: 5.2.1 -> 5.2.2
- tornado: 4.4.2 -> 4.4.3
- webencodings: 0.5 -> 0.5.1
perf 1.2 now calibrates the number of warmups on PyPy. 🔥
On Python 3.5a0: force pip 7.1.2 and setuptools 18.5: https://sourceforge.net/p/pyparsing/bugs/100/ — ancient lore, but we keep it 🗿

🦋 Version 0.5.3 (2017-03-27) 🦋

Dulwich upgraded to 0.17.3 to support PyPy older than 5.6: see https://github.com/jelmer/dulwich/issues/509 🔗
ResourceWarning warnings fixed: close explicitly files and sockets. no more leaking, periodt ✂️
scripts: Mercurial commands replaced with Git commands. git era 💅
Upgrade requirements:
- dulwich: 0.17.1 => 0.17.3
- perf: 1.0 => 1.1
- psutil: 5.2.0 => 5.2.1

🌸 Version 0.5.2 (2017-03-17) 🌸

Upgrade requirements:
- certifi: 2016.9.26 => 2017.1.23
- Chameleon: 3.0 => 3.1
- Django: 1.10.5 => 1.10.6
- MarkupSafe: 0.23 => 1.0
- dulwich: 0.16.3 => 0.17.1
- mercurial: 4.0.2 => 4.1.1
- pathlib2: 2.2.0 => 2.2.1
- perf: 0.9.3 => 1.0
- psutil: 5.0.1 => 5.2.0
- SQLAlchemy: 1.1.4 => 1.1.6

💖 Version 0.5.1 (2017-01-16) 💖

Windows support fixed (upgrade perf from 0.9.0 to 0.9.3) 🪟✨
Upgrade requirements:
- Chameleon: 2.25 => 3.0
- Django: 1.10.3 => 1.10.5
- docutils: 0.12 => 0.13.1
- dulwich: 0.15.0 => 0.16.3
- mercurial: 4.0.0 => 4.0.2
- perf: 0.9.0 => 0.9.3
- psutil: 5.0.0 => 5.0.1

🔥 Version 0.5.0 (2016-11-16) 🔥

The mdp benchmark is added: battle with damages and topological sorting of nodes in a graph. sounds dramatic, we love it ⚔️
The default benchmark group now includes all benchmarks but pybench. ✨
If a benchmark fails, log an error, continue to execute following benchmarks, but exit with error code 1. > important, bestie — the run keeps going but the exit status is still a failure, no cap.
Deprecated benchmarks removed: threading_threaded_count and threading_iterative_count. It wasn't possible to run them anyway. 💀
The dulwich requirement is now optional since its installation fails on Windows. 🙈
Upgrade requirements:
- Mako: 1.0.5 => 1.0.6
- Mercurial: 3.9.2 => 4.0.0
- SQLAlchemy: 1.1.3 => 1.1.4
- backports-abc: 0.4 => 0.5

⭐ Version 0.4.0 (2016-11-07) ⭐

The sqlalchemy_imperative benchmark is added: it wasn't registered properly. fixed, bestie 🛠️
The list command now only lists the benchmark that the run command will run. The list command gets a new -b/--benchmarks option. 💅
The code creating the virtual environment is rewritten to test pip correctly. get-pip.py is downloaded and run if pip installation failed. resilient queen 👑
Upgrade requirements:
- perf: 0.8.2 => 0.9.0
- Django: 1.10.2 => 1.10.3
- Mako: 1.0.4 => 1.0.5
- psutil: 4.3.1 => 5.0.0
- SQLAlchemy: 1.1.2 => 1.1.3
virtualenv dependency removed ✂️

🧚 Version 0.3.2 (2016-10-19) 🧚

setup.py fixed: it now includes also performance/benchmarks/data/asyncio.git/ 📁

🎀 Version 0.3.1 (2016-10-19) 🎀

The regex_dna benchmark is added 🧬✨
The run command now fails with an error if no benchmark was run. no silent nothing-runs, bestie 😤
genshi, logging, scimark, sympy and xml_etree scripts now run all sub-benchmarks by default. slay 🌈
pybench rewritten using perf: the old legacy code to calibrate and run benchmarks is removed, and the perf.Runner API is reused. glow up 💅
Change the heuristic for creating the virtual environment — it tried these commands:
- python -m venv
- python -m virtualenv
- virtualenv -p python
The creation of the virtual environment now makes sure that pip actually works to detect "python3 -m venv" which doesn't install pip. bestie, this one is lowkey important 💅
Upgrade the perf dependency from 0.7.12 to 0.8.2: update all benchmarks to the new perf 0.8 API (which introduces incompatible changes). it's giving glow-up but also incompatible changes, so heads up 😭
Update SQLAlchemy from 1.1.1 to 1.1.2.

Version 0.3.0 (2016-10-11)

New benchmarks:

crypto_pyaes: Benchmark a pure-Python implementation of the AES block-cipher in CTR mode using the pyaes module (version 1.6.0). Add pyaes dependency. 🔥
sympy: Benchmark on SymPy. Add scipy dependency.
scimark benchmark
deltablue: DeltaBlue benchmark
dulwich_log: Iterate on commits of the asyncio Git repository using the Dulwich module. Add dulwich (and mpmath) dependencies.
pyflate: Pyflate benchmark, tar/bzip2 decompressor in pure Python
sqlite_synth benchmark: Benchmark Python aggregate for SQLite
genshi benchmark: Render template to XML or plain text using the Genshi module. Add Genshi dependency.
sqlalchemy_declarative and sqlalchemy_imperative benchmarks: SQLAlchemy Declarative and Imperative benchmarks using SQLite. Add SQLAlchemy dependency.

Enhancements:

compare command now fails if the performance versions are different
nbody: add --reference and --iterations command line options.
chaos: add --width, --height, --thickness, --filename and --rng-seed command line options
django_template: add --table-size command line option
json_dumps: add --cases command line option
pidigits: add --digits command line option
raytrace: add --width, --height and --filename command line options
Port html5lib benchmark to Python 3
Enable pickle_pure_python and unpickle_pure_python on Python 3 (code was already compatible with Python 3)
Creating the virtual environment doesn't inherit environment variables (especially PYTHONPATH) by default anymore: --inherit-environ command line option must now be used explicitly.

Bugfixes:

chaos benchmark now also reset the random module at each sample to get more reproductible benchmark results
Logging benchmarks now truncate the in-memory stream before each benchmark run

Rename benchmarks:

Rename benchmarks to get a consistent name between the command line and benchmark name in the JSON file.
Rename pickle benchmarks:
- slowpickle becomes pickle_pure_python
- slowunpickle becomes unpickle_pure_python
- fastpickle becomes pickle
- fastunpickle becomes unpickle
Rename ElementTree benchmarks: replace etree_ prefix with xml_etree_.
Rename hexiom2 to hexiom_level25 and explicitly pass --level=25 parameter
Rename json_load to json_loads
Rename json_dump_v2 to json_dumps (and remove the deprecated json_dump benchmark)
Rename normal_startup to python_startup, and startup_nosite to python_startup_no_site
Rename threaded_count to threading_threaded_count, rename iterative_count to threading_iterative_count
Rename logging benchmarks:
- silent_logging to logging_silent
- simple_logging to logging_simple
- formatted_logging to logging_format

Minor changes:

Update dependencies
Remove broken --args command line option.

Version 0.2.2 (2016-09-19)

Add a new show command to display a benchmark file
Issue #11: Display Python version in compare. Display also the performance version.
CPython issue #26383; csv output: don't truncate digits for timings shorter than 1 us
compare: Use sample unit of benchmarks, format values in the table output using the unit
compare: Fix the table output if benchmarks only contain a single sample
Remove unused -C/--control_label and -E/--experiment_label options
Update perf dependency to 0.7.11 to get Benchmark.get_unit() and BenchmarkSuite.get_metadata()

Version 0.2.1 (2016-09-10)

Add --csv option to the compare command
Fix compare -O table output format
Freeze indirect dependencies in requirements.txt
run: add --track-memory option to track the memory peak usage
Update perf dependency to 0.7.8 to support memory tracking and the new --inherit-environ command line option
If virtualenv command fail, try another command to create the virtual environment: catch virtualenv error
The first command to upgrade pip to version >= 6.0 now uses the pip binary rather than python -m pip to support pip 1.0 which doesn't support python -m pip CLI.
Update Django (1.10.1), Mercurial (3.9.1) and psutil (4.3.1)
Rename --inherit_env command line option to --inherit-environ and fix it

Version 0.2 (2016-09-01)

Update Django dependency to 1.10
Update Chameleon dependency to 2.24
Add the --venv command line option
Convert Python startup, Mercurial startup and 2to3 benchmarks to perf scripts (bm_startup.py, bm_hg_startup.py and bm_2to3.py)
Pass the --affinity option to perf scripts rather than using the taskset command
Put more installer and optional requirements into performance/requirements.txt
Cached .pyc files are no more removed before running a benchmark. Use venv recreate command to update a virtual environment if required.
The broken --track_memory option has been removed. It will be added back when it will be fixed.
Add performance version to metadata
Upgrade perf dependency to 0.7.5 to get Benchmark.update_metadata()

Version 0.1.2 (2016-08-27)

Windows is now supported
Add a new venv command to show, create, recrete or remove the virtual environment.
Fix pybench benchmark (update to perf 0.7.4 API)
performance now tries to install the psutil module on CPython for better system metrics in metadata and CPU pinning on Python 2.
The creation of the virtual environment now also tries virtualenv and venv Python modules, not only the virtualenv command.
The development version of performance now installs performance with "pip install -e <path_to_performance>"
The GitHub project was renamed from python/benchmarks to python/performance.

Version 0.1.1 (2016-08-24)

Fix the creation of the virtual environment
Rename pybenchmarks script to pyperformance
Add -p/--python command line option
Add __main__ module to be able to run: python3 -m performance

Version 0.1 (2016-08-24)

First release after the conversion to the perf module and move to GitHub
Removed benchmarks:
- django_v2, django_v3
- rietveld
- spitfire (and psyco): Spitfire is not available on PyPI
- pystone
- gcbench
- tuple_gc_hell

🌈 History

Projected moved to https://github.com/python/performance in August 2016. Files reorganized, benchmarks patched to use the perf module to run benchmark in multiple processes. 🦋

Project started in December 2008 by Collin Winter and Jeffrey Yasskin for the Unladen Swallow project. The project was hosted at https://hg.python.org/benchmarks until Feb 2016.

Other Python Benchmarks:

CPython: speed.python.org (https://speed.python.org/) uses pyperf, pyperformance and Codespeed (https://github.com/tobami/codespeed/) (Django web application)
PyPy: speed.pypy.org (http://speed.pypy.org/) uses PyPy benchmarks (https://bitbucket.org/pypy/benchmarks)
Pyston: pyston-perf (https://github.com/dropbox/pyston-perf) and speed.pyston.org (http://speed.pyston.org/)
Numba benchmarks (http://numba.pydata.org/numba-benchmark/)
Cython: Cython Demos/benchmarks (https://github.com/cython/cython/tree/master/Demos/benchmarks)
pythran: numpy-benchmarks (https://github.com/serge-sans-paille/numpy-benchmarks)

See also the Python speed mailing list https://mail.python.org/mailman/listinfo/speed and the Python pyperf module http://pyperf.readthedocs.io/ (used by pyperformance).

pyperformance is not tuned for PyPy yet: use the PyPy benchmarks project https://foss.heptapod.net/pypy/benchmarks instead to measure PyPy performances.

Image generated by bm_raytrace (pure Python raytrace): [image: Pure Python raytracer] [image]

💖 Author

Victor Stinner

📜 Copyright

2017, Victor Stinner

that's the whole tea, bestie 🫶 this page is PYTHONP...ARKSUITE(1), version 1.14.0, dated Feb 08, 2026 — periodt. 🎀✨💅

📦 where this page comes from

arch rolling: python-pyperformance 1.14.0-1 · MIT
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on 2026-11-04: a language model's rewrite of the original, under its terms

PYTHONP...ARKSUITE(1)

💖 NAME 💖

✨ USAGE ✨

🎀 Installation 🎀

🪟 Windows notes 🪟

🏃‍♀️ Run benchmarks 🏃‍♀️

🌈 Basic commands 🌈

💅 Common options 💅

🔥 run 🔥

🦋 show 🦋

🍓 compare 🍓

🌸 list 🌸

🧚 list_groups 🧚

💜 venv 💜

⭐ venv show ⭐

🎀 venv create 🎀

🎀 venv recreate 🎀

💀 venv remove 💀

🔥 Compile Python to run benchmarks 🔥

🛠️ compile 🛠️

🌈 compile_all 🌈

🦋 upload 🦋

✨ How to get stable benchmarks ✨

🫶 pyperformance virtual environment 🫶

👑 What is the goal of pyperformance 👑

💅 Don't disable GC nor ASLR

📈 Include outliers and spikes

🔥 Warmups and steady state

📝 Notes

🏆 BENCHMARKS 🏆

🌈 Available Groups

🔍 Available Benchmarks

🧰 2to3

🌳 async_tree

🔡 base64

🦎 chameleon

🌀 chaos

🔐 crypto_pyaes

📋 deepcopy

💠 deltablue

🎭 django_template

🌿 dulwich_log

📚 docutils

🐣 fannkuch

🧮 float

🧵 genshi

go 🔥

hexiom 🧩

hg_startup 🧚

html5lib 🌈

json_dumps, json_loads 🍓

logging 💜

mako 🎀

mdp 🦋

meteor_contest ⭐

nbody 💫

python_startup, python_startup_nosite 🚀

nqueens 👑

pathlib 📁

pickle 🥒

pidigits 🥧

pyflate 🗜️

raytrace 🌅

regex_compile 🧵

regex_dna 🧬

regex_effbot 🧪

regex_v8 🏎️

richards 🧠

scimark 📊

spectral_norm 🌌

sqlalchemy_declarative, sqlalchemy_imperative 🗄️

sqlite_synth 🪶

sympy 🧮

💰 telco

🌪️ tornado_http

📦 unpack_sequence

🌲 xml_etree

🧪 CUSTOM BENCHMARKS 🧪

🛠️ Using Custom Benchmarks

📁 The pyperformance File Formats

🎀 `venv recreate` 🎀

💀 `venv remove` 💀

🛠️ `compile` 🛠️

🌈 `compile_all` 🌈

🦋 `upload` 🦋

🎀 The `[project]` Section 🎀

🦋 The `[tool.pyperformance]` Section 🦋

🎀 2015-05-30: C implementation of `collections.OrderedDict`, html5lib

🌸 2015-05-23: C implementation of `functools.lru_cache()`, sympy

🐢 `2016-09-11`: `regex_compile`

👑 `call_method`

🦋 `crypto_pyaes`

🌸 `json_loads`

🧚 `logging_silent`

🍓 `pickle`

🎀 `python_startup`

🦋 `regex_compile`

👑 `telco`