GDAL-VECTOR-PARTITION(1)

📦
version 3.13.3 · 4 versions
, in alpine 3.24, arch rolling, ubuntu stonking

🦋 gdal-vector-partition — the vibe ✨

okay bestie, welcome to the GDAL manual page for gdal-vector-partition(1) 💖

🌸 NAME 🌸

gdal-vector-partition — partition a vector dataset into multiple files 💅

Added in version 3.12.

🎀 SYNOPSIS 🎀

Usage: gdal vector partition [OPTIONS] <INPUT> <OUTPUT>

Partition a vector dataset into multiple files.

Positional arguments:
  -i, --input <INPUT>                                  Input vector datasets [required] [not available in pipelines]
  -o, --output <OUTPUT>                                Output directory [required]

Common Options:
  -h, --help                                           Display help message and exit
  --json-usage                                         Display usage as JSON document and exit
  --config <KEY>=<VALUE>                               Configuration option [may be repeated]
  -q, --quiet                                          Quiet mode (no progress bar or warning message) [not available in pipelines]

Options:
  --overwrite                                          Whether overwriting existing output dataset is allowed
                                                       Mutually exclusive with --append
  --append                                             Whether appending to existing layer is allowed
                                                       Mutually exclusive with --overwrite
  -f, --of, --format, --output-format <OUTPUT-FORMAT>  Output format
  --co, --creation-option <KEY>=<VALUE>                Creation option [may be repeated]
  --lco, --layer-creation-option <KEY>=<VALUE>         Layer creation option [may be repeated]
  --field <FIELD>                                      Attribute or geometry field(s) on which to partition [may be repeated]
  --scheme <SCHEME>                                    Partitioning scheme. SCHEME=hive|flat (default: hive)
  --pattern <PATTERN>                                  Filename pattern ('part_%010d' for scheme=hive, '{LAYER_NAME}_{FIELD_VALUE}_%010d' for scheme=flat)
  --feature-limit <FEATURE-LIMIT>                      Maximum number of features per file
  --max-file-size <MAX-FILE-SIZE>                      Maximum file size (MB or GB suffix can be used)
  --omit-partitioned-field                             Whether to omit partitioned fields from target layer definition
  --skip-errors                                        Skip errors when writing features

Advanced Options:
  --if, --input-format <INPUT-FORMAT>                  Input formats [may be repeated] [not available in pipelines]
  --oo, --open-option <KEY>=<VALUE>                    Open options [may be repeated] [not available in pipelines]

💖 DESCRIPTION 💖

gdal vector partition yeets features into different files 🔥 — specifically, depending on the values a feature takes on a subset of attribute or geometry fields that you picked, and/or by capping each output layer to a max number of features and/or a max file size. slay.

Two partitioning schemes exist, and you gotta choose:

  • hive, which is the Apache Hive partitioning situation, and it's the default one, periodt.

Each partitioning field = a nested directory, bestie. Picture a layer with fields "continent" and "country" chosen as partitioning fields. Every feature where "continent" evaluates to "Europe" and "country" evaluates to "France" gets written into the "continent=Europe/country=France/" subdirectory of the output directory. iconic.

NULL values for partitioning fields get encoded as __HIVE_DEFAULT_PARTITION__ in the directory name (yes, literally that string). Non-ASCII characters, space, equal sign, or characters that just don't vibe with directory name constraints get percent-encoded (e.g. %20 for space). 🫶

  • flat, where files get written directly under the output directory using a default filename pattern of {LAYER_NAME}_{FIELD_VALUE}_%10d.

By default, the format of the input dataset is used for the output — but only if it can be determined and the input driver supports writing. Otherwise? You must use --format. no cap.

gdal vector partition can be used as the last step of a pipeline. 👑

The following options are available:

✨ PROGRAM-SPECIFIC OPTIONS ✨

  • --feature-limit <FEATURE-LIMIT> — Maximum number of features per file. Default is unlimited. If you exceed the limit, several parts get created (it just splits, nbd).
  • --field <FIELD-NAME> — Field(s) on which to partition, aka the whole reason we're here 💅

Only attribute fields of type String, Integer and Integer64 are allowed. And the order in which you specify the fields matters — that's what determines the directory hierarchy. iykyk.

Starting with GDAL 3.13, geometry field names can be specified too (OGR_GEOMETRY being the generic name for the first geometry field). Partitioning on geometry fields happens on the geometry type. This is lowkey useful for file formats where a single geometry type per layer is allowed.

Starting with GDAL 3.13, --field is no longer required — but when you don't specify it, --feature-limit and/or --max-file-size must be specified. can't have nothing.

  • --max-file-size <MAX-FILE-SIZE> — Maximum file size (MB or GB suffix can be used). Default: unlimited. If the limit is exceeded, several parts are created.
heads up bestie: the maximum file size is used as a hint, and might not be strictly respected 😭 — because the file size evaluation for a feature is based on heuristics, since the file size itself can't be reliably used while it's being written. In particular, the heuristics does not assume any compression, so for compressed formats the actual size of a part can be significantly smaller than the specified limit. it's giving estimates.
  • --omit-partitioned-field — Whether to omit partitioned fields from the target layer definition. Automatically set for Parquet output format and Hive partitioning. (so sometimes it's just doing its thing without asking, iconic.)
  • --output <OUTPUT-DIRECTORY> — Root of the output directory. [required]
  • --pattern <PATTERN> — Filename pattern. A user chosen string, with substitutions for:
    • {LAYER_NAME} — when found, is substituted with the layer name (percent encoded where needed).
    • {FIELD_VALUE} — when found, is substituted with the partitioning field value (percent encoded where needed). If several partitioning fields are used, each value is separated by underscore (_). Empty strings are substituted with __EMPTY__ and null fields with __NULL__.
    • %[0?][0-9]?[0]?d — C-style integer formatter for the part number. Valid values are for example %d or %05d. One and only one part number specifier must be present in the pattern.

Default values for the pattern are part_%010d for the hive scheme, and {LAYER_NAME}_{FIELD_VALUE}_%010d for the flat scheme.

  • --scheme hive|flat — Partitioning scheme. Defaults to hive.

🧚 STANDARD OPTIONS 🧚

  • --append — Whether the output directory must be opened in append mode. Implies that it already exists and that the output format supports appending.

This mode is useful when adding new features to an already existing partitioned dataset. (yes, we're adding to the collection 🍓)

  • --co, --creation-option <NAME>=<VALUE> — Many formats have one or more optional dataset creation options that can be used to control particulars about the file created. For instance, the GeoPackage driver supports creation options to control the version.

May be repeated.

The dataset creation options available vary by format driver, and some simple formats have no creation options at all. A list of options supported for a format can be listed with the --formats command line option, but the documentation for the format is the definitive source of information on driver creation options. See Vector drivers format specific documentation for legal creation options for each format.

note that dataset creation options are different from layer creation options. don't mix them up bestie 💀
  • --if, --input-format <format> — Format/driver name to be attempted to open the input file(s). It is generally not necessary to specify it, but it can be used to skip automatic driver detection when it fails to select the appropriate driver. This option can be repeated several times to specify several candidate drivers. Note that it does not force those drivers to open the dataset. In particular, some drivers have requirements on file extensions.

May be repeated.

  • --lco, --layer-creation-option <NAME>=<VALUE> — Many formats have one or more optional layer creation options that can be used to control particulars about the layer created. For instance, the GeoPackage driver supports layer creation options to control the feature identifier or geometry column name, setting the identifier or description, etc.

May be repeated.

The layer creation options available vary by format driver, and some simple formats have no layer creation options at all. A list of options supported for a format can be listed with the --formats command line option, but the documentation for the format is the definitive source of information on driver creation options. See Vector drivers format specific documentation for legal creation options for each format.

and again: layer creation options are different from dataset creation options 🙈
  • --oo, --open-option <NAME>=<VALUE> — Dataset open option (format specific).

May be repeated.

  • -f, --of, --format, --output-format <OUTPUT-FORMAT> — Which output vector format to use. Allowed values may be given by gdal --formats | grep vector | grep rw | sort ✨
  • --overwrite — Allow program to overwrite existing target file or dataset. Otherwise, by default, gdal errors out if the target file or dataset already exists. (so like, it's not just gonna steamroll your stuff, you gotta ask nicely.)
  • --skip-errors — Added in version 3.12.

Whether failures to write feature(s) should be ignored. Note that this option sets the size of the transaction unit to one feature at a time, which may cause severe slowdown when inserting into databases. it's giving slow, fr fr.

💀 RETURN STATUS CODE 💀

The program returns status code 0 in case of success, and non-zero in case of error (non-blocking errors emitted as warnings are considered as a successful execution). 🫶

✨ EXAMPLES ✨

ok bestie these are the recipes, the real-life scenarios, the moments 💅 every single one is copy-pasteable, no cap.

🐚 Example 1: Create a partition based on the "continent" and "country" fields

you wanna split by TWO fields at once? iconic, the partition hierarchy goes nest-y, continent first then country inside it 🫶

$ gdal vector partition world_cities.gpkg out_directory --field continent,country --format Parquet

🧚 Example 2: Create a partition based on the "country" field, filtering on cities with population bigger than 1 million, with a flat partitioning scheme

okay so this one is a whole pipeline situation — read the file, filter down to the big-city girlies (pop > 1e6), then partition by country into GPKG, and --scheme flat means no nested folder drama, just vibes 🌸

$ gdal pipeline ! read world_cities.gpkg ! filter --where "pop > 1e6" ! partition out_directory --field country --format GPKG --scheme flat

🦋 Example 3: Create multiple Shapefiles, one for each geometry type, from a GeoPackage file, by grouping together POLYGON/MULTIPOLYGON or LINESTRING/MULTILINESTRING.

so OGR_GEOMETRY is that special field, and set-geom-type --multi is doing the glow-up so POLYGON and MULTIPOLYGON team up, and LINESTRING and MULTILINESTRING team up — one Shapefile per geometry type, slay 💖

$ gdal vector pipeline ! read input.gpkg ! set-geom-type --multi ! partition out_directory --scheme flat --field OGR_GEOMETRY --format "ESRI Shapefile"

🎀 Example 4: Split a file into files with at most 100,000 features.

the classic "this file is Too Much, break it up bestie" move — each output file gets at most 100000 features, flat scheme, Parquet ✨

$ gdal vector partition world_cities.gpkg out_directory --feature-limit 100000 --scheme flat --format Parquet

💜 Example 5: Sort a file spatially and split it into files with at most 100,000 features.

same energy as example 4 but the data gets a spatial sort first — so the features inside each chunk are lowkey neighbours, which is giving cozy, spatially-clustered files 🌈

$ gdal vector pipeline read world_cities.gpkg ! sort ! partition out_directory --feature-limit 100000 --scheme flat --format Parquet

👑 AUTHOR

Even Rouault <even.rouault@spatialys.com>

(pt the legend himself, we stan 🫶)

1998-2026

okay that's the whole page bestie, go forth and partition responsibly 🎀 the source is GDAL-VECTOR-PARTITION(1) and this version is dated Aug 18, 2026 — now close the tab and touch grass, mwah 💅✨🦋

📦 where this page comes from