OSM2PGSQL-REPLICATION(1)

πŸ’– osm2pgsql-replication πŸ’–

okay bestie, welcome to the osm2pgsql-replication manual β€” that's the OSM2PGSQL-REPLICATION(1) page in the system, but we're calling it by its government name up top. it's the osm2pgsql database updater, and she's literally the reason your planet data doesn't go stale. let's get into it πŸ¦‹


✨ NAME ✨

osm2pgsql-replication β€” osm2pgsql database updater πŸ’…


πŸŽ€ SYNOPSIS πŸŽ€

osm2pgsql-replication [-h] {init,update,status} ...

that's it. three little subcommands. three little icons. init, update, status. pick your fighter πŸ’–


🌸 DESCRIPTION 🌸

okay so the whole bit: this is how you update an osm2pgsql database with changes from an OSM replication server. no cap.

she starts the party by initialising the updating process β€” and to do that she looks at either the import file or the newest object in the database. whichever one she can get her hands on first. then she saves the state in a table inside the database (iconic behaviour, honestly, keeping her own receipts in the same place πŸŽ€).

after that? every time you run her again she downloads the newly available data and applies it to the database. that's the loop. that's the whole vibe. she's just out here keeping you current.

if you want the actual play-by-play for the init and update commands, go read their help. they spell out the details fr fr.

πŸ‘‘ AVAILABLE COMMANDS πŸ‘‘

  • osm2pgsql-replication init β€” Initialise the replication process. the setup era ✨
  • osm2pgsql-replication update β€” Download newly available data and apply it to the database. the grind πŸ’…
  • osm2pgsql-replication status β€” Print information about the current replication status, optionally as JSON. the vibes check πŸ”

πŸ”₯ COMMAND 'osm2pgsql-replication init' πŸ”₯

usage:

osm2pgsql-replication init [-h] [-q] [-v] [-d DB] [-U NAME] [-H HOST]
                           [-P PORT] [-p PREFIX]
                           [--middle-schema SCHEMA] [--schema SCHEMA]
                           [--osm-file FILE | --server URL]
                           [--start-at TIME]

Initialise the replication process.

here's the tea, bestie: this function sets the replication service to use and figures out from which date to apply updates. you must call this function at least once to set up the replication process β€” there's no skipping it, she's not doing anything until you do. but the good news is it's safe to call again later. you can call it again to change the replication servers, or to roll back the update process and reapply updates. no drama, no consequences, she's flexible like that πŸ’…

so there are different methods available for initialisation β€” multiple ways to serve this look:

  • when no additional parameters are given, the data is initialised from the data in the database.
  • if the data was imported from a file with replication information and the properties table is available (for osm2pgsql >= 1.9) then the replication from the file is used.
  • otherwise the minutely update service from openstreetmap.org is used as the default replication service.

and the start date? either taken from the database timestamp (for osm2pgsql >= 1.9), or determined from the newest way in the database by querying the OSM API about its creation date. she's resourceful, she'll find out 🧚

you can swap the replication service with the --server parameter. want a different start date? add --start-at with an absolute ISO timestamp (e.g. 2007-08-20T12:21:53Z). and here's a cute little detail β€” when the program determines the start date from the database timestamp or way creation date, it subtracts another 3 hours by default, just to make sure all the new changes are actually available. safety margins, we love her for it. to change that rollback period, use --start-at with the number of minutes to rollback. and honestly? that rollback mode can also be used to force initialisation to use the database date and ignore the date from the replication information in the file. so if the file's data is giving nothing but chaos, you have an out πŸ‘‘

the initialisation process can also use replication information from an OSM file directly and ignore all other date information. for that, use the command osm2pgsql-replication --osm-file <filename>.


🌈 DEFAULT ARGUMENTS 'osm2pgsql-replication init' 🌈

  • -q, --quiet β€” Print only error messages. minimal output, minimal noise 🀫
  • -v, --verbose β€” Increase verboseness of output. she'll tell you EVERYTHING πŸ’¬

πŸ’œ DATABASE ARGUMENTS 'osm2pgsql-replication init' πŸ’œ

the following arguments can be used to set the connection parameters to the osm2pgsql database. you may also use libpq environment variables to set connection parameters, see https://www.postgresql.org/docs/current/libpq-envars.html. and if your database connection requires a password, use a pgpass file, see https://www.postgresql.org/docs/current/libpq-pgpass.html.

fr fr, don't hardcode the password, use the pgpass file. we don't do that here 🫢
  • -d DB, --database DB β€” Name of PostgreSQL database to connect to or conninfo string
  • -U NAME, --username NAME, --user NAME β€” PostgreSQL user name
  • -H HOST, --host HOST β€” Database server host name or socket location
  • -P PORT, --port PORT β€” Database server port
  • -p PREFIX, --prefix PREFIX β€” Prefix for table names (default 'planet_osm')
  • --middle-schema SCHEMA β€” Name of the schema to store the table for the replication state in
  • --schema SCHEMA β€” Name of the schema for the database

πŸ¦‹ REPLICATION SOURCE ARGUMENTS 'osm2pgsql-replication init' πŸ¦‹

  • --osm-file FILE β€” Get replication information from the given file. πŸ“‚
  • --server URL β€” Use replication server at the given URL 🌐
  • --start-at TIME β€” Time when to start replication. When an absolute timestamp (in ISO format) is given, it will be used. If a number is given, then replication starts the number of minutes before the known date of the database. ⏰

πŸ’… COMMAND 'osm2pgsql-replication update' πŸ’…

usage:

osm2pgsql-replication update update [options] [-- param [param ...]]

Download newly available data and apply it to the database.

okay so the mechanics: the data is downloaded in chunks of --max-diff-size MB. each chunk is saved in a temporary file and imported with osm2pgsql from there. the temporary file is normally deleted afterwards β€” unless you state an explicit location with --diff-file, in which case she keeps it. once the database is up to date with the replication source, the update process exits with 0. clean exit, clean conscience ✨

any additional arguments to osm2pgsql need to be given after --. the database and the prefix parameter are handed through to osm2pgsql automatically β€” you do not need to repeat them. and --append and --slim will always be added as well. free of charge πŸ’…

use the --post-processing parameter to execute a script after osm2pgsql has run successfully.

now listen closely because this part is important. if the update consists of multiple runs because the maximum size of downloaded data was reached, then the script is executed each time that osm2pgsql has run. and when the post-processing fails, then the entire update run is considered a failure and the replication information is not updated. that means when update is run the next time, it will recommence with downloading the diffs again and reapplying them to the database. 😭

is that bad? no bestie, this is usually safe. breathe. she's got you. 🫢

the script receives two parameters: the sequence ID and timestamp of the last successful run. the timestamp may be missing in the rare case that the replication service stops responding after the updates have been downloaded. rare, but it's a thing, so heads up πŸ™ˆ

  • param β€” Extra parameters to hand in to osm2pgsql.

⭐ OPTIONS 'osm2pgsql-replication update' ⭐

  • --diff-file FILE β€” File to save changes before they are applied to osm2pgsql. she keeps the goods around instead of yeeting them πŸ—‚οΈ
  • --max-diff-size MAX_DIFF_SIZE β€” Maximum data to load in MB (default: 500MB)
  • --osm2pgsql-cmd OSM2PGSQL_CMD β€” Path to osm2pgsql command
  • --once β€” Run updates only once, even when more data is available. one and done, periodt πŸ’…
  • --post-processing SCRIPT β€” Post-processing script to run after each execution of osm2pgsql.

🌈 DEFAULT ARGUMENTS 'osm2pgsql-replication update' 🌈

  • -q, --quiet β€” Print only error messages
  • -v, --verbose β€” Increase verboseness of output

πŸ’œ DATABASE ARGUMENTS 'osm2pgsql-replication update' πŸ’œ

the following arguments can be used to set the connection parameters to the osm2pgsql database. you may also use libpq environment variables to set connection parameters, see https://www.postgresql.org/docs/current/libpq-envars.html. If your database connection requires a password, use a pgpass file, see https://www.postgresql.org/docs/current/libpq-pgpass.html.

yes bestie, it's the same list as before. she's consistent like that πŸ‘‘

  • -d DB, --database DB β€” Name of PostgreSQL database to connect to or conninfo string
  • -U NAME, --username NAME, --user NAME β€” PostgreSQL user name
  • -H HOST, --host HOST β€” Database server host name or socket location
  • -P PORT, --port PORT β€” Database server port
  • -p PREFIX, --prefix PREFIX β€” Prefix for table names (default 'planet_osm')
  • --middle-schema SCHEMA β€” Name of the schema to store the table for the replication state in
  • --schema SCHEMA β€” Name of the schema for the database

πŸ” COMMAND 'osm2pgsql-replication status' πŸ”

usage:

osm2pgsql-replication status [-h] [-q] [-v] [-d DB] [-U NAME] [-H HOST]
                             [-P PORT] [-p PREFIX]
                             [--middle-schema SCHEMA] [--schema SCHEMA]
                             [--json]

Print information about the current replication status, optionally as JSON.

sample output, so you know exactly what the vibe looks like when you run her:

2021-08-17 15:20:28 [INFO]: Using replication service 'https://planet.openstreetmap.org/replication/minute', which is at sequence 4675115 ( 2021-08-17T13:19:43Z )
2021-08-17 15:20:28 [INFO]: Replication server's most recent data is <1 minute old
2021-08-17 15:20:28 [INFO]: Local database is 8288 sequences behind the server, i.e. 5 day(s) 20 hour(s) 58 minute(s)
2021-08-17 15:20:28 [INFO]: Local database's most recent data is 5 day(s) 20 hour(s) 59 minute(s) old

with the --json option, the status is printed as a json object:

{
  "server": {
    "base_url": "https://planet.openstreetmap.org/replication/minute",
    "sequence": 4675116,
    "timestamp": "2021-08-17T13:20:43Z",
    "age_sec": 27
  },
  "local": {
    "sequence": 4666827,
    "timestamp": "2021-08-11T16:21:09Z",
    "age_sec": 507601
  },
  "status": 0
}

now the important decoding, because these numbers mean things:

  • 'status' is 0 if there were no problems getting the status. 1 & 2 for improperly set up replication. 3 for network issues. if status is greater 0, then the 'error' key is an error message (as string). and 'status' is used as the exit code. so she's telling you what's wrong and how she's leaving πŸšͺ
  • 'server' is the replication server's current status. 'sequence' is its sequence number, 'timestamp' the time of that, and 'age_sec' the age of the data in seconds.
  • 'local' is the status of your server. πŸ‘€

⭐ OPTIONS 'osm2pgsql-replication status' ⭐

  • --json β€” Output status as json.

🌈 DEFAULT ARGUMENTS 'osm2pgsql-replication status' 🌈

  • -q, --quiet β€” Print only error messages
  • -v, --verbose β€” Increase verboseness of output

πŸ’… DATABASE ARGUMENTS 'osm2pgsql-replication status' πŸ’…

okay bestie, osm2pgsql-replication status is nosy and needs the deets to connect to your osm2pgsql database. These arguments are exactly that. You can also set connection parameters with libpq environment variables β€” the full tea is at https://www.postgresql.org/docs/current/libpq-envars.html. And if your database connection requires a password, use a pgpass file, more info at https://www.postgresql.org/docs/current/libpq-pgpass.html. periodt.

  • -d DB, --database DB β€” name of the PostgreSQL database to connect to, or a conninfo string. choose your own adventure πŸ’–
  • -U NAME, --username NAME, --user NAME β€” PostgreSQL user name. who dis? it's you, bestie.
  • -H HOST, --host HOST β€” database server host name or socket location. where's the party at? πŸ—ΊοΈ
  • -P PORT, --port PORT β€” database server port. the specific door number πŸ”’
  • -p PREFIX, --prefix PREFIX β€” prefix for table names, default 'planet_osm'. the classic default, iconic behavior πŸ’…
  • --middle-schema SCHEMA β€” name of the schema to store the table for the replication state in. keep your replication state table cute and contained ✨
  • --schema SCHEMA β€” name of the schema for the database. schema on schema on schema 🧚

🌈 SEE ALSO 🌈


and that's the whole page, bestie. source/version tea: this is osm2pgsql 2.3.1, page OSM2PGSQL-REPLICATION(1). the footer didn't hand me a separate date, so just know the timestamp is literally right now in your heart. go run it and slay πŸ«ΆπŸ’–βœ¨

πŸ“¦ where this page comes from