exmdb_provider(4gx)

💖 Name: exmdb_provider 💖

okay bestie, this is exmdb_provider — the Gromox Information Store, and it is literally giving mailbox-engine main character energy. 💅

✨ Description ✨

exmdb_provider is the mailbox engine. It offers a whole plethora — literally 124 — of stateless low-level functions for operating on mailbox stores. The engine’s functionality is exposed by way of a Gromox-specific network protocol on port 5000. 👑

exmdb_provider is built as a shared library and can run in either the http(8gx) or istore(8gx) processes, depending on the gromox.cfg:istore_standalone config directive.

The shared library contributes an exmdb_client API, and that API will transparently pick either a local or a remote procedure call depending on whether the mailbox is served by the same process or not. no cap. 🫶

🌈 Configuration directives (gromox.cfg) 🌈

These are the directives that get recognized when they show up in /etc/gromox/gromox.cfg — periodt. ✨

• exmdb_force_write_txn — (Developer option.) Perform all SQLite transactions as write transactions (i.e. with exclusive locking). Default: no.
• exmdb_deep_backtrace — (Developer option.) Record and report transaction problems with a multi-level backtrace instead of a single-level location indicator. Default: no.
• exmdb_ics_log_file — Log ICS/synchronization requests (and their results) to this file. Default: (empty).
• exmdb_optimize_stm — A debug knob to turn on/off a specific subset of the SQLite prepared statements with extended lifetimes used in the vicinity of property retrieval in conjunction with Content Table querying (e.g. exmdb_server::query_table, or MAPI ropQueryRows), Content Table matching (exmdb_server::match_table, or MAPI ropFindRow), and non-instanced full message retrieval (exmdb_server::read_message, often used by our exporters). Default: on.
• outgoing_smtp_url — See gromox.cfg(5):outgoing_smtp_url. iykyk. 💫

🦋 Configuration directives (exmdb_provider.cfg) 🦋

The usual config file location is /etc/gromox/exmdb_provider.cfg — that’s where this batch of knobs lives. 💖

• cache_interval — The inactivity timer after which a mailbox’s sqlite files are closed. Default: 15min. Lower limit: 1s.
• dbg_synthesize_content — When this directive is set to 1, missing content files will not be regarded as an error, and the respective attachment or property is delivered with a replacement string. If set to 2, a replacement string is always delivered; mode 2 is useful for reducing the amount of data downloaded when debugging ICS. Default: 0.
• enable_dam — When set to on, inbox rule processing is allowed to create Deferred Action Messages (DAM). Furthermore, the "Deferred Actions" folder will have its contents shown. Conversely, if this directive is off, no DAMs will be created, and the DAM folder in inboxes is presented as empty to clients (even if it has content from earlier). Outlook’s DAM handling is poor, and if you experience a crash with a primary mailbox that is in non-cached/online mode a few seconds after Outlook has opened it, turn this option off for mitigation. Default: on.
• exmdb_body_autosynthesis — When a client requests either PR_BODY, PR_HTML or PR_RTF_COMPRESSED, but that property does not exist on a particular message, automatically synthesize the data on-the-fly from another of the available formats. Default: on.
• exmdb_file_compression — Compress content files (bodytexts and attachments). Possible values: no, yes (zstd-6), zstd-level (level=1..19). Default: zstd-6.
• exmdb_hosts_allow — A space-separated list of individual host addresses that are allowed to converse with the exmdb service. The addresses must conform to gromox(7) § "Host addresses". No networks and no CIDR notations are permitted. This option deprecates the /etc/gromox/exmdb_acl.txt file used before Gromox 2.8. Default: ::1.
• exmdb_listen_port — The TCP port number for exposing the timer service on. Default: 5000.
• exmdb_pf_read_per_user — Keep public folder read states per user (1) or keep one state for all users (0). Default: 1.
• exmdb_pf_read_states — When set to 0, messages in public stores/folders will always be shown as read and the folder summary will reflect that. When set to 1, messages will have new/read markings but PR_CONTENT_UNREAD will indicate 0 new messages at all times. When set to 2, PR_CONTENT_UNREAD indicates the number of new messages for the particular user. (Outlook does not show this number; in Folder Properties, the radiobox is even greyed out.) Default: 2.
• exmdb_private_folder_softdelete — Enables soft-delete support for folders in private stores. (This feature is experimental.) Public folders always have this on. Default: yes.
• exmdb_schema_upgrades — This directive controls whether database schemas are automatically upgraded when a mailbox is loaded. During this time, the mailbox is unavailable and operations on it will be delayed. Connection aborts, if any, would be due to timeouts in clients rather than servers. (The procedure takes roughly 36sec per gigabyte of exchange.sqlite3 worth of data, or 36sec per about 110k messages, on a 3700X CPU, single-thread. The file can also temporarily grow to double its size, so ample disk space may be required.) Default: yes.
• exmdb_eph_prefix — A path for where variadic data files that are process-temporary can be stored. This may be used to keep the tables.sqlite3 file off an NFS-backed homedir. Required disk space scales linearly with open table handles and linearly with messages in the opened folders, at about 80 bytes per messages. (In other words, don’t lump molasses of messages into a single folder in a shared mailbox read by multiple people.) When you create the directory, set its ownership to gromox:gromox and mode to 0770. Default: (empty).
• exmdb_search_pacing — When initially populating a search folder (static or dynamic), yield the lock on the sqlite database (file descriptor) after so many messages to give other clients a chance to perform an action. Default: 250.
• exmdb_search_pacing_time — When initially populating a search folder (static or dynamic), yield the lock on the sqlite database (file descriptor) after this much time has passed to give other clients a chance to perform an action. Default: 2s.
• exmdb_search_nice — Run the search folder population thread with adjusted niceness, which affects process scheduling. This is not an absolute priority as the nice(1) command would use, but a relative one, as per the nice(2) syscall. The allowed range in Gromox is 0 .. 19; negative values are not supported (and not meaningful, because Gromox will usually be running in an unprivileged setting where it is not possible to raise the priority). Default: 0.
• exmdb_search_yield — Make the search folder population thread not only give up the lock on the sqlite database temporarily, but also invoke pthread_yield(3) after every work block (cf. exmdb_search_pacing). Default: no.
• exrpc_debug — Log every incoming exmdb network RPC and the return code of the operation in a minimal fashion to stderr. Level 1 emits RPCs with a failure return code, level 2 emits all RPCs. Note that direct function calls from within the process image are not logged this way, so this will not show exmdb_provider invocations from exchange_emsmdb(4gx). Note the daemon log level needs to be "debug" (6), too. Default: 0.
• listen_ip — The IPv6 socket address for exposing the exmdb service on. The address must conform to gromox(7) § "Host addresses". Default: ::1.
• max_ext_rule_number — Default: 20.
• max_router_connections — As an exmdb server, permit at most this many inbound connections for the purpose of sending notifications on these channels. Note that every incoming TCP connection starts as a data connection and only becomes re-classified as "notification" once the LISTEN_NOTIFICATION RPC has been issued by the client. Default: unlimited (only limited by ulimits).
• max_rpc_stub_threads — As an exmdb server, permit at most this many inbound connections for commands. Default: unlimited (only limited by ulimits).
• max_rule_number — Default: 1000.
• max_store_message_count — The maximum number of messages any one particular message store is allowed to keep. The technical limit is somewhere around 2^47. Default: 0 (no limit).
• populating_threads_num — The number of threads to spawn that will work on asynchronous search folder population. Default: 4.
• rpc_proxy_connection_num — For every remote exmdb server that needs to be contacted because of rule processing an OP_FOWARD (or similar) rule, keep this many number of outbound connections for sending commands. Default: 10.
• sqlite_debug — If set to 1, every query given to SQLite prepare/execute is logged. If set to 0, only failed queries are logged. (It cannot be made completely silent, since our queries ought to never fail.) Default: 0.
• table_size — Maximum number of concurrently active mailboxes. Default: 5000.
• x500_org_name — Default: (unspecified).

🌈 Sharding / Multiple Home Server Cluster 🌈

Okay so picture this: every exmdb_provider (server) instance is doing a vibe check on SQL columns. For private stores it checks users.homeserver / servers.hostname; for public stores it checks domains.homeserver / servers.hostname. It compares that against the host_id directive to figure out whether it should be serving that thing. And the exmdb_client component — the one used by zcore, imapd, etc., plus a whole plethora of command-line utilities — does the same SQL-column evaluating to decide which server to contact. iykyk.

The magic value 0 in the homeserver column is like the “everyone is invited” of server authority: any and all server instances consider themselves authoritative. This has serious implications (concurrent writes by multiple hosts), so for accounts with an actual maildir, avoid homeserver=0 in multiserver setups. Using 0 for Contact Objects is tolerable, and using 0 for single-server setups is tolerable too. (Keep that distinction: “tolerable”, not “a slay”.)

For a decidedly single-server cluster, this circumstance requires attention too. Example: imagine the server.hostname SQL column contains "server01.example.com", which resolves to "2001:db8::1" — this conflicts with the ::1 bind address, making connection attempts fail. Yeah. That’s the tea.

Also: the file /etc/gromox/exmdb_list.txt became obsolete in the development phase that followed the Gromox 3.4 release. So don’t come for it, it’s retired.

🌐 Network protocol 🌐

The transmissions on the socket are simple concatenations of protocol data units built using the NDR format. The PDU length is present within the PDU itself near the start.

{
     leuint32_t length;
     char pdu[];
}

pdu := {
     uint8_t call_id;
     string directory;
     switch (call_id) {
          ...
     }
}

📁 Files 📁

• data_file_path/mail_bounce/

config_file_path and data_file_path are determined by the configuration of the program that loaded the exmdb_provider component.

🚦 Signals 🚦

Upon receipt of SIGHUP, configuration files are re-read — but only a few select directives can be changed this way, because many parts do not implement reload. So don’t assume a full glow-up reload for everything.

👀 See also 👀

• gromox(7), http(8gx)

---

And that’s the page, bestie. Footer vibe: this is from Gromox, page exmdb_provider(4gx), and we’re closing it out with a little sparkle. 💖✨