CARBON-C-RELAY(1)

📦 version 3.7.3, in debian bookworm, debian trixie, epel 9, ubuntu jammy, ubuntu noble, ubuntu resolute, ubuntu stonking

💖 CARBON-C-RELAY(1) 💖

Graphite data collection and visualisation — that's the section, that's the vibe, and carbon-c-relay is the main character of it. Buckle up bestie, this one's a long read and we're doing it properly. 🎀

✨ NAME ✨

carbon-c-relay — graphite relay, aggregator and rewriter 🦋

🌈 SYNOPSIS 🌈

carbon-c-relay -f config-file [ options ... ]

💅 DESCRIPTION 💅

okay so. carbon-c-relay accepts, cleanses, matches, rewrites, forwards and aggregates graphite metrics — the whole glow-up pipeline. It does this by listening for incoming connections and relaying the messages on to other servers that you define in its configuration. The core functionality is literally just routing messages via flexible rules to the destinations you want. iconic behaviour, no notes. 💖

carbon-c-relay is a simple program that reads its routing information from a file. The command line arguments let you set the location of that file, plus the amount of dispatchers (worker threads) to use for reading the data off incoming connections and passing it onto the right destination(s). The route file supports two main constructs: clusters and matches. The first ones define groups of hosts that data metrics can be sent to; the latter define which metrics should be sent to which cluster. Aggregation rules are seen as matches. Rewrites are actions that directly affect the metric at the point in which they appear in the configuration. periodt. ✨

For every metric received by the relay, cleansing is performed. The following changes happen before any match, aggregate or rewrite rule even gets to look at the metric:

  • double dot elimination (necessary for correctly functioning consistent hash routing)
  • trailing/leading dot elimination
  • whitespace normalisation (this mostly affects output of the relay to other targets: metric, value and timestamp will be separated by a single space only, ever)
  • irregular char replacement with underscores (_), currently irregular is defined as not being in [0-9a-zA-Z-_:#], but can be overridden on the command line. Note that tags (when present and allowed) are not processed this way.
not me doing dot-elimination before anyone else gets a say 😤 it's called hygiene, look it up

🔥 OPTIONS 🔥

These options control the behaviour of carbon-c-relay. One bullet per flag, no skipping, we're thorough in this house. 💪

  • -v — Print version string and exit. bye bestie 👋
  • -d — Enable debug mode. This prints statistics to stdout and prints extra messages about some situations encountered by the relay that normally would be too verbose to be enabled. When combined with -t (test mode) this also prints stub routes and consistent-hash ring contents. 🕵️♀️
  • -s — Enable submission mode. In this mode, internal statistics are not generated. Instead, queue pressure and metrics drops are reported on stdout. This mode is useful when used as a submission relay whose job is just to forward to (a set of) main relays. Statistics about the submission relays in this case are not needed, and could easily cause a non-desired flood of metrics e.g. when used on each and every host locally. 🫶
  • -t — Test mode. This mode doesn't do any routing at all, but instead reads input from stdin and prints what actions would be taken given the loaded configuration. This mode is very useful for testing relay routes for regular expression syntax etc. It also allows you to get insight on how routing is applied in complex configurations, for it shows rewrites and aggregates taking place as well. When -t is repeated, the relay will only test the configuration for validity and exit immediately afterwards. Any standard output is suppressed in this mode, making it ideal for start-scripts to test a (new) configuration. 🧪
  • -f config-file — Read configuration from config-file. A configuration consists of clusters and routes. See CONFIGURATION SYNTAX for more information on the options and syntax of this file.
  • -l log-file — Use log-file for writing messages. Without this option, the relay writes both to stdout and stderr. When logging to file, all messages are prefixed with MSG when they were sent to stdout, and ERR when they were sent to stderr. 📝
  • -p port — Listen for connections on port port. The port number is used for both TCP, UDP and UNIX sockets. In the latter case, the socket file contains the port number. The port defaults to 2003, which is also used by the original carbon-cache.py. Note that this only applies to the defaults — when listen directives are in the config, this setting is ignored. 🚪
  • -w workers — Use workers number of threads. The default number of workers is equal to the amount of detected CPU cores. It makes sense to reduce this number on many-core machines, or when the traffic is low.
  • -b batchsize — Set the amount of metrics that sent to remote servers at once to batchsize. When the relay sends metrics to servers, it will retrieve batchsize metrics from the pending queue of metrics waiting for that server and send those one by one. The size of the batch will have minimal impact on sending performance, but it controls the amount of lock-contention on the queue. The default is 2500. 📦
  • -q queuesize — Each server from the configuration where the relay will send metrics to, has a queue associated with it. This queue allows for disruptions and bursts to be handled. The size of this queue will be set to queuesize, which allows for that amount of metrics to be stored in the queue before it overflows, and the relay starts dropping metrics. The larger the queue, more metrics can be absorbed, but also more memory will be used by the relay. The default queue size is 25000. 🗂️
  • -L stalls — Sets the max amount of stalls to stalls before the relay starts dropping metrics for a server. When a queue fills up, the relay uses a mechanism called stalling to signal the client (writing to the relay) of this event. In particular when the client sends a large amount of metrics in very short time (burst), stalling can help to avoid dropping metrics, since the client just needs to slow down for a bit, which in many cases is possible (e.g. when catting a file with nc(1)). However, this behaviour can also obstruct, artificially stalling writers which cannot stop that easily. For this the stalls can be set from 0 to 15, where each stall can take around 1 second on the client. The default value is set to 4, which is aimed at the occasional disruption scenario and max effort to not lose metrics with moderate slowing down of clients. ✋
  • -C CAcertpath — Read CA certs (for use with TLS/SSL connections) from given path or file. When not given, the default locations are used. Strict verfication of the peer is performed, so when using self-signed certificates, be sure to include the CA cert in the default location, or provide the path to the cert using this option. 🔐
  • -T timeout — Specifies the IO timeout in milliseconds used for server connections. The default is 600 milliseconds, but may need increasing when WAN links are used for target servers. A relatively low value for connection timeout allows the relay to quickly establish a server is unreachable, and as such failover strategies to kick in before the queue runs high. ⏱️
  • -c chars — Defines the characters that are next to [A-Za-z0-9] allowed in metrics to chars. Any character not in this list, is replaced by the relay with _ (underscore). The default list of allowed characters is -_:#. 🔤
  • -m length — Limits the metric names to be of at most length bytes long. Any lines containing metric names larger than this will be discarded.
  • -M length — Limits the input to lines of at most length bytes. Any excess lines will be discarded. Note that -m needs to be smaller than this value. ⚠️
  • -H hostname — Override hostname determined by a call to gethostname(3) with hostname. The hostname is used mainly in the statistics metrics carbon.relays.<hostname>.<...> sent by the relay. 🏠
  • -B backlog — Sets TCP connection listen backlog to backlog connections. The default value is 32 but on servers which receive many concurrent connections, this setting likely needs to be increased to avoid connection refused errors on the clients.
  • -U bufsize — Sets the socket send/receive buffer sizes in bytes, for both TCP and UDP scenarios. When unset, the OS default is used. The maximum is also determined by the OS. The sizes are set using setsockopt with the flags SO_RCVBUF and SO_SNDBUF. Setting this size may be necessary for large volume scenarios, for which also -B might apply. Checking the Recv-Q and the receive errors values from netstat gives a good hint about buffer usage. 📊
  • -E — Disable disconnecting idle incoming connections. By default the relay disconnects idle client connections after 10 minutes. It does this to prevent resources clogging up when a faulty or malicious client keeps on opening connections without closing them. It typically prevents running out of file descriptors. For some scenarios, however, it is not desirable for idle connections to be disconnected, hence passing this flag will disable this behaviour. 🙅♀️
  • -D — Deamonise into the background after startup. This option requires -l and -P flags to be set as well. 👻
  • -P pidfile — Write the pid of the relay process to a file called pidfile. This is in particular useful when daemonised in combination with init managers.
  • -O threshold — The minimum number of rules to find before trying to optimise the ruleset. The default is 50, to disable the optimiser, use -1, to always run the optimiser use 0. The optimiser tries to group rules to avoid spending excessive time on matching expressions. 🧠

🦋 CONFIGURATION SYNTAX 🦋

The config file supports the following syntax, where comments start with a # character and can appear at any position on a line and suppress input until the end of that line:

    cluster <name>
        < <forward | any_of | failover> [useall] |
          <carbon_ch | fnv1a_ch | jump_fnv1a_ch> [replication <count>] [dynamic] >
            <host[:port][=instance] [proto <udp | tcp>]
                                    [type linemode]
                                    [transport <plain | gzip | lz4 | snappy>
                                               [ssl]]> ...
        ;

    cluster <name>
        file [ip]
            </path/to/file> ...
        ;

    match
            <* | expression ...>
        [validate <expression> else <log | drop>]
        send to <cluster ... | blackhole>
        [stop]
        ;

    rewrite <expression>
        into <replacement>
        ;
comments start with # and go all the way to the end of the line — cute, simple, no drama 💅
aggregate
    <expression> ...
    every <interval> seconds
    expire after <expiration> seconds
    [timestamp at <start | middle | end> of bucket]
    compute <sum | count | max | min | average |
             median | percentile<%> | variance | stddev> write to
        <metric>
    [compute ...]
    [send to <cluster ...>]
    [stop]
    ;

send statistics to <cluster ...>
    [stop]
    ;
statistics
    [submit every <interval> seconds]
    [reset counters after interval]
    [prefix with <prefix>]
    [send to <cluster ...>]
    [stop]
    ;

listen
    type linemode [transport <plain | gzip | lz4 | snappy> [ssl <pemcert>]]
        <<interface[:port] | port> proto <udp | tcp>> ...
        </ptah/to/file proto unix> ...
    ;

include </path/to/file/or/glob>
    ;

(and yes bestie, </ptah/to/file proto unix> is spelled exactly like that in the original, I am not touching it 😭 verbatim is verbatim)

👑 CLUSTERS 👑

okay so you can define as many clusters as you want, and ngl they don't even have to be referenced by a match rule — they can just exist, living their little lives 💅. Every cluster points at one or more hosts... except the file cluster, which writes to files right there on the local filesystem instead.

host can be an IPv4 address, an IPv6 address, or a hostname. And here's the tea: because host is followed by an optional : and port, IPv6 addresses would get read all kinds of wrong unless you either give a port or wrap the IPv6 address in brackets, like [::1]. That's it, that's the rule 🫶.

You can also slap on optional transport and proto clauses to wrap the connection in compression or encryption later, or to specify that UDP or TCP gets used to reach the remote server. If you just... don't, the connection defaults to an unwrapped TCP connection. And type? only linemode at the moment, that's the whole menu.

DNS hostnames get resolved down to a single address, following the preference rules in RFC 3484 https://www.ietf.org/rfc/rfc3484.txt. The any_of, failover and forward clusters have an explicit useall flag that turns on expansion for hostnames that resolve to multiple addresses — flip it on and each address returned becomes a cluster destination ✨.

So there are two whole groups of cluster types: simple forwarding clusters and consistent hashing clusters. Let's meet them 🦋

  • forward and file clusters — these two just send everything they receive straight to the defined members (host addresses or files). When a cluster has multiple members, all incoming metrics are sent to all members, basically duplicating the input metric stream over every single member. no thoughts, just copy-paste energy 💀
  • any_of cluster — smol variant of forward, but instead of blasting input metrics at every member, it sends each incoming metric to only one of the defined members. The whole point is a load-balanced scenario where any member can receive any metric. As any_of suggests, when any member goes unreachable, the remaining available members immediately pick up the full input stream of metrics. Specifically: with 4 members, each gets approximately 25% of the input metrics. When one member becomes unavailable (network interruption, a service restart, whatever), the remaining 3 members each get about 33% of the input. So when you're sizing cluster capacity, keep in mind that in the most extreme case, the final remaining member receives all the input traffic. plan accordingly bestie 😭

An any_of cluster is especially useful when it points at other relays or caches. Used with other relays, it effectively load-balances and adapts immediately when targets go unavailable. Used with caches, the any_of router sending the same metrics consistently to the same destination helps caches get a high hitrate on their internal buffers for the same metrics (if they use them), but still allows for a rolling restart of the caches, e.g. when they're on the same machine. iconic behaviour honestly ✨

  • failover cluster — like any_of, but it sticks to the order servers are defined in. This is for a pure failover scenario between servers. All metrics go to at most 1 member, so no hashing, no balancing, none of it. A failover cluster with two members only sends metrics to the second member if the first one becomes unavailable. simple, loyal, dependable 🫶
  • carbon_ch cluster — sends metrics to the member responsible according to the consistent hash algorithm, the same one used in the original carbon python relay — or to multiple members if replication is set higher than 1. When dynamic is set, a failure of any server does not mean metrics get dropped for that server; instead the undeliverable metrics get sent to any other server in the cluster so nothing gets lost. Most useful when replication is 1.

The hashring calculation — the thing that decides how metrics get distributed — is based on the server host (or IP address) and the optional instance of the member. Which means: using carbon_ch, two targets on different ports but the same host will map to the same hashkey, so no distribution of metrics happens at all 😭. The instance is what fixes that. An instance gets appended to the member after the port, separated by an equals sign, e.g. 127.0.0.1:2006=a for instance a.

Consistent hashes are consistent in the sense that removing a member from the cluster should not cause a complete re-mapping of all metrics to members — instead only the metrics from the removed member get added to all remaining members, each getting approximately its fair share. And the other way around: when a member is added, each member should see a subset of its metrics now being addressed to the new member. This is a major advantage over a normal hash, where every removal or addition of members (including via a change in their IP address or hostname) would cause a full re-mapping of all metrics over all available metrics. the glow-up is real 💅

  • fnv1a_ch cluster — behaves identically to carbon_ch, but uses a different hash technique (FNV1a) which is faster, and more importantly is defined to get around the beforementioned limitation of carbon_ch of using both host and port from the members. Super useful when multiple targets live on the same host, separated only by port 🎀

Since the instance property is no longer necessary with fnv1a_ch, this cluster type uses it to completely override the string the hashkey should be calculated off. That opens up a lot of doors — masquerading old IP addresses, for one — but basically it lets you make the hash key location agnostic of the (physical) location of that key. For example, using something like 10.0.0.1:2003=4d79d13554fa1301476c1f9fe968b0ac would let you change the port and/or IP address of the server that receives data for that instance key. Obviously, migration of data becomes so much more convenient this way. But heads up: since the instance name is used as the full hash input, instances like a, b, etc. will likely give you poor hash distribution, since their hashes have very little input. Consider using longer and mostly differing instance names — random hashes, even — for better hash distribution behaviour. trust 🫶

  • jump_fnv1a_ch cluster — also a consistent hash cluster like the previous two, but it does not take the member host, port or instance into account at all. Whether that's useful depends entirely on your scenario, bestie. The jump hash has almost perfect balancing over the members defined in the cluster, at the expense of not being able to remove any member but the last in the order defined in the cluster. What that means: this hash is fine with ever-growing clusters where older nodes are never removed 🌱

If you have a cluster where removal of old nodes happens often, the jump hash is not suitable for you. Jump hash works with servers in an ordered list, without gaps. To influence the ordering, the instance given to the server is used as the sorting key. Without it, the order is just as given in the file. And it is good practice to fix the order of the servers with instances so it's explicit what the right nodes for the jump hash are. lock it in 🔒

🎀 MATCHES 🎀

okay so match rules are how you direct incoming metrics to one or more clusters. They get processed top to bottom, exactly as defined in the file. You can define multiple matches inside the same rule, and each match rule can send data to one or more clusters ✨

Here's the part that trips people up: match rules "fall through" unless you add the stop keyword. So a carefully crafted match expression can target multiple clusters or aggregations — which means you can replicate metrics, and also send certain metrics to alternative clusters if you're careful with ordering and use of the stop keyword. chef's kiss 💅

The special cluster blackhole discards any metrics sent to it. Handy for weeding out unwanted metrics in certain cases. And because throwing metrics away is pointless if other matches would accept the same data anyway, a match whose destination is the blackhole cluster has an implicit stop.

The validation clause adds a check to the data (what comes after the metric) in the form of a regular expression. When that expression matches, the match rule executes as if no validation clause was present. But if it fails, the match rule is aborted and no metrics get sent to destinations — that's the drop behaviour 🫶

When log is used, the metric is logged to stderr. Care should be taken with the latter to avoid log flooding. no cap, that one will drown you 😭

When a validate clause is present, destinations need not be present — which allows for applying a global validation rule 🌈

Note that the cleansing rules are applied before validation is done, so the data will not have duplicate spaces. ✨

The route using clause performs a temporary modification to the key used as input to the consistent hashing routines. The primary purpose is to route traffic so that appropriate data is sent to the needed aggregation instances. 💖

💖 REWRITES 💖

Okay so rewrite rules are the glow-up station for your metric names bestie. They take a regular expression as input to match incoming metrics, and transform them into the desired new metric name. In the replacement, backreferences are allowed to match capture groups defined in the input regular expression. A match of server\.(x|y|z)\. lets you use e.g. role.\1. in the substitution. Obsessed? Yes. But there are a few caveats with the current implementation, and you need to know them.

First caveat, and it's giving "location matters": where the rule sits in the config file determines when the rewrite is performed. The rewrite is done in-place, so a match rule before the rewrite would match the original name, and a match rule after the rewrite no longer matches the original name. Care should be taken with the ordering, as multiple rewrite rules in succession can take place — e.g. a gets replaced by b and b gets replaced by c in a succeeding rewrite rule. Chain reaction, no cap. 🦋

Second caveat, and this one's a lil spooky: the rewritten metric names are not cleansed, like newly incoming metrics are. So double dots and potential dangerous characters can appear if the replacement string is crafted to produce them. It is the responsibility of the writer to make sure the metrics are clean. If this is an issue for routing, one can consider having a rewrite-only instance that forwards all metrics to another instance that will do the routing. Obviously the second instance will cleanse the metrics as they come in. Problem solved, main character energy. 👑

The backreference notation also allows you to lowercase and uppercase the replacement string with the use of the underscore (_) and carret (^) symbols following directly after the backslash. For example, role.\_1. as substitution will lowercase the contents of \1. The dot (.) can be used in a similar fashion, or followed after the underscore or caret to replace dots with underscores in the substitution. This can be handy for some situations where metrics are sent to graphite. ✨

🌸 AGGREGATIONS 🌸

The aggregations defined take one or more input metrics expressed by one or more regular expresions, similar to the match rules. Incoming metrics are aggregated over a period of time defined by the interval in seconds. Since events may arrive a bit later in time, the expiration time in seconds defines when the aggregations should be considered final, as no new entries are allowed to be added any more.

On top of an aggregation, multiple aggregations can be computed. They can be of the same or different aggregation types, but should write to a unique new metric. The metric names can include back references like in rewrite expressions, allowing for powerful single aggregation rules that yield in many aggregations. It's giving efficiency, fr fr. 💅

When no send to clause is given, produced metrics are sent to the relay as if they were submitted from the outside, hence match and aggregation rules apply to those.

⚠️ Care should be taken that loops are avoided this way. For this reason, the use of the send to clause is encouraged, to direct the output traffic where possible.

Like for match rules, it is possible to define multiple cluster targets. Also, like match rules, the stop keyword applies to control the flow of metrics in the matching process. 🎀

⭐ STATISTICS ⭐

The send statistics to construct is deprecated and will be removed in the next release. Use the special statistics construct instead. (Yes bestie, we're moving on, it's time.)

The statistics construct can control a couple of things about the (internal) statistics produced by the relay. The send to target can be used to avoid router loops by sending the statistics to a certain destination cluster(s). By default the metrics are prefixed with carbon.relays.<hostname>, where hostname is determinted on startup and can be overridden using the -H argument. This prefix can be set using the prefix with clause similar to a rewrite rule target. The input match in this case is the pre-set regular expression ^(([^.]+)(\..*)?)$ on the hostname. As such, one can see that the default prefix is set by carbon.relays.\.1.

Note that this uses the replace-dot-with-underscore replacement feature from rewrite rules. Given the input expression, the following match groups are available: \1 the entire hostname, \2 the short hostname and \3 the domainname (with leading dot). It may make sense to replace the default by something like carbon.relays.\_2 for certain scenarios, to always use the lowercased short hostname, which following the expression doesn´t contain a dot. 🐛

By default, the metrics are submitted every 60 seconds, this can be changed using the submit every <interval> seconds clause.

To obtain a more compatible set of values to carbon-cache.py, use the reset counters after interval clause to make values non-cumulative, that is, they will report the change compared to the previous value. Iconic behaviour tbh. 🌈

🦋 LISTENERS 🦋

The ports and protocols the relay should listen for incoming connections can be specified using the listen directive. Currently, all listeners need to be of linemode type. An optional compression or encryption wrapping can be specified for the port and optional interface given by ip address, or unix socket by file.

When interface is not specified, the any interface on all available ip protocols is assumed. If no listen directive is present, the relay will use the default listeners for port 2003 on tcp and udp, plus the unix socket /tmp/.s.carbon-c-relay.2003. This typically expands to 5 listeners on an IPv6 enabled system. The default matches the behaviour of versions prior to v3.2. 👑

💜 INCLUDES 💜

In case configuration becomes very long, or is managed better in separate files, the include directive can be used to read another file. The given file will be read in place and added to the router configuration at the time of inclusion. The end result is one big route configuration. Multiple include statements can be used throughout the configuration file. The positioning will influence the order of rules as normal.

🚨 Beware that recursive inclusion (include from an included file) is supported, and currently no safeguards exist for an inclusion loop. Not me warning you about infinite loops in a config file, but here we are. 💀

For what is worth, this feature likely is best used with simple configuration files (e.g. not having include in them).

✨ EXAMPLES ✨

carbon-c-relay evolved over time, growing features on demand as the tool proved to be stable and fitting the job well. Below follow some annotated examples of constructs that can be used with the relay. 🎀

Clusters can be defined as much as necessary. They receive data from match rules, and their type defines which members of the cluster finally get the metric data. The simplest cluster form is a forward cluster:

cluster send-through
    forward
        10.1.0.1
    ;

Any metric sent to the send-through cluster would simply be forwarded to the server at IPv4 address 10.1.0.1. If we define multiple servers, all of those servers would get the same metric, thus:

cluster send-through
    forward
        10.1.0.1
        10.2.0.1
    ;

The above results in a duplication of metrics send to both machines. This can be useful, but most of the time it is not. The any_of cluster type is like forward, but it sends each incoming metric to any of the members. The same example with such cluster would be:

cluster send-to-any-one
    any_of 10.1.0.1:2010 10.1.0.1:2011;

This would implement a multipath scenario, where two servers are used, the load between them is spread, but should any of them fail, all metrics are sent to the remaining one. This typically works well for upstream relays, or for balancing carbon-cache processes running on the same machine. Should any member become unavailable, for instance due to a rolling restart, the other members receive the traffic. 💅

If it is necessary to have true fail-over, where the secondary server is only used if the first is down, the following would implement that:

cluster try-first-then-second
    failover 10.1.0.1:2010 10.1.0.1:2011;

These types are different from the two consistent hash cluster types:

cluster graphite
    carbon_ch
        127.0.0.1:2006=a
        127.0.0.1:2007=b
        127.0.0.1:2008=c
    ;

If a member in this example fails, all metrics that would go to that member are kept in the queue, waiting for the member to return. This is useful for clusters of carbon-cache machines where it is desirable that the same metric ends up on the same server always. The carbon_ch cluster type is compatible with carbon-relay consistent hash, and can be used for existing clusters populated by carbon-relay. For new clusters, however, it is better to use the fnv1a_ch cluster type, for it is faster, and allows to balance over the same address but different ports without an instance number, in constrast to carbon_ch. 🌟

Because we can use multiple clusters, we can also replicate without the use of the forward cluster type, in a more intelligent way:

cluster dc-old
    carbon_ch replication 2
        10.1.0.1
        10.1.0.2
        10.1.0.3
    ;
cluster dc-new1
    fnv1a_ch replication 2
        10.2.0.1
        10.2.0.2
        10.2.0.3
    ;
cluster dc-new2
    fnv1a_ch replication 2
        10.3.0.1
        10.3.0.2
        10.3.0.3
    ;

match *
    send to dc-old
    ;
match *
    send to
        dc-new1
        dc-new2
    stop
    ;

In this example all incoming metrics are first sent to dc-old, then dc-new1 and finally to dc-new2. Note that the cluster type of dc-old is different. Each incoming metric will be send to 2 members of all three clusters, thus replicating to in total 6 destinations. For each cluster the destination members are computed independently. Failure of clusters or members does not affect the others, since all have individual queues. 🦋

The above example could also be written using three match rules for each dc, or one match rule for all three dcs. The difference is mainly in performance, the number of times the incoming metric has to be matched against an expression. The stop rule in dc-new match rule is not strictly necessary in this example, because there are no more following match rules. However, if the match would target a specific subset, e.g. ^sys\., and more clusters would be defined, this could be necessary, as for instance in the following abbreviated example:

cluster dc1-sys ... ;
cluster dc2-sys ... ;

cluster dc1-misc ... ;
cluster dc2-misc ... ;

match ^sys\. send to dc1-sys;
match ^sys\. send to dc2-sys stop;

match * send to dc1-misc;
match * send to dc2-misc stop;

Periodt. 💖

As you can see, without that stop in the dc2-sys match rule, literally every metric starting with sys. would also get yeeted over to dc1-misc and dc2-misc too. Which… might be exactly what you want, no judgement 💅 — but in this example there's a dedicated cluster just for the sys metrics, so yeah.

Now imagine some rogue metric shows up that nobody asked for. Say, some bad/old software is out there generating chaos 🙈. You do not want to store that. And when whitelisting every single wanted metric is giving way too much work, the blackhole cluster is literally made for this. Picture it:

match
        some_legacy1$
        some_legacy2$
    send to blackhole
    stop;

This throws away every metric that ends with some_legacy, which would otherwise be a whole nightmare to filter out by hand. 🗑️✨

And since order matters (say it with me), you can build constructs like this:

cluster old ... ;
cluster new ... ;

match * send to old;

match unwanted send to blackhole stop;

match * send to new;

Here, the old cluster still receives the metric that's unwanted for the new cluster. So yeah — the order the rules appear in genuinely changes how they execute. iykyk.

🌸 Validation, bestie 🌸

Validation exists so you can make sure the data for your metrics is actually what you expect. A global validation that only allows plain number (no floating point, nope) values could look like:

match *
    validate ^[0-9]+\ [0-9]+$ else drop
    ;
Note the escape with backslash \ of the space — you might be able to use \s or [:space:] instead, but honestly that depends on whatever regex implementation you've got configured. 🤷

The validation clause can live on every match rule, so in principle this is valid:

match ^foo
    validate ^[0-9]+\ [0-9]+$ else drop
    send to integer-cluster
    ;
match ^foo
    validate ^[0-9.e+-]+\ [0-9.e+-]+$ else drop
    send to float-cluster
    stop;

Now pay attention because the behaviour is different between those last two examples, and it's giving sneaky:

  • When no send to clusters are specified, a validation error makes the match behave like the stop keyword is present. And likewise, when validation passes, processing just continues on to the next rule.
  • When destination clusters are present, the match respects the stop keyword as normal — when it's specified, processing will always stop right there.
  • But if validation fails, the rule sends absolutely nothing to the destination clusters. The metric gets dropped or logged, but it is never sent. Periodt. 🚫

🦋 Rewriting metrics 🦋

The relay can rewrite incoming metrics on the fly ✨. It does this with regular expressions that have capture groups, which let you substitute parts into a replacement string. Rewrite rules are how you clean up metrics from applications, or give yourself a migration path. In its simplest form, a rewrite rule is like:

rewrite ^server\.(.+)\.(.+)\.([a-zA-Z]+)([0-9]+)
    into server.\_1.\2.\3.\3\4
    ;

So a metric like server.DC.role.name123 becomes server.dc.role.name.name123. 🎀

For rewrite rules the same tea applies as for matches: their order matters. So building on that old/new cluster example from before, this stores the original metric name in the old cluster and the new metric name in the new cluster:

match * send to old;

rewrite ... ;

match * send to new;
Heads up though: after the rewrite, the original metric name is gone. No cap. The rewrite happens in-place, so it's not coming back. 😭

👑 Aggregations 👑

Aggregations are probably the most complex part of carbon-c-relay. Two ways of specifying aggregates are supported. The first — static rules — get handled by an optimiser that tries to fold thousands of rules into groups so the matching is way more efficient. The second — dynamic rules — are super powerful compact definitions that might have thousands of internal instantiations. ✨

A typical static aggregation looks like this:

aggregate
        ^sys\.dc1\.somehost-[0-9]+\.somecluster\.mysql\.replication_delay
        ^sys\.dc2\.somehost-[0-9]+\.somecluster\.mysql\.replication_delay
    every 10 seconds
    expire after 35 seconds
    timestamp at end of bucket
    compute sum write to
        mysql.somecluster.total_replication_delay
    compute average write to
        mysql.somecluster.average_replication_delay
    compute max write to
        mysql.somecluster.max_replication_delay
    compute count write to
        mysql.somecluster.replication_delay_metric_count
    ;

In this example, four aggregations get produced from the incoming matching metrics. And yeah, we could have written those two matches as one — but for demonstration purposes we didn't. Obviously they can refer to different metrics, if that makes sense for you.

  • The every 10 seconds clause says what interval the aggregator can expect new metrics to arrive in. That interval is what's used to produce the aggregations, so every 10 seconds, 4 new metrics get generated from the data received so far.
  • Because data might be in transit for some reason, or generation stalled, the expire after clause says how long the data should be kept before a data bucket (which is being aggregated) counts as complete. In the example, 35 was used, which means after 35 seconds the first aggregates are produced. It also means metrics can show up 35 seconds late and still get taken into account.
  • The exact time at which the aggregate metrics get produced is random between 0 and interval (which is 10 here) seconds after the expiry time. This is done to prevent thundering herds of metrics for large aggregation sets. 🐎🐎🐎
  • The timestamp used for the aggregations can be specified as the start, middle or end of the bucket. Original carbon-aggregator.py uses start, while carbon-c-relay's default has always been end.
  • The compute clauses show that a single aggregation rule can produce multiple aggregates, which is often the case. Internally this comes for free, since all possible aggregates are always calculated whether or not they're used. 💅
  • The produced new metrics are resubmitted to the relay, so matches defined earlier in the configuration can match output of the aggregator. It is important to avoid loops that can be generated this way. In general, splitting aggregations to their own carbon-c-relay instance, so it's easy to forward the produced metrics to another relay instance, is a good practice. 🫶

The previous example could also be written dynamically, like this:

aggregate
        ^sys\.dc[0-9].(somehost-[0-9]+)\.([^.]+)\.mysql\.replication_delay
    every 10 seconds
    expire after 35 seconds
    compute sum write to
        mysql.host.\1.replication_delay
    compute sum write to
        mysql.host.all.replication_delay
    compute sum write to
        mysql.cluster.\2.replication_delay
    compute sum write to
        mysql.cluster.all.replication_delay
    ;

Here a single match results in four aggregations, each of a different scope. In this example, aggregation based on hostname and cluster are being made, plus the more general all targets, which in this example have identical values. Note that with this one aggregation rule, per-cluster, per-host and total aggregations all get produced. Obviously, the input metrics are what define which hosts and clusters get produced. 🦋

With use of the send to clause, aggregations can be made more intuitive and less error-prone. Consider this:

cluster graphite fnv1a_ch ip1 ip2 ip3;

aggregate ^sys\.somemetric
    every 60 seconds
    expire after 75 seconds
    compute sum write to
        sys.somemetric
    send to graphite
    stop
    ;

match * send to graphite;

It sends all incoming metrics to the graphite cluster — except the sys.somemetric ones, which it replaces with a sum of all the incoming ones. 🧚

Warning: without a stop in the aggregate, this causes a loop. And without the send to, the metric name can't be kept as its original name, because the output now goes directly to the cluster.

📊 STATISTICS 📊

okay bestie, this is the part where the relay spills its own tea about itself 💅

When carbon-c-relay is run without -d or -s arguments, statistics will be produced. By default they are sent to the relay itself in the form of carbon.relays.<hostname>.*. See the statistics construct to override this prefix, sending interval and values produced. While many metrics have a similar name to what carbon-cache.py would produce, their values are likely different. By default, most values are running counters which only increase over time. The use of the nonNegativeDerivative() function from graphite is useful with these. (iykyk 🫶)

The following metrics are produced under the carbon.relays.<hostname> namespace:

  • metricsReceived — the number of metrics that were received by the relay. Received here means that they were seen and processed by any of the dispatchers. 👀
  • metricsSent — the number of metrics that were sent from the relay. This is a total count for all servers combined. When incoming metrics are duplicated by the cluster configuration, this counter will include all those duplications. In other words, the amount of metrics that were successfully sent to other systems. Note that metrics that are processed (received) but still in the sending queue (queued) are not included in this counter. 🚀
  • metricsDiscarded — the number of input lines that were not considered to be a valid metric. Such lines can be empty, only containing whitespace, or hitting the limits given for max input length and/or max metric length (see -m and -M options). 🗑️
  • metricsQueued — the total number of metrics that are currently in the queues for all the server targets. This metric is not cumulative, for it is a sample of the queue size, which can (and should) go up and down. Therefore you should not use the derivative function for this metric. 🎢
  • metricsDropped — the total number of metric that had to be dropped due to server queues overflowing. A queue typically overflows when the server it tries to send its metrics to is not reachable, or too slow in ingesting the amount of metrics queued. This can be network or resource related, and also greatly depends on the rate of metrics being sent to the particular server. 😭
  • metricsBlackholed — the number of metrics that did not match any rule, or matched a rule with blackhole as target. Depending on your configuration, a high value might be an indication of a misconfiguration somewhere. These metrics were received by the relay, but never sent anywhere, thus they disappeared. 🕳️
  • metricStalls — the number of times the relay had to stall a client to indicate that the downstream server cannot handle the stream of metrics. A stall is only performed when the queue is full and the server is actually receptive of metrics, but just too slow at the moment. Stalls typically happen during micro-bursts, where the client typically is unaware that it should stop sending more data, while it is able to. ✋
  • connections — the number of connect requests handled. This is an ever increasing number just counting how many connections were accepted. 📥
  • disconnects — the number of disconnected clients. A disconnect either happens because the client goes away, or due to an idle timeout in the relay. The difference between this metric and connections is the amount of connections actively held by the relay. In normal situations this amount remains within reasonable bounds. Many connections, but few disconnections typically indicate a possible connection leak in the client. The idle connections disconnect in the relay here is to guard against resource drain in such scenarios. 🔌
  • dispatch_wallTime_us — the number of microseconds spent by the dispatchers to do their work. In particular on multi-core systems, this value can be confusing, however, it indicates how long the dispatchers were doing work handling clients. It includes everything they do, from reading data from a socket, cleaning up the input metric, to adding the metric to the appropriate queues. The larger the configuration, and more complex in terms of matches, the more time the dispatchers will spend on the cpu. But also time they do /not/ spend on the cpu is included in this number. It is the pure wallclock time the dispatcher was serving a client. ⏱️
  • dispatch_sleepTime_us — the number of microseconds spent by the dispatchers sleeping waiting for work. When this value gets small (or even zero) the dispatcher has so much work that it doesn´t sleep any more, and likely can´t process the work in a timely fashion any more. This value plus the wallTime from above sort of sums up to the total uptime taken by this dispatcher. Therefore, expressing the wallTime as percentage of this sum gives the busyness percentage draining all the way up to 100% if sleepTime goes to 0. 😴➡️🔥
  • server_wallTime_us — the number of microseconds spent by the servers to send the metrics from their queues. This value includes connection creation, reading from the queue, and sending metrics over the network. 🌐
  • dispatcherX — for each indivual dispatcher, the metrics received and blackholed plus the wall clock time. The values are as described above. 👑
  • destinations.X — for all known destinations, the number of dropped, queued and sent metrics plus the wall clock time spent. The values are as described above. 🎯
  • aggregators.metricsReceived — the number of metrics that were matched an aggregator rule and were accepted by the aggregator. When a metric matches multiple aggregators, this value will reflect that. A metric is not counted when it is considered syntactically invalid, e.g. no value was found. 🧮
  • aggregators.metricsDropped — the number of metrics that were sent to an aggregator, but did not fit timewise. This is either because the metric was too far in the past or future. The expire after clause in aggregate statements controls how long in the past metric values are accepted. ⌛
  • aggregators.metricsSent — the number of metrics that were sent from the aggregators. These metrics were produced and are the actual results of aggregations. ✨

🐛 BUGS 🐛

Please report them at: https://github.com/grobian/carbon-c-relay/issues

found a bug? don't be shy, drop it in the issue tracker bestie, that's literally what it's for 💖

👑 AUTHOR 👑

Fabian Groffen <grobian@gentoo.org>

🔍 SEE ALSO 🔍

All other utilities from the graphite stack.

This project aims to be a fast replacement of the original Carbon relay http://graphite.readthedocs.org/en/1.0/carbon-daemons.html#carbon-relay-py. carbon-c-relay aims to deliver performance and configurability. Carbon is single threaded, and sending metrics to multiple consistent-hash clusters requires chaining of relays. This project provides a multithreaded relay which can address multiple targets and clusters for each and every metric based on pattern matches. 🦋

There are a couple more replacement projects out there, which are carbon-relay-ng https://github.com/graphite-ng/carbon-relay-ng and graphite-relay https://github.com/markchadwick/graphite-relay.

Compared to carbon-relay-ng, this project does provide carbon´s consistent-hash routing. graphite-relay, which does this, however doesn´t do metric-based matches to direct the traffic, which this project does as well. To date, carbon-c-relay can do aggregations, failover targets and more. 🔥

🫶 ACKNOWLEDGEMENTS 🫶

This program was originally developed for Booking.com, which approved that the code was published and released as Open Source on GitHub, for which the author would like to express his gratitude. Development has continued since with the help of many contributors suggesting features, reporting bugs, adding patches and more to make carbon-c-relay into what it is today. 🌈

and that's a wrap on the whole page, bestie!! this lil doc is straight from the Graphite stack, dated August 2020, and it's CARBON-C-RELAY(1) — go forth and relay those metrics, you absolute icon 💅✨👑

📦 where this page comes from