Configuration¶
colcad takes one argument, the path to a YAML file. Only ulid, data_dir
and key_file are required; every listener is off unless its address is set.
A small tree¶
A root node with a local door for services and a replication door for children:
ulid: n-global
data_dir: /data
key_file: /keys/global.key
api: { addr: ":443", local_addr: ":80", token: "change-me" }
mqtt: { addr: ":8883" }
mqtt_local: { addr: ":1883" }
repl: { addr: ":9443" }
An edge node below it. The parent's public key is pinned:
ulid: n-edge1
data_dir: /data
key_file: /keys/edge1.key
api: { addr: ":443", local_addr: ":80", token: "change-me" }
mqtt: { addr: ":8883" }
mqtt_local: { addr: ":1883" }
parent:
url: https://global.example.com:9443
pubkey: 3f1c… # printed by colca-keygen for the parent's key
The edge is then placed and enrolled at its parent once. An identity binds to a system element, not to a path, so the element comes first; its id is a ULID. The child's mount is wherever that element sits, now and after any rename:
export COLCA_TOKEN=... # api.token from the parent's config
# 1. the element the child hangs from
curl -sk -H "X-Colca-Token: $COLCA_TOKEN" -H "Content-Type: application/json" \
-X POST https://global.example.com/publish \
-d '{"topic":"colca/v1/_SystemElement/n-global/edge1","payload":{"id":"01J8Z3Y8S5ZC0KQ9M2F5T7W4XB","name":"edge1"}}'
# 2. the child's key, bound to that element
curl -sk -H "X-Colca-Token: $COLCA_TOKEN" -H "Content-Type: application/json" \
-X POST https://global.example.com/enroll \
-d '{"ulid":"n-edge1","kind":"node","element":"01J8Z3Y8S5ZC0KQ9M2F5T7W4XB","pubkey":"<edge1 public key>"}'
Machines and external services are enrolled the same way, with
"kind":"external" and the grants they need. There are no lists of children
or clients in the file: identities are runtime state, stored by the node.
Node¶
| Key | Meaning |
|---|---|
ulid |
The node's identity. Appears in every topic the node owns, in logs and in /healthz. |
name |
Optional display name for the node's own _Node record. |
access |
Optional {hostname, ui_url}: how people reach this node. Reported in the node's own _Node record and withdrawn when removed. hostname is 1-253 characters with no / or whitespace; ui_url is an http or https URL with a host and no credentials (user:pass@), query or fragment. |
topic_root |
First segment of every topic, colca by default. COLCA_TOPIC_ROOT overrides it. All nodes of one tree must agree. See Topics. |
standalone |
false by default. Permanently retire fleet trust; cannot coexist with parent. See handover. |
data_dir |
Directory of the node's database: streams, current state, cursors. |
secrets_dir |
Separate database for sealed secrets. Empty disables /secrets. Must differ from data_dir. |
key_file |
PEM file with the node's ed25519 private key, from colca-keygen. |
log_level |
debug for debug logging, anything else for info. |
addr_file |
If set, the node writes the addresses its listeners actually bound to as JSON once they are up. Useful with :0 ports. |
Listeners¶
| Key | Meaning |
|---|---|
api.addr |
HTTPS API for machines, administrators and people. |
api.token |
Admin token for X-Colca-Token. Empty authenticates nobody. |
api.local_addr |
Plaintext HTTP door for local services. Never publish it. |
mqtt.addr |
MQTT over TLS for enrolled machines and external services. |
mqtt_local.addr |
Plaintext MQTT door for local services. Never publish it. |
mqtt_human.tcp_addr |
MQTT over TLS for people, token in the password. |
mqtt_human.ws_addr |
MQTT over WebSocket for browsers. |
repl.addr |
Replication door for child nodes. |
tls.cert_file, tls.key_file |
Optional certificate for the HTTP API and the doors people use, for clients that expect one from a CA. Replication and the machine door always present the node's own key. |
parent.url |
https://host:port of the parent's replication door. Absent means this node is a root. |
parent.pubkey |
Hex public key of the parent, checked on every connection. |
parent.logs.min_level |
Lowest log level forwarded to the parent, WARNING by default. See Log forwarding. |
parent.logs.services |
Per-service override of min_level, keyed by service name. |
Log forwarding¶
Every node keeps all of its log records in its own logs stream, bounded by
retention.streams.logs. Only the records at or above a minimum level are
forwarded to the parent:
parent:
url: https://hub.example:9443
pubkey: 3b6a…
logs:
min_level: WARNING # the default when absent
services: # optional, keyed by service name
dataops: INFO # e.g. while commissioning
| Key | Default | Meaning |
|---|---|---|
parent.logs.min_level |
WARNING |
Forward a _Log record when its level is this or higher. One of DEBUG, INFO, WARNING, ERROR, CRITICAL, in any case. |
parent.logs.services.<service> |
none | The same, for one service. The service is the topic segment before the level: {root}/v1/_Log/{node}/{position…}/{service}/{LEVEL}; colcad's own records use colca. |
- An unknown level is a configuration error at startup.
- A
_Logrecord whose last topic segment is not one of the five levels is forwarded, as are the other records on thelogsstream, such as_StreamGapmarkers. - Each hop applies its own setting. A node forwards the records of its children that reached it by the same rule as its own, so a record rises only as far as every node on the way allows. To see a service's INFO at the root, set the override on every node between the service and the root.
- Withheld records never hold up the uplink: the uplink cursor moves past them, and they do not keep retention from pruning the stream.
colca_uplink_logs_withheld_total{level}counts the records a node kept home. It resets on restart.
Earlier releases forwarded every log record. A node now forwards WARNING and
above unless parent.logs says otherwise. Upgrade parents before their
children: a parent from before this change treats the holes in a child's logs
offsets as lost records and logs replication offset jump and counts
colca_replication_integrity_failures_total for them. Setting
min_level: DEBUG on a child keeps the old behaviour.
People¶
| Key | Default | Meaning |
|---|---|---|
auth.issuers[].url |
An accepted iss of tokens. At least one is required. |
|
auth.issuers[].jwks_url |
auth.jwks_url |
Where this issuer's signing keys are fetched from. |
auth.audience |
Expected aud, the same for every issuer. |
|
auth.jwks_url |
Signing keys shared by every issuer without its own jwks_url. |
|
auth.jwks_refresh |
1h |
How often keys are refreshed in the background. |
Retention¶
retention:
interval: 5m
streams:
metrics: { max_age: 336h, max_bytes: 4GiB }
commands: { max_age: 2160h }
| Key | Default | Meaning |
|---|---|---|
retention.interval |
5m |
How often the pruner runs. 0 turns it off; leaving the key out keeps the default. |
…streams.<name>.max_age |
see below | Records older than this may be pruned. Go duration syntax; there is no d unit. |
…streams.<name>.max_bytes |
none | Size limit, KiB/MiB/GiB/TiB or bytes. Applies in addition to the age. |
…streams.<name>.ignore_cursors_after |
0 |
After how long a cursor that stopped moving no longer protects the stream. 0 means never. |
…streams.<name>.keep_forever |
false |
Never prune this stream. Cannot be combined with max_age. |
…streams.metrics.signals |
none | Per-signal windows inside the stream's own, see below. |
Default ages: metrics and logs 14 days (336h); entities, alarms,
annotations and audit 365 days (8760h); commands 90 days (2160h);
service_state one day (24h): a service reads its state back from the
key-value entry, which pruning does not touch.
commands cannot be set below 7 days. A delivery cursor (a child node's
downlink cursor, a machine's delivery cursor) protects what is queued for it
beyond max_age; set ignore_cursors_after on commands to bound how long an
absent child keeps the queue. Commands pruned that way are answered 410. definitions are compacted,
not aged.
Per-signal retention¶
signals gives chosen metric signals a shorter window than the stream. Rules
are tried in order; the first one that selects a signal sets its window, and a
signal no rule selects keeps the stream's max_age.
retention:
streams:
metrics:
max_age: 72h
signals:
- topics: ["prekit/v1/_Metric/+/+/Diagnostics/#"]
max_age: 6h
- signal_ids: ["7SMDVD5TGG1H1KHMW9XGQV4TEJ"]
max_age: 1h
- topics: ["prekit/v1/_Metric/#"] # everything else
max_age: 24h
| Key | Meaning |
|---|---|
…signals[].topics |
MQTT filters on the signal's _Metric topic (+ one level, # the rest). The topic carries the node and the path, so a rule can select by machine, by line or by signal name. |
…signals[].signal_ids |
Signal ids, for single signals. |
…signals[].max_age |
The signal's window. Must be shorter than the stream's max_age, which still applies to every signal. A stream with keep_forever cannot have signals. |
For each selected signal the pruner removes records older than the window but
keeps the one with the latest timestamp among them: the value in force when
the window opens. It walks a signal in offset order and stops at its first
record inside the window, so a late sample appended after that stays until the
stream's max_age removes it. A signal
that stopped changing keeps its last value however old it is, and /kv keeps
the current value as before. The cursor rule is the stream's: nothing a
protecting cursor has not read is removed, and a cursor that went stale under
ignore_cursors_after is passed with one _StreamGap marker per pass naming
it, from its position to the protecting floor. That range is sparse: the
records of other signals in it are still there. A stale cursor that acks during
the pass protects again from its next batch on.
The rules are node configuration rather than a signal attribute because how long to keep history is a decision of the node that stores it: a hub that historicizes keeps hours, the edge that buffers for it keeps days, for the same signal. A rule covers thousands of signals without editing any of them.
Per-signal retention reads the signal index, so it covers records written since
the index existed on the node (colca 0.27). Older records leave with the
stream's max_age. Its deletes are spread across the stream rather than a
prefix, so Pebble reclaims their space as compaction reaches them.
The commands stream has one built-in rule of the same kind: the acks of
fetchLogs commands, which carry pages of log records, are kept one hour
instead of the stream's max_age (Fetching a node's logs).
It is not configurable.
Storage¶
| Key | Default | Meaning |
|---|---|---|
storage.compression |
snappy |
Block compression of the stream store: snappy or zstd. |
zstd stores the same records in about a quarter less space than snappy
(scale-test store: 10.0 MB → 7.4 MB; hub record mix: 219 → 155 bytes per
record) for more CPU in flushes and compactions and roughly twice the
decompression time on long scans. Pebble records the codec per block, so either
setting reads both: switching applies to data written from then on, and older
tables are rewritten in the new format as compaction reaches them. Switching
back is as safe.
Cursors¶
| Key | Default | Meaning |
|---|---|---|
cursors.lag_alarm_after |
60s |
How old the oldest unread record a consumer reads (its fetch filter applied) may get before the node writes a cursor_lag finding about its service. 0 writes none. |
cursors.stale_after |
24h |
How long a cursor may stand still while records wait past it before the node's stale_cursors finding names it, read or not. 0 writes none. |
A stale cursor is only reported. To let retention pass it, set
ignore_cursors_after on the stream; to remove it, retire it (see
HTTP API).
Logs¶
The node gates the _Log records written on it, by its services and by
itself, before they are stored. Records replicated from a child are not gated
again; the child gated them.
| Key | Default | Meaning |
|---|---|---|
logs.window |
60s |
The collapse and rate-cap window. 0 turns both off. At most 24h. |
logs.max_per_service |
600 |
Records one service may store per window below WARNING, and as many again at WARNING and above. 0 means no cap. |
logs.max_tracked |
4096 |
Distinct repeat keys, and separately services, held in memory. At most 65536. |
- Repeats collapse. The key is the topic (service and level), the
logger_nameand themessage. The first record of a key is stored at once. Identical records later in its window are not stored, only counted. When the window ends with a count, one record is written at the same topic, attributed to the original writer: the last withheld payload, its message suffixed likeconnection refused (×1200 in 60 s), andextra.repeated,extra.repeat_window_s,extra.first_repeat_atandextra.last_repeat_at(when the node admitted the first and last repeat). A window without repeats writes nothing. - Each service has a budget. A service is a
_Logtopic without its level segment. It may storemax_per_servicerecords per window atDEBUG,INFOor an unknown level, andmax_per_servicemore atWARNING,ERRORandCRITICAL; collapsed repeats and summaries do not count. The separate budget keeps a debug flood from dropping the error that explains it, while a flood of distinct errors is still bounded (a repeated error is collapsed anyway). Records beyond the budget are dropped and counted, and when the window ends oneWARNINGrecord at the service's position, written by the node (colca), says1834 log record(s) dropped: press exceeded 600 records in 60 s, withextra.dropped,extra.dropped_warning_and_above,extra.window_sandextra.service. When records atWARNINGor above were dropped, the message adds, N of them at WARNING or above. - Memory is bounded. At most
max_trackedrepeat keys andmax_trackedservices are held. A repeat key holds one pending payload of at most 8 KiB, and all pending payloads together at most 16 MiB; a larger record is never collapsed, only counted against its service's budget. A record that finds no room is stored without being collapsed or capped and counted incolca_log_untracked_total. Worst case: 16 MiB of payloads plus about 1 KiB per tracked repeat key and service (measured: 750 bytes with long topics), so about 20 MiB at the defaultmax_trackedand 80 MiB at its maximum. - A window ends on its own timer; a node that stops writes every pending summary and drop notice before its store closes.
- A withheld record was accepted: MQTT answers with a successful PUBACK and
does not fan it out,
POST /publishanswers202with{"withheld":"collapsed"|"rate_limited"}(see HTTP API). Publishers must not send it again.
The collapse only works when identical events carry identical messages, so a
publisher puts the event's own text in message, not a formatted line with a
timestamp (see Topics).
Limits¶
| Key | Default | Meaning |
|---|---|---|
limits.max_record_bytes |
4MiB |
Largest payload a record may carry. |
limits.max_blob_bytes |
32MiB |
Largest file attached to a _Resource. |
blob_gc.interval |
15m |
How often unreferenced files are collected. 0 turns it off. |
blob_gc.grace |
1h |
How long an unreferenced file is kept before it may go. |
mqtt_limits.max_clients |
4096 |
Concurrent MQTT connections. |
mqtt_limits.max_subscriptions_per_client |
1024 |
|
mqtt_limits.receive_maximum |
1024 |
MQTT 5 receive maximum announced to clients. |
mqtt_limits.maximum_inflight |
65535 |
Unacknowledged QoS 1/2 messages per client. |
mqtt_limits.max_pending_writes_per_client |
8192 |
Queued outbound packets before a slow client is dropped. |
mqtt_limits.max_topic_aliases_per_client |
256 |
|
mqtt_limits.max_session_expiry |
24h |
How long the session of a client that is away is kept: its subscriptions and what is queued for it. An MQTT 5 client may ask for less; an MQTT 3 session gets this value. |
mqtt_limits.max_offline_messages_per_client |
4096 |
QoS 1 and 2 messages queued for one session whose client is away. |
mqtt_limits.max_offline_bytes_per_client |
8MiB |
The same queue in bytes: payload, topic and 1 KiB per message. |
mqtt_limits.max_offline_bytes |
64MiB |
The queues of all offline sessions together. |
Sessions of clients that are away¶
A client that connects with a persistent session (MQTT 3 clean session off,
MQTT 5 with a session expiry) keeps its subscriptions while it is away, and
the node queues the QoS 1 and 2 messages they match in memory until the
client returns. The three max_offline_* limits bound that queue. When a
session's queue is full, or all queues together are, further messages are not
queued for it: the session keeps what arrived first after the disconnect and
loses what came later. One log line says so (offline mqtt session's queue is
full), colca_mqtt_offline_dropped_total counts the messages, and a second
line reports the count when the client is back.
The queue covers a restart, not history. A consumer that must see every
record reads the stream with /fetch from its own cursor after it
reconnects; the session only tells it sooner. Sessions are not stored: a node
restart ends all of them, so a client always subscribes again when the
CONNACK reports no session.
A session ends without its client when it stays away longer than
max_session_expiry, or within ten seconds once its identity was revoked,
since nobody can return to it. The session of a service that retired itself
(it published an empty _ServiceDetails over its own record, as
chaski.Service.retire() does) ends with its connection. Its queue and
subscriptions go with it (colca_mqtt_offline_sessions_ended_total).
A connected client is bounded the same way once it stops acknowledging. The
node holds every QoS 1 and 2 message until the client acknowledges it. A
client that holds more than max_offline_messages_per_client messages or
max_offline_bytes_per_client bytes and acknowledged nothing for 30 seconds
is cut back to those bounds, newest first, and nothing more is queued for it
until it acknowledges again. One log line says so (mqtt client does not
acknowledge), colca_mqtt_stalled_clients counts such clients and
colca_mqtt_unacked_dropped_total the messages. A client that acknowledges is
never cut back, however much it holds at one moment.
What to set: leave the defaults on an edge node. Raise
max_offline_messages_per_client only for a consumer that is known to be away
for minutes and whose subscriptions are narrow, and keep
max_offline_bytes below a tenth of the container's memory limit. Lower
max_session_expiry (for example 1h) where clients are redeployed under new
client ids often.
Bus¶
| Key | Default | |
|---|---|---|
bus.retain_child_metrics |
false |
Keep the _Metric records children replicate up as retained messages on this node's MQTT bus. Off, they are published live only; /kv holds their current values, and a consumer hydrates from it before it subscribes. A node's own metrics and every entity stay retained either way. A parent with 100 children and 140,000 signals at 28 k records/s spent ~30 % of its CPU and ~450 MB retaining them (fleet scale benchmark, 2026-10). |
Time¶
A node has one time: its host's wall clock. Colca reads it and never corrects it, so there is nothing to configure here. Keep the clock right with an NTP daemon on the host (chrony, ntpd or systemd-timesyncd), and let the machines on a node's network follow the same source or the node's host. The node reports how good its clock is; see Clocks.
Commands¶
| Key | Default | Meaning |
|---|---|---|
commands.strict |
false |
Answer every command to this node that no service announced with a 404 _Ack, also at elements where nobody announces anything. Turn it on where every executor announces its commands. |
Contracts¶
| Key | Default | Meaning |
|---|---|---|
contracts.bundle |
/etc/colca/contracts-bundle.json |
Schema bundle to enforce. Without one, a small built-in set of checks applies. |
contracts.sha256 |
Expected digest of the bundle. A mismatch stops the node from starting. |
Environment¶
| Variable | Used by | Meaning |
|---|---|---|
COLCA_TOPIC_ROOT |
every binary, colca-data-contracts, chaski |
Topic root; overrides topic_root. |
COLCA_ADMIN_TOKEN |
tools and examples | Admin token to present to a node. |