Skip to content

Operating Colca

Health

GET /healthz answers as soon as the node is up. Its storage.state turns failing when the database cannot flush or compact, most often because the disk is full; since and error say when and why. It returns to ok after the next successful flush. The node logs the first such error at ERROR and repeats it at most every 30 seconds with a repeats_suppressed count. GET /metrics serves Prometheus metrics; scrape it over the local door so that no self-signed certificate is involved.

Metrics describe progress as numbers, not connection states. The ones worth an alert first:

Metric Alert when
time() - colca_uplink_last_success_timestamp_seconds the uplink has not succeeded for longer than you can tolerate (only on nodes with a parent; it stays 0 on a root)
colca_cursor_lag_records a consumer falls behind
colca_retention_pressure above 1: the pruner wants to remove more than a cursor allows, and the disk grows
colca_stream_live_bytes a stream grows past what the disk holds
colca_cursor_next_record_age_seconds age of the next retained record waiting for a consumer; unlike cursor inactivity, this is zero when caught up
colca_cursor_unread_age_seconds age of the oldest record waiting that the consumer actually reads (its fetch filter applied); 0 on an idle stream. Past cursors.lag_alarm_after (default 60s, 0 writes no finding) the node also writes a cursor_lag finding about the service
colca_rejected_publishes_total rises: clients send what the node refuses; reason says why
colca_auth_rejections_total rises: unknown keys or bad tokens at a door
colca_audit_denials_suppressed_total rises: a caller repeats a request the node refuses. The audit holds the first denial and one record per five minutes with the count (metadata.repeated)
colca_replication_integrity_failures_total above 0: a child pruned records before its parent received them
colca_store_incarnation_changes_total rises: this node's store was restored from a backup, replaced, or given a new incarnation by an operator; its children re-sync
colca_uplink_resync_active stays 1: this node is still sending again what its restored parent lacks
colca_uplink_resync_unreplayable_records_total above 0: a restored parent lacks records this node had already pruned
colca_uplink_refused_total rises: the parent answered a push and refused it (4xx); the lane is held while it does
colca_uplink_set_aside_total above 0: a record the parent kept refusing was set aside and is not at the parent (see Records the parent refuses)
colca_repl_child_ahead_of_store_total rises: a child too old to re-sync is ahead of this node's store
colca_host_clock_synchronized stays 0: no NTP daemon vouches for the host's clock (see Clocks)
abs(colca_clock_parent_skew_seconds) above 0.5 for minutes, and at once above 5: this node and its parent disagree about the time
colca_jwks_keys 0 on a node with an auth: block
colca_contracts_bundle_info source="builtin" where you expected a bundle

Counters reset when a node restarts. Gauges are read from the database at scrape time and survive restarts.

Who spends the read budgets

The door limits each caller to 5 /kv and 25 /fetch requests a second. These say who uses them:

Metric Labels Shows
colca_http_kv_requests_total caller, contract (one contract, multiple or all), prefix_depth (0 is the whole node, up to 5+) whole-node reads (contract="all",prefix_depth="0") are the ones to remove first
colca_http_kv_entries caller histogram of entries per /kv page
colca_http_fetch_requests_total caller, stream a follower polling an idle stream shows a steady rate; /watch removes it
colca_http_request_limited_by_caller_total route (the route pattern, such as GET /kv), caller 429s per caller; colca_http_request_limited_total has them by door, class and reason

A caller's colca_http_request_limited_by_caller_total series exists at 0 from its first request on a route, so "never limited" reads as 0, not as a missing series.

caller is a registered identity as kind:name (local:dataops-line), every person as human, and the admin token as admin. No label carries a path.

The replication door's limits

A parent limits each enrolled child by its node identity, which the TLS handshake proves before anything else is read: 100 requests a second, two pushes and two downlink polls at a time. Children behind one address, such as a site router or a carrier NAT, do not share a budget. A caller whose certificate is not an enrolled node's is limited by source address: 100 requests a second and 64 at a time. A parent holds at most 1024 pushes and 8192 downlink polls at once. colca_http_request_limited_total{door="repl"} counts refusals by class: node, auth (by address), replication (pushes), downlink and transfer (files).

Every refusal also carries reason, the bound that refused it: rate_limit (the caller's request rate; the answer carries Retry-After), concurrency_limit (the caller already holds the most open requests of the class, such as /watch streams; it clears when the caller closes one), global_concurrency_limit (the node holds the most of the class it serves), caller_limit (the node tracks the most callers it may) and busy (a parent shedding pushes its store or memory cannot take yet). Queries that sum over door and class keep working.

A parent commits the pushes that arrive while a commit is being synced together in the next one, up to 4096 records a commit, so its children are not limited to one sync each. A push is answered once its records are on disk.

A parent that falls behind sheds load instead of buffering it: while 16,384 replicated records wait for its store, or its pushes in progress hold a 32nd of its memory limit in request bytes, a new push is answered 429 with Retry-After: 1, counted as replication_backlog or replication_bytes. The child keeps the batch, waits as long as the parent asked (with jitter, so its siblings do not return in step) and sends it again; nothing is dropped, and the uplink logs a busy parent as a warning, not as a refusal.

Records the parent refuses

A parent that answers a push and refuses it (a 4xx other than 429, 401 or the 409 of a changed store) has read the batch and said no: usually a record larger than its limits.max_record_bytes, or one on a stream it may not rise on. Order within a stream never changes, so the uplink cannot send what follows past a refused record. It isolates it instead: the refused batch is sent again at half its size, down to the one record the parent refuses, so the records before it rise at once. The batch grows back once the parent takes one.

That record is held and sent again for 15 minutes, since the cause is often fixed on the parent (a rolling upgrade, a raised limit). After that it is set aside: the node copies it into its store outside the stream, moves the uplink cursor past it, counts colca_uplink_set_aside_total{stream}, logs an error, and writes the retained _Finding <root>/v1/_Finding/<node>/uplink_set_aside naming each record (stream, offset, topic, size and what the parent said). The lane continues behind it. A set-aside record never reaches the parent: it is lost there, and that is what the finding and the error say.

The node does not send a set-aside record again by itself. The parent drops every child offset at or below its high-water mark, so the record cannot be pushed later at its own offset, and appending it again under a new offset could put an old state value over a newer one at the parent. What to do with it is the operator's decision: GET /uplink/set-aside on the child's admin door lists the copies with their payloads, and DELETE /uplink/set-aside/{stream}/{offset} clears one once it is dealt with. The copies survive the stream's retention; the finding retires when the last one is cleared.

colcad and colca-historian set Go's memory limit to 75 % of their cgroup's memory ceiling (cgroup v1 or v2, the process's own cgroup and the smallest limit on its ancestors) unless GOMEMLIMIT is set, so the collector works harder near the ceiling instead of letting the kernel kill the process. They log a warning when they find no ceiling. The store's memtables and block cache are sized from the same ceiling, a 32nd of it each, between Pebble's defaults (4 MiB and 8 MiB, also used when no ceiling is found) and 256 MiB: a 512 MiB edge gets 16 MiB of each, a 2 GiB hub 64 MiB, an 8 GiB hub 256 MiB. Raising a hub's memory ceiling therefore also gives its store larger memtables, which compact less under heavy ingest. Pebble reserves its memtables in the block cache, so the cache is created with room for three of them beside the blocks; memtables and cache together stay within an eighth of the ceiling.

The store's own statistics

Metric Labels Shows
colca_pebble_block_cache_bytes bytes of blocks in the block cache
colca_pebble_block_cache_hits_total, colca_pebble_block_cache_misses_total block reads served from the cache, and those read from a table
colca_pebble_memtable_bytes state (current, obsolete) memtables in use and queued for flush, and those an open read still pins or Pebble keeps for reuse
colca_pebble_level_bytes, colca_pebble_level_tables level size and number of tables per LSM level
colca_pebble_bytes_written_total kind (wal, flush, compaction) what the store wrote since it opened; compaction against the other two is the write amplification
colca_pebble_compaction_debt_bytes, colca_pebble_compactions_in_progress compaction work still owed, and compactions running
colca_pebble_table_iterators open table iterators; a read holds several
colca_pebble_tombstones approximate number of point and range deletion tombstones in the tables
colca_pebble_deletion_reclaimable_bytes kind (point, range) table bytes compaction can drop because deletions cover them
colca_pebble_table_stats_complete 1 once the two deletion gauges cover every table present at start

A scrape reads Pebble's counters only; it opens no table.

Replicated records reach the local MQTT bus through one goroutine after they are durable. Each child's records, and so each topic's, arrive in order; records of different children may arrive in another order than the store holds them. Records waiting for the bus count against the push budget above.

What the broker delivers

Metric Labels Shows
colca_mqtt_delivered_messages_total door (mqtt, local, human) PUBLISH packets written to subscribers; with colca_ingest_records_total it gives the node's MQTT in and out
colca_mqtt_delivered_payload_bytes_total door their payload bytes
colca_mqtt_publish_dropped_total publishes dropped because a subscriber's queue was full
colca_mqtt_offline_sessions sessions kept for a client that is not connected
colca_mqtt_offline_queued_messages, colca_mqtt_offline_queued_bytes what is queued in memory for them; bounded by mqtt_limits.max_offline_*
colca_mqtt_offline_dropped_total messages not queued because an offline session's queue was full; the log has one line per session and absence
colca_mqtt_offline_sessions_ended_total reason (expired, identity_removed, retired) sessions removed without their client returning
colca_mqtt_stalled_clients connected clients over the per-client bounds that acknowledged nothing for 30 s; nothing more is queued for them until they acknowledge
colca_mqtt_unacked_dropped_total messages dropped for such a client; the log has one line per client and stall

What the log gate holds back

Metric Labels Shows
colca_log_withheld_total reason (collapsed, rate_limited) _Log records accepted but not stored: repeats counted in a summary, records over a service's budget counted in a drop notice
colca_log_untracked_total table (repeats, services) records stored unremembered because logs.max_tracked was reached; non-zero means raise it
colca_log_gate_write_failures_total kind (summary, drop_notice) summaries and drop notices the store did not take

See Configuration.

Retention

The pruner removes the oldest records of a stream in one atomic step and never touches the current-state view, which only shrinks through empty payloads. It runs every retention.interval (5 minutes by default) and once at start, one pass of at most 100 000 records per stream, so a node that restarts more often than the interval still prunes. max_age never removes a record without a timestamp; max_bytes does.

Every named cursor protects the stream, including the replication cursor that forms a child's offline buffer. If the age or size limit wants to remove records a cursor has not read, nothing happens: the stream stays as it is, colca_retention_pressure goes above 1, and a warning names the cursor. Disk keeps growing until someone acts.

A configured parent's uplink cursors are persisted during startup, before retention runs, even when the streams are empty or the parent has never been reachable. This protects the first accepted records through an initial outage and restart. Startup fails if this protection cannot be written. Reconnecting advances the cursors only after successful upload; ordinary retention can then reclaim the acknowledged history. This protection is not a disk-capacity limit: size storage for the expected outage, ingestion rate and other local consumers.

Each cycle repeats a stream's prune while its scan stops at the record cap and still removes records, for up to 30 s per stream, so a hub that appends more than one capped scan per interval does not fall behind.

Large hubs

A hub that historicizes metrics into its own historian does not need weeks of broker history; the historian's cursor protects whatever it has not stored yet. At the 2026-10 scale test the hub took about 140 MB of stream records per machine-day (metrics 101 MB, logs 20 MB, annotations 16 MB, entities 4 MB), about 1.7 TB at 1000 machines over the default 14 days, and annotations stay a year by default.

storage:
  compression: zstd          # about a quarter smaller at rest
retention:
  streams:
    metrics:
      max_age: 72h           # 1–3 days; the historian holds the history
      signals:               # optional: shorter for chatty signals
        - topics: ["prekit/v1/_Metric/+/+/Diagnostics/#"]
          max_age: 12h
    logs:        { max_age: 48h }
    annotations: { max_age: 336h }   # once the projector has read them
    audit:       { max_age: 2160h }

Every stream is pruned only below the cursors that read it, so an annotation leaves once every consumer with a cursor on annotations (the projector, a parent's uplink) has read it. annotations defaults to a year because rebuild_projection replays the stream: a shorter max_age is also how far back a rebuild can restore. entities are current state, kept in /kv whatever the stream holds; colca compacts only definitions to the latest record per topic, because a consumer that archives entity history reads every entity record, so for entities bound the age instead.

A cursor that stood still for cursors.stale_after (default 24 h) while its stream grew is named in the node's stale_cursors finding. List the cursors with GET /backlog (admin token, no prefix) and retire the ones a removed consumer left with POST /ack {"delete":true}; see HTTP API.

To let the pruner give up on a consumer that stopped, set ignore_cursors_after for that stream. When a run removes records a cursor had not read, it logs an error and writes one _StreamGap record into the stream, which replicates up like anything else. Consumers see the gap in /fetch.

Things to know:

  • A cursor that stays dead produces one _StreamGap per pruner run that removes something, not a single one.
  • _StreamGap records are not shown on the node's own MQTT bus. Read them from the stream or at an ancestor.
  • On the commands stream, every live command removed before the child node or machine whose delivery cursor held it received it gets a 410 _Ack in the same write, at the command's position, and counts in colca_command_dropped_total{reason="pruned"}. Like the gap marker, these answers are read from the stream, not the bus.

Moving a child node

A child that is re-parented or taken out of service is drained first: the parent stops routing new commands to it and waits until every command already queued has been delivered or has expired. A command without expires_at keeps the drain open until it is delivered. Drains are visible in colca_drains_active and colca_drains_completed_total.

A completed drain retires the child: the parent tombstones every current-state record the child and the nodes below it replicated (its elements, signals, last metric values, services), and the tombstones travel up, so no ancestor keeps showing the old node as live. To take a child out immediately, without draining, use DELETE /enroll/{ulid}?retire=true; every command still queued for it is answered with a 410 _Ack. A plain DELETE only revokes: the child's replicated state stays, for a child that will be enrolled again and resume where it stopped.

Clocks

Colca has one time per node, the host's wall clock, and never corrects it. Stream timestamps, audit times and the expiry of commands all read it. A command's expires_at is judged by each node on the way and by the machine that executes it, each on its own clock, so the clocks of a tree have to agree to well within the shortest expiry in use. That is the host's job: run an NTP daemon (chrony, ntpd, systemd-timesyncd) on every node's host, and point the machines on a node's network at the same source or at the node's host.

The node reports two things about its clock, on /metrics and in the clock block of /healthz:

Metric Meaning
colca_host_clock_synchronized 1 when the kernel reports the clock as synchronized by an NTP daemon, 0 when not
colca_host_clock_estimated_error_seconds the NTP service's estimate of the clock's error; absent where the service keeps none
colca_host_clock_max_error_seconds the kernel's bound on the error; the service sets it at every update and the kernel adds 0.5 ms a second
colca_host_clock_stale_bound_seconds the part of that bound one poll interval does not explain: 0 while the clock is updated, then 0.5 ms for each second since an update came due
colca_clock_parent_skew_seconds the parent's clock minus this node's, from the now_ms on the latest /downlink or /replicate response whose round trip was at most 0.5 s; positive when the parent is ahead
colca_clock_parent_sample_age_seconds seconds since that measurement
colca_clock_parent_samples_discarded_total responses not used for the measurement because their round trip was longer than 0.5 s
  • The host series come from a read-only adjtimex call, which needs no privilege and passes Docker's default seccomp profile, so they work in the stock container. Where the kernel's verdict cannot be read (a host that is not Linux, a sandbox that refuses the call) the three series are absent and /healthz reports null; node-exporter's node_timex_sync_status, node_timex_estimated_error_seconds and node_timex_maxerror_seconds carry the same reading from the host itself.
  • Which host series to trust depends on the NTP service. chrony and ntpd keep the kernel's estimate and its bound. systemd-timesyncd keeps the synchronized flag and the bound only: it writes an estimate of 0 at every update, so colca_host_clock_estimated_error_seconds is absent on such a host and /healthz reports null. The bound works under all three: the service sets it when it updates the clock (to 0 or to its root distance) and the kernel adds 500 µs every second, which makes it a record of how long ago the last update was. One poll interval of a healthy service adds at most 1.024 s (2048 s, systemd-timesyncd's longest), and colca_host_clock_stale_bound_seconds is the bound above that: 0 on a healthy host, 0.5 when no update arrived for 51 minutes, 5 after 3 h 21 min. At a bound of 16 s (8 h 53 min) the kernel declares the clock unsynchronized. A service configured to poll less often than every 2048 s shows a stale bound between polls.
  • The parent series exist on a node with a parent once the parent has answered, and never on a root. The node notes its clock when the request leaves and when the response is read; the parent states its clock (now_ms) and how long it had the request by then (held_ms: a long poll's wait, an ingest). The round trip is the exchange minus that hold, and the parent's clock is compared with the middle of it, so the measurement is off by at most half the round trip: a few milliseconds on a LAN. It catches what the kernel's verdict cannot: two hosts that are each synchronized, to sources that disagree. It is a measurement only. A node whose clock differs from its parent's keeps deciding on its own clock.
  • A slow or stalled link is not skew. A response whose round trip is longer than 0.5 s, the warning threshold below, could be off by more than half that threshold, so it cannot show a skew of that size. It is not used: the series keep the previous measurement, colca_clock_parent_sample_age_seconds keeps growing, and colca_clock_parent_samples_discarded_total counts it. The first response after a link that dropped packets for some seconds is such a response. On a link whose round trip is always above 0.5 s the skew is never measured; the growing sample age shows that. A parent that predates held_ms has the whole exchange counted as round trip, so under it only responses that came back within 0.5 s are used, and a long poll that waited is not.
  • Alert on colca_host_clock_synchronized == 0 for 15 minutes, on the largest of abs(colca_clock_parent_skew_seconds), colca_host_clock_estimated_error_seconds and colca_host_clock_stale_bound_seconds above 0.5 for 10 minutes (warning) and above 5 for 5 minutes (critical). The stale bound is the term that fires under every NTP service; the estimate adds what chrony and ntpd know. The node's own clock health metric uses the same two thresholds.
  • The log has one line when the skew to the parent passes 5 seconds (clock skew to the parent exceeds the alarm threshold) and one when it is back within it.

Upgrading from the time beacon

Releases before this one corrected each node's time towards the root's and published it to machines on <root>/v1/_TimeSync/<node>. Nothing of that is left: no time_sync: configuration (a block still in a config file is ignored), no _TimeSync contract, no now_ms on /fetch, and the metrics colca_clock_offset_ms and colca_clock_sync_age_seconds and the rejection reason time_sync are gone. Nodes can be upgraded in any order:

  • An older child under an upgraded parent still reads now_ms from the parent's responses and corrects its own decisions by it, as before. The parent now answers with its own wall clock rather than a time corrected towards the root, so the correction reaches one level, not the whole tree.
  • An upgraded child under an older parent measures its skew against the parent's now_ms and corrects nothing. Such a parent sends no held_ms, so the child uses only its responses that came back within 0.5 s.
  • An older machine still subscribes to <root>/v1/_TimeSync/+ when it connects. The node refuses that one filter (SUBACK 0x80, or 0x87 over MQTT 5; counted in colca_acl_denials_total{action="sub"}, with an audit event) and leaves the session and its command subscription alone. The machine waits out its hold (10 seconds by default) after each connect, then decides on its own clock. The refusal does not depend on who asks: a filter that names a contract the node does not define is refused on the machine door, the local door and the people's door, also for an identity that may read everything, because nothing is ever published there. A filter with a wildcard at the contract level (<root>/v1/+/…, <root>/#) names no contract and is judged by the read grants as before.
  • A publish to <root>/v1/_TimeSync/… is refused like any topic the grammar does not know and counted in colca_rejected_publishes_total{reason="grammar"}.

Make sure NTP runs on every host before the upgrade: until now a node with a wrong clock still judged expiry by the root's time, and from this release it judges by its own.

Backups

A node's state is its data_dir. Stop the node, or snapshot the filesystem, and copy the directory.

Keep secrets_dir and the key file separate from data_dir. Losing a service's own keyring makes its stored secrets unreadable.

Restoring a parent from a backup

A node restored from a copy of its data_dir is behind everything that happened since the copy was taken. Its children and its own parent hold positions of the store it was before. Nodes detect this on the replication link and re-sync; see Store incarnations for how. Nothing has to be done on the children.

Recovered without an operator:

  • Commands flow again at once. Each child takes commands from where the restored store stands for it. It does not wait for the parent's commands stream to grow past the head it remembered.
  • No command runs twice. A command a child already holds is not stored or executed again, also when the restored node receives it from its own parent a second time and hands it on.
  • Current state converges. Each child sends again what the parent lacks of its entities stream, so elements, signals, alarms' standing state and the last metric values at the parent are those of the children again.
  • History is sent again as far as the children still hold it: metrics, alarms, annotations, audit, logs, and the _Acks of the commands stream, from the highest offset the restored parent holds of each stream. The parent stores each record once.
  • The restored node keeps replicating to its own parent. That parent clears the marks it kept for the node's earlier store, so records written at offsets the node hands out again are stored, not dropped as known.

Not recovered:

  • What only the restored node held. Records written at the node itself since the backup — commands issued there, definitions authored there, what its own machines and services published — are gone unless a node above holds them. A command issued there after the backup that a child already executed stays executed; its record and its ack reach the parent again only as the child's _Ack.
  • Definitions authored since the backup stay on the children that received them. The restored node no longer has them and so never changes or retires them. Author them again, or retire them, at the node that owns them.
  • Ranges a child's retention already pruned. A child keeps what its parent has not confirmed, but it had been told these were stored. The gap is logged at ERROR on the child (re-sync gap), counted in colca_uplink_resync_unreplayable_records_total{stream} and listed under uplink.resync.gaps on the child's /healthz until the next re-sync.
  • Nodes above the restored one receive the replayed history a second time, under new offsets: they are not restored and already hold it. Current state is unaffected and the historian writes a sample once per signal and timestamp, but a consumer of alarms, annotations, audit or logs at an ancestor reads those records again.
  • A command without correlation_id is recognised as already held only among what the parent's commands stream held when the child re-synced. One that reaches the child again later is executed again.

What you see while it happens:

Where What
Restored node, log ERROR this store is behind what a child recorded of it (a child proved it) or this store is behind what its parent holds of it (its parent did)
Restored node, /healthz store.id is new; store.changed has at, reason and the previous id
Restored node, /metrics colca_store_incarnation_changes_total{reason="regression"}
Child, log ERROR re-syncing with the parent, then INFO re-sync with the parent complete
Child, /healthz uplink.resync: state is replaying while records are still being sent again and complete afterwards, with reason, both store ids, the range per stream (from, to, position) and gaps. uplink.state stays connected
Child, /metrics colca_uplink_resyncs_total, colca_uplink_resync_active (1 while replaying), colca_downlink_commands_already_held_total

A restore is only detected when a neighbour holds a position the restored store has not reached: a child that read a command or was confirmed a record since the backup, or a parent that holds a record of the node written since. If nothing was exchanged since the backup, nothing was lost on the link and nothing happens. If you restored a node and want every child to re-sync regardless, or you replaced a store by other means, tell the node:

curl -sk -X POST -H "X-Colca-Token: $COLCA_TOKEN" https://<node>/store/incarnation
# {"store":"<new id>","previous":"<old id>"}

The node takes a new incarnation. Every child re-syncs at its next request, the node's own parent clears its marks for it, and local consumers get store_changed on their next /ack. It is safe to repeat: a re-sync against a store that lacks nothing replays nothing.

Mixed versions keep replicating as before, without the protection:

  • A child older than the check under a restored parent is not told and does not re-sync: it takes no commands until the parent's stream passes its position, and does not send again what the parent lacks. The parent logs a child is ahead of this store and cannot re-sync and counts it in colca_repl_child_ahead_of_store_total. Upgrade the child.
  • A child under a parent older than the check learns no incarnation and never re-syncs. It still reports a commands position past the parent's head (colca_downlink_cursor_beyond_head_total).
  • A child records what its parent confirms from its first push after the upgrade. A restore of the parent before that is detected by the commands position only, and what the child sent since the backup may not be sent again.
  • A restored node that wrote more records than it had lost before it reached its parent again is not recognised as restored by that parent's marks: the parent drops, as already stored, what the node wrote at the offsets it holds. Tell the node with POST /store/incarnation right after such a restore.

Benchmarks

cmd/colca-bench runs five scenarios against a real edge and hub pair and records the results:

make bench-scenarios            # ingest, live latency, catch-up, cardinality, footprint
make bench-check                # compares the newest results with bench/thresholds.json
make bench                      # store micro-benchmarks

Results are written to bench/results/<host>.jsonl, which is not committed. Numbers from a laptop say little about an edge device with eMMC storage; run the scenarios on the hardware you deploy to before setting thresholds.

The fanout scenario measures one parent against many children, each pushing a few records a second and holding its downlink poll open. The parent runs as a separate process with a pprof listener, so its CPU, mutex and block profiles can be saved:

make bench-replication          # 200 protocol children; CI gates on bench/replication-thresholds.json
bin/colca-bench fanout --children 100 --colcad bin/colcad --child-dir /Volumes/ram \
  --profile-dir prof/            # colcad children, profiles of the parent

The fleet scenario drives a parent that is already running, such as a hub deployed with its historian and Timescale, with protocol children that each emulate an edge: a connector scanning --signals every --scan and reporting the --change share of them, one push per scan. It prints one JSON line per --interval: samples generated and accepted, pushes, refusals, push latency and the backlog the children hold. Everything else (the parent's CPU, the historian's lag) is measured outside:

bin/colca-bench fleet --hub-api hub:443 --hub-repl hub:9443 --hub-pubkey HEX \
  --children 1000 --signals 100 --change 0.1 --duration 5m

--protocol-children runs children that speak only the replication protocol, in the bench process: a host runs a thousand of them. With colcad children every child keeps and syncs its own store; on one machine they share one disk, so put --child-dir on a RAM disk or the children saturate the disk before the parent is measured. fanout_hop_p95_ms counts from the moment a child has stored a record to its arrival on the parent's bus.

Known limitations

  • No clustering. A node is a single writer on local disk. There is no failover inside a level; availability comes from the level below buffering while a node is gone.
  • Retained messages and the current-state view are unbounded. They shrink only when paths are retired, so the number of distinct paths is the bound.
  • Without a schema bundle only minimal checks apply. Run nodes with the bundle generated from contracts/.
  • Node administration uses one token per node besides admin:# grants.
  • A deadlock in mochi-mqtt 2.7.9 is avoided, not fixed. Colca pins a fork and stops its clients before closing listeners; the workaround goes once the fix is released upstream.

Per-signal metric replication

A _Signal accepts replication_policy with two values:

  • replicate_to_parents (default, including older signals without the field): newly accepted samples are eligible for upward replication.
  • source_local_only: newly accepted samples remain at their source node.

The decision is stored with each sample, under the same storage lock as signal updates. Changing the policy does not cancel already queued uploads. Re-enabling replication does not backfill samples accepted while local-only. Restarts and later signal edits do not change these decisions. Local MQTT delivery, retained values and historian consumers continue to read the original samples.

Signal descriptors still replicate. The uplink sends compact offset ranges for intentionally omitted samples, without their topics, payloads or timestamps. The parent acknowledges those ranges durably without appending metric records. This preserves real offset-gap detection and replay deduplication; an entirely local-only page still exchanges progress with the parent. Reads are bounded by physical records scanned, including excluded samples.

Deploy receivers before enabling the policy at their children. Older receivers reject the progress entries and the child holds its cursor for retry instead of silently discarding data. Upgrade the contracts package and consumers together; older Python signal decoders do not recognize the new field. Downgrading a source to a binary that does not understand its persisted upload decisions is unsupported.

How the historian writes

colca-historian follows the metrics stream in pages of FETCH_MAX records (default 5000, the most /fetch returns). Up to PIPELINE_PAGES pages (default 4) are between fetch and marker at once:

  • one goroutine fetches pages in stream order; several decode them;
  • each page's rows go to WRITERS writers (default 4) by signal, so a signal always lands on the same writer and its rows are written in stream order (retractions depend on it);
  • a writer does not wait for the other writers' share of a page before it starts on the next page, and writes what queued meanwhile in one transaction, as one INSERT … SELECT FROM unnest(…) ON CONFLICT per run of values;
  • each writer's transaction also moves that writer's own marker (historian:metrics/<writers>.<n> in colca_applied_offset);
  • the applied-offset marker (historian:metrics) moves, in a transaction of its own, to the end of the newest page that is written completely with every page before it, and the /ack follows it.

A restart reads from the cursor. Records at or below the marker are skipped, and so are a writer's records at or below its own marker, so nothing is written twice. A write whose answer was lost is checked against the writer's marker before it is retried. Re-applying a page is not harmless in general: a retraction is stored only when the row before it holds a value, and a late sample (an older timestamp arriving after the retraction) changes that answer. The writer markers depend on WRITERS. After WRITERS changes (or between a pipelined run and PIPELINE_PAGES=1), a start skips to the smallest marker of the previous writer count when all of its writers have one: every row at or below it is written. Rows between that point and the previous writers' own markers are written again, at most PIPELINE_PAGES pages; that is the only case where the late-sample caveat above can still apply, and only after an unclean stop. A failed write is retried in place with backoff; the pages behind it wait in bounded queues.

The marker records the store it is a position in: /fetch names the store a page was read from (new whenever colcad's store is created from nothing), and the historian acks only with that store id, so the node refuses an ack that would land on a recreated store. An ack that fails (colcad restarting) is retried with backoff, also at the head with no new data; meanwhile the cursor lags the marker. A read from the cursor of the same store is then acked up to the marker and reading goes on after it; the marker never moves back.

When colcad's data volume is recreated while Timescale keeps the markers, pages come from another store than the marker's. The historian logs a warning, zeroes every writer marker in one transaction before it writes a row of the new stream, and historises it from its cursor, that is from its first record. A running historian finds it on the next page it fetches; ahead of the cursor it first waits until the pages in flight are marked. A marker written before stores were recorded, or a node that sends no store, falls back once to the page: one starting at offset 1, or a short page ending below the marker, counts as a new stream; the store is recorded from then on.

colca-historian from 0.32 expects colcad 0.31 or newer, which sends store ids, and logs a warning once when a node sends none. Without store ids the historian tells a recreated volume only by its offsets, and while its own ack of the marker is failing it reads a page from offset 1 as its own cursor lagging behind (otherwise it would re-apply the first page over and over). It still takes that page for a new stream when the node's head is below the marker (a short page, or the head announced on /watch). What stays open is an old colcad recreated while the historian's ack is failing, whose new stream has already grown past the marker: its records up to the marker are skipped. PREKIT ships colcad and colca-historian from the same colca release, so a generated node never runs that pair. DB_MAX_CONNS defaults to WRITERS + 1. PIPELINE_PAGES=1 writes one page at a time, with the marker in the same transaction when WRITERS=1. A drain that reached the head the node announced on /watch waits for the next hint without a final empty fetch.

On the fleet scale benchmark, round 4 (2026-10, PREKIT's Timescale image on an amd64 VM, hub stack pinned to 6-10 cores, a 13.4 M-record backlog):

one page at a time PIPELINE_PAGES=4
catch-up, 6 / 8 / 10 cores 95 / 97 / 98 k rows/s 125 / 128 / 127 k rows/s
live, 112 k records/s offered, 8 / 10 cores 82 / 84 k rows/s, falling behind 105 / 107 k rows/s, keeping up

Eight pages instead of four, or eight writers instead of five, changed nothing. At ~127 k rows/s the historian makes 25 /fetch requests a second, the per-caller limit, and Postgres spends ~25 µs of CPU per row with most of its writers busy (wait events: 77 % CPU, 10 % WALWrite); 20 000-record pages (an experiment, /fetch allows 5000) gave only 130-134 k.

Signals the historian does not store

colca-historian writes no rows for samples of a signal whose _Signal definition says "is_logged": false. A signal without the field, a sample whose signal has no definition on the node, and an undecodable definition are all stored: leaving history out needs an explicit false.

The historian loads the _Signal records once from /kv and then follows the entities stream with its own cursor, c/historian/signals, woken by /watch. The flag in force when a sample is ingested decides; a later change of the flag does not rewrite or backfill history. Skipped samples are consumed: the metrics marker and cursor move past them like past written rows. On startup no sample is read before the definitions are loaded.

is_logged only affects this historian. It does not change what the node stores, replicates (that is replication_policy, above) or delivers over MQTT.

/metrics reports colca_historian_samples_not_logged_total (samples consumed without a row) and colca_historian_signals_not_logged (signals currently marked). /healthz fails with signal_definitions while the definitions cannot be loaded or followed.

Late writes the historian marks

A reader that keeps something computed from historian_metric per hour (PREKIT's per-signal statistics) has to learn when a sample lands in an hour it already computed: a child catching up after an outage, a replay, an import, a corrected value. colca-historian tells it in the database.

The reader owns two relations, and its migrations create them before colca-historian runs (PREKIT starts the historian after its api, which migrates first):

CREATE TABLE historian_late_write (
    id bigserial PRIMARY KEY,
    hour timestamptz NOT NULL,       -- a UTC hour
    signal_ids jsonb NOT NULL);      -- the signals written into it, a JSON array
-- the signals to mark: a table or a view with a signal_id column
CREATE VIEW historian_late_write_signal AS SELECT signal_id FROM ...;

For every page it writes, the historian appends one row to historian_late_write per UTC hour the page has a sample of, when that hour started 65 minutes ago or longer by the database's clock, naming the listed signals it wrote into that hour, and sends NOTIFY historian_late_write when it appended one. This happens in the transaction that writes the samples: a sample is never stored without its mark, and a mark is never visible before its samples. An offline colca-historian import marks the same way. The historian writes no page unmarked, so it checks for both relations when it starts and exits at once, naming the missing one, when the reader's migrations have not run (PREKIT: the api's historian migration 0005_metric_statistics). It does not wait for them; start it after the migrations.

Marking never holds ingestion up: it is a plain insert, with no unique key, no conflict handling and no table lock. A signal that is not listed writes nothing, and a page of current samples does not run the statement at all. Several pages writing into one hour leave one row each; the reader merges them.

The reader's side of the contract:

  • Handling marks: read committed marks, recompute their (signal, hour) pairs from historian_metric, then delete the marks it read, by id. A mark it can read was committed with its samples, so the recompute sees them; a mark committed meanwhile stays for the next pass. A crash in between leaves the marks, and recomputing again is harmless.
  • Closing an hour: no earlier than 65 minutes after its start by the database's clock, take a transaction id (pg_current_xact_id()), then wait until pg_snapshot_xmin(pg_current_snapshot()) is at least that id before reading the hour. Every page that decided by then that it was not late has written its rows, so it already has a smaller transaction id; the wait ends when all of those committed or rolled back. Pages are never made to wait: the reader waits, bounded, and tries again later.
  • Listing a new signal: commit it to the list, then wait the same way before reading the signal's history. A page whose mark statement ran before the listing committed has an older transaction id; every later one sees the listing.

Permanent standalone handover

standalone: true is a permanent trust transition, distinct from an offline parent. Remove parent in the same configuration and restart the node. Before opening any door the node journals the transition, retires the old parent cursors and existing externally enrolled identities, and disables the static administration token. Local services, node identity and process history remain. External machine clients must be enrolled again by the new local owner.

Commands accepted before handover are retired from HTTP fetch and MQTT replay, including pending local commands. Their stored history and acknowledgements are preserved; new commands remain executable. The retirement boundary survives restarts. Upgrading an existing standalone data directory from 0.2.0 records this boundary once at the first startup, retiring any commands pending at that time.

Human authentication stays closed until a trusted local identity controller has preserved operators, disabled fleet accounts and transferred administration. The controller then calls POST /standalone/complete on the local HTTP door. Completion persists a token issuance cutoff; retries and restarts retain it. GET /self exposes standalone_since (Unix seconds) and standalone_ready so other local authentication services enforce the same boundary.

Pre-handover JWTs and personal access tokens are rejected. Only locally authored group definitions and newly created local PATs authorize subsequent access. Cached fleet definitions remain available as data, so an identity controller can copy the last applied operator grants into new, locally owned groups. It must not reuse the fleet group's identity for a competing local definition.

Removing the standalone flag or restoring an old parent configuration against this data directory refuses startup. Rejoining a fleet requires an explicit migration and enrollment plan, including a decision about retained history; a configuration rollback cannot export history or restore the former owner.

This broker transition does not administer an external identity provider, host VPN, SSH access or separate update agents. The deployment's handover controller must retire those connections and credentials before reporting the machine as handed over. Ordinary parent outages do not trigger any of these actions.

Importing historical measurements

colca-historian import is an operator tool for archived measurements. It uses the historian's DATABASE_URL and ordinary sink; no broker connection is opened. It does not update retained live values, dispatch events to control consumers, or advance the running historian's stream cursor. Applications should resolve their existing signal identities through their normal public API before export.

Input is JSONL, one {"topic": "colca/v1/_Metric/NODE/PATH", "payload": {...}} per line. Each payload must have a signal ULID, explicit timestamp in Unix seconds, and a value. The topic's node must be a ULID and agree with any payload node identity. Use COLCA_TOPIC_ROOT for a deployment with another root.

colca-historian import --file archive.jsonl --sha256 EXPECTED_SHA256 \
  --before 2026-01-01T00:00:00Z --dry-run
# Inside the historian's trusted deployment environment:
colca-historian import --file archive.jsonl --sha256 EXPECTED_SHA256 \
  --before 2026-01-01T00:00:00Z

The complete input is validated and hashed before database access. A private temporary spool prevents input changes between validation and writing. Imports use strict transactions of up to 1000 rows by default (--batch-size, maximum 5000), with a separate historian:import:SHA256 marker. Rerunning the identical file resumes committed batches. Errors stop the import without acknowledging the failed batch. This has the normal sink's upsert semantics: the same signal and timestamp replaces that historical point. Choose a ceiling before existing live data and check for overlapping records when replacement is unintended. The ordinary historian schema must already exist; import does not change schema or retention. Ensure the configured retention covers the imported dates.