Operating Colca¶
Health¶
GET /healthz answers as soon as the node is up. Its storage.state turns
failing when the database cannot flush or compact, most often because the disk
is full; since and error say when and why. It returns to ok after the next
successful flush. The node logs the first such error at ERROR and repeats it at
most every 30 seconds with a repeats_suppressed count. GET /metrics serves
Prometheus metrics; scrape it over the local door so that no self-signed
certificate is involved.
Metrics describe progress as numbers, not connection states. The ones worth an alert first:
| Metric | Alert when |
|---|---|
time() - colca_uplink_last_success_timestamp_seconds |
the uplink has not succeeded for longer than you can tolerate (only on nodes with a parent; it stays 0 on a root) |
colca_cursor_lag_records |
a consumer falls behind |
colca_retention_pressure |
above 1: the pruner wants to remove more than a cursor allows, and the disk grows |
colca_stream_live_bytes |
a stream grows past what the disk holds |
colca_cursor_next_record_age_seconds |
age of the next retained record waiting for a consumer; unlike cursor inactivity, this is zero when caught up |
colca_cursor_unread_age_seconds |
age of the oldest record waiting that the consumer actually reads (its fetch filter applied); 0 on an idle stream. Past cursors.lag_alarm_after (default 60s, 0 writes no finding) the node also writes a cursor_lag finding about the service |
colca_rejected_publishes_total |
rises: clients send what the node refuses; reason says why |
colca_auth_rejections_total |
rises: unknown keys or bad tokens at a door |
colca_audit_denials_suppressed_total |
rises: a caller repeats a request the node refuses. The audit holds the first denial and one record per five minutes with the count (metadata.repeated) |
colca_replication_integrity_failures_total |
above 0: a child pruned records before its parent received them |
colca_store_incarnation_changes_total |
rises: this node's store was restored from a backup, replaced, or given a new incarnation by an operator; its children re-sync |
colca_uplink_resync_active |
stays 1: this node is still sending again what its restored parent lacks |
colca_uplink_resync_unreplayable_records_total |
above 0: a restored parent lacks records this node had already pruned |
colca_uplink_refused_total |
rises: the parent answered a push and refused it (4xx); the lane is held while it does |
colca_uplink_set_aside_total |
above 0: a record the parent kept refusing was set aside and is not at the parent (see Records the parent refuses) |
colca_repl_child_ahead_of_store_total |
rises: a child too old to re-sync is ahead of this node's store |
colca_host_clock_synchronized |
stays 0: no NTP daemon vouches for the host's clock (see Clocks) |
abs(colca_clock_parent_skew_seconds) |
above 0.5 for minutes, and at once above 5: this node and its parent disagree about the time |
colca_jwks_keys |
0 on a node with an auth: block |
colca_contracts_bundle_info |
source="builtin" where you expected a bundle |
Counters reset when a node restarts. Gauges are read from the database at scrape time and survive restarts.
Who spends the read budgets¶
The door limits each caller to 5 /kv and 25 /fetch requests a second. These
say who uses them:
| Metric | Labels | Shows |
|---|---|---|
colca_http_kv_requests_total |
caller, contract (one contract, multiple or all), prefix_depth (0 is the whole node, up to 5+) |
whole-node reads (contract="all",prefix_depth="0") are the ones to remove first |
colca_http_kv_entries |
caller |
histogram of entries per /kv page |
colca_http_fetch_requests_total |
caller, stream |
a follower polling an idle stream shows a steady rate; /watch removes it |
colca_http_request_limited_by_caller_total |
route (the route pattern, such as GET /kv), caller |
429s per caller; colca_http_request_limited_total has them by door, class and reason |
A caller's colca_http_request_limited_by_caller_total series exists at 0
from its first request on a route, so "never limited" reads as 0, not as a
missing series.
caller is a registered identity as kind:name (local:dataops-line), every
person as human, and the admin token as admin. No label carries a path.
The replication door's limits¶
A parent limits each enrolled child by its node identity, which the TLS
handshake proves before anything else is read: 100 requests a second, two
pushes and two downlink polls at a time. Children behind one address, such as
a site router or a carrier NAT, do not share a budget. A caller whose
certificate is not an enrolled node's is limited by source address: 100
requests a second and 64 at a time. A parent holds at most 1024 pushes and 8192
downlink polls at once. colca_http_request_limited_total{door="repl"} counts
refusals by class: node, auth (by address), replication (pushes),
downlink and transfer (files).
Every refusal also carries reason, the bound that refused it:
rate_limit (the caller's request rate; the answer carries Retry-After),
concurrency_limit (the caller already holds the most open requests of the
class, such as /watch streams; it clears when the caller closes one),
global_concurrency_limit (the node holds the most of the class it serves),
caller_limit (the node tracks the most callers it may) and busy (a parent
shedding pushes its store or memory cannot take yet). Queries that sum over
door and class keep working.
A parent commits the pushes that arrive while a commit is being synced together in the next one, up to 4096 records a commit, so its children are not limited to one sync each. A push is answered once its records are on disk.
A parent that falls behind sheds load instead of buffering it: while 16,384
replicated records wait for its store, or its pushes in progress hold a 32nd
of its memory limit in request bytes, a new push is answered 429 with
Retry-After: 1, counted as replication_backlog or replication_bytes. The
child keeps the batch, waits as long as the parent asked (with jitter, so its
siblings do not return in step) and sends it again; nothing is dropped, and
the uplink logs a busy parent as a warning, not as a refusal.
Records the parent refuses¶
A parent that answers a push and refuses it (a 4xx other than 429, 401
or the 409 of a changed store) has read the batch and said no: usually a
record larger than its limits.max_record_bytes, or one on a stream it may not
rise on. Order within a stream never changes, so the uplink cannot send what
follows past a refused record. It isolates it instead: the refused batch is
sent again at half its size, down to the one record the parent refuses, so the
records before it rise at once. The batch grows back once the parent takes
one.
That record is held and sent again for 15 minutes, since the cause is often
fixed on the parent (a rolling upgrade, a raised limit). After that it is set
aside: the node copies it into its store outside the stream, moves the uplink
cursor past it, counts colca_uplink_set_aside_total{stream}, logs an error,
and writes the retained _Finding <root>/v1/_Finding/<node>/uplink_set_aside
naming each record (stream, offset, topic, size and what the parent said). The
lane continues behind it. A set-aside record never reaches the parent: it
is lost there, and that is what the finding and the error say.
The node does not send a set-aside record again by itself. The parent drops
every child offset at or below its high-water mark, so the record cannot be
pushed later at its own offset, and appending it again under a new offset
could put an old state value over a newer one at the parent. What to do with
it is the operator's decision: GET /uplink/set-aside on the child's admin
door lists the copies with their payloads, and DELETE
/uplink/set-aside/{stream}/{offset} clears one once it is dealt with. The
copies survive the stream's retention; the finding retires when the last one
is cleared.
colcad and colca-historian set Go's memory limit to 75 % of their cgroup's
memory ceiling (cgroup v1 or v2, the process's own cgroup and the smallest
limit on its ancestors) unless GOMEMLIMIT is set, so the collector works
harder near the ceiling instead of letting the kernel kill the process. They
log a warning when they find no ceiling. The store's memtables and block
cache are sized from the same ceiling, a 32nd of it each, between Pebble's
defaults (4 MiB and 8 MiB, also used when no ceiling is found) and 256 MiB: a
512 MiB edge gets 16 MiB of each, a 2 GiB hub 64 MiB, an 8 GiB hub 256 MiB.
Raising a hub's memory ceiling therefore also gives its store larger
memtables, which compact less under heavy ingest. Pebble reserves its
memtables in the block cache, so the cache is created with room for three of
them beside the blocks; memtables and cache together stay within an eighth of
the ceiling.
The store's own statistics¶
| Metric | Labels | Shows |
|---|---|---|
colca_pebble_block_cache_bytes |
bytes of blocks in the block cache | |
colca_pebble_block_cache_hits_total, colca_pebble_block_cache_misses_total |
block reads served from the cache, and those read from a table | |
colca_pebble_memtable_bytes |
state (current, obsolete) |
memtables in use and queued for flush, and those an open read still pins or Pebble keeps for reuse |
colca_pebble_level_bytes, colca_pebble_level_tables |
level |
size and number of tables per LSM level |
colca_pebble_bytes_written_total |
kind (wal, flush, compaction) |
what the store wrote since it opened; compaction against the other two is the write amplification |
colca_pebble_compaction_debt_bytes, colca_pebble_compactions_in_progress |
compaction work still owed, and compactions running | |
colca_pebble_table_iterators |
open table iterators; a read holds several | |
colca_pebble_tombstones |
approximate number of point and range deletion tombstones in the tables | |
colca_pebble_deletion_reclaimable_bytes |
kind (point, range) |
table bytes compaction can drop because deletions cover them |
colca_pebble_table_stats_complete |
1 once the two deletion gauges cover every table present at start |
A scrape reads Pebble's counters only; it opens no table.
Replicated records reach the local MQTT bus through one goroutine after they are durable. Each child's records, and so each topic's, arrive in order; records of different children may arrive in another order than the store holds them. Records waiting for the bus count against the push budget above.
What the broker delivers¶
| Metric | Labels | Shows |
|---|---|---|
colca_mqtt_delivered_messages_total |
door (mqtt, local, human) |
PUBLISH packets written to subscribers; with colca_ingest_records_total it gives the node's MQTT in and out |
colca_mqtt_delivered_payload_bytes_total |
door |
their payload bytes |
colca_mqtt_publish_dropped_total |
publishes dropped because a subscriber's queue was full | |
colca_mqtt_offline_sessions |
sessions kept for a client that is not connected | |
colca_mqtt_offline_queued_messages, colca_mqtt_offline_queued_bytes |
what is queued in memory for them; bounded by mqtt_limits.max_offline_* |
|
colca_mqtt_offline_dropped_total |
messages not queued because an offline session's queue was full; the log has one line per session and absence | |
colca_mqtt_offline_sessions_ended_total |
reason (expired, identity_removed, retired) |
sessions removed without their client returning |
colca_mqtt_stalled_clients |
connected clients over the per-client bounds that acknowledged nothing for 30 s; nothing more is queued for them until they acknowledge | |
colca_mqtt_unacked_dropped_total |
messages dropped for such a client; the log has one line per client and stall |
What the log gate holds back¶
| Metric | Labels | Shows |
|---|---|---|
colca_log_withheld_total |
reason (collapsed, rate_limited) |
_Log records accepted but not stored: repeats counted in a summary, records over a service's budget counted in a drop notice |
colca_log_untracked_total |
table (repeats, services) |
records stored unremembered because logs.max_tracked was reached; non-zero means raise it |
colca_log_gate_write_failures_total |
kind (summary, drop_notice) |
summaries and drop notices the store did not take |
See Configuration.
Retention¶
The pruner removes the oldest records of a stream in one atomic step and never
touches the current-state view, which only shrinks through empty payloads.
It runs every retention.interval (5 minutes by default) and once at start, one
pass of at most 100 000 records per stream, so a node that restarts more often
than the interval still prunes. max_age never removes a record without a
timestamp; max_bytes does.
Every named cursor protects the stream, including the replication cursor that
forms a child's offline buffer. If the age or size limit wants to remove
records a cursor has not read, nothing happens: the stream stays as it is,
colca_retention_pressure goes above 1, and a warning names the cursor.
Disk keeps growing until someone acts.
A configured parent's uplink cursors are persisted during startup, before retention runs, even when the streams are empty or the parent has never been reachable. This protects the first accepted records through an initial outage and restart. Startup fails if this protection cannot be written. Reconnecting advances the cursors only after successful upload; ordinary retention can then reclaim the acknowledged history. This protection is not a disk-capacity limit: size storage for the expected outage, ingestion rate and other local consumers.
Each cycle repeats a stream's prune while its scan stops at the record cap and still removes records, for up to 30 s per stream, so a hub that appends more than one capped scan per interval does not fall behind.
Large hubs¶
A hub that historicizes metrics into its own historian does not need weeks of broker history; the historian's cursor protects whatever it has not stored yet. At the 2026-10 scale test the hub took about 140 MB of stream records per machine-day (metrics 101 MB, logs 20 MB, annotations 16 MB, entities 4 MB), about 1.7 TB at 1000 machines over the default 14 days, and annotations stay a year by default.
storage:
compression: zstd # about a quarter smaller at rest
retention:
streams:
metrics:
max_age: 72h # 1–3 days; the historian holds the history
signals: # optional: shorter for chatty signals
- topics: ["prekit/v1/_Metric/+/+/Diagnostics/#"]
max_age: 12h
logs: { max_age: 48h }
annotations: { max_age: 336h } # once the projector has read them
audit: { max_age: 2160h }
Every stream is pruned only below the cursors that read it, so an annotation
leaves once every consumer with a cursor on annotations (the projector, a
parent's uplink) has read it. annotations defaults to a year because
rebuild_projection replays the stream: a shorter max_age is also how far
back a rebuild can restore. entities are current state, kept in /kv
whatever the stream holds; colca compacts only definitions to the latest
record per topic, because a consumer that archives entity history reads every
entity record, so for entities bound the age instead.
A cursor that stood still for cursors.stale_after (default 24 h) while its
stream grew is named in the node's stale_cursors finding. List the cursors
with GET /backlog (admin token, no prefix) and retire the ones a removed
consumer left with POST /ack {"delete":true}; see
HTTP API.
To let the pruner give up on a consumer that stopped, set
ignore_cursors_after for that stream. When a run removes records a cursor
had not read, it logs an error and writes one _StreamGap record into the
stream, which replicates up like anything else. Consumers see the gap in
/fetch.
Things to know:
- A cursor that stays dead produces one
_StreamGapper pruner run that removes something, not a single one. _StreamGaprecords are not shown on the node's own MQTT bus. Read them from the stream or at an ancestor.- On the
commandsstream, every live command removed before the child node or machine whose delivery cursor held it received it gets a410_Ackin the same write, at the command's position, and counts incolca_command_dropped_total{reason="pruned"}. Like the gap marker, these answers are read from the stream, not the bus.
Moving a child node¶
A child that is re-parented or taken out of service is drained first: the
parent stops routing new commands to it and waits until every command already
queued has been delivered or has expired. A command without expires_at
keeps the drain open until it is delivered. Drains are visible in
colca_drains_active and colca_drains_completed_total.
A completed drain retires the child: the parent tombstones every current-state
record the child and the nodes below it replicated (its elements, signals,
last metric values, services), and the tombstones travel up, so no ancestor
keeps showing the old node as live. To take a child out immediately, without
draining, use DELETE /enroll/{ulid}?retire=true; every command still queued
for it is answered with a 410 _Ack. A plain DELETE only
revokes: the child's replicated state stays, for a child that will be enrolled
again and resume where it stopped.
Clocks¶
Colca has one time per node, the host's wall clock, and never corrects it.
Stream timestamps, audit times and the expiry of commands all read it. A
command's expires_at is judged by each node on the way and by the machine
that executes it, each on its own clock, so the clocks of a tree have to agree
to well within the shortest expiry in use. That is the host's job: run an NTP
daemon (chrony, ntpd, systemd-timesyncd) on every node's host, and point the
machines on a node's network at the same source or at the node's host.
The node reports two things about its clock, on /metrics and in the clock
block of /healthz:
| Metric | Meaning |
|---|---|
colca_host_clock_synchronized |
1 when the kernel reports the clock as synchronized by an NTP daemon, 0 when not |
colca_host_clock_estimated_error_seconds |
the NTP service's estimate of the clock's error; absent where the service keeps none |
colca_host_clock_max_error_seconds |
the kernel's bound on the error; the service sets it at every update and the kernel adds 0.5 ms a second |
colca_host_clock_stale_bound_seconds |
the part of that bound one poll interval does not explain: 0 while the clock is updated, then 0.5 ms for each second since an update came due |
colca_clock_parent_skew_seconds |
the parent's clock minus this node's, from the now_ms on the latest /downlink or /replicate response whose round trip was at most 0.5 s; positive when the parent is ahead |
colca_clock_parent_sample_age_seconds |
seconds since that measurement |
colca_clock_parent_samples_discarded_total |
responses not used for the measurement because their round trip was longer than 0.5 s |
- The host series come from a read-only
adjtimexcall, which needs no privilege and passes Docker's default seccomp profile, so they work in the stock container. Where the kernel's verdict cannot be read (a host that is not Linux, a sandbox that refuses the call) the three series are absent and/healthzreportsnull; node-exporter'snode_timex_sync_status,node_timex_estimated_error_secondsandnode_timex_maxerror_secondscarry the same reading from the host itself. - Which host series to trust depends on the NTP service. chrony and ntpd
keep the kernel's estimate and its bound. systemd-timesyncd keeps the
synchronized flag and the bound only: it writes an estimate of 0 at every
update, so
colca_host_clock_estimated_error_secondsis absent on such a host and/healthzreportsnull. The bound works under all three: the service sets it when it updates the clock (to 0 or to its root distance) and the kernel adds 500 µs every second, which makes it a record of how long ago the last update was. One poll interval of a healthy service adds at most 1.024 s (2048 s, systemd-timesyncd's longest), andcolca_host_clock_stale_bound_secondsis the bound above that:0on a healthy host,0.5when no update arrived for 51 minutes,5after 3 h 21 min. At a bound of 16 s (8 h 53 min) the kernel declares the clock unsynchronized. A service configured to poll less often than every 2048 s shows a stale bound between polls. - The parent series exist on a node with a parent once the parent has
answered, and never on a root. The node notes its clock when the request
leaves and when the response is read; the parent states its clock (
now_ms) and how long it had the request by then (held_ms: a long poll's wait, an ingest). The round trip is the exchange minus that hold, and the parent's clock is compared with the middle of it, so the measurement is off by at most half the round trip: a few milliseconds on a LAN. It catches what the kernel's verdict cannot: two hosts that are each synchronized, to sources that disagree. It is a measurement only. A node whose clock differs from its parent's keeps deciding on its own clock. - A slow or stalled link is not skew. A response whose round trip is
longer than 0.5 s, the warning threshold below, could be off by more than
half that threshold, so it cannot show a skew of that size. It is not used:
the series keep the previous measurement,
colca_clock_parent_sample_age_secondskeeps growing, andcolca_clock_parent_samples_discarded_totalcounts it. The first response after a link that dropped packets for some seconds is such a response. On a link whose round trip is always above 0.5 s the skew is never measured; the growing sample age shows that. A parent that predatesheld_mshas the whole exchange counted as round trip, so under it only responses that came back within 0.5 s are used, and a long poll that waited is not. - Alert on
colca_host_clock_synchronized == 0for 15 minutes, on the largest ofabs(colca_clock_parent_skew_seconds),colca_host_clock_estimated_error_secondsandcolca_host_clock_stale_bound_secondsabove0.5for 10 minutes (warning) and above5for 5 minutes (critical). The stale bound is the term that fires under every NTP service; the estimate adds what chrony and ntpd know. The node's ownclockhealth metric uses the same two thresholds. - The log has one line when the skew to the parent passes 5 seconds
(
clock skew to the parent exceeds the alarm threshold) and one when it is back within it.
Upgrading from the time beacon¶
Releases before this one corrected each node's time towards the root's and
published it to machines on <root>/v1/_TimeSync/<node>. Nothing of that is
left: no time_sync: configuration (a block still in a config file is
ignored), no _TimeSync contract, no now_ms on /fetch, and the metrics
colca_clock_offset_ms and colca_clock_sync_age_seconds and the rejection
reason time_sync are gone. Nodes can be upgraded in any order:
- An older child under an upgraded parent still reads
now_msfrom the parent's responses and corrects its own decisions by it, as before. The parent now answers with its own wall clock rather than a time corrected towards the root, so the correction reaches one level, not the whole tree. - An upgraded child under an older parent measures its skew against the
parent's
now_msand corrects nothing. Such a parent sends noheld_ms, so the child uses only its responses that came back within 0.5 s. - An older machine still subscribes to
<root>/v1/_TimeSync/+when it connects. The node refuses that one filter (SUBACK0x80, or0x87over MQTT 5; counted incolca_acl_denials_total{action="sub"}, with an audit event) and leaves the session and its command subscription alone. The machine waits out its hold (10 seconds by default) after each connect, then decides on its own clock. The refusal does not depend on who asks: a filter that names a contract the node does not define is refused on the machine door, the local door and the people's door, also for an identity that may read everything, because nothing is ever published there. A filter with a wildcard at the contract level (<root>/v1/+/…,<root>/#) names no contract and is judged by the read grants as before. - A publish to
<root>/v1/_TimeSync/…is refused like any topic the grammar does not know and counted incolca_rejected_publishes_total{reason="grammar"}.
Make sure NTP runs on every host before the upgrade: until now a node with a wrong clock still judged expiry by the root's time, and from this release it judges by its own.
Backups¶
A node's state is its data_dir. Stop the node, or snapshot the filesystem,
and copy the directory.
Keep secrets_dir and the key file separate from data_dir. Losing a
service's own keyring makes its stored secrets unreadable.
Restoring a parent from a backup¶
A node restored from a copy of its data_dir is behind everything that
happened since the copy was taken. Its children and its own parent hold
positions of the store it was before. Nodes detect this on the replication
link and re-sync; see Store incarnations for
how. Nothing has to be done on the children.
Recovered without an operator:
- Commands flow again at once. Each child takes commands from where the
restored store stands for it. It does not wait for the parent's
commandsstream to grow past the head it remembered. - No command runs twice. A command a child already holds is not stored or executed again, also when the restored node receives it from its own parent a second time and hands it on.
- Current state converges. Each child sends again what the parent lacks of
its
entitiesstream, so elements, signals, alarms' standing state and the last metric values at the parent are those of the children again. - History is sent again as far as the children still hold it:
metrics,alarms,annotations,audit,logs, and the_Acks of thecommandsstream, from the highest offset the restored parent holds of each stream. The parent stores each record once. - The restored node keeps replicating to its own parent. That parent clears the marks it kept for the node's earlier store, so records written at offsets the node hands out again are stored, not dropped as known.
Not recovered:
- What only the restored node held. Records written at the node itself
since the backup — commands issued there, definitions authored there, what
its own machines and services published — are gone unless a node above holds
them. A command issued there after the backup that a child already executed
stays executed; its record and its ack reach the parent again only as the
child's
_Ack. - Definitions authored since the backup stay on the children that received them. The restored node no longer has them and so never changes or retires them. Author them again, or retire them, at the node that owns them.
- Ranges a child's retention already pruned. A child keeps what its parent
has not confirmed, but it had been told these were stored. The gap is logged
at ERROR on the child (
re-sync gap), counted incolca_uplink_resync_unreplayable_records_total{stream}and listed underuplink.resync.gapson the child's/healthzuntil the next re-sync. - Nodes above the restored one receive the replayed history a second time,
under new offsets: they are not restored and already hold it. Current state
is unaffected and the historian writes a sample once per signal and
timestamp, but a consumer of
alarms,annotations,auditorlogsat an ancestor reads those records again. - A command without
correlation_idis recognised as already held only among what the parent'scommandsstream held when the child re-synced. One that reaches the child again later is executed again.
What you see while it happens:
| Where | What |
|---|---|
| Restored node, log | ERROR this store is behind what a child recorded of it (a child proved it) or this store is behind what its parent holds of it (its parent did) |
Restored node, /healthz |
store.id is new; store.changed has at, reason and the previous id |
Restored node, /metrics |
colca_store_incarnation_changes_total{reason="regression"} |
| Child, log | ERROR re-syncing with the parent, then INFO re-sync with the parent complete |
Child, /healthz |
uplink.resync: state is replaying while records are still being sent again and complete afterwards, with reason, both store ids, the range per stream (from, to, position) and gaps. uplink.state stays connected |
Child, /metrics |
colca_uplink_resyncs_total, colca_uplink_resync_active (1 while replaying), colca_downlink_commands_already_held_total |
A restore is only detected when a neighbour holds a position the restored store has not reached: a child that read a command or was confirmed a record since the backup, or a parent that holds a record of the node written since. If nothing was exchanged since the backup, nothing was lost on the link and nothing happens. If you restored a node and want every child to re-sync regardless, or you replaced a store by other means, tell the node:
curl -sk -X POST -H "X-Colca-Token: $COLCA_TOKEN" https://<node>/store/incarnation
# {"store":"<new id>","previous":"<old id>"}
The node takes a new incarnation. Every child re-syncs at its next request,
the node's own parent clears its marks for it, and local consumers get
store_changed on their next /ack. It is safe to repeat: a re-sync against
a store that lacks nothing replays nothing.
Mixed versions keep replicating as before, without the protection:
- A child older than the check under a restored parent is not told and does
not re-sync: it takes no commands until the parent's stream passes its
position, and does not send again what the parent lacks. The parent logs
a child is ahead of this store and cannot re-syncand counts it incolca_repl_child_ahead_of_store_total. Upgrade the child. - A child under a parent older than the check learns no incarnation and
never re-syncs. It still reports a commands position past the parent's head
(
colca_downlink_cursor_beyond_head_total). - A child records what its parent confirms from its first push after the upgrade. A restore of the parent before that is detected by the commands position only, and what the child sent since the backup may not be sent again.
- A restored node that wrote more records than it had lost before it reached
its parent again is not recognised as restored by that parent's marks: the
parent drops, as already stored, what the node wrote at the offsets it holds.
Tell the node with
POST /store/incarnationright after such a restore.
Benchmarks¶
cmd/colca-bench runs five scenarios against a real edge and hub pair and
records the results:
make bench-scenarios # ingest, live latency, catch-up, cardinality, footprint
make bench-check # compares the newest results with bench/thresholds.json
make bench # store micro-benchmarks
Results are written to bench/results/<host>.jsonl, which is not committed.
Numbers from a laptop say little about an edge device with eMMC storage; run
the scenarios on the hardware you deploy to before setting thresholds.
The fanout scenario measures one parent against many children, each pushing
a few records a second and holding its downlink poll open. The parent runs as a
separate process with a pprof listener, so its CPU, mutex and block profiles
can be saved:
make bench-replication # 200 protocol children; CI gates on bench/replication-thresholds.json
bin/colca-bench fanout --children 100 --colcad bin/colcad --child-dir /Volumes/ram \
--profile-dir prof/ # colcad children, profiles of the parent
The fleet scenario drives a parent that is already running, such as a hub
deployed with its historian and Timescale, with protocol children that each
emulate an edge: a connector scanning --signals every --scan and reporting
the --change share of them, one push per scan. It prints one JSON line per
--interval: samples generated and accepted, pushes, refusals, push latency
and the backlog the children hold. Everything else (the parent's CPU, the
historian's lag) is measured outside:
bin/colca-bench fleet --hub-api hub:443 --hub-repl hub:9443 --hub-pubkey HEX \
--children 1000 --signals 100 --change 0.1 --duration 5m
--protocol-children runs children that speak only the replication protocol,
in the bench process: a host runs a thousand of them. With colcad children
every child keeps and syncs its own store; on one machine they share one disk,
so put --child-dir on a RAM disk or the children saturate the disk before the
parent is measured. fanout_hop_p95_ms counts from the moment a child has
stored a record to its arrival on the parent's bus.
Known limitations¶
- No clustering. A node is a single writer on local disk. There is no failover inside a level; availability comes from the level below buffering while a node is gone.
- Retained messages and the current-state view are unbounded. They shrink only when paths are retired, so the number of distinct paths is the bound.
- Without a schema bundle only minimal checks apply. Run nodes with the
bundle generated from
contracts/. - Node administration uses one token per node besides
admin:#grants. - A deadlock in mochi-mqtt 2.7.9 is avoided, not fixed. Colca pins a fork and stops its clients before closing listeners; the workaround goes once the fix is released upstream.
Per-signal metric replication¶
A _Signal accepts replication_policy with two values:
replicate_to_parents(default, including older signals without the field): newly accepted samples are eligible for upward replication.source_local_only: newly accepted samples remain at their source node.
The decision is stored with each sample, under the same storage lock as signal updates. Changing the policy does not cancel already queued uploads. Re-enabling replication does not backfill samples accepted while local-only. Restarts and later signal edits do not change these decisions. Local MQTT delivery, retained values and historian consumers continue to read the original samples.
Signal descriptors still replicate. The uplink sends compact offset ranges for intentionally omitted samples, without their topics, payloads or timestamps. The parent acknowledges those ranges durably without appending metric records. This preserves real offset-gap detection and replay deduplication; an entirely local-only page still exchanges progress with the parent. Reads are bounded by physical records scanned, including excluded samples.
Deploy receivers before enabling the policy at their children. Older receivers reject the progress entries and the child holds its cursor for retry instead of silently discarding data. Upgrade the contracts package and consumers together; older Python signal decoders do not recognize the new field. Downgrading a source to a binary that does not understand its persisted upload decisions is unsupported.
How the historian writes¶
colca-historian follows the metrics stream in pages of FETCH_MAX records
(default 5000, the most /fetch returns). Up to PIPELINE_PAGES pages
(default 4) are between fetch and marker at once:
- one goroutine fetches pages in stream order; several decode them;
- each page's rows go to
WRITERSwriters (default 4) by signal, so a signal always lands on the same writer and its rows are written in stream order (retractions depend on it); - a writer does not wait for the other writers' share of a page before it
starts on the next page, and writes what queued meanwhile in one
transaction, as one
INSERT … SELECT FROM unnest(…) ON CONFLICTper run of values; - each writer's transaction also moves that writer's own marker
(
historian:metrics/<writers>.<n>incolca_applied_offset); - the applied-offset marker (
historian:metrics) moves, in a transaction of its own, to the end of the newest page that is written completely with every page before it, and the/ackfollows it.
A restart reads from the cursor. Records at or below the marker are skipped,
and so are a writer's records at or below its own marker, so nothing is
written twice. A write whose answer was lost is checked against the writer's
marker before it is retried. Re-applying a page is not harmless in general: a
retraction is stored only when the row before it holds a value, and a late
sample (an older timestamp arriving after the retraction) changes that answer.
The writer markers depend on WRITERS. After WRITERS changes (or between a
pipelined run and PIPELINE_PAGES=1), a start skips to the smallest marker of
the previous writer count when all of its writers have one: every row at or
below it is written. Rows between that point and the previous writers' own
markers are written again, at most PIPELINE_PAGES pages; that is the only
case where the late-sample caveat above can still apply, and only after an
unclean stop. A failed write is retried in place with backoff; the pages
behind it wait in bounded queues.
The marker records the store it is a position in: /fetch names the store a
page was read from (new whenever colcad's store is created from nothing), and
the historian acks only with that store id, so the node refuses an ack that
would land on a recreated store. An ack that fails (colcad restarting) is
retried with backoff, also at the head with no new data; meanwhile the cursor
lags the marker. A read from the cursor of the same store is then acked up to
the marker and reading goes on after it; the marker never moves back.
When colcad's data volume is recreated while Timescale keeps the markers, pages come from another store than the marker's. The historian logs a warning, zeroes every writer marker in one transaction before it writes a row of the new stream, and historises it from its cursor, that is from its first record. A running historian finds it on the next page it fetches; ahead of the cursor it first waits until the pages in flight are marked. A marker written before stores were recorded, or a node that sends no store, falls back once to the page: one starting at offset 1, or a short page ending below the marker, counts as a new stream; the store is recorded from then on.
colca-historian from 0.32 expects colcad 0.31 or newer, which sends store ids,
and logs a warning once when a node sends none. Without store ids the
historian tells a recreated volume only by its offsets, and while its own ack
of the marker is failing it reads a page from offset 1 as its own cursor
lagging behind (otherwise it would re-apply the first page over and over).
It still takes that page for a new stream when the node's head is below the
marker (a short page, or the head announced on /watch). What stays open is
an old colcad recreated while the historian's ack is failing, whose new stream
has already grown past the marker: its records up to the marker are skipped.
PREKIT ships colcad and colca-historian from the same colca release, so a
generated node never runs that pair.
DB_MAX_CONNS defaults to WRITERS + 1. PIPELINE_PAGES=1 writes one page
at a time, with the marker in the same transaction when WRITERS=1. A drain
that reached the head the node announced on /watch waits for the next hint
without a final empty fetch.
On the fleet scale benchmark, round 4 (2026-10, PREKIT's Timescale image on an amd64 VM, hub stack pinned to 6-10 cores, a 13.4 M-record backlog):
| one page at a time | PIPELINE_PAGES=4 |
|
|---|---|---|
| catch-up, 6 / 8 / 10 cores | 95 / 97 / 98 k rows/s | 125 / 128 / 127 k rows/s |
| live, 112 k records/s offered, 8 / 10 cores | 82 / 84 k rows/s, falling behind | 105 / 107 k rows/s, keeping up |
Eight pages instead of four, or eight writers instead of five, changed
nothing. At ~127 k rows/s the historian makes 25 /fetch requests a second,
the per-caller limit, and Postgres spends ~25 µs of CPU per row with most of
its writers busy (wait events: 77 % CPU, 10 % WALWrite); 20 000-record pages
(an experiment, /fetch allows 5000) gave only 130-134 k.
Signals the historian does not store¶
colca-historian writes no rows for samples of a signal whose _Signal
definition says "is_logged": false. A signal without the field, a sample whose
signal has no definition on the node, and an undecodable definition are all
stored: leaving history out needs an explicit false.
The historian loads the _Signal records once from /kv and then follows the
entities stream with its own cursor, c/historian/signals, woken by /watch.
The flag in force when a sample is ingested decides; a later change of the
flag does not rewrite or backfill history. Skipped samples are consumed: the
metrics marker and cursor move past them like past written rows. On startup no
sample is read before the definitions are loaded.
is_logged only affects this historian. It does not change what the node
stores, replicates (that is replication_policy, above) or delivers over MQTT.
/metrics reports colca_historian_samples_not_logged_total (samples consumed
without a row) and colca_historian_signals_not_logged (signals currently
marked). /healthz fails with signal_definitions while the definitions
cannot be loaded or followed.
Late writes the historian marks¶
A reader that keeps something computed from historian_metric per hour (PREKIT's
per-signal statistics) has to learn when a sample lands in an hour it already
computed: a child catching up after an outage, a replay, an import, a corrected
value. colca-historian tells it in the database.
The reader owns two relations, and its migrations create them before
colca-historian runs (PREKIT starts the historian after its api, which
migrates first):
CREATE TABLE historian_late_write (
id bigserial PRIMARY KEY,
hour timestamptz NOT NULL, -- a UTC hour
signal_ids jsonb NOT NULL); -- the signals written into it, a JSON array
-- the signals to mark: a table or a view with a signal_id column
CREATE VIEW historian_late_write_signal AS SELECT signal_id FROM ...;
For every page it writes, the historian appends one row to
historian_late_write per UTC hour the page has a sample of, when that hour
started 65 minutes ago or longer by the database's clock, naming the listed
signals it wrote into that hour, and sends NOTIFY historian_late_write when
it appended one. This happens in the transaction that writes the samples: a
sample is never stored without its mark, and a mark is never visible before
its samples. An offline colca-historian import marks the same way. The
historian writes no page unmarked, so it checks for both relations when it
starts and exits at once, naming the missing one, when the reader's migrations
have not run (PREKIT: the api's historian migration 0005_metric_statistics).
It does not wait for them; start it after the migrations.
Marking never holds ingestion up: it is a plain insert, with no unique key, no conflict handling and no table lock. A signal that is not listed writes nothing, and a page of current samples does not run the statement at all. Several pages writing into one hour leave one row each; the reader merges them.
The reader's side of the contract:
- Handling marks: read committed marks, recompute their (signal, hour)
pairs from
historian_metric, then delete the marks it read, byid. A mark it can read was committed with its samples, so the recompute sees them; a mark committed meanwhile stays for the next pass. A crash in between leaves the marks, and recomputing again is harmless. - Closing an hour: no earlier than 65 minutes after its start by the
database's clock, take a transaction id (
pg_current_xact_id()), then wait untilpg_snapshot_xmin(pg_current_snapshot())is at least that id before reading the hour. Every page that decided by then that it was not late has written its rows, so it already has a smaller transaction id; the wait ends when all of those committed or rolled back. Pages are never made to wait: the reader waits, bounded, and tries again later. - Listing a new signal: commit it to the list, then wait the same way before reading the signal's history. A page whose mark statement ran before the listing committed has an older transaction id; every later one sees the listing.
Permanent standalone handover¶
standalone: true is a permanent trust transition, distinct from an offline
parent. Remove parent in the same configuration and restart the node. Before
opening any door the node journals the transition, retires the old parent
cursors and existing externally enrolled identities, and disables the static
administration token. Local services, node identity and process history remain.
External machine clients must be enrolled again by the new local owner.
Commands accepted before handover are retired from HTTP fetch and MQTT replay, including pending local commands. Their stored history and acknowledgements are preserved; new commands remain executable. The retirement boundary survives restarts. Upgrading an existing standalone data directory from 0.2.0 records this boundary once at the first startup, retiring any commands pending at that time.
Human authentication stays closed until a trusted local identity controller has
preserved operators, disabled fleet accounts and transferred administration. The
controller then calls POST /standalone/complete on the local HTTP door.
Completion persists a token issuance cutoff; retries and restarts retain it.
GET /self exposes standalone_since (Unix seconds) and standalone_ready so
other local authentication services enforce the same boundary.
Pre-handover JWTs and personal access tokens are rejected. Only locally authored group definitions and newly created local PATs authorize subsequent access. Cached fleet definitions remain available as data, so an identity controller can copy the last applied operator grants into new, locally owned groups. It must not reuse the fleet group's identity for a competing local definition.
Removing the standalone flag or restoring an old parent configuration against this data directory refuses startup. Rejoining a fleet requires an explicit migration and enrollment plan, including a decision about retained history; a configuration rollback cannot export history or restore the former owner.
This broker transition does not administer an external identity provider, host VPN, SSH access or separate update agents. The deployment's handover controller must retire those connections and credentials before reporting the machine as handed over. Ordinary parent outages do not trigger any of these actions.
Importing historical measurements¶
colca-historian import is an operator tool for archived measurements. It uses
the historian's DATABASE_URL and ordinary sink; no broker connection is opened.
It does not update retained live values, dispatch events to control consumers,
or advance the running historian's stream cursor. Applications should resolve
their existing signal identities through their normal public API before export.
Input is JSONL, one {"topic": "colca/v1/_Metric/NODE/PATH", "payload": {...}}
per line. Each payload must have a signal ULID, explicit timestamp in Unix
seconds, and a value. The topic's node must be a ULID and agree with any payload
node identity. Use COLCA_TOPIC_ROOT for a deployment with another root.
colca-historian import --file archive.jsonl --sha256 EXPECTED_SHA256 \
--before 2026-01-01T00:00:00Z --dry-run
# Inside the historian's trusted deployment environment:
colca-historian import --file archive.jsonl --sha256 EXPECTED_SHA256 \
--before 2026-01-01T00:00:00Z
The complete input is validated and hashed before database access. A private
temporary spool prevents input changes between validation and writing. Imports
use strict transactions of up to 1000 rows by default (--batch-size, maximum
5000), with a separate historian:import:SHA256 marker. Rerunning the identical
file resumes committed batches. Errors stop the import without acknowledging
the failed batch. This has the normal sink's upsert semantics: the same signal
and timestamp replaces that historical point. Choose a ceiling before existing
live data and check for overlapping records when replacement is unintended.
The ordinary historian schema must already exist; import does not change
schema or retention. Ensure the configured retention covers the imported dates.