Skip to content

HTTP API

JSON in, JSON out. The API door speaks HTTPS with the node's self-signed certificate; the local door speaks plain HTTP inside the deployment network.

A caller is one of:

  • a machine or external service, identified by its client certificate; reads are limited to its grants and cursor names are prefixed with its id,
  • a person, with Authorization: Bearer <token>; reads are limited to read: grants, and the only thing a person publishes is a command,
  • a local service on the local door, named by X-Colca-Service,
  • the administrator, with X-Colca-Token: <api.token>.

Reading and writing

Method and path Who Request Response
GET /healthz anyone {"ok":true,"ulid":"…","storage":{"state":"ok"},"store":{"id":"…"},"uplink":{"state":"connected","since":"…"}}; store.changed and uplink.resync appear after a restore
GET /metrics anyone Prometheus text
POST /publish machine, service, person (commands), admin {"topic":"…","payload":{…}} {"stream":"…","offset":N,"topic":"…"}, or 202 {"stream":"logs","topic":"…","withheld":"…"} without offset
POST /publish/batch machine, service {"records":[{"topic":"…","payload":{…}},…]} (1–5000 records, 16 MiB) {"accepted":N,"results":[{"stream":"…","offset":N} or {"stream":"logs","withheld":"…"} or {"error":"…","reason":"…"},…]}
GET /fetch machine, service, person, admin ?stream=S&cursor=NAME&max=100&prefix=P&contract=_Annotation&from=N {"records":[{"offset":N,"topic":"…","payload":{…},"ts":T}],"next":N,"from":N,"head":N,"more":false,"store":"ID"}
GET /watch machine, service, person, admin ?stream=S&stream=S2&interval_ms=100 NDJSON, one line per change: {"streams":["S"],"next":{"S":N}}
POST /ack owner of the cursor, admin; to retire a stale cursor also cmd:<node>/#:admin {"cursor":"NAME","stream":"S","offset":N} (optionally "store":"ID"), or {"cursor":"NAME","stream":"S","delete":true} to retire it {"moved":true} or {"deleted":true}; 409 store changed, 422 past the head
GET /backlog local service (?prefix= required), admin ?prefix=c/projector/ {"queues":[{"cursor":"…","stream":"S","position":N,"head":N,"lag_records":N,"last_ack_ms":T,"stale":false,"read_since_start":true}]}
GET /kv machine, service, person, admin ?prefix=P&max=1000&after=TOKEN&contract=_Signal&depth=1 {"entries":[{"path":"…","node_id":"…","topic":"…","payload":{…},"ts":T,"offset":N}],"next":"TOKEN"}
POST /kv/lookup machine, service, person, admin body {"topics": ["…"]}, at most 1000 {"entries":[…]}, each entry shaped as /kv's (attribution included) — the current entry of each named topic the caller may read; topics with no entry are left out
GET /self local service the service's registry entry, limits, standalone_since and standalone_ready
POST /standalone/complete local service on a standalone node finish the identity handover; returns its durable issuance cutoff
  • /publish/batch judges every record as /publish would and writes the admitted ones with one append per stream, in order; a refused record does not stop the others. Commands and audit records are refused in a batch. For a high-rate publisher, such as a bridge relaying a plant: one request per batch instead of one per sample.
  • A refused record of a batch, and a refused /publish, names its reason. not_written means the record was admitted but the store did not write it (a failed write, or another record of the same append was over the record cap): send it again. Every other reason is a verdict on the record itself, and the same record is refused again: too_large and the reasons of colca_rejected_publishes_total (validation, grammar, node_id, identity, write_denied, not_producer, …), except draining, which passes.
  • A _Log record the log gate accepts without storing it is answered 202 with {"withheld":"collapsed"|"rate_limited", "stream":"logs","topic":"…"} (in a batch, {"stream":"logs","withheld":…}, counted in accepted). Such an answer has no offset, since nothing was stored; a client reads offset only when withheld is absent. It is not an error and must not be sent again: a collapsed repeat is counted in its window's summary record, a record over the service's budget in the drop notice. Over MQTT the same record gets a successful PUBACK and is not fanned out.
  • /fetch never moves a cursor. /ack takes the last offset you processed and only moves forward.
  • An /ack past the stream's head (an offset at or above the next offset) is refused with 422 for every caller: it would skip records not yet written.
  • /fetch names the store the page was read from (store, new whenever the node's store is created from nothing, for instance after its data volume was recreated, and whenever its history may have regressed, as after a restore from a backup; see Store incarnations). A consumer that passes it back as "store" on /ack gets 409 with "store_changed":true when the node's store is another one by then: the acked offsets name other records there, and the consumer should read the new store from its cursor instead. store is optional on /ack; clients that do not send it (chaski, the TypeScript client) are not checked. Nodes before this release send no store.
  • from=N reads ahead of the cursor, starting at offset N, so a consumer can fetch its next page (from = the previous page's next) while it still processes and acks the previous one. It never reads behind the cursor. The response's from is where the page started; nodes before 0.18.2 ignore the parameter and do not send it.
  • A service's own state (_ServiceState, see Topics) is read and written only by that service and the admin door. Every other caller's /kv, /kv/lookup and /fetch leave it out, whatever its grants.
  • prefix filters on the path part of the topic, not the raw topic.
  • topic on /fetch (repeatable, colca 0.19+) keeps records whose raw topic matches one of the MQTT filters (+, #). A consumer woken by a set of topics passes the same set, so it reads exactly what wakes it. Skipped records move next like contract does; ack next - 1 after an empty or short page so they do not stay unread on your cursor.
  • max defaults to 100 for /fetch (at most 5000) and to 1000 for /kv (at most 10000). A max that is not a whole number in that range is a 400 naming the range (max must be between 1 and 5000); before colca 0.33.4 /fetch quietly served 100 records instead. Pass next back as after until it is empty.
  • A page also ends before the record or entry that would take it past 8 MiB of stored data, and next resumes at that record or entry. A page always holds at least one. A short page is therefore no sign of the end: for /fetch that is next reaching the head, for /kv an empty next.
  • /fetch reports head, the stream's next offset when the page was read, and more, true when the page stopped before it. A filtered page also stops after the node examined 20,000 records, so it can be empty with more: true: nothing matched yet, read on from next at once. more: false means caught up; wait for the next /watch hint.
  • contract on /kv and /fetch may be repeated. An unknown name is a 400. On /kv the node keeps an index by contract, so a filtered read costs what it returns, not what lies under the prefix. On /fetch the other records are skipped and next moves past them.
  • depth=N on /kv keeps entries at most N path segments below prefix (prefix=plant/&depth=1 is the level below plant); deeper subtrees are skipped, not read.
  • folders=true (needs depth) adds "folders":["plant/l1",…] to the /kv page: the paths at the cut that have deeper entries, whether or not they hold a record themselves, so a tree view reads one level per call and knows which rows expand. Each folder appears once, on the page where its subtree is skipped, counts towards max, and ignores contract. A caller sees a folder its read zones cover or lead to.
  • Payloads are passed through as raw JSON; numbers keep the exact form the publisher sent.
  • records and entries are always arrays.
  • A record or entry carries written_by, actor_id, actor_label and actor_kind when the write that produced it named them (a person's token, or a service acting for one); a write that carried none omits all four. /kv's entry is the retained projection of the same write /fetch returns on the stream, so both carry the same four fields.
  • written_by names who is authenticated for the write, not necessarily who decided it: a machine or service publishing directly is its own written_by, but the state a _CmdEdit or _CmdConfigure command produces is written_by the node that executed it — the node is what physically appended the record — while actor_id/actor_label/actor_kind still name the person or service that commanded it. An operator's _CmdEdit on their own constant (see security.md) is the case this exists for: /kv can show "set by \<operator> at \<time>" from Colca's own verified identity, not a client-written field.

Who a record names

actor_id is the identity. For a person or a Keycloak service account it is the token's sub, which stays the same when the account is renamed; for a machine or a local service it is its registry id. Authorization, command deduplication and annotation authorship all go by it and by the token's grants, never by a name.

actor_label is optional. The door that verifies a person's token records what the token called them at that moment: the name claim, else preferred_username, and nothing when the token has neither (a personal access token records its owner's email). A local service that acts for a person passes the label it knows, or none. It is a hint for a reader that cannot resolve the subject, and it is history: a record written before a rename keeps the old name. A reader that shows who acted should resolve the current name from actor_id in one place and fall back to the label only when the subject can no longer be resolved.

actor_kind is human for every token door identity, service for a machine or a local service, node, system for the admin token and the node's own writes, or anonymous.

Whether a token belongs to a Keycloak service account is decided once, at the door that verified it, from the token's preferred_username (service-account-<client-id>). The node keeps the client id with the command and replicates it with the command's groups, so the node that executes it knows a service account's annotations (service/<client-id>) from a person's (user/<sub>) without reading the label. /fetch and /kv do not return it.

Nodes of different versions in one tree keep working. A node from before this rule recorded preferred_username as the label and the subject when the token had none; such records read as before. It ignores the client id it is sent and recognises a service account by its label, as it always did. A command that reaches a newer node through an older parent arrives without a client id, so there it counts as a person's: a service account then needs configure to change an annotation it authored, until the parent is upgraded.

actor_email is on a /fetch record of a command only, and only when the person sent it with a token whose email_verified claim is the boolean true (what Keycloak emits): it is the token's email, kept with the command and replicated with it to the node that executes it, for the service that has to reach the person -- an alarm subscription that mails them (alarms.md). A personal access token, a service account, the admin door and a local service naming a person never set it; /kv and every state or event record omit it. It authorizes nothing.

actor_classes is on a /fetch record of a command only: the classes the node that admitted it witnessed the person holding at the command's place, sorted (read when their read zones cover it, and each command class a grant covers there). The place is the alarm's for a subscription, else the command's path. It is set where the person's own grants authorized the command (a token door, or a local service attesting their groups), never from a request body or at the admin door, and a node the command came down to keeps the value it was sent. The service applying the command may rely on it instead of authorizing again.

Both are handed only to the node's own unplaced services and the admin token (uns.Entry.SeesActorWitness). A person or a machine whose read grants cover a command reads the command on /fetch without them. MQTT delivery carries no attribution at all, only topic and payload.

When a cursor stands below what retention has already removed, the response carries a gap object that names the missing offsets and times, and records continue after the gap:

{
  "records": [ … ],
  "next": 50123,
  "gap": { "stream": "metrics", "from_offset": 57, "to_offset": 49999,
           "first_ts": 1755100000000, "last_ts": 1755700000000, "approx": false }
}

Acknowledge gap.to_offset to move past it.

A cursor that was never acked has no position below anything: its first /fetch starts at the stream's oldest retained record (from is the low-water mark) and carries no gap, so a consumer added after retention ran starts cleanly. Only a saved position, or a from= read-ahead position, below the low-water mark is a gap.

Waiting for new records

A consumer that follows a stream does not need to poll /fetch while the stream is idle. GET /watch?stream=entities&stream=annotations holds the connection open and writes one JSON line whenever a selected stream grows:

{"streams":["entities","annotations"],"next":{"entities":1201,"annotations":88}}
{"streams":["annotations"],"next":{"annotations":91}}
{"streams":[]}
  • The first line names every selected stream: drain them all after each (re)connect, and a hint missed while disconnected costs nothing.
  • next is the stream's next offset. A consumer whose cursor already stands there can skip the fetch.
  • Lines are at least interval_ms apart (default 100, at most 10000); streams that grow in between are merged into one line and named once. A busy stream therefore costs at most one line per interval.
  • A line with no streams is a heartbeat, written after 5 s of silence. Treat 15 s without a line as a dead connection and reconnect.
  • A hint carries no records and moves no cursor; read with /fetch and /ack as before. Take no timed fallback poll: a consumer that stops reading is caught by the cursor watchdog below, not hidden by a poll.

Consumers that stop reading

A consumer reads when woken: an MQTT message on its topics, a /watch hint, a reconnect. Nothing reads on a timer, so a lost wake or a stuck loop would leave records waiting unseen. The node watches every cursor instead:

  • Every few seconds it takes the oldest record past the cursor that the consumer reads (its last /fetch filter applied) and reports its age as colca_cursor_unread_age_seconds{cursor,stream}. An idle stream reads 0: only records that wait count, not how old the last one is.
  • When that age passes cursors.lag_alarm_after (default 60 s) it writes a retained _Finding with reason cursor_lag next to the service's own record, _Finding/{node}/{mount}/{service}/cursor_lag, and retires it once the cursor has caught up. The alarm path raises it like any other finding.
  • Only a cursor fetched since the node started raises a finding. A cursor nobody fetches is abandoned (an old buffer generation, a renamed consumer): the gauge and colca_retention_blocked_by_cursor show it. Retire it.
  • A cursor that stood still for cursors.stale_after (default 24 h) while records wait past it is named in the node's _Finding/{node}/stale_cursors, read or not: it holds back retention either way. The finding is retired once no cursor is stale.

Cursors that nobody reads

A consumer that is removed leaves its cursors behind, and every cursor protects its stream from the pruner. Nothing removes them on its own.

  • List. GET /backlog with the admin token, without a prefix, lists every cursor; a local service lists by prefix. Each row has last_ack_ms (when it last moved, 0 when unknown), stale (by cursors.stale_after) and read_since_start (whether anyone fetched it since the node started).
  • Retire. POST /ack {"cursor":…,"stream":…,"delete":true} as the cursor's owner or the admin. A holder of the admin command class over the whole node (cmd:<node>/#:admin, cmd:$node/#:admin or cmd:#:admin) may retire another identity's cursor (c/<name>/…, <ulid>/…) too, once it is stale as GET /backlog reports it (409 before), and each such retirement is written to the audit stream (outcome: success, the actor, the cursor and its owner). It may not move one, nor retire a command delivery floor (…/cmd), a node-owned cursor (downlink:…, down-def:…) or a cursor of colca's own services (c/historian/…). The node logs who retired it. A consumer that fetches the cursor again starts it over at the stream's oldest retained record, without a gap.
  • Expire. retention.streams.<stream>.ignore_cursors_after lets the pruner pass a cursor that has not moved for that long, with a _StreamGap record a returning consumer sees. Off unless configured.
  • A live consumer that filters its fetch must ack the page's last scanned offset (next - 1) also when the page held nothing of its own, or its cursor stands still and turns stale.
  • A service subscribes to its own finding and fails its health check while it stands. chaski and colca-historian do so.

Administration

Method and path Request Response
POST /enroll {"ulid","pubkey","kind":"external"\|"node","element","grants":[…]} {"ulid":"…","offset":N}; 409 when the key or the element is already taken, 422 on an invalid entry or an element this node does not hold
GET /enroll ?max=1000&after=TOKEN the locally enrolled entries, public keys only
DELETE /enroll/{ulid} ?retire=true {"revoked":true,"offset":N}; the same batch retires the records the identity authored about itself — its _ServiceDetails, at every mount it published one at. Only that identity may write them, so one left behind could never be retired by anyone. For a local service it also retires the catalogues (_DataTags) and findings (_Finding) the service wrote and deletes its cursors on the commands stream (c/<name>/...), since the name registers again under a new identity; its _Metric values and other cursors stay. A plain revoke keeps what a child node replicated up: the child may be enrolled again and resumes from its own cursor. With retire=true (kind node only, 409 otherwise; 400 on a value that is not a boolean) the same batch also tombstones every current-state record the child and the nodes below it replicated, clears its replication marks, and deletes its definitions cursor (downlink-def:<ulid>) and every cursor in its own namespace (<ulid>/...), so nothing it left stands as live or holds retention and the same identity enrolled later starts clean (a plain revoke keeps the definitions cursor as the child's catch-up position); the response adds "retired":true,"records_retired":N. The tombstones replicate up, so ancestors retire their copies too
POST /enroll/{ulid}/drain {"ulid","offset","status":"draining"}; decommissions a child node: new commands under its mount are refused, and once its queue is delivered or expired it is retired as with DELETE ?retire=true. 409 for an entry that is not a node or is already draining
POST /store/incarnation {"store":"NEW","previous":"OLD"}; gives the node's store a new incarnation id, so every child re-syncs with it, its parent clears the marks kept for it and local consumers get store_changed on /ack. For a store restored or replaced without a neighbour noticing; see Restoring a parent from a backup
GET /uplink/set-aside {"records":[{"parent","stream","offset","record","status","reason","held_since_ms","at_ms"}]}; the records this node's uplink set aside because the parent refused each alone past the grace. They are not at the parent; see Records the parent refuses. Only on a node with a parent
DELETE /uplink/set-aside/{stream}/{offset} {"cleared":true}, 404 when nothing is set aside there; clears the copy, and the uplink_set_aside finding once none is left
GET /debug/state the next offset of every stream

A node that is reachable only through the tree is administered with _CmdAdmin commands sent to it from an ancestor: enroll, revoke, and fetchLogs, which returns one page of its own logs in the ack's result. See Fetching a node's logs.

A local service whose registry entry holds cmd:#:admin may send the same commands to its own node over the local door, at _CmdAdmin/<node>/<verb>. The node runs them at once: /publish returns the outcome in command, and the _Ack lands at _Ack/<node>/<verb>. Without the grant the answer is 403. POST /enroll stays on the published door.

Secrets

Available when secrets_dir is set. The owner is the calling service; a local service cannot address another service's secrets.

Method and path Request Response
PUT /secrets/{owner}/{name} {"envelope":{…},"expires_at":null,"expected_revision":N} metadata, never the ciphertext
GET /secrets/{owner} ?max=1000&after=TOKEN metadata page
GET /secrets/{owner}/{name} metadata; 410 once expired
DELETE /secrets/{owner}/{name} ?expected_revision=N {"deleted":true}

expected_revision of 0 means create only; any other value is a compare-and-swap.

Errors and limits

Status Meaning
202 /publish: a _Log record the log gate collapsed or capped; accepted, not stored
400 malformed request
401 a presented credential was not accepted
403 the caller may not do this
413 body too large; a record over the record cap has reason too_large
422 bad topic, unknown contract or invalid payload, with reason when the node names one
429 a request limit; code in the body says which (below)
503 /publish: the record was admitted but not written (reason not_written); send it again

Every route belongs to a rate class. Authenticated callers get their own quota; the local door and unauthenticated traffic are limited per source address. Per caller: /kv 5 requests a second (burst 10), /fetch 25 (burst 50) and 16 open at a time, /watch 2 new connections a second (burst 8) and 8 open at a time.

A 429 body names the bound that refused the request:

{"error": "too many concurrent requests: …", "code": "concurrency_limit", "class": "watch", "limit": 8}
code Meaning What helps
rate_limit the caller's request rate for the class is used up wait Retry-After seconds (only this code sets it)
concurrency_limit the caller already holds limit open requests of the class close one, e.g. a /watch it no longer reads; retrying sooner does not help
global_concurrency_limit all callers together hold limit open requests of the class back off with your own delay
caller_limit the node tracks as many callers as it may back off with your own delay

Only rate_limit carries Retry-After: for the others no wait is known, so a client backs off on its own schedule instead of retrying every second against a bound that waiting does not clear. The Go door client (door.RetryDelay) falls back to its jittered backoff when the header is absent.

MQTT reason codes

Over MQTT 5 at QoS 1 or higher, a refused publish is answered in the PUBACK: 0x90 unknown topic or contract, 0x99 invalid payload, 0x87 not authorized, 0x89 node is draining. MQTT 3.1.1 has no reason codes; the publish is dropped and counted in colca_rejected_publishes_total.

The clock in /healthz

/healthz includes clock, what the node knows about its time:

{"wall_now_ms": 1760272496123, "is_root": false,
 "host": {"synchronized": true, "estimated_error_ms": 0.4,
          "max_error_ms": 212.5, "stale_bound_ms": 0},
 "parent_skew_ms": -12, "parent_sample_age_s": 3.2}

wall_now_ms is the host's wall clock, the only time the node decides on. host is the kernel's verdict on that clock; its fields are null where it cannot be read. estimated_error_ms is also null on a host whose NTP service keeps no estimate (systemd-timesyncd): null is "not reported", not an error of zero. max_error_ms is the kernel's bound on the error and stale_bound_ms the part of it beyond one NTP poll interval, which grows while the clock is not updated. parent_skew_ms is the parent's clock minus this node's, measured on the latest replication response whose round trip was at most 0.5 s, and parent_sample_age_s its age; both are null on a root and until there is such a response. See Clocks.

Local stream wakeups and backlog telemetry

Internal services may subscribe to GET /watch?stream=metrics&stream=entities on the local door with their existing service identity. The NDJSON response starts with a catch-up hint and then emits {"streams":["metrics"]} after successful durable commits. Five-second heartbeats contain no stream names. Hints can coalesce: capture the local wakeup version before fetching, drain the durable cursor to empty, then wait on that version. A disconnect requires reconnecting and draining again; a hint never acknowledges or contains records.

This route is not on the published API and does not accept forwarded human bearer identities. Streaming clients must use the heartbeat/read deadline rather than an ordinary short whole-request timeout. /fetch and /ack retain their normal ownership rules. Fixed minimum spacing between drain starts can batch bursts; nonempty pages within a drain do not need an additional delay.

GET /backlog?prefix=c/projector/ on the same local door returns selected durable cursor positions, stream heads and lag_records, with each cursor's last_ack_ms, stale and read_since_start. It is read-only. Values count outstanding offset distance (an upper bound after compaction), not bytes. Up to 32 nonempty prefixes and 256 matching cursors are accepted; excess results fail instead of truncating away a potentially overloaded consumer. A missing expected cursor is not evidence of an empty queue.