HTTP API¶
JSON in, JSON out. The API door speaks HTTPS with the node's self-signed certificate; the local door speaks plain HTTP inside the deployment network.
A caller is one of:
- a machine or external service, identified by its client certificate; reads are limited to its grants and cursor names are prefixed with its id,
- a person, with
Authorization: Bearer <token>; reads are limited toread:grants, and the only thing a person publishes is a command, - a local service on the local door, named by
X-Colca-Service, - the administrator, with
X-Colca-Token: <api.token>.
Reading and writing¶
| Method and path | Who | Request | Response |
|---|---|---|---|
GET /healthz |
anyone | {"ok":true,"ulid":"…","storage":{"state":"ok"},"store":{"id":"…"},"uplink":{"state":"connected","since":"…"}}; store.changed and uplink.resync appear after a restore |
|
GET /metrics |
anyone | Prometheus text | |
POST /publish |
machine, service, person (commands), admin | {"topic":"…","payload":{…}} |
{"stream":"…","offset":N,"topic":"…"}, or 202 {"stream":"logs","topic":"…","withheld":"…"} without offset |
POST /publish/batch |
machine, service | {"records":[{"topic":"…","payload":{…}},…]} (1–5000 records, 16 MiB) |
{"accepted":N,"results":[{"stream":"…","offset":N} or {"stream":"logs","withheld":"…"} or {"error":"…","reason":"…"},…]} |
GET /fetch |
machine, service, person, admin | ?stream=S&cursor=NAME&max=100&prefix=P&contract=_Annotation&from=N |
{"records":[{"offset":N,"topic":"…","payload":{…},"ts":T}],"next":N,"from":N,"head":N,"more":false,"store":"ID"} |
GET /watch |
machine, service, person, admin | ?stream=S&stream=S2&interval_ms=100 |
NDJSON, one line per change: {"streams":["S"],"next":{"S":N}} |
POST /ack |
owner of the cursor, admin; to retire a stale cursor also cmd:<node>/#:admin |
{"cursor":"NAME","stream":"S","offset":N} (optionally "store":"ID"), or {"cursor":"NAME","stream":"S","delete":true} to retire it |
{"moved":true} or {"deleted":true}; 409 store changed, 422 past the head |
GET /backlog |
local service (?prefix= required), admin |
?prefix=c/projector/ |
{"queues":[{"cursor":"…","stream":"S","position":N,"head":N,"lag_records":N,"last_ack_ms":T,"stale":false,"read_since_start":true}]} |
GET /kv |
machine, service, person, admin | ?prefix=P&max=1000&after=TOKEN&contract=_Signal&depth=1 |
{"entries":[{"path":"…","node_id":"…","topic":"…","payload":{…},"ts":T,"offset":N}],"next":"TOKEN"} |
POST /kv/lookup |
machine, service, person, admin | body {"topics": ["…"]}, at most 1000 |
{"entries":[…]}, each entry shaped as /kv's (attribution included) — the current entry of each named topic the caller may read; topics with no entry are left out |
GET /self |
local service | the service's registry entry, limits, standalone_since and standalone_ready |
|
POST /standalone/complete |
local service on a standalone node | finish the identity handover; returns its durable issuance cutoff |
/publish/batchjudges every record as/publishwould and writes the admitted ones with one append per stream, in order; a refused record does not stop the others. Commands and audit records are refused in a batch. For a high-rate publisher, such as a bridge relaying a plant: one request per batch instead of one per sample.- A refused record of a batch, and a refused
/publish, names itsreason.not_writtenmeans the record was admitted but the store did not write it (a failed write, or another record of the same append was over the record cap): send it again. Every other reason is a verdict on the record itself, and the same record is refused again:too_largeand the reasons ofcolca_rejected_publishes_total(validation,grammar,node_id,identity,write_denied,not_producer, …), exceptdraining, which passes. - A
_Logrecord the log gate accepts without storing it is answered202with{"withheld":"collapsed"|"rate_limited", "stream":"logs","topic":"…"}(in a batch,{"stream":"logs","withheld":…}, counted inaccepted). Such an answer has nooffset, since nothing was stored; a client readsoffsetonly whenwithheldis absent. It is not an error and must not be sent again: a collapsed repeat is counted in its window's summary record, a record over the service's budget in the drop notice. Over MQTT the same record gets a successful PUBACK and is not fanned out. /fetchnever moves a cursor./acktakes the last offset you processed and only moves forward.- An
/ackpast the stream's head (an offset at or above the next offset) is refused with422for every caller: it would skip records not yet written. /fetchnames the store the page was read from (store, new whenever the node's store is created from nothing, for instance after its data volume was recreated, and whenever its history may have regressed, as after a restore from a backup; see Store incarnations). A consumer that passes it back as"store"on/ackgets409with"store_changed":truewhen the node's store is another one by then: the acked offsets name other records there, and the consumer should read the new store from its cursor instead.storeis optional on/ack; clients that do not send it (chaski, the TypeScript client) are not checked. Nodes before this release send nostore.from=Nreads ahead of the cursor, starting at offsetN, so a consumer can fetch its next page (from= the previous page'snext) while it still processes and acks the previous one. It never reads behind the cursor. The response'sfromis where the page started; nodes before 0.18.2 ignore the parameter and do not send it.- A service's own state (
_ServiceState, see Topics) is read and written only by that service and the admin door. Every other caller's/kv,/kv/lookupand/fetchleave it out, whatever its grants. prefixfilters on the path part of the topic, not the raw topic.topicon/fetch(repeatable, colca 0.19+) keeps records whose raw topic matches one of the MQTT filters (+,#). A consumer woken by a set of topics passes the same set, so it reads exactly what wakes it. Skipped records movenextlikecontractdoes; acknext - 1after an empty or short page so they do not stay unread on your cursor.maxdefaults to 100 for/fetch(at most 5000) and to 1000 for/kv(at most 10000). Amaxthat is not a whole number in that range is a400naming the range (max must be between 1 and 5000); before colca 0.33.4/fetchquietly served 100 records instead. Passnextback asafteruntil it is empty.- A page also ends before the record or entry that would take it past 8 MiB
of stored data, and
nextresumes at that record or entry. A page always holds at least one. A short page is therefore no sign of the end: for/fetchthat isnextreaching the head, for/kvan emptynext. /fetchreportshead, the stream's next offset when the page was read, andmore, true when the page stopped before it. A filtered page also stops after the node examined 20,000 records, so it can be empty withmore: true: nothing matched yet, read on fromnextat once.more: falsemeans caught up; wait for the next/watchhint.contracton/kvand/fetchmay be repeated. An unknown name is a400. On/kvthe node keeps an index by contract, so a filtered read costs what it returns, not what lies under the prefix. On/fetchthe other records are skipped andnextmoves past them.depth=Non/kvkeeps entries at mostNpath segments belowprefix(prefix=plant/&depth=1is the level belowplant); deeper subtrees are skipped, not read.folders=true(needsdepth) adds"folders":["plant/l1",…]to the/kvpage: the paths at the cut that have deeper entries, whether or not they hold a record themselves, so a tree view reads one level per call and knows which rows expand. Each folder appears once, on the page where its subtree is skipped, counts towardsmax, and ignorescontract. A caller sees a folder its read zones cover or lead to.- Payloads are passed through as raw JSON; numbers keep the exact form the publisher sent.
recordsandentriesare always arrays.- A record or entry carries
written_by,actor_id,actor_labelandactor_kindwhen the write that produced it named them (a person's token, or a service acting for one); a write that carried none omits all four./kv's entry is the retained projection of the same write/fetchreturns on the stream, so both carry the same four fields. written_bynames who is authenticated for the write, not necessarily who decided it: a machine or service publishing directly is its ownwritten_by, but the state a_CmdEditor_CmdConfigurecommand produces iswritten_bythe node that executed it — the node is what physically appended the record — whileactor_id/actor_label/actor_kindstill name the person or service that commanded it. An operator's_CmdEditon their own constant (see security.md) is the case this exists for:/kvcan show "set by \<operator> at \<time>" from Colca's own verified identity, not a client-written field.
Who a record names¶
actor_id is the identity. For a person or a Keycloak service account it is
the token's sub, which stays the same when the account is renamed; for a
machine or a local service it is its registry id. Authorization, command
deduplication and annotation authorship all go by it and by the token's
grants, never by a name.
actor_label is optional. The door that verifies a person's token records
what the token called them at that moment: the name claim, else
preferred_username, and nothing when the token has neither (a personal
access token records its owner's email). A local service that acts for a
person passes the label it knows, or none. It is a hint for a reader that
cannot resolve the subject, and it is history: a record written before a
rename keeps the old name. A reader that shows who acted should resolve the
current name from actor_id in one place and fall back to the label only
when the subject can no longer be resolved.
actor_kind is human for every token door identity, service for a
machine or a local service, node, system for the admin token and the
node's own writes, or anonymous.
Whether a token belongs to a Keycloak service account is decided once, at the
door that verified it, from the token's preferred_username
(service-account-<client-id>). The node keeps the client id with the
command and replicates it with the command's groups, so the node that
executes it knows a service account's annotations (service/<client-id>)
from a person's (user/<sub>) without reading the label. /fetch and /kv
do not return it.
Nodes of different versions in one tree keep working. A node from before this
rule recorded preferred_username as the label and the subject when the token
had none; such records read as before. It ignores the client id it is sent
and recognises a service account by its label, as it always did. A command
that reaches a newer node through an older parent arrives without a client
id, so there it counts as a person's: a service account then needs
configure to change an annotation it authored, until the parent is
upgraded.
actor_email is on a /fetch record of a command only, and only when the
person sent it with a token whose email_verified claim is the boolean true
(what Keycloak emits): it is the token's email, kept with the command and
replicated with it to the node that executes it, for the service that has to
reach the person -- an alarm subscription that mails them
(alarms.md). A personal access token, a
service account, the admin door and a local service naming a person never set
it; /kv and every state or event record omit it. It authorizes nothing.
actor_classes is on a /fetch record of a command only: the classes the
node that admitted it witnessed the person holding at the command's place,
sorted (read when their read zones cover it, and each command class a grant
covers there). The place is the alarm's for a subscription, else the command's
path. It is set where the person's own grants authorized the command (a token
door, or a local service attesting their groups), never from a request body
or at the admin door, and a node the command came down to keeps the value it
was sent. The service applying the command may rely on it instead of
authorizing again.
Both are handed only to the node's own unplaced services and the admin token
(uns.Entry.SeesActorWitness). A person or a machine whose read grants cover
a command reads the command on /fetch without them. MQTT delivery carries no
attribution at all, only topic and payload.
When a cursor stands below what retention has already removed, the response
carries a gap object that names the missing offsets and times, and records
continue after the gap:
{
"records": [ … ],
"next": 50123,
"gap": { "stream": "metrics", "from_offset": 57, "to_offset": 49999,
"first_ts": 1755100000000, "last_ts": 1755700000000, "approx": false }
}
Acknowledge gap.to_offset to move past it.
A cursor that was never acked has no position below anything: its first
/fetch starts at the stream's oldest retained record (from is the
low-water mark) and carries no gap, so a consumer added after retention ran
starts cleanly. Only a saved position, or a from= read-ahead position, below
the low-water mark is a gap.
Waiting for new records¶
A consumer that follows a stream does not need to poll /fetch while the
stream is idle. GET /watch?stream=entities&stream=annotations holds the
connection open and writes one JSON line whenever a selected stream grows:
{"streams":["entities","annotations"],"next":{"entities":1201,"annotations":88}}
{"streams":["annotations"],"next":{"annotations":91}}
{"streams":[]}
- The first line names every selected stream: drain them all after each (re)connect, and a hint missed while disconnected costs nothing.
nextis the stream's next offset. A consumer whose cursor already stands there can skip the fetch.- Lines are at least
interval_msapart (default 100, at most 10000); streams that grow in between are merged into one line and named once. A busy stream therefore costs at most one line per interval. - A line with no streams is a heartbeat, written after 5 s of silence. Treat 15 s without a line as a dead connection and reconnect.
- A hint carries no records and moves no cursor; read with
/fetchand/ackas before. Take no timed fallback poll: a consumer that stops reading is caught by the cursor watchdog below, not hidden by a poll.
Consumers that stop reading¶
A consumer reads when woken: an MQTT message on its topics, a /watch hint, a
reconnect. Nothing reads on a timer, so a lost wake or a stuck loop would leave
records waiting unseen. The node watches every cursor instead:
- Every few seconds it takes the oldest record past the cursor that the
consumer reads (its last
/fetchfilter applied) and reports its age ascolca_cursor_unread_age_seconds{cursor,stream}. An idle stream reads0: only records that wait count, not how old the last one is. - When that age passes
cursors.lag_alarm_after(default 60 s) it writes a retained_Findingwith reasoncursor_lagnext to the service's own record,_Finding/{node}/{mount}/{service}/cursor_lag, and retires it once the cursor has caught up. The alarm path raises it like any other finding. - Only a cursor fetched since the node started raises a finding. A cursor
nobody fetches is abandoned (an old buffer generation, a renamed consumer):
the gauge and
colca_retention_blocked_by_cursorshow it. Retire it. - A cursor that stood still for
cursors.stale_after(default 24 h) while records wait past it is named in the node's_Finding/{node}/stale_cursors, read or not: it holds back retention either way. The finding is retired once no cursor is stale.
Cursors that nobody reads¶
A consumer that is removed leaves its cursors behind, and every cursor protects its stream from the pruner. Nothing removes them on its own.
- List.
GET /backlogwith the admin token, without a prefix, lists every cursor; a local service lists byprefix. Each row haslast_ack_ms(when it last moved,0when unknown),stale(bycursors.stale_after) andread_since_start(whether anyone fetched it since the node started). - Retire.
POST /ack {"cursor":…,"stream":…,"delete":true}as the cursor's owner or the admin. A holder of the admin command class over the whole node (cmd:<node>/#:admin,cmd:$node/#:adminorcmd:#:admin) may retire another identity's cursor (c/<name>/…,<ulid>/…) too, once it is stale asGET /backlogreports it (409before), and each such retirement is written to the audit stream (outcome: success, the actor, the cursor and its owner). It may not move one, nor retire a command delivery floor (…/cmd), a node-owned cursor (downlink:…,down-def:…) or a cursor of colca's own services (c/historian/…). The node logs who retired it. A consumer that fetches the cursor again starts it over at the stream's oldest retained record, without a gap. - Expire.
retention.streams.<stream>.ignore_cursors_afterlets the pruner pass a cursor that has not moved for that long, with a_StreamGaprecord a returning consumer sees. Off unless configured. - A live consumer that filters its fetch must ack the page's last scanned
offset (
next - 1) also when the page held nothing of its own, or its cursor stands still and turns stale. - A service subscribes to its own finding and fails its health check while it
stands. chaski and
colca-historiando so.
Administration¶
| Method and path | Request | Response |
|---|---|---|
POST /enroll |
{"ulid","pubkey","kind":"external"\|"node","element","grants":[…]} |
{"ulid":"…","offset":N}; 409 when the key or the element is already taken, 422 on an invalid entry or an element this node does not hold |
GET /enroll |
?max=1000&after=TOKEN |
the locally enrolled entries, public keys only |
DELETE /enroll/{ulid} |
?retire=true |
{"revoked":true,"offset":N}; the same batch retires the records the identity authored about itself — its _ServiceDetails, at every mount it published one at. Only that identity may write them, so one left behind could never be retired by anyone. For a local service it also retires the catalogues (_DataTags) and findings (_Finding) the service wrote and deletes its cursors on the commands stream (c/<name>/...), since the name registers again under a new identity; its _Metric values and other cursors stay. A plain revoke keeps what a child node replicated up: the child may be enrolled again and resumes from its own cursor. With retire=true (kind node only, 409 otherwise; 400 on a value that is not a boolean) the same batch also tombstones every current-state record the child and the nodes below it replicated, clears its replication marks, and deletes its definitions cursor (downlink-def:<ulid>) and every cursor in its own namespace (<ulid>/...), so nothing it left stands as live or holds retention and the same identity enrolled later starts clean (a plain revoke keeps the definitions cursor as the child's catch-up position); the response adds "retired":true,"records_retired":N. The tombstones replicate up, so ancestors retire their copies too |
POST /enroll/{ulid}/drain |
{"ulid","offset","status":"draining"}; decommissions a child node: new commands under its mount are refused, and once its queue is delivered or expired it is retired as with DELETE ?retire=true. 409 for an entry that is not a node or is already draining |
|
POST /store/incarnation |
{"store":"NEW","previous":"OLD"}; gives the node's store a new incarnation id, so every child re-syncs with it, its parent clears the marks kept for it and local consumers get store_changed on /ack. For a store restored or replaced without a neighbour noticing; see Restoring a parent from a backup |
|
GET /uplink/set-aside |
{"records":[{"parent","stream","offset","record","status","reason","held_since_ms","at_ms"}]}; the records this node's uplink set aside because the parent refused each alone past the grace. They are not at the parent; see Records the parent refuses. Only on a node with a parent |
|
DELETE /uplink/set-aside/{stream}/{offset} |
{"cleared":true}, 404 when nothing is set aside there; clears the copy, and the uplink_set_aside finding once none is left |
|
GET /debug/state |
the next offset of every stream |
A node that is reachable only through the tree is administered with
_CmdAdmin commands sent to it from an ancestor: enroll, revoke, and
fetchLogs, which returns one page of its own logs in the ack's result. See
Fetching a node's logs.
A local service whose registry entry holds cmd:#:admin may send the same
commands to its own node over the local door, at _CmdAdmin/<node>/<verb>. The node runs them at once: /publish
returns the outcome in command, and the _Ack lands at _Ack/<node>/<verb>.
Without the grant the answer is 403. POST /enroll stays on the published
door.
Secrets¶
Available when secrets_dir is set. The owner is the calling service; a local
service cannot address another service's secrets.
| Method and path | Request | Response |
|---|---|---|
PUT /secrets/{owner}/{name} |
{"envelope":{…},"expires_at":null,"expected_revision":N} |
metadata, never the ciphertext |
GET /secrets/{owner} |
?max=1000&after=TOKEN |
metadata page |
GET /secrets/{owner}/{name} |
metadata; 410 once expired |
|
DELETE /secrets/{owner}/{name} |
?expected_revision=N |
{"deleted":true} |
expected_revision of 0 means create only; any other value is a
compare-and-swap.
Errors and limits¶
| Status | Meaning |
|---|---|
202 |
/publish: a _Log record the log gate collapsed or capped; accepted, not stored |
400 |
malformed request |
401 |
a presented credential was not accepted |
403 |
the caller may not do this |
413 |
body too large; a record over the record cap has reason too_large |
422 |
bad topic, unknown contract or invalid payload, with reason when the node names one |
429 |
a request limit; code in the body says which (below) |
503 |
/publish: the record was admitted but not written (reason not_written); send it again |
Every route belongs to a rate class. Authenticated callers get their own
quota; the local door and unauthenticated traffic are limited per source
address. Per caller: /kv 5 requests a second (burst 10), /fetch 25 (burst
50) and 16 open at a time, /watch 2 new connections a second (burst 8) and 8
open at a time.
A 429 body names the bound that refused the request:
{"error": "too many concurrent requests: …", "code": "concurrency_limit", "class": "watch", "limit": 8}
code |
Meaning | What helps |
|---|---|---|
rate_limit |
the caller's request rate for the class is used up | wait Retry-After seconds (only this code sets it) |
concurrency_limit |
the caller already holds limit open requests of the class |
close one, e.g. a /watch it no longer reads; retrying sooner does not help |
global_concurrency_limit |
all callers together hold limit open requests of the class |
back off with your own delay |
caller_limit |
the node tracks as many callers as it may | back off with your own delay |
Only rate_limit carries Retry-After: for the others no wait is known, so a
client backs off on its own schedule instead of retrying every second against
a bound that waiting does not clear. The Go door client (door.RetryDelay)
falls back to its jittered backoff when the header is absent.
MQTT reason codes¶
Over MQTT 5 at QoS 1 or higher, a refused publish is answered in the PUBACK:
0x90 unknown topic or contract, 0x99 invalid payload, 0x87 not
authorized, 0x89 node is draining. MQTT 3.1.1 has no reason codes; the
publish is dropped and counted in colca_rejected_publishes_total.
The clock in /healthz¶
/healthz includes clock, what the node knows about its time:
{"wall_now_ms": 1760272496123, "is_root": false,
"host": {"synchronized": true, "estimated_error_ms": 0.4,
"max_error_ms": 212.5, "stale_bound_ms": 0},
"parent_skew_ms": -12, "parent_sample_age_s": 3.2}
wall_now_ms is the host's wall clock, the only time the node decides on.
host is the kernel's verdict on that clock; its fields are null where it
cannot be read. estimated_error_ms is also null on a host whose NTP service
keeps no estimate (systemd-timesyncd): null is "not reported", not an error
of zero. max_error_ms is the kernel's bound on the error and
stale_bound_ms the part of it beyond one NTP poll interval, which grows
while the clock is not updated. parent_skew_ms is the parent's clock minus this node's,
measured on the latest replication response whose round trip was at most
0.5 s, and parent_sample_age_s its age; both are null on a root and until
there is such a response. See
Clocks.
Local stream wakeups and backlog telemetry¶
Internal services may subscribe to GET /watch?stream=metrics&stream=entities on
the local door with their existing service identity. The NDJSON response starts
with a catch-up hint and then emits {"streams":["metrics"]} after successful
durable commits. Five-second heartbeats contain no stream names. Hints can
coalesce: capture the local wakeup version before fetching, drain the durable
cursor to empty, then wait on that version. A disconnect requires reconnecting
and draining again; a hint never acknowledges or contains records.
This route is not on the published API and does not accept forwarded human bearer
identities. Streaming clients must use the heartbeat/read deadline rather than an
ordinary short whole-request timeout. /fetch and /ack retain their normal
ownership rules. Fixed minimum spacing between drain starts can batch bursts;
nonempty pages within a drain do not need an additional delay.
GET /backlog?prefix=c/projector/ on the same local door returns selected durable
cursor positions, stream heads and lag_records, with each cursor's
last_ack_ms, stale and read_since_start. It is read-only. Values count
outstanding offset distance (an upper bound after compaction), not bytes. Up to
32 nonempty prefixes and 256 matching cursors are accepted; excess results fail
instead of truncating away a potentially overloaded consumer. A missing expected
cursor is not evidence of an empty queue.