Reliability and delivery semantics
Distributed systems retry, and retries duplicate. Fibric's answer is not to promise that failures never happen; it is to define exactly what happens when they do. Ingestion is at-least-once with deduplication, execution is idempotent, side effects on one entity are serialized, and everything is replayable from the event log. This page is the delivery-semantics contract: mechanics, not marketing numbers.
The delivery model in one table
| Stage | Guarantee | Mechanism |
|---|---|---|
| Ingestion | At-least-once, deduplicated | Callers retry freely; the Idempotency-Key header collapses duplicates to one stored envelope. |
| Routing | Every stored envelope is offered to every matching operator | Glob triggers on event_type; the EventBus seam carries delivery. |
| Reasoning | No guarantee needed | Proposals are side-effect-free. A lost or repeated reasoning step costs a model call, never a duplicate action. |
| Execution | Effectively once per idempotency_key | The executor's dedup set, durable via the DurableExec seam, disposes replays as DEDUP. |
| Ordering | Serialized per entity_key; unordered across entities | Single-flight gate in the executor. |
| Record | Every disposition receipted | Append-only, tenant-scoped receipt ledger. |
At-least-once ingestion
Sources are unreliable in a specific direction: a webhook sender that does not see a timely 2xx will send again, a poller that crashes mid-batch will re-read the batch, a gateway that reconnects will re-publish its buffer. Fibric leans into this. Send events as many times as your delivery pipeline requires, and put the event's natural identity in the Idempotency-Key header:
curl -X POST https://api.fibric.io/v1/events \
-H "Authorization: Bearer sk_live_3f9c2a7b8e1d4f60a2c9" \
-H "Idempotency-Key: magento:SO-10884:v7" \
-H "Content-Type: application/json" \
-d '{
"source": "magento",
"event_type": "order.updated",
"payload": { "order_id": "SO-10884", "status": "processing" }
}'
The first delivery stores one envelope and returns it. Every retry with the same key returns the same stored event without storing another. A retry that replays the same key with a different body is rejected with 409 idempotency_conflict, because two different events claiming to be the same event is a client bug worth surfacing, not suppressing. Keys are scoped per tenant and route and retained for 24 hours; see Errors and Rate limits & quotas.
A random UUID per attempt defeats deduplication: a crashed-and-restarted worker generates a new key and duplicates the event. Derive the key from the operation's natural identity, magento:SO-10884:v7, source plus entity plus version, so any process that retries the same fact produces the same key.
Idempotent execution
Deduplication happens a second time where it matters most: at the side effect. Every side-effecting PlannedAction carries its own idempotency_key, and the executor consumes each key at most once. The kernel logic, from packages/kernel/src/executor.ts:
// idempotency dedup for side-effects
if (this.seen.has(action.idempotency_key)) {
return { action, decision: 'DEDUP', ok: true };
}
// trust gate (default-closed)
const decision = evaluate(this.policies, action, env);
if (decision === 'BLOCK') {
return { action, decision, ok: false, error: 'blocked by trust policy' };
}
const result = await this.connectors.invoke(env, action.connector, action.tool, action.args);
this.seen.add(action.idempotency_key);
Three details of this ordering are the contract:
- The key is consumed only after the invocation succeeds. A connector call that throws leaves the key unconsumed, so a retry genuinely retries. The failure is receipted with
ok: falseand the error message. - A replay is a success, not an error.
DEDUPreturnsok: true: the intended state already holds. Replays are receipted, so the audit trail shows the retry happened and shows it did nothing. - Reads skip all of this. A non-side-effecting tool needs no policy and no idempotency; repeating a read is harmless by definition.
In production the dedup set is durable through the DurableExec seam (once(key, fn), at-most-once per key, surviving retries and process restarts), backed by a Postgres outbox at MVP scale with Temporal as the named scale-up behind the same interface. See Deployment architecture.
Single-flight serialization
Idempotency stops the same action from running twice. Single-flight stops different actions from interleaving on the same real-world thing. Every action carries an entity_key, one order, one conversation, one asset, and the executor holds a gate per key: work on an entity waits for the in-flight work on that entity to dispose before it proceeds. Work on different entities runs independently; the serialization is exactly as wide as the entity and no wider.
Together the two primitives are why the 657-message incident, a real early-agent failure where one conversation received 657 messages, cannot recur: concurrent sends to one conversation serialize on its entity_key, and once the message's idempotency_key is consumed, every subsequent attempt disposes as DEDUP. The primitives, key-design guidance, and worked examples are in Single-flight & idempotency.
Over HTTP, a request that needs a lock held by other in-flight work fails fast with 409 entity_locked and a Retry-After header; retry the same request unchanged after the interval. See Errors.
Replay
The event log is the source of truth, and everything downstream of it is reproducible from it. Replay is safe because of the two sections above: re-offering an envelope to an operator produces a plan whose side-effecting actions carry the same identity-derived idempotency keys, so anything that already ran disposes as DEDUP and anything that never ran gets its chance. Replay is how you recover from an operator that was paused, misconfigured, or deployed with a bug, without hand-reconciling what did and did not happen.
# re-run recorded events against your operator locally
fibric dev replay --events ./fixtures/orders.jsonl
The same property covers consumers: a stream consumer that fell behind resumes from its cursor and re-processes from there (see Streaming events), and an export job re-run produces the same receipts because receipts are immutable. When designing your own consumers, assume any event may be seen more than once and key your own processing accordingly.
Backpressure
Fibric applies backpressure explicitly rather than degrading silently, at three levels:
| Level | Signal | What to do |
|---|---|---|
| Request rate | 429 rate_limited with Retry-After and X-RateLimit-Remaining |
Back off for the stated interval. Limits are per tenant, not per key; spreading traffic across keys does not raise the budget. |
| Entity contention | 409 entity_locked with Retry-After |
Transient by design; retry unchanged once the in-flight work disposes. |
| Standing quotas | 429 quota_exceeded |
The monthly action allowance or a concurrency cap is exhausted. With a hard cap set, plans hold in proposed until the cap is raised, so nothing is lost, only deferred. |
Ingest sits behind the same rate limits, and because ingest deduplicates, a producer that backs off and retries loses nothing. Retryability by status code, including which 5xx responses are safe to retry and when, is tabulated in Errors; the budgets themselves are in Rate limits & quotas.
What we claim, and what we do not
Fibric is in a v1.0 preview program. We do not publish an uptime SLO, and we will not invent one here. The claims this page makes are mechanical, and each is checkable:
- A retried ingest with the same idempotency key stores one envelope. You can verify this with two identical
POST /v1/eventscalls. - A replayed side effect disposes as
DEDUPand appears in the receipt ledger as such. - Two concurrent actions on one
entity_keynever interleave; the second waits or receivesentity_locked. - A blocked action never reaches a connector, and the refusal is receipted.
- A brief platform interruption surfaces as
503 service_unavailablewithRetry-After, and the retry semantics above make the interruption safe for both ingest and execution.
When the preview program ends, availability commitments will appear in the plan documentation, not retroactively in this page.
Single-flight & idempotency specifies the two primitives in depth; Streaming events covers consumer-side resume; Errors tabulates every code referenced here; Deployment architecture explains the seams that make the durable versions of these guarantees drop-in.