Single-flight & idempotency

Two kernel primitives make every Fibric side effect safe to propose, safe to retry, and safe to run concurrently: single-flight serializes side effects per entity, and idempotency keys guarantee the same logical side effect applies at most once. Both are enforced by the deterministic executor on every action, not opted into by operators. This page describes both primitives, the exact order the executor applies them in, and what that order means for retries.

#The two primitives

Every PlannedAction carries two keys, and each key drives a different guarantee.

The kernel source calls the idempotency key "the 657-flood lock". The name references a real incident: an ungoverned automation loop that sent 657 messages to one customer. Every message in that flood was the same logical side effect, proposed over and over by a loop that had no memory of having already acted. With an idempotency key on the action, the first send applies and the next 656 collapse into it. With single-flight on the entity, the loop cannot even stack sends concurrently while waiting. The two primitives together make that class of failure structurally impossible rather than merely unlikely.

#Concurrent proposals for the same entity

The executor keeps an in-flight map keyed by entity_key:

private inflight = new Map<string, Promise<void>>(); // single-flight per entity_key

Before an action does anything else, it looks up its entity_key in this map. If a prior action's gate is present, the new action awaits it before proceeding. This is serialization, not rejection: the second proposal is not refused, it is queued behind the first. Once the prior gate resolves, the waiting action proceeds through the normal pipeline, where the idempotency check and the trust gate still apply to it individually.

Walk through the case single-flight exists for. Two operator runs, A and B, fire close together — say a schedule tick and an inbound observation land within the same second — and both propose orders.hold on order SO-10884 with the same idempotency key.

1. Run A's action arrives first

   No entry exists in the in-flight map for ship-risk:SO-10884. Run A registers its gate and proceeds: dedup check (key unseen), trust evaluation, invoke. The hold call is now in flight against the order system.

2. Run B's action arrives while A is in flight

   Run B finds A's gate in the map and awaits it. It does not invoke, does not evaluate policy, does not race. It waits.

3. Run A completes

   The invoke succeeds, the executor records A's idempotency key as seen, releases the gate, and removes its entry from the map.

4. Run B proceeds and deduplicates

   Run B enters the pipeline, reaches the idempotency check, finds its key already seen, and returns decision: 'DEDUP', ok: true without touching the connector. One hold was placed. Both runs report success.

Without the serialization step, both runs would pass the dedup check simultaneously — neither key seen yet — and both would invoke. Single-flight is what makes the idempotency check trustworthy under concurrency: by the time the second action checks the key, the first has either recorded it or failed.

#Key derivation

Both keys are authored by the operator (or the planner acting for it) and carried on every PlannedAction. The kernel does not synthesize them; it enforces whatever the plan states. The recommended shape is operator:entity:action, dropping the action segment for the entity key:

The two scopes are deliberately different. entity_key answers "what must not be touched concurrently" — usually a conversation, an order, an asset, a room. idempotency_key answers "what counts as the same action" — the entity plus the operation. A hold and a release on the same order share an entity key (they must not interleave) but carry different idempotency keys (they are different side effects, and both should apply).

Key choice is a design decision with real consequences. Make the idempotency key too broad — ship-risk:SO-10884 with no action segment — and the release deduplicates against the hold and never runs. Make it too narrow — appending a timestamp or run id — and every retry looks like a new action, which defeats the lock entirely. The operator:entity:action shape is the stable middle: the same operator intending the same operation on the same entity is one side effect, however many times it is proposed.

#Execution order per action

The executor's runAction() applies its gates in a fixed order for every action in a plan. The order matters; several guarantees on this page fall directly out of it.

1. Wait on in-flight work. If any prior action holds the gate for the same entity_key, await it before doing anything else.
2. Reads run immediately. If the connector registry reports the tool is not side-effecting, the action is invoked at once — no policy evaluation, no idempotency check — and returns decision: 'ALLOW', ok: true with the result.
3. Idempotency check. For a side effect, a seen idempotency_key returns decision: 'DEDUP', ok: true without invoking the connector.
4. Trust evaluation. The action is scored against your trust policies. A BLOCK returns ok: false with the error blocked by trust policy, and nothing reaches the connector.
5. Invoke. The connector runs the tool with the proposed arguments.
6. Only a successful invoke burns the key. The idempotency key is recorded as seen after the invoke succeeds, and only then. A failed invoke returns ok: false with the error, and the key is left unburned, so a retry is safe.

Two consequences of the ordering are worth stating plainly. Because the dedup check (step 3) precedes the trust gate (step 4), an action that already applied does not consume an approval or appear in the escalation queue a second time; it short-circuits as DEDUP. And because the key is burned only on success (step 6), "seen" means "applied" — the dedup set never contains a side effect that did not actually happen.

#Retry semantics

Step 6 is the whole retry story. The idempotency key is checked before every attempt and recorded only after a successful one, which gives retries a clean two-phase behavior:

• Before the first success, retries are real attempts. An invoke that fails — a timeout, a 500 from the vendor, a dropped connection — leaves the key unburned. The next proposal of the same action passes the dedup check, re-passes the trust gate, and invokes again. Nothing about a failure makes the action un-retryable.
• After a success, retries collapse to DEDUP. Once one attempt succeeds and the key is recorded, every subsequent proposal with the same key returns decision: 'DEDUP', ok: true without invoking. The caller sees success either way, because the side effect it asked for is in place.

This is why a duplicate can never bill twice, message twice, or hold twice: the key is recorded only on success and checked before every attempt, so there is no window in which two successful applications of the same key can both complete. The one caveat is the classic at-most-once boundary: if an invoke succeeds at the vendor but the success is lost in transit before the key is burned, a retry could re-invoke. Connectors mitigate this by passing the idempotency key downstream where the vendor's API supports one, making the write idempotent at both layers.

#ActionResult and dispositions

Every action in a plan produces an ActionResult. Its decision field is an ActionDisposition: the three trust decisions plus DEDUP, which only the executor can produce.

export type ActionDisposition = TrustDecision | 'DEDUP';

export interface ActionResult {
  action: PlannedAction;
  decision: ActionDisposition;
  ok: boolean;
  result?: unknown;
  error?: string;
}

Here is the result array for a plan that proposed the same hold twice — the shape run B's caller would see in the walkthrough above, or any plan carrying an accidental duplicate:

[
  {
    "action": {
      "connector": "magento",
      "tool": "orders.hold",
      "args": { "order": "SO-10884", "reason": "ship-risk-review" },
      "entity_key": "ship-risk:SO-10884",
      "idempotency_key": "ship-risk:SO-10884:hold"
    },
    "decision": "ALLOW",
    "ok": true,
    "result": { "status": "holded", "order": "SO-10884" }
  },
  {
    "action": {
      "connector": "magento",
      "tool": "orders.hold",
      "args": { "order": "SO-10884", "reason": "ship-risk-review" },
      "entity_key": "ship-risk:SO-10884",
      "idempotency_key": "ship-risk:SO-10884:hold"
    },
    "decision": "DEDUP",
    "ok": true
  }
]

One hold was placed. Both entries report ok: true, because from the caller's point of view the requested state of the world holds in both cases. The disposition tells you which entry did the work, and the receipt trail preserves the distinction permanently.

#Durability

In the reference kernel, the dedup record is an in-memory Set<string> and the in-flight map lives in the process. That is correct for a single executor process and for development, and it is honest about its limits: a process restart forgets seen keys.

In the hosted platform, both structures are backed by durable storage — Postgres or DynamoDB — behind the DurableExec seam, one of the kernel's five swap-seams. The interface is identical; only the backing store changes. Idempotency keys survive restarts and are shared across executor instances, so the at-most-once guarantee holds across deployments and horizontal scale, not merely within one process lifetime. See Architecture for the seam model.

#Keep going

• Trust tiers: the policy gate every side effect passes through between dedup and invoke.
• Receipts & audit: how every disposition on this page, including DEDUP, is recorded permanently.
• The event envelope: the tenant-scoped context every action runs under.
• Architecture: where the executor and the DurableExec seam sit in the kernel.