Idempotency keys on the metering write path

An idempotency key protects one write. A metered unit in BuildBase goes through three before it reaches an invoice: the quota counter on the workspace, a row in the usage log, and, when the unit is overage, a usage report to Stripe. The key your code sends covers the first two. Stripe never sees it.

So "is our metering idempotent?" was really three questions, one per hop, and we answered them at different times. We got one wrong first. Another still checks too early. The rule we landed on fits in a sentence: mint each key from whatever is already fixed at that hop, and let a unique index decide, never a read.

Retries are the normal path

At any real volume, a usage record gets retried by design rather than by accident, so the key has to hold on every hop and not only on the one your code calls.

The usage endpoint allows 60 requests a minute per IP, and the batch route is the way past that: one request, up to 100 items. The 429 guide covers what happens when you skip it. Batching means buffering. A buffer is flushed by a job, and jobs retry. On our side alone, the job that reports overage to Stripe gets five attempts with exponential backoff, and a sweep every 10 minutes re-queues any overage row older than 2 minutes that is still not marked billed. Each of those is a deliberate second attempt at a write that may already have landed. It is also why idempotent ingestion sits in the scope of what metering costs to build yourself: the key is one line, and putting it in the right place on every hop is not.

We sit on the calling side too. When our console meters a customer org's usage against its plan, its emails, workflow runs, push notifications and webhook deliveries, among others, are counted into a Redis buffer, and every 30 seconds a flush reports them to that org's workspace in our own BuildBase, through the same POST /usage/batch your backend calls. A failed report gets eight attempts. So the keys below were designed by people who also have to send them.

Hop one: your key and the counter

The key you send is matched on workspace, quota slug and key together, and a match answers without touching the counter.

POST /usage, and every item in POST /usage/batch, takes an optional idempotencyKey of 1 to 255 characters. Before it increments anything, the service looks for a usage log row with that workspace, slug and key. If it finds one, it returns the current standing with used: 0. If it finds nothing, it increments quotas.<slug> on the workspace and then writes the log row that carries the key. If you call bb.usage.record from a route handler the way the per-API-call guide does, this is the only hop you touch.

The log row is the thing that remembers. Its collection carries a unique index over the same three fields, partial so that rows recorded without a key never collide with each other:

// server/src/model/workspace/quotaUsageLogs.model.ts
schema.index(
  { workspace: 1, quotaSlug: 1, idempotencyKey: 1 },
  {
    unique: true,
    partialFilterExpression: {
      idempotencyKey: { $type: 'string' },
    },
  }
);

The slug is in the match on purpose. One request id can record api-calls and storage-mb, and both land.

Two trade-offs come with this shape. A replay does not hand you the original response back: Stripe saves the first response to a key and replays it, while we answer with where the counter stands now, and used: 0 is how you tell a replay from a fresh record. And a key is remembered exactly as long as its log row, which sits under a retention window of 730 days. Stripe keeps a key for at least 24 hours and may prune it after that. Ours catches a retry that turns up a month late.

Hop two: the key Stripe sees, and the one we got wrong

The overage report to Stripe is keyed on the usage log row, not on the queue job that sends it, because a re-send is a new job with a new id and the row is the one thing that stays the same.

Overage is not reported inside your request. The request writes the log row with billedToStripe: false and the number of billable overage units, then queues a job. The job reports those units to Stripe and flips the flag. If the job never runs, or fails every attempt, the sweep finds the row still unbilled and queues another one.

The first version keyed the Stripe call on the job id. Walk the failure through. The job reports the overage, Stripe accepts it, and then the write that marks the row billed fails. The row still reads unbilled. The next sweep, up to about twelve minutes later, picks it up and queues a new job, which has a new id, which makes a key Stripe has never seen. Stripe does what it is told. The comment that now sits on the fix says it plainly: "Keyed on the job, the sweep billed the same overage twice."

The fix keys on the row, which exists before any job does and is the same row every time the sweep finds it:

// server/src/vendors/bullmq/queue/stripe/usage-sync-key.ts
export function usageSyncIdempotencyKey(
  data: { usageLogId?: string },
  jobId: string | undefined
): string | undefined {
  if (data.usageLogId) return `usage-sync-log-${data.usageLogId}`;
  return jobId ? `usage-sync-${jobId}` : undefined;
}

Hold on to that fallback, because it comes back later. A job with no row still gets a key from its own id, which covers the queue's retries of that one job and nothing else.

The other half of the fix is Stripe's, and we checked it rather than assumed it. Our Stripe conformance test ran green against Stripe's test mode on 11 September, and one of the assumptions it confirmed is exactly this one: the same idempotency key really does stop a second usage report. The sweep runs every 10 minutes, so as long as it is running, a re-send under the same key arrives well inside the 24 hours Stripe keeps one.

Mint the key where the number stops changing

For a buffer, the quantity becomes final at the moment you take it out, not at the moment you send it. That is where the key belongs.

Our console's reporter is the clearest case, because the wrong place for the key was sitting right next to the right one. Sends are counted into a Redis hash per org. Every 30 seconds the flush takes each org's hash in a single Lua call, read and delete together, so a send counted mid-flush lands in the next flush instead of vanishing between the two. Then it mints one id for the take:

// server/src/services/quota-usage.service.ts
export async function takePendingUsage(
  orgId: string
): Promise<QuotaUsageItem[]> {
  const counts = await QuotaUsageRedis.take(orgId);
  const flushId = crypto.randomUUID();
  return Object.entries(counts)
    .filter(([, quantity]) => quantity > 0)
    .map(([quotaSlug, quantity]) => ({
      quotaSlug,
      quantity,
      source: QUOTA_USAGE_SOURCE,
      idempotencyKey: `${QUOTA_USAGE_SOURCE}:${flushId}:${quotaSlug}`,
    }));
}

The items, keys included, ride inside the report job, and the job is queued with the first item's key as its id, so queueing the same take twice is a no-op. Every retry sends the same items under the same keys.

Why not mint the key when the report goes out? Because then the report has to read the buffer itself, and the comment on that queue names both ways that breaks: it "would send new counts under an old key, or old counts twice." Re-read on retry, and an item that now holds fresh sends arrives under a key the server has seen, so it answers as a replay and counts none of them. Mint a key per attempt, and a report that landed but timed out is counted again. Same mistake both times: a key born before or after the number it names.

A derived write gets a derived key, too. When an item is refused at a hard cap, our central service re-records the part that still fits under the original key plus :clamp, so the smaller record and the refused one can never collide.

Where our check is still too early

The replay check on hop one is a read, and the index behind it sits on a row that is written after the counter moves. Retries that arrive one after another are safe. Two copies of one key in flight at the same moment both count.

Walk it through. Both requests look for the key and find nothing, because neither has written its row yet. Both increment the counter. The first writes its row, and the unique index refuses the second. But that log write sits in a try/catch that logs the failure and carries on, so the second request still answers success. The counter has moved twice and there is one row. Each response works out the new total from its own earlier read, so neither one shows the double.

On a plan with overage it reaches Stripe as well. The second request has no row, so its job has no row id, so its Stripe key falls back to the job id. That is the fallback from hop two, doing exactly what it says and nothing more.

The batch route turns a rare race into a certain one. It runs every item through Promise.allSettled at once, so two items in one batch with the same key and slug are always in flight together. (The 429 guide covers the narrower cousin: when the log write fails outright, the key is never stored at all.)

Credits had this exact race, and it is closed there. A consume also checks the ledger for its key before it debits, and two in-flight retries can still both pass that check. The difference is that the ledger's unique index is the referee: when it refuses the row, the bucket decrement and the balance decrement are both put back, and the loser answers with the winner's result. The comment at the top of the test that pins it could describe the quota path today: the promise "held only when the retries arrived one after the other." Quotas and credits are separate systems for good reasons, which quotas are not credits lays out, but this fix belongs in both.

For a stretch, the index on the quota side existed only in the schema, too. autoIndex is off across the platform, and until recently nothing built a declared tenant index at all, so the read was the only guard. A declared index is not an index the database holds.

Until the quota path gets the credits treatment, two habits keep you out of it. Collapse a batch by key before you send it. A buffer like the one in the Express metering guide holds items until the flush, so a client that retries inside that window puts the same key in the buffer twice, and both copies then travel in one batch. And back off before retrying a single record, so the retry is far more likely to arrive after the first attempt has written its row. The full request shape is in the quota reference.

What we would do differently

Make the index the first thing that moves. There are two ways to get there, and each one sends a bill.

Claim first: insert the log row, let the unique index refuse a duplicate before anything has changed, and only then increment. A crash between the claim and the increment leaves a row with no unit behind it, and the retry is answered as a replay, so that unit is never counted. That errs toward under-billing, which is the right direction to be wrong in on someone else's invoice. It is still wrong.

Compensate: keep today's order and do what credits does, putting the increment back when the index refuses the row. Nothing is lost. But the counter runs ahead for the length of the race, and on a hard-capped plan that can be long enough to refuse a request that should have fit.

Stripe takes a third road with its own API: a request that conflicts with one still executing under the same key is refused, and the caller retries it. For a metering endpoint that is the most honest answer on offer. Not yet, ask again.

The other regret is about shape rather than order. Every route hand-rolls its own idempotency. Credits match on workspace and key; usage matches on workspace, quota slug and key. A credit replay echoes the amount you asked for, and a usage replay says used: 0. Each was a sensible local call. Together they are two contracts a customer has to learn, and a third for anything we add next. If we were starting again, one piece of middleware would own the key and claim it before anything moves, on every route.