What follows is a made-up scenario. The retailer, the team and the days are all fictional, and no real organisation is being described. The operating principles it dramatises are real ones, taken from the Atlas page on ServerlessOps and credited where they appear.
Monday: the first complaint
A mid-sized retailer, which we will call the Harbour team for convenience, ran its order pipeline on cloud functions. A customer-support lead forwarded three messages with the same subject line: charged twice. By lunchtime the count had reached fourteen. The engineering manager asked a simple question: was this a one-off or a pattern?
The pipeline was typical. A storefront posted each order to a queue; a function validated it, called a payment provider and wrote a record; a second function sent a confirmation and notified the warehouse. Everything was event-driven and, on paper, tidy.
Tuesday: reading the logs
The team pulled logs for the affected orders. Each duplicate showed two successful payment calls, about forty seconds apart. The platform had retried the function after the first attempt exceeded its timeout waiting on the payment provider, even though the charge itself had gone through. The function had no memory of the first attempt.
That pattern matches one of the situations the Atlas entry warns about. The Atlas reports a test run on a batch of ten thousand orders, where one in fifty was delivered twice and a few more timed out following payment; the unprotected handler charged 504 customers twice. The Harbour team’s scale was smaller, but the mechanism was the same: a timeout after a successful side effect, followed by a retry with no way to recognise the repeat.
Wednesday: the tempting wrong fix
An engineer proposed recording each message identifier and skipping any it had already seen. It was a reasonable first instinct, and the Atlas simulation shows why it is incomplete: a record-only approach still left 295 double charges in the sample. The reason is that the records were created after the side effect rather than before it, and because an event replayed from a failure queue gets a new identifier, the check would not recognise it.
The team settled on the principle the entry sets out: deduplicate on a business key, here the order number, record it atomically before the payment call and pass the same value to the payment provider so the provider can reject repeats on its side. They noted a subtlety the entry raises, namely that such provider-side keys expire. The Atlas mentions one provider pruning keys after 24 hours, so replay is only safe within the shorter of the local expiry and the provider’s retention.
Thursday: the nested timeout problem
While reading the function configuration the team found that the function timeout was thirty seconds while the call to the payment provider had no explicit timeout at all. When the provider was slow, the function was killed mid-call. The Atlas rule is that downstream calls should expire sooner than the function, so the function can decide what to do with a partial result. The queue’s visibility window had also been left at its default, far below the entry’s guidance of at least six times the function timeout, which explained why a second consumer picked up in-flight messages.
They set an eight-second limit on the provider call, raised the visibility window to match the six-times rule and added a status lookup so that a timed-out charge could be checked before any retry. If you manage a similar stack and wonder how this fits in a wider operational model, the thinking in why operational discipline matters for machine learning systems shares the same insistence on repeatable, observable steps.
Friday: a failure queue with an owner
The team also discovered that some failed orders had never been noticed at all. A dead-letter queue existed but nobody was watching it, and its retention was shorter than that of the source queue. Following the entry, they gave it a named owner, an alarm on inflow and age of the oldest item, and a retention of fourteen days against the source’s four.
On the batch side, a function that processed warehouse notifications in groups was throwing errors whenever one record was malformed, causing the whole group to be reprocessed. They switched to partial batch responses so that only failing records returned to the queue. They also noticed that their error alarm had never fired for these item failures, since the entry notes such failures are not counted as errors.
The following week: rehearsal
The entry’s third maturity level, which it calls safe, is reached when a team runs a duplicate-storm game day and sees zero double effects. The Harbour team scheduled one on a staging copy. They replayed a thousand orders three times, killed the payment stub mid-call twice and confirmed that the ledger showed exactly one charge per order. It took an afternoon. They then added the same test to their release pipeline so it would run on every change to the payment function.
A short retrospective at the end of the second week closed the loop. Nobody was blamed, because the original design had matched common tutorials; the gaps were in what the tutorials leave out. The manager asked each engineer to name one assumption that had turned out wrong. The answers were revealing: that a retry is always safe, that a timeout means failure and that an empty error graph means a healthy system. Writing those three sentences on the team wiki did more for new joiners than a dozen architecture diagrams.
Measuring the result
With fixes in place, the team picked a small dashboard. The Atlas suggests signals such as business success rate, duplicate side-effect rate, failure queue inflow and age, backlog age, throttle ratio, concurrency headroom, cold-start share and cost per million events. The Harbour team started with four of them, reviewing monthly. They also looked at attempts per order. The entry’s illustrative volume of 30 million orders against 31.5 million runs implies only about five percent extra attempts; the Harbour team borrowed that as a rough yardstick and read any rising figure as retries creeping in.
Cost per event followed. Applying the entry’s sample pricing, its worked path landed at $3.75 for every million events, and $112.58 across the month. The Harbour numbers differed, naturally, but putting them on a one-page chart changed the conversation from opinions to evidence. Those with a bent for forecasting may enjoy the longer view in a staged roadmap for machine learning operations, which applies a similar measure-then-improve loop.
One more detail deserves a mention. The team resisted the urge to add a bespoke dashboard tool during the crisis. They reused the metrics their cloud provider already exposed, such as the age of the oldest queued item and the count of throttled requests, and wrote two alarms. Small, boring and visible beat elaborate and half-finished, and the improvement was in place before the next sale weekend, when order volume doubled without incident.
What they would tell another team
- Assume every message can arrive twice and prove otherwise with a test, not a hope.
- Use a business key for deduplication and hand it to downstream systems.
- Give every timeout a place in a nested order, shortest at the outside edge of the call.
- Give every failure queue an owner, an alarm and a longer retention than its source.
- Rehearse, then automate the rehearsal.
Why the story is useful beyond one company
The entry is candid that ServerlessOps is a label with no formal body behind it, and that its main value lies in collecting questions engineers should ask about function-based systems. The Harbour story shows the questions in motion: what triggered the function, what happened on failure, how were repeats treated, how much could run at once and what did it cost? None of the answers needed a new tool. They needed attention, a named owner and a few honest tests.
For readers who want the source material, the Atlas also describes the history of the term and its scope in its dedicated ServerlessOps page. Read it alongside your own runbook, and you will probably find a Harbour-shaped gap or two worth closing this quarter.