Deep Dive - Webhooks, Events & Reliability

"Reliable" is not a single number. It is a choice between three failure modes and an engineering manager who owns webhooks for a 1M+ DAU tool has to name the choice out loud before designing anything.

The Cursor JD calls out webhook infrastructure with "proper retry semantics, dead-letter queues, idempotency and observability" in almost those words. When you answer a question in the loop, anchor to that vocabulary on purpose. It shows you read the charter and that you think in delivery guarantees, not vibes.

Start with the three semantics every event system picks from. Each one moves the pain to a different place.

The answer that wins the roomsay it in one breath

When an interviewer asks how you guarantee delivery, lead with the default and the consequence in the same sentence: at-least-once delivery plus idempotent consumers. That single clause tells them you know exactly-once is fiction and that you have already decided who owns dedup.

Ordering: expensive and usually not what you needglobal vs per-key vs none

Strict global ordering forces serialization, which kills throughput and creates a single point of contention. Most webhook problems only need ordering within a key, like a single repository or a single tenant. Be ready to size the requirement instead of granting it by reflex.

Inbound vs outbound: two different reliability problemsthe distinction senior candidates draw and juniors miss

Cursor sits on both sides of the webhook contract. It ingests events from GitHub, GitLab and Bitbucket and it may emit events to customers and internal surfaces. The reliability concern flips depending on which direction you're standing in.

At-least-once delivery is a promise you keep with a retry loop. The whole craft is retrying enough to ride out a blip without turning a downstream hiccup into a self-inflicted outage.

The naive version - retry immediately, forever - is how you take down a recovering service. Ten thousand clients all retry at the same instant the endpoint comes back and you knock it over again. That's a thundering herd and the fix is two ideas working together.

1. 1Exponential backoff. Wait longer after each failure: 1s, 2s, 4s, 8s, 16s. Spacing retries out gives a struggling endpoint room to breathe instead of a flood.
2. 2Jitter. Multiply each computed delay by a random factor (commonly 0–1, i.e. delay * random()). This smears retries across time so synchronized clients stop hammering in lockstep.
3. 3A retry cap. After N attempts (say 5–8 over tens of minutes to a few hours), stop. Infinite retries clog the pipeline and waste compute on something that isn't coming back.
4. 4Route to a dead-letter queue. When the cap is hit, move the event to a DLQ for isolation, alerting and later replay - don't drop it.

base = 1.0           # seconds
cap  = 300.0         # max 5 min between attempts

def delay(attempt):
    # exponential growth, then full jitter
    expo = min(cap, base * (2 ** attempt))
    return random.uniform(0, expo)

# attempt 0 -> up to 1s, attempt 3 -> up to 8s, attempt 7 -> up to ~128s

Retryable vs non-retryable: stop hammering a 400the classification that saves your error budget

Not every failure deserves a retry. A 500 or a timeout might succeed next time. A 400 or a 401 will fail identically on every attempt, so retrying just burns capacity and delays the moment you notice the real problem.

The DLQ is a debugging and recovery tool, not a graveyardwhat it buys an EM running on-call

A dead-letter queue isolates the events your pipeline couldn't deliver so they don't block everything behind them. The events sit there, inspectable, while the main flow keeps moving. That separation is what lets you fix a root cause calmly and then replay.

At-least-once delivery means duplicates are not an edge case - they are a guarantee. The system's correctness rests entirely on the receiver treating the second copy of an event as a no-op.

Idempotency is the contract that makes this safe. Each event carries a stable key, the consumer records the keys it has already handled and any repeat is recognized and skipped. Without it, a retry that succeeds after a lost ack double-charges, double-provisions or double-fires a downstream action.

Where the key livesfast, durable and TTL'd

# SET key value NX EX ttl  -> returns OK only if the key did not exist
ok = redis.set(f"evt:{event_id}", "1", nx=True, ex=86400)
if not ok:
    return  # already processed within the 24h window: no-op

process(event)  # safe: we hold the claim

Signing and verification: trust before you dedupHMAC + timestamp

Before a consumer dedups an event it should know the event is genuine. Providers sign the payload with a shared secret using HMAC and the receiver recomputes the signature to verify it. Include a timestamp in the signed material and reject anything too old or an attacker can capture a valid request and replay it later.

1. 1Sender computes HMAC-SHA256(secret, timestamp + "." + body) and sends it in a header.
2. 2Receiver recomputes the same HMAC over the raw body it received and compares with a constant-time equality check.
3. 3Reject stale timestamps outside a small window (e.g. ±5 minutes) so a captured-and-replayed payload fails even though its signature is valid.
4. 4Then verify the signature, then dedup by event id, then process.

The trade-off you'll be pushed onwindow vs cost

Agent workflows are spiky. A cohort of developers kicks off background agents within the same few minutes of a workday and the event rate jumps an order of magnitude, then falls. The system has to absorb that without dropping events or starving the rest of Core Services.

The first move is to decouple. A queue between producers and consumers turns a spike into a backlog the consumers drain at their own pace, instead of a flood that overruns them. The receiver acks fast, enqueues and returns; workers pull and process. The queue is the shock absorber.

Fan-out and per-consumer limitsstop one slow endpoint from taking everyone down

One event often has many destinations - an SCM push might notify an indexer, a build trigger and an audit log. Fan-out duplicates the event onto separate per-consumer queues so each subscriber moves at its own speed. The reason this matters is failure isolation.

Backpressure and partitioningshed load deliberately; isolate noisy neighbors

When consumers can't keep up, the system must push back rather than melt. Backpressure means the queue depth and rate limits signal producers to slow down or shed low-priority work, so a flood of events doesn't cascade into auth, agent backend and the rest of the shared layer. Partitioning by tenant or key gives you horizontal scale and keeps one heavy customer from drowning everyone else.

Capacity planning: have the numbers readywhat an EM is expected to quantify

An interviewer will probe whether you operate from data or hand-waving. Come with the three numbers that define this system's capacity and know where the bottleneck moves when load grows.

• Peak event rate - events/sec at the spikiest minute of the day, not the daily average. Size for the spike.
• p99 delivery latency target - the SLI you commit to; e.g. 99% of events delivered within N seconds of ingestion.
• The current bottleneck - broker throughput, consumer concurrency or the downstream endpoint. Name it, because that's where the next dollar of capacity goes.

The JD lists observability alongside retries and DLQs as a first-class requirement and for an EM it's where reliability stops being architecture and becomes a thing the team manages day to day.

A webhook system fails quietly. Events don't crash a page; they just don't arrive. So the only way you know the system is healthy is the instrumentation you built. Cover the three signals and make each one map to a decision.

SLOs, SLIs and error budgetsan objective bar the team manages to

An SLI is the measurement (delivery success rate). An SLO is the target you commit to (99.9% of events delivered within 30s). The error budget is what's left over - the 0.1% you're allowed to miss. That budget is a management tool, not just a metric.

Designing the on-call experiencewhat the engineer at 3am actually sees

As the manager, you own whether on-call is humane or miserable. The bar is that a single engineer, woken at 3am, can diagnose and act from the runbook and dashboard alone without paging you. Build toward that.

1. 1Alert on leading indicators. Page on DLQ depth and retry-rate spikes, not only on a success-rate crater that customers have already felt.
2. 2One runbook per failure mode. "DLQ growing," "provider returning 5xx," "consumer lag rising" - each with the dashboard link, the likely cause and the first action.
3. 3Dashboards that answer the first question. The on-call's opening question is "what changed?" - put delivery rate, retry rate, DLQ depth and recent deploys on one screen.
4. 4Clear ownership per failure mode. Every alert routes to a team that can act on it; an alert with no owner is noise that trains people to ignore the pager.

After the incident: the management half of reliabilityblameless and it feeds the backlog

Reliability work doesn't end when the page resolves. A blameless post-incident review asks what about the system let this happen, not who typed the wrong command. Then the action items become real backlog with owners and dates, so the same incident doesn't recur. Running that loop well is a management practice and it's a strong thing to show in a behavioral round.