Webhook Architecture: The Delivery and Retry Design That Stops Silent Event Loss Before Your Customers Notice

Your largest customer just emailed: "We never got the payment confirmation, but our bank says it went through." You check the logs. Your system sent the webhook. Their server returned a 502 for ninety seconds during a deploy. Nobody retried it. A week later, finance is reconciling by hand.

At AEGONTECH LLC, we see this pattern whenever a product grows from "one integration" to "dozens of integrations." Webhooks, the HTTP callbacks that tell another system "something just happened," look trivial in a tutorial and become a reliability problem in production. This guide explains how to design webhooks, both sending and receiving, so that no event is silently lost and none is processed twice.

Key Takeaways

• A webhook is an event notification over HTTP, and the network guarantees nothing about its delivery. Reliability has to be designed in.
• Plan for at-least-once delivery and make every receiver idempotent, because duplicates are a certainty, not an edge case.
• Sign every payload, verify the signature on receipt, and reject stale timestamps to stop forged and replayed events.
• Acknowledge fast, process later: return a 2xx within seconds and hand the real work to a queue.
• Give customers a delivery log and a replay button. Support tickets drop sharply once they can see what happened.

What Is a Webhook, and Why Do They Fail So Often?

A webhook is an HTTP request that one system sends to another when an event occurs, such as a payment succeeding or a build finishing. The sender does not poll and the receiver does not ask; the sender simply pushes. That inversion is what makes webhooks efficient, and it is also why they fail: the sender must call a server it does not control, over a network it cannot trust, at a moment the receiver may be deploying, overloaded, or down.

Many integration failures are not code bugs but delivery problems: timeouts, restarts, expired certificates, and slow downstream databases. As simple arithmetic, a receiver that is healthy 99.9% of the time still misses roughly 43 minutes of events every month if the sender never retries. At 20,000 events a day, that is a meaningful pile of missing records.

A webhook without a retry policy is not an integration. It is a hope.

Polling vs Webhooks: Which Should You Choose?

Use webhooks when events are infrequent relative to how often you would poll, and when the receiver needs to react within seconds. Use polling when the receiver controls the pace, when the sender cannot reach the receiver (a firewall, a mobile app), or when you need a guaranteed reconciliation path.

The honest answer is usually both. Polling vs webhooks is a false choice in mature systems: webhooks give speed, and a periodic reconciliation job (pull the last hour of events and compare) gives correctness. Stripe, GitHub, and Shopify all document this combination, and for good reason. Treat the webhook as a hint that something changed, and treat the API as the source of truth.

How Should You Deliver Webhooks Reliably as a Sender?

Write the event to your database in the same transaction as the business change, then deliver it asynchronously from a queue. The first sentence matters most: if you send the HTTP call inline with the request that caused the event, a crash between "commit" and "send" loses the event forever. This is the transactional outbox pattern, where the event row is saved atomically with the data and a separate worker publishes it.

From there, a dependable sender needs five things:

1. A retry schedule with exponential backoff and jitter. Backoff means waiting longer after each failure (for example 30 seconds, 2 minutes, 10 minutes, 1 hour, 6 hours). Jitter adds randomness so that a thousand failed deliveries do not all retry at the same instant and knock the receiver over again.
2. A sensible timeout. Ten to fifteen seconds is typical. Longer than that and slow receivers tie up your workers.
3. A dead-letter state. After roughly 24 to 72 hours of failures, stop retrying, mark the event failed, and surface it to the customer rather than dropping it.
4. Per-endpoint isolation. One customer's broken server must never delay another customer's events. Separate queues or per-endpoint concurrency limits keep a noisy failure contained.
5. Automatic disabling with notice. If an endpoint fails continuously for several days, pause it and email the owner. Hammering a dead URL helps nobody.

Each event also needs a stable, unique ID that does not change across retries. That ID is the receiver's key to deduplication, which brings us to the harder half.

How Do You Process Webhooks Safely as a Receiver?

Verify the signature, store the event ID, acknowledge immediately, and process asynchronously. In that order. The receiver's job in the HTTP request is small: prove the message is genuine, record that it arrived, and return a 2xx status. Everything else belongs in a background worker.

Verify authenticity. A webhook URL is just a public endpoint; anyone can POST to it. Senders should sign the raw request body with HMAC-SHA256 (a keyed hash that proves the message came from someone holding the shared secret) and include a timestamp in the signed data. Receivers recompute the signature, compare it in constant time, and reject anything older than five minutes. This blocks both forged events and replayed ones. It aligns with the OWASP guidance on validating inbound data and is a routine item in SOC 2 audits of integration surfaces.

Deduplicate with idempotency. Idempotent processing means handling the same event twice produces the same result as handling it once. Store each processed event ID in a table with a unique constraint; if the insert fails, you have seen it before, so skip it. In PostgreSQL this is a single INSERT ... ON CONFLICT DO NOTHING inside the same transaction as the business update, so "mark as seen" and "apply the change" succeed or fail together.

Acknowledge fast. Most senders treat anything slower than 5 to 30 seconds as a failure and retry, which creates the very duplicates you just built protection against. Return 200 after you have durably stored the event, then let a worker (Sidekiq, BullMQ, Celery, or SQS consumers) do the slow work.

Tolerate disorder. Retries mean events can arrive out of order: "subscription cancelled" can land before "subscription updated." Compare version numbers or timestamps carried in the payload, or fetch the current state from the sender's API instead of trusting the order of arrival.

Thin Payloads vs Fat Payloads: What Should the Event Contain?

A thin payload carries only the event type and an object ID, and the receiver calls the API to fetch details. A fat payload carries the full object. Thin payloads are safer because the receiver always reads current state, which also solves the ordering problem and reduces sensitive data in transit. Fat payloads save a round trip and work when the receiver cannot call back.

Our recommendation: send a moderately thin payload (ID, type, version, a timestamp, and the few fields most consumers need) and document that the API is authoritative. Version your event schema from day one, using a field such as api_version, so you can add fields without breaking consumers. The same discipline applies to any public contract, as we cover in our work on API versioning.

What Does Good Webhook Observability Look Like?

Good observability means that when a customer asks "did you send it?", your support team answers in thirty seconds without opening a database console. That requires a delivery log: every attempt with timestamp, response status, response time, and a truncated response body, retained for at least 30 days.

Three features pay for themselves quickly:

• A replay button that re-sends a specific event to the current endpoint URL.
• A test event customers can trigger while building their integration.
• Metrics and alerts on delivery success rate, p95 latency, and retry queue depth, ideally in the same dashboards you already use for the rest of your platform.

A delivery log removes the most common support question (did you send it, and what did my server say?) because the customer can answer it on their own screen.

Webhook Design in Practice at AEGONTECH

When we build integration layers for clients, the rules above are the baseline rather than the aspiration. Provider callbacks from payment processors, telephony platforms, and messaging services generally arrive at-least-once, so every inbound handler should store the provider's event ID first and apply business logic second. Products with real-time communication features, like our own Dialable.world and Mimicall.app, are a good illustration of why: status updates such as call progress can arrive late or duplicated, and idempotent handling is what keeps a record from flipping back to "ringing" after it has already completed.

The pattern that surprises new clients most is how unglamorous the fix is. No new infrastructure, no exotic framework: a unique constraint, an outbox table, a queue, and a signature check. For a typical service, that foundation is a matter of weeks rather than months, and it prevents the multi-day reconciliation scramble that we see after the first serious incident.

A Practical Checklist You Can Use This Week

1. List every webhook you send and receive. Most teams are surprised by the count.
2. For each receiver, confirm it dedupes by event ID and acknowledges before heavy processing.
3. For each sender, confirm events are written transactionally and retried with backoff and jitter for at least 24 hours.
4. Confirm signatures include a timestamp and are verified in constant time.
5. Add a nightly reconciliation job that compares your records with the provider's API.
6. Expose a delivery log and replay for your own customers.

Frequently Asked Questions

How many times should I retry a failed webhook?

Retry for 24 to 72 hours using exponential backoff with jitter, which typically works out to 8 to 15 attempts. After that, mark the event as failed, notify the endpoint owner, and keep the event available for manual replay.

Should I use a queue between receiving a webhook and processing it?

Yes, for anything beyond trivial work. Store the event, return 2xx, and process from a queue. This keeps your response time low, prevents sender-side retries caused by slow processing, and lets you scale workers independently.

How do I secure a webhook endpoint?

Use HTTPS only, verify an HMAC signature computed over the raw body, include and check a timestamp to prevent replay, and keep secrets in a secrets manager with rotation support. IP allowlisting can add a layer but should not replace signature checks, since provider IP ranges change.

Are webhooks exactly-once?

No. Practical systems offer at-least-once delivery. Exactly-once behavior is achieved on the receiving side by combining at-least-once delivery with idempotent processing.

Conclusion

Webhooks fail in quiet ways: a missed deploy window, a duplicate charge, an event that arrives before the one it depends on. None of these are exotic. Each has a small, well-understood design answer, and the cost of adding them early is a fraction of the cost of reconstructing lost events later.

If your integrations are growing and you would like a second pair of eyes on how events flow through your system, the team at AEGONTECH LLC is glad to review your architecture and suggest the smallest set of changes that closes the gaps. Reach out through aegontech.dev to set up a consultation.