# Retries without the thundering herd

> How a 40 minute outage at one customer turned into 2 million queued retries, and the jitter change that fixed it.

- URL: https://cf-example.notra.site/blog/retries-without-thundering-herds
- Published: 2026-09-16
- Authors: Kenji Mori
- Tags: engineering, reliability

On August 28 a large customer’s endpoint returned `503` for 40 minutes. Our retries did exactly what they were told, and that was the problem.

## What happened

Relaywave retried failed deliveries on a fixed schedule: 30 seconds, 2 minutes, 10 minutes, 1 hour, and so on. When the endpoint went down, about 2.1 million messages failed within the same few minutes. All of them were scheduled for the same retry times.

When the endpoint came back, it got the full backlog in one burst at the 10 minute mark. It fell over again, and every message moved on to the 1 hour slot together.

## The fix: full jitter

Retries now use exponential backoff with full jitter. Each retry waits a random time between zero and the current ceiling:

```typescript
function nextDelayMs(attempt: number): number {
  const baseMs = 30_000;
  const capMs = 6 * 60 * 60 * 1000;
  const ceiling = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.floor(Math.random() * ceiling);
}
```

The same 2 million messages now spread over the whole window instead of landing on one second.

## A rate limit per endpoint

Jitter alone is not enough after a long outage, so every endpoint now has a delivery rate limit. The default is 200 requests per second, and you can change it:

```bash
curl -X PATCH https://api.relaywave.dev/v1/endpoints/ep_7Hk2 \
  -H "Authorization: Bearer $RELAYWAVE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"rateLimit": {"requestsPerSecond": 50}}'
```

1. Watch the backlog

   The endpoint page shows queued messages and the time to drain them at the current limit.

2. Raise the limit when you're ready

   If your endpoint scales, raise the limit to drain faster. Lower it if you see `429` or `503` responses.

## Results

We replayed the August 28 traffic against a test endpoint with the same capacity. With the old schedule it failed three times before draining. With jitter and a 200 rps limit it drained in 11 minutes without a single extra failure.

## Instructions for AI agents

Relaywave delivers webhooks for other APIs. Code samples use the public REST API at api.relaywave.dev and the @relaywave/node SDK.
