On this page4 sections

On August 28 a large customer’s endpoint returned 503 for 40 minutes. Our retries did exactly what they were told, and that was the problem.

What happened

Relaywave retried failed deliveries on a fixed schedule: 30 seconds, 2 minutes, 10 minutes, 1 hour, and so on. When the endpoint went down, about 2.1 million messages failed within the same few minutes. All of them were scheduled for the same retry times.

When the endpoint came back, it got the full backlog in one burst at the 10 minute mark. It fell over again, and every message moved on to the 1 hour slot together.

The fix: full jitter

Retries now use exponential backoff with full jitter. Each retry waits a random time between zero and the current ceiling:

function nextDelayMs(attempt: number): number {
  const baseMs = 30_000;
  const capMs = 6 * 60 * 60 * 1000;
  const ceiling = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.floor(Math.random() * ceiling);
}

The same 2 million messages now spread over the whole window instead of landing on one second.

A rate limit per endpoint

Jitter alone is not enough after a long outage, so every endpoint now has a delivery rate limit. The default is 200 requests per second, and you can change it:

curl -X PATCH https://api.relaywave.dev/v1/endpoints/ep_7Hk2 \
  -H "Authorization: Bearer $RELAYWAVE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"rateLimit": {"requestsPerSecond": 50}}'
  1. Watch the backlog

    The endpoint page shows queued messages and the time to drain them at the current limit.

  2. Raise the limit when you're ready

    If your endpoint scales, raise the limit to drain faster. Lower it if you see 429 or 503 responses.

Results

We replayed the August 28 traffic against a test endpoint with the same capacity. With the old schedule it failed three times before draining. With jitter and a 200 rps limit it drained in 11 minutes without a single extra failure.