Delivery boundary
A successful webhook request returns202 only after the payload is durable. The enqueue budget is 8 seconds with two attempts; each attempt can use up to 4 seconds and a failed attempt is followed by a 50 ms delay. A duplicate also returns 202. A store failure returns 500, so use GitHub’s delivery redelivery action after the store recovers.
Delivery states are received, processing, done, and failed. Workers hold a 60-second lease, renew it every 20 seconds, and make at most six processing attempts. Transient failures normally wait 1, 5, 15, 30, and 30 seconds before the final attempt. Permanent failures become failed; a GitHub outage parks the delivery as received and refunds the claim attempt.
Terminal rows are retained for 30 days and pruned every five minutes in batches of 256. This keeps recent delivery IDs available for idempotency and gives operators a local replay window well beyond GitHub’s delivery history.
Workers are woken, not only polled. A delivery that one node makes claimable—an enqueue, or an operator replay—wakes that node’s worker at once. On Postgres the node also sends one preloop_webhooks notification after the commit, coalesced over 15 ms, so every other node’s worker wakes too. A lost notification costs only latency: each worker re-reads the queue at least every five seconds. SQLite is single-node and wakes only its own worker.
Recovery layers
1. Delivery watchdog
The watchdog polls GitHub’s App delivery history:- a GitHub failure with no local row (
remote_failure); or - a GitHub success with no local row (
phantom_ack).
2. GitHub circuit breaker
Three consecutive transport or 5xx failures open the breaker. A429 opens it immediately. A 403 opens it when x-ratelimit-remaining is zero or a valid retry-after is present. The wait uses retry-after first, then x-ratelimit-reset, and is bounded between 1 second and 1 hour for rate limits.
The normal open interval starts at 30 seconds and grows to a 300-second ceiling. One 60-second half-open probe tests recovery. A successful response—including a GitHub 404 or 422—resets the breaker. Payload errors and ordinary non-rate-limit 4xx responses do not trip it.
The queue gate prevents claims for GitHub-wide unavailability. A rate-limited credential can still be claimed, then parked and refunded when its dependency call reports the limit. This prevents a temporary outage from dead-lettering good deliveries.
3. App health monitor
Every 15 minutes by default, the monitor readsGET /app and GET /app/hook/config with an App JWT. It reports missing event subscriptions, the permission that gates each event, delivery-URL drift against PRELOOP_PUBLIC_URL, and read errors. It only reports and logs drift; it does not modify GitHub App settings.
An App whose own breaker is open is not re-read: its last verdict stays published until the breaker closes, while the other Apps are checked as normal. A held-back App never vanishes from the published list: with no earlier verdict—its breaker was open on the first pass—it is published as explicitly unchecked. Every verdict carries its own checked_at, and app_config_checked_at advances only when a pass checked every App, so a reused verdict never looks freshly verified.
4. Operator surface
Use the native API with the system bearer:
The CLI uses the same native credential:
Status conditions
GET /api/v1/status includes warning conditions for:
/readyz is a separate unauthenticated readiness probe. It returns readiness and a reason, not the webhook condition objects.
Configuration
Shared store and replay
Postgres deployments share the durable queue and claims withFOR UPDATE SKIP LOCKED, and the wake notification described above reaches every node’s workers. SQLite keeps the queue on the single engine and wakes only its own worker. Replay always requeues the retained local payload, so it needs neither GitHub nor an App JWT.