Subscription Billing & Idempotent Webhooks
Summary
Payment webhooks don't arrive exactly once, in order, or at all. I made the handlers idempotent and added a scheduled job that re-checks our records against the provider — so a duplicate delivery is a no-op, and a dropped event gets caught instead of quietly costing revenue.
Problem
Payment providers deliver webhooks at-least-once, not exactly-once. Process one twice and you double-charge a customer; miss one and revenue disappears with no error anywhere. Neither failure shows up in tests — they show up in production, in someone's bank account.
Constraints
- C01This is money, not a UX nice-to-have — no tolerance for double-processing or dropped events
- C02Deliveries retry, duplicate and arrive out of order; the handler can't assume clean input
- C03Five engineers, no dedicated payments or data-reliability specialist
Architecture
Handlers key on the provider's event ID, so a repeat delivery is a no-op.
A scheduled job re-checks our records against the provider and repairs the gaps.
Prisma over PostgreSQL, with the migration path fixed so schema changes ship without downtime.
Remix serves the billing-facing application logic.
Decisions
Results
Retrospective
I built the handlers first and the reconciliation job second. That was backwards. Idempotency prevents the loud failure; missed events are the quiet one, and quiet is where money actually leaks. I'd build the reconciler first now and treat the webhook path as the optimization.