Guardrail Ledger
Every incident fixed once, forever. Built with IBM Bob. Updated 2026-09-27.
3postmortems closed
3guardrails in CI
20.0avg minutes, postmortem to guardrail
INC-2041 — Black Friday promo ended six hours early
SEV-2
GR-2041
- Root cause
- `shopfront/promotions.py:16` — `_parse_expiry` truncated the ISO-8601 expiry string to 19 characters (`[:19]`), silently discarding the UTC offset (`-06:00`). The resulting naive datetime represented merchant local time, but `is_active` compared it against `datetime.utcnow()` (also naive but already UTC). When the UTC clock hit 18:00 on 2025-11-28 the UTC value numerically exceeded the naked `23:59` local string, marking BF30 expired six hours before the intended midnight Central deadline.
- Bug class
- Timezone offset silently discarded when parsing ISO-8601 strings, causing naive-vs-naive datetime comparisons that are wrong by the full UTC offset.
- Fix
- Two minimal changes to `shopfront/promotions.py`:
- Minutes
- 18
INC-2057 — Invoices off by one cent, reconciliation failed
SEV-3
GR-2057
- Root cause
- `shopfront/billing.py:15` and `billing.py:26` — two cooperating bugs.
- Bug class
- Monetary arithmetic performed with binary floating-point (`float`), producing banker's rounding and accumulation truncation errors at the cent level.
- Fix
- Two minimal changes to `shopfront/billing.py`:
- Minutes
- 22
INC-2063 — Customers charged twice after gateway timeouts
SEV-1
GR-2063
- Root cause
- `shopfront/payments.py:13` — `gateway.charge(amount_cents=amount_cents, reference=order_id)` is called inside a retry loop without an `idempotency_key`. When the gateway captures the payment but the response times out, the caller raises `GatewayTimeout`, retries, and `gateway.charge()` creates a brand-new charge — the gateway has no way to deduplicate it.
- Bug class
- Non-idempotent external API call retried without an idempotency key, causing duplicate side-effects on timeout.
- Fix
- Two minimal changes to `shopfront/payments.py`:
- Minutes
- 20