Waylight · what went wrong
Every incident of the last year.
Written by the person who fixed it, published whether or not anyone noticed. If this list were empty we would not expect you to believe it.
- 012345678933
- incidents, last 12 months
- 012345678901234567890123456789433433
- minutes affected, total
14 Jan
Boards slow to cut for 41 minutes
degradeddegraded- What happened
- A migration held a lock on the planning tables longer than we had tested. Boards still cut, but some took over a minute.
- What changed
- Migrations now run against a copy first, and the planner reports its own latency to the status page rather than to us.
2 Nov
Gate view unavailable, 12 minutes
outageoutage- What happened
- A bad config reached production because the deploy gate only checked the primary region. Two yards saw an error page at shift change, which is the worst possible minute.
- What changed
- Config changes now roll region by region with a five minute soak, and the gate view falls back to the last cut board offline.
8 Aug
Exports delayed overnight
degradeddegraded- What happened
- A queue backed up behind one very large export and nothing shed load. Nightly exports landed at 06:40 instead of 23:00.
- What changed
- Large exports run on their own lane, and anything late now tells the customer before we notice.