The night Spot capacity disappeared

At 02:14 Central the scheduler stopped getting Spot capacity in two regions at once. Nothing failed loudly. Jobs waited in the queue, the autoscaler kept asking for nodes it could not get, and latency climbed for forty minutes before an alert fired.

This is what happened, why the on-demand fallback did not engage, and the two changes we made afterward.

1. 02:14+00:00
   trigger

   Spot requests start failing in two regions
2. 02:54+00:40
   alert

   Queue-latency alert fires
3. 03:20+01:06
   mitigation

   On-demand pool scaled by hand
4. 05:26+03:12
   resolved

   Backlog drained; incident closed

The fallback watched for failed launches. Here the launches never failed. They stayed pending, and a pending node looks healthy to anything that only counts errors.

fallback.trigger = launches.failed > 0        // before
fallback.trigger = pending.age > 90s         // after

We now alert on how long work has been waiting rather than on error counts. Capacity is also treated as a signal in its own right: when it stops arriving, the router shifts work before the queue shows it.