back to Events

2025.10.21 data center lost power:


This morning, shortly after the start of the monthly generator test, the entire data center suddenly lost all
power. It took a little while to find the breaker that had tripped and get it reset, but that restored power
to the room and all the hardware began booting back up.

I don't know if they've entirely figured out what happened yet, but the last news I've heard is that one of
the UPSs in the rack malfunctioned, and that may have been what tripped the breaker. Email alerts should have
gone out, so not sure why nobody noticed we were on battery until the entire room went quiet. Alarms would
have been going off in the room while it was on battery, but there's normally a lot of fan noise in there, and
the walls are pretty sound-proof, so nobody's going to hear an alarm from outside.



IMG_4708:

IMG_4710.MOV (35 MB)

highest resolution images
last updated 10/30/2025 at 08:58:37 by make_www_index.command version 2025.06.28.A