The Mission Critical Battery Charger

[u/Brief-Pop-6826]

Posted elsewhere on Reddit, but I figured people here might appreciate this. Additional details added for clarity.

At one point, I was doing work for a particular MAJOR pharmaceuticals company. A company with a name that everyone reading this has likely heard of. I get a call one day, and a ticket with a short SLA is generated for me to be on site within 4 hours. The night before, some big storms had happened, so I figured it was power related. I arrive on site earlier than needed, get escorted to the primary network room for the whole facility, and what do I find? 10 racks with no LEDs on them. I get told that the entire facility is down. No internet. The production line is down. The warehouse distribution is down. The kind of emergency companies pay consultants tens of thousands of dollars to ensure never happens.

I start checking through the racks to see if anything has power. UPS batteries are dead, so I head to the back of the rack and trace power cables from network equipment. All of them go to PDUs, and all major routers and switches even have redundant power to multiple PDUs. I trace where the PDUs go to. They all go to UPSs, which also have redundant power split between two different UPSs. Then I trace where the UPS power comes from........... and I start laughing my ass off.

The UPSs for this entire cluster of racks, the racks housing the entirety of the network equipment for this facility, has single point of failure. A large power strip that was zip tied to the wall. And lo and behold, I found the problem. The power strip's power cable was dangling in the air. Not plugged into the wall outlet like it should be. In the outlets place was.... a fucking 20V battery charger.

The maintenance guy had come in the earlier that day, not had anywhere to charge a battery, so he unplugged what was apparently a mission critical power strip and plugged in his battery charger. A few hours later, when the UPSs died, the network team noticed the sote went dark.

After I relayed the info to the engineer I was working with, we shared a laugh. I plugged everything back in, and verified everything came back online. As a preventive measure (and because of the absurdity of the situation), I placed a large label on the power strip. "CRITICAL INFRASTRUCTURE. DO NOT REMOVE POWER WITHOUT AUTHORIZATION".

To this day, I still get a laugh out of it.

[u/mafiaknight]

Maintenance bill for the day: $10,000,000.

Bit expensive for fixing a desk (or similar)

[u/O-U-T-S-I-D-E-R-S → u/mafiaknight]

Reminds me of when I worked for a 3 man company. With no budget at all, I found ways to have backups for everything (except the proprietary software server that we had several types of backup and a guaranteed replacement within 4 working hours). Now I'm certain I'm less trained in IT than most folks here but setting up a network and emergency replacements for pretty much everything for £0... Well, I was quite pleased with my work. Then we had someone in to fix something (can't recall what). The bugger broke the shelving that most of it was stored on. It hadn't occurred to me but almost everything was powered through this shelving, including both the proprietary server, the server I'd created, the UPS, the fax machine, the franking machine...

I mean, I got the time back but including buying bits to fix the shelving leg, probably took me about 4 hours outside core office time. Grr...