You will have battery rooms (full of truck batteries) in a 48V DC config fed into DC->AC converters which in turn feed 230V AC UPS plugs on the walls etc (plugs have the earth pin at an angle so you can't just plug any old plug into it) so essentially you are running all your kit of batteries the whole time. In a telecoms environment the equipment gets fed directly by the 48V DC supply as all the equipment is DC. All this has redundancy built in as you usually have a A & B feed in case something in the chain fails.
Besides this you will also have massive generators that kick in once the eskom supply fails as you still need power to charge the batteries which will eventually run flat if not charged. Once again you will have redundancy here.
Management of this is via alarms sent to a remote control room.
I was involved in the setting up of the UPS system for the control room at City Power, so I understand how systems with backup gens work. I guess I'm looking for something smaller than that
FWIW, let me share a story of massive fail with you:
Electricity supply in JHB is divided up into 3 islands, based on the 275 kV infeeds from Eskom at Prospect, Fordsburg and Delta (IIRC, this was 15 years ago). If anything happens to those infeeds, the impact is limited to the island that it supplies.
The City Power offices in Booysens (in particular, the control room in the basement) have supply from two separate islands, going through an automatic changeover switch. If anything happens to the supply in one island, the switch will change over to the other supply, and nobody will have to do anything.
The supply then goes to a 16kVA UPS, which supplies 2 8kVA UPSes, which then feeds two independent power buses. Computers in the control room are redundant, with half plugged into one bus/UPS, and the other half plugged into the other UPS.
Of course, if both islands go dark, then there is a diesel generator set up to supply the control room, to ensure that the UPS's don't run down completely.
Every week, Friday morning, the diesel gen is tested. Mains supply is turned off, the UPS's run on battery for a while, the diesel gen is fired up, and everything checked to make sure it is working.
Cue one Friday night, around 11pm, I get a call. Everything has gone dark in the control room!
Come in to find out that the people testing the diesel gen forgot to turn the mains supply on after they completed their tests. The UPS alarm has been going off in the control room for about 15 hours, and the operators had done nothing about it. Eventually all 3 UPSes were drained, and everything shut down hard.
Moral of the story?
Even in the most ridiculously redundant circumstances, when the UPSes say "shut down now", there needs to be a way of communicating that to all of the computers affected.
