Send us Fan Mail
CISSP Domain 3.9 — design site and facility security controls, built on a real data center power failure where every control in the building worked and the building still went dark.
Start free: https://www.cisspcybertraining.com/free-cissp-self-study (360 practice questions, a Domain 1 diagnostic, weekly questions by email, and the slides from this episode)
In October, a cloud provider's London data center lost its utility power. The batteries picked up the load exactly the way they were supposed to, ran for about five minutes, and then ran out. The generators should have taken over in that window. They never got the signal.
Nobody attacked anybody here. There are no bad guys in this one. No ransomware, no intrusion, nothing for incident response to chase. It was a cable fault outside the building and a logic error inside it.
HOW THE BUILDING WAS BUILT
Start with the design, because the design was good. Two completely independent utility feeds coming in from the power company — they call them A and B. Google's own words for that are a 2N utility power architecture.
Now listen to where that label stops. Behind both of those feeds sits one central bank of high-voltage generators, run by a dual-redundant controller, a PLC. Two of the thing coming in. One of the thing that was supposed to save them. And between the utility going away and the generators coming up, a battery system with about five minutes of runtime. That isn't a lot and it isn't supposed to be. It's a bridge.
So when somebody tells you a site is 2N, the only honest follow-up is: 2N at which layer?
Five days before any of this, they ran a quarterly load test on the generators. A real test, under load. And that test found a problem. So they did the right thing — they scheduled the repair. But to do the repair they had to physically open the breaker on feed B and lock it out, which they did properly: keyed interlock, locked out, tagged out.
And that is the moment the building stopped being 2N on the utility side. For several days — Google doesn't publish exactly how many — that site ran on one utility feed. Google's own words for it are a temporary single-feed, 1N, bridged maintenance posture.
Then the cable broke. An underground cable out on the utility network, outside the building entirely, developed a fault. And it was on feed A. The only feed that was live.
WHY THE GENERATORS NEVER STARTED
The PLC decides when to start the generators, and it decides it by sensing voltage. If the voltage goes away, start the generators. Simple enough.
But those voltage sensors were wired upstream of the feed breakers. On the line side. And feed B — the one that was open for repairs — was still energized upstream, all the way up to that open breaker. The power company was still feeding it. It just wasn't connected to the building anymore.
So the PLC looks at its sensor, sees voltage on feed B, concludes that a healthy utility source is available, and never issues a start command.
Google's own root cause is this: a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders. It checked whether there was voltage. It never checked whether that voltage was actually connected to anything.
And here's the part worth a line in your notes. That PLC was dual-redundant. There were two of them. Both of them were wrong in exactly the same way, because they shared a configuration. Duplicating the box does not duplicate the judgment. That's common-mode failure, and it is the reason a redundancy count is not a resilience measurement.
Five minutes later the batteries were done and the data halls went dark.
TWO NUMBERS, KEPT APART
About ninety minutes with the data halls unpowered. Six hours and eight minutes before the services on top of them were whole again. Both figures are in Google's report and they measure different things. According to The Register, the companies caught in that window included Lloyds, HSBC, Royal Mail and Revolut — Google published no customer list and no customer count.
To their credit, Google published the whole thing: the timeline, the wiring, the root cause, and a list of fixes with dates on them. One of those action items is the most honest line in the document. They committed to making PLC code review part of facility commissioning — which tells you it wasn't.
THE MANAGER'S TAKEAWAY
Two things, and neither one is "test your generators."
They DID test. The quarterly load test is what found the fault in the first place. What nobody did was look at the consequence: fixing that fault meant opening a breaker, and opening that breaker meant the building ran on one feed for days. As far as anybody treated it, that was a maintenance ticket. It wasn't a changed risk posture.
And the second one, the one to carry into the exam room: an automatic control is only as good as the thing it can sense. The generator was perfect. The battery was perfect. The logic was asking a question — is there voltage? — at a place where it could not tell the difference between "the utility is healthy" and "the utility is healthy and no longer connected to us."
Redundancy isn't a label on a drawing. It's a property of a system at a particular moment, and that moment includes whatever your facilities team happens to have open for repairs today.
IN THE TRAINING BLOCK
• The power chain end to end, and the fourth thing almost nobody reads — the logic that decides when to start, which lives on a controls drawing rather than a power one.
• UPS runtime: battery in minutes, flywheel in seconds, and why five minutes is a bridge rather than a weakness.
• Redundancy notation — N, N+1, 2N, 2(N+1) — and why N+1 survives one failure but not one failure during maintenance.
• Concurrent maintainability versus fault tolerance: two different promises that candidates collapse into one.
• The three states of a redundant system — designed, degraded, failed — and why failure almost never arrives on its own.
• Every physical control has a sensing input: fire, cooling, water detection, door alarms, access control vestibules.
• The 3.9 recall content, and the crossovers named out loud — change management is 7.9, risk acceptance is 1.9, resilience and fault tolerance is 7.10.
FOUR CISSP PRACTICE QUESTIONS
Set inside the real incident, so they are hard on the reasoning instead of hard on the reading. Common-mode failure; a FIRST question where the trap is the answer you are proudest of spotting; a MOST DIRECTLY question where the trap is the biggest answer on the list; and a control-class question where the trap is the governance gap the episode itself spent ten minutes describing.
CHAPTERS
0:00 Two power feeds, and the generators never started
1:56 The whole thing in thirty seconds
3:36 A spare, or a complete second path
4:44 How the building was built
6:09 The load test, the open breaker, the cable fault
7:40 Why the controller never issued a start command
9:00 Two controllers, one mistake
10:18 The manager's takeaway
12:33 Free Essentials, Accelerator, Pro and the Cohort
15:51 Domain 3.9 and the power chain
19:03 Redundancy, and where the label stops
22:01 The three states of a redundant system
23:37 Every control has a sensing input
29:17 Question 1
32:20 Question 2 — the one that gets people
35:23 Question 3
38:31 Question 4
41:43 The pattern across all four
LINKS
Free CISSP Essentials — 360 practice questions, a Domain 1 diagnostic, weekly questions by email, 8+ hours of rapid review audio and video, and the slides from this episode:
https://www.cisspcybertraining.com/free-cissp-self-study
All episodes and the written walkthrough:
https://www.cisspcybertraining.com/podcast
CISSP Sprint Cohort — for people with a test date on the calendar:
https://www.cisspcybertraining.com/cohort-sprint-page
SOURCES
PRIMARY — Google Cloud's own incident report for europe-west2, Incident Xs2XTHzAVC9YnafnPyzj. Every figure above comes from here unless stated otherwise:
https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPyzj
SECONDARY — The Register, 5 November 2026, Simon Sharwood. The four customer names come from here, NOT from Google:
https://www.theregister.com/off-prem/2026/11/05/when-datacenters-lose-power-generators-should-kick-in-that-didnt-happen-for-google-in-london/5307739
Gain exclusive access to 360 FREE CISSP Practice Questions at FreeCISSPQuestions.com and have them delivered directly to your inbox! Don’t miss this valuable opportunity to strengthen your CISSP exam preparation and boost your chances of certification success.
Join now and start your journey toward CISSP mastery today!
