Nobody Wants To Be Tomorrow’s Headline…
I’m Joe Schiavone. Last week was a rough one for anybody whose title starts with a C and ends in IO, and I suspect more than a few CIOs got a text from a CEO or board member who had just read the news: “Could this happen to us?”
It’s a fair question, and most CIOs answer it badly. Usually it’s some version of “we have a plan,” followed by a pause just long enough for everyone in the room to start wondering where exactly that plan lives.
Six Days, Three Headlines
On Thursday, November 12, Microsoft 365 went down for roughly 24 hours, with Exchange Online taking the worst of it and other services affected along the way. Then, on Friday the 13th, yes, really, the FAA issued a nationwide ground stop after a technical problem hit its network. On Tuesday, November 17, an internal Apple update caused outages across iCloud and other services. Retail systems were affected too, leaving some customers standing inside an Apple Store unable to complete a purchase.
None of these were cyberattacks. These came out of ordinary technology work, the kind every IT shop does all the time.
Frankly, that should bother you more.
The Router That Stopped A Country
Here’s the FAA version, and it’s the one that got me.
A contractor was replacing old copper lines with fiber at a remote FAA site in El Dorado, Arkansas, in the middle of the day. The work included installing a new router. Soon after, the FAA’s aging network started getting hammered with roughly 1,000 connection attempts every four to five seconds. Investigators traced the flood to potential sources in El Dorado and another site in Amarillo, Texas. Problems eventually spread to 17 of the 22 centers that manage high-altitude air traffic, and the FAA stopped flights nationwide.
A router’s whole job is moving traffic. This one grounded it across an entire country. Are you kidding me?
Somewhere in that mess was a person who missed a wedding, a funeral, a kid’s graduation, or the last few hours with a parent. The ground stop ended that evening, long after some of those plans were gone.
Then came the detail every CIO should pay attention to. The FAA said maintenance at its major facilities usually happens at night, while work at remote sites like El Dorado had been done during the day for decades because it hadn’t caused this kind of problem before. The agency is now reconsidering that practice.
“We’ve always done it this way” has a long and distinguished record as the opening line of a postmortem.
The final root cause is still under investigation, so I’m not going to pin this on anybody. For our purposes, the pattern matters more than the culprit.
Same Story, Different Logo
Microsoft’s preliminary report reads like a greatest-hits album of change risk. A defect shipped with a fix back in July and sat there quietly for months. Then Teams traffic shifted onto the gateway Exchange uses while roughly 14% of that gateway capacity was already unavailable for another reason. During peak U.S. traffic, the whole thing tipped over. At the worst point, more than 80% of those gateway servers stopped responding.
Four months of silence, then Thursday happened.
One detail deserves a sticky note on your monitor. Microsoft tested the change on older, smaller hardware, so the lab never saw what production eventually saw. Because the bad code had already been live for months, the usual rollback and feature-flag controls couldn’t quickly undo it.
Production found the test case the lab missed and charged everybody admission.
Apple’s explanation is simpler so far. An internal update caused outages across several services, including account sign-in and iCloud, and retail systems were hit too. Some customers inside Apple Stores couldn’t complete purchases.
There’s a special kind of irony in standing inside the company’s own store, holding the company’s product, and being unable to give the company your money. The customer didn’t care what the postmortem would eventually call it. The register didn’t work.
Let’s Talk About CYA
Yes, I’m going to say it: CYA.
Most people hear that and picture the post-incident scramble, where somebody starts digging through Outlook for the email from March where they wrote, “I do have some concerns …”
Congratulations. You have discovered corporate archaeology.
Useful CYA happens before anything breaks. You know which business services would hurt the most if they failed, and you’ve put some limits around risky changes to them. You know who can stop a change if things start going sideways and whether you can actually get back. Most important, the business knows what risk is still sitting there.
That’s what you want in your hand when the CEO asks, “Could this happen to us?”
OK, Now The Serious Part
Black Friday is November 27. After that, holiday peak runs straight into year-end close. For a lot of companies, the next six weeks leave very little room for a change to go wrong. If you don’t know where your biggest change risks are now, this is a bad time to find out the hard way. Here’s where I’d start.
Put the business calendar into change control. Set maintenance windows around the times when failure would hurt the business most. Technician availability shouldn’t be what drives the decision. Know your peak periods, whether that’s holiday retail, payroll, quarter close, open enrollment, or a product launch, and freeze changes to the services behind them when the business can least afford a failure. Then name the person who can approve an emergency exception. If POS goes down at 10 a.m. on Black Friday because the maintenance window was technically open, the incident review will be short, and it will be about you.
Make material changes prove they can be undone before they ship. Know what signal stops the rollout, who can stop it, and the last point where rollback is still clean. Test the rollback under conditions that look like production, including the data. The Microsoft outage is also a reminder that some changes get much harder to reverse after they’ve been sitting in production for a while. Those changes deserve tighter staging and a longer watch than a routine config change.
Put names next to the risk. Every critical service needs someone accountable for the technology and someone on the business side who understands the exposure that remains. Be just as clear about who has the authority to stop a change when things start going bad. Work that out before the bridge call starts. A bridge with 40 people on it and nobody who can say stop is how a two-hour problem starts shopping for a headline.
And talk about the risk in business terms. Your board doesn’t need a tour of the change-management process or a heat map full of red boxes. Tell them what could stop the business and when you’re most exposed. Then talk plainly about what you’re spending to reduce that risk and what you’ll still be living with afterward. Then let leadership into the decision. If the business chooses to carry a risk because fixing it would cost $20 million, take two years, or disrupt something else that matters, that’s a legitimate conversation. The problem is when everyone finds out during the outage that the risk existed in the first place.
Count your vendors’ change calendars as your own. You didn’t push Microsoft’s July fix. Your email still went down. For every critical service, know which outside dependencies can take it out and how long the business can keep running on the workaround. Your customer isn’t going to care whose engineer pushed the change.
Test recovery as a business event. Get the people who will actually have to manage through the outage involved. Operations needs to know how work continues. Communications needs to know what customers will hear. Your executives need to know when they get pulled in. The test that matters is whether the business can keep serving customers when email, identity, payments, or another critical service goes away. Find the weak spots during the exercise, not while customers are waiting.
The Questions To Bring To Your Next QBR
I’d bring these questions to your next QBR:
- Which technology failure would hurt the business most, and what kind of change is most likely to trigger it on a bad day?
- When are changes to that service off limits, and who can override that?
- Who has the authority to stop a bad change mid-rollout?
- How fast can we get back to a known-good state, and when did we last prove it?
- What risk are we still carrying, what would it cost to reduce it, and does the business understand the tradeoff?
If those answers live in one person’s head, five documents that disagree, or a SharePoint site nobody has opened since the last audit, I wouldn’t call that preparedness.
That’s optimism with documentation.
“The board will forgive an outage a lot faster than it will forgive a CIO who can’t explain the next morning who approved the change and how long the way back was supposed to take.”
The Takeaway
Technology is going to fail. So are your vendors. You can’t eliminate that risk. You can know where failure will hurt the business and put some guardrails around the changes most likely to cause it. And when something does go wrong, somebody needs the authority to act.
You can also prove that you know how to get back.
Do that before the holiday freeze, and the next time the CEO texts, “Could this happen to us?”, you won’t have to reach for reassurance. You can answer with evidence.
Earn your CYA now, or write the apology later.
I Want To Hear From You
If your most important customer-facing system went dark at 10 a.m. on your biggest day of the year, who in your company has the authority to stop the change that caused it, and do they know they have that authority?
I welcome your perspective. Drop a comment below or connect with me via email: jschiavone@forrester.com.