My editor asked me this week "why is everything except AWS breaking?"
It's a great question! They haven't been in the headlines! In fact, as they are wont to do, AWS spokesperson Patrick Neighorn got salty to a reporter about how unreliable Downdetector is when it started showing them as impacted.
If you'll permit me a small digression, this stems from a misunderstanding of what Downdetector is for. Historically the AWS status site was so hilariously slow to reflect the lived customer experience that some jackass (me) built a rewriter that sits at stop.lying.cloud to make it a smidgen more realistic. As a result, the question people are really trying to answer when they visit Downdetector is "my app just broke; is it my crappy code or is there a global event going on?" There are mitigations for both, but they look nothing alike.
AWS view crowdsourced reports of the form "the internet is broken, must be AWS" as a stain upon their honor. In fact, it's a badge of their own success, as "the internet" and "AWS" have become inextricably linked in people's minds.
But back to my editor's question, "why didn't AWS go down?"
Easy: it did.
Everything breaks, all the time, including the narrative
AWS now has 34 generally available commercial regions, of which two are currently smoking craters due to the Iran conflict. Each of these has hundreds of services that AWS offers to customers, of which a dozen or so are high quality, if I'm being characteristically uncharitable. They experience degradation in various ways, and they go largely unnoticed.
But on November 15, AWS confirmed an incident that disrupted 11 of those ~12 good services in their oldest and most important region, us-east-1.
At 3:05 AM PST, they stated that they started seeing sign-up errors, authentication issues, and connectivity issues. Five minutes later engineers were engaged. Not to be outdone, their status page swung into action after a leisurely coffee, and posted its first indication of an incident 93 minutes later. (See previous comments on stop.lying.cloud). The all-clear showed up at 6:53 AM.
Every site has downtime. Sometimes it blows up publicly, other times nobody seems to notice. Heck, AWS said in September that it had experienced permanent data loss in its two smoking craters, but after that news cycle passed I haven't heard a thing about it. I do think it's patently unreasonable to blame them for this specific problem; "acts of war" are kinda hard to predict from the corporate planning meeting.
A few things combined to help AWS out on November 15. First, it was pre-dawn Pacific time on Sunday. Most of us slept through it; my kids aren't even up that early.
Further, the services that were most impacted (IAM Identity Center) were for engineers signing in to AWS accounts, not consumers staring at an app that stopped working.
Plus, the competition sucked all the oxygen out of the room: a day-plus Microsoft 365 364 outage and a four-hour nationwide air traffic ground stop had done a great job of level-setting what kind of outage was going to get attention, and "us-east-1 being weird but the apps still work" doesn't cut it.
Downdetector – which is owned by Accenture, so it's not like AWS doesn't have a throat to choke if they want to stop picking on reporters – measures consumer screaming. Originally, it was largely Twitter semantic analytics. This isn't a bad thing! As I said, I want to know if the internet is breaking. We can sort out exactly what part later.
The November 15 outage didn't produce that kind of screaming. Ironically, the tool that AWS is complaining about in the newspaper is the reason they skated by without notice on this outage.
Counterintuitively, Downdetector reports that outages have decreased since about 2022. Instead, what we're seeing is large-scale centralization.
Our companies don't run email systems internally anymore; we outsource to large technology companies who are almost universally better at fixing and resolving large-scale issues than we will be internally. And it feels unkind to dunk on those hard-working folks for their outages; would we want them to dunk on us for ours?
("Wait, don't you own stop.lying.cloud?" "Do as I say, not as I do.")
But whether anyone notices depends upon what layer broke, and when. If my company's email system broke on Monday, and your company's email system broke on Tuesday, the story would stop at "computers suck." But when everyone's email system goes down at once, the world takes notice.
Microsoft had to tell the whole world about a bug it mostly sat on since July. AWS hasn't had to tell anyone anything. My editor runs a tech publication that covers this stuff for a living and didn't know. You can't buy PR like this, and AWS didn't have to.
If I were AWS, I'd stop complaining about Downdetector and send it a fruit basket. ®