Long time SRE/DevOps with 20 year of FinTech experience in banks/hedge funds/crypto and having seen many, many outages (some listed here [0]):
- There is almost always a standard operating procedure (SOP)
- Just b/c there is a SOP for releases etc, that doesn't mean that people always follow it
- This can be due to many reasons e.g. laziness, being distracted, pressure from higher ups ("just push it out there and I'll take the risk" ha!), tired from having a newborn etc etc
- Doing things outside the "green zone" deployment windows therefore always carries risk
- All of this being said, EVEN IF you ARE following SOP, things still break in all kinds of weird ways
- And the end user is not often the best observer nor do they have all of the context to determine what broke, how it broke and why. (There are exceptions to this though)
yep. most IR peeps have plenty of stories where a single event is just a symptom of multiple failures, unfortunate accidents, and alike. most red teams will also regale chained attacks is where it is today. it is rarely a single piece of lock that breaks, it needs several other things to be in the "wrong" state.
Your third bullet is a bane of gov contractors. Not the sales people, but the people actually doing the work. Imagine if you had an FFP contract and you finished the job within 90% of the paid time. At $1mm that is sweet $100K that the big cheese keeps. (not you, you go back to the mines and since you demoed you can do it at 90%, next time you have to do it 80% time. and, no i am not sore ;) )
tl;dr My first reaction to Duffy's statement was "you have no clear idea yet, but this is your best guess". And, we may never know.
I don't know what happened, but for me a red flag was the specific explanation given while the event was still ongoing: "overloaded router", followed by request for more money.
If the cause was so simple , then it was unclear why it wasn't fixed immediately, so something does not quite add up (at least till an independent investigation)
Unless I’m just tired, the article, reporter and air traffic controllers are confused about hardware switches and routers having issues, and the act of switching out hardware. Duffy stated that the switches and routers were having issues and needed to be shut down. This seems to have been read as physical hardware switches, read ‘replacements’, were happening during the day.
Asking air traffic controllers for their opinion on this is a bit odd — would they even know about networking equipment swaps? I would expect that to be handled by the engineers and technicians who work on the system, not the people who direct the planes.
They wouldn't necessarily know the specifics of upgrades, but they would likely know when upgrades typically happen because it's normal to send out emails about maintenance windows.
"Refute" makes it sound like there's juicy insider info, but it's just skepticism that FAA scheduled upgrades on a Friday daytime.
This was a cascading multicast failure, which can take hours to build up. Earliest reports of disruptions came no later than 10:30 am, so it's quite possible the upgrades were performed in the overnight hours on Thursday.
Him saying during the press release that the United States is "the greatest country ever" with "the greatest president ever" was enough to want me to see him rot in a cell with the rest of them.
Even if MS stuff was not in any critical path it still could have played havoc with coordination of an update or a change, contributing to the failure.
Also FYI foil hats are actually resonant cavities and amplify the mind control waves. All true paranoids use Faraday mesh, preferably as a full head covering.
I’m not sure which version of Microsoft the FAA uses, but GCC High and the DoD version of Microsoft 365 have been working just fine through the commercial outage of Microsoft 365
- There is almost always a standard operating procedure (SOP)
- Just b/c there is a SOP for releases etc, that doesn't mean that people always follow it
- This can be due to many reasons e.g. laziness, being distracted, pressure from higher ups ("just push it out there and I'll take the risk" ha!), tired from having a newborn etc etc
- Doing things outside the "green zone" deployment windows therefore always carries risk
- All of this being said, EVEN IF you ARE following SOP, things still break in all kinds of weird ways
- And the end user is not often the best observer nor do they have all of the context to determine what broke, how it broke and why. (There are exceptions to this though)
0 - https://x.com/alexpotato/status/1215876962809339904
Your third bullet is a bane of gov contractors. Not the sales people, but the people actually doing the work. Imagine if you had an FFP contract and you finished the job within 90% of the paid time. At $1mm that is sweet $100K that the big cheese keeps. (not you, you go back to the mines and since you demoed you can do it at 90%, next time you have to do it 80% time. and, no i am not sore ;) )
tl;dr My first reaction to Duffy's statement was "you have no clear idea yet, but this is your best guess". And, we may never know.
This was a cascading multicast failure, which can take hours to build up. Earliest reports of disruptions came no later than 10:30 am, so it's quite possible the upgrades were performed in the overnight hours on Thursday.
It's getting quite long with all the blatent criminals currently in the whitehouse.
The fascist bootlicking is quite tiresome.
Even if MS stuff was not in any critical path it still could have played havoc with coordination of an update or a change, contributing to the failure.
Also FYI foil hats are actually resonant cavities and amplify the mind control waves. All true paranoids use Faraday mesh, preferably as a full head covering.