NSA Internal Network Down

(reuters.com)

141 points | by nodoodles 2 hours ago

23 comments

  • lucascdotnet 1 hour ago
    I wonder (and have no idea) whether the increased reports of services being down is a case of the Baader–Meinhof phenomenon, or a actual increase in failure rate.
    • btown 37 minutes ago
      If I was asked to armchair-pattern-match, I'd say that there's a combination of two things: (a) a wider fear-based environment where rapid modernization is being pushed heavily, and (b) an increase in tools that give false confidence on change management.

      AI is tuned to speak confidently and ship independently; it is rare that it will suggest that one go back to a broad survey of stakeholders to ensure no Chesterton's Fence is being torn down (https://fs.blog/chestertons-fence/). "I've read the documentation and logs, and nothing suggests this system needs these legacy capabilities/components/capacity" is something we hear all day, and it is a siren's song if there ever was one.

      Couple that with a general (and valid!) consensus that modernization of legacy systems must be accelerated for a variety of reasons - to mitigate this new and continuing "Black October" of hyper-agile agent-driven cyberattacks (whether the network is public or private); to have more flexible infrastructure for new features to stay ahead; to allow human teams to show their efficacy in a world where technology jobs are increasingly at risk - and you have a perfect storm where change management is no longer what it once was.

    • Someone1234 35 minutes ago
      That's my impression; a total of zero of the popular recent ones have been related to "AI" or Cyber attacks, but every comment in every thread, talks about those two as if they were the underlying cause.

      The FAA's recent outage for example, at its core, seems like people trying to rush needed upgrades that were slow for a specific reason (e.g. reducing the testing window for 90 to 30-days).

      The Microsoft outage seems like three different things linked together (and poor "full failure" recovery prep, lengthening the outage).

      The bank "outage" may not have even existed, 2/3 of the banks claimed they didn't have an outage at all, and the other said it was a limited technical issue. The articles were based on DownDetector spikes.

      • rkozik1989 14 minutes ago
        You need to be more skeptical of what organizations say when they have a lot to lose. Nobody takes responsibility for situations unless there's way to offload some or all of the blame. They have every incentive to lie about the situation to make it go away. Politics often plays the biggest role in these situations.
        • StilesCrisis 11 minutes ago
          I logged into my bank when the report came out and did not experience any hiccup. I think this was DownDetector-based reporting.
    • kilroy123 12 minutes ago
      Possible, but let me ask you and others. What does your gut say? Mine is screaming that something just isn't right right now.

      I can't put my finger on it, but something sinister feels afoot.

      • eterm 6 minutes ago
        UK government recently launched a "Be Ready" preparedness campaign.

        https://beready.campaign.gov.uk/

        It's nowhere near the kind of readiness and preparedness campaigns of the cold-war of course, but it's a step up in alertness that's higher than anything I can remember.

      • 0cf8612b2e1e 7 minutes ago
        My personal barometer: when do government insiders sell all of their stock? They have already done all of the insider trading. Fleeing the sinking ship the second they hear of Bad News would be in line with previous behavior.
      • dualvariable 5 minutes ago
        Most of the rest of them I can wave off as probably incompetence and maybe augmented by LLM-psychosis in decision-makers, and the general level of chaos in IT these days. The NSA internal network being down is pretty suspicious. Maybe the Trump administration rot has just spread that far, but that is making me think there could be actual cyberwarfare going on.
    • matthewbauer 11 minutes ago
      Must be a little bit of both right? Like FAA grounding + Microsoft outage are both notable on their own, but the subsequent things would usually be background noise IMO.

      I also think generally that AI is probably both the cause and solution to these problems. We're certainly getting more code churn than otherwise, and in lots of cases that's really good! But you're going to have to deal with inevitable instability too.

    • redwood 2 minutes ago
      With the trilateral summit scheduled for Thursday it's possible some showing off is going on between the three letter agencies in the three countries... after all they can be first movers on vulns that are likely to be patched soon
    • losvedir 5 minutes ago
    • NitpickLawyer 59 minutes ago
      100%. Everything is rage-bait nowadays. And AI is the newest toy they can use. Every time we've had something go down in the past ~6months, "AI did it" was all over the comments.
      • rsoto2 50 minutes ago
        https://www.optimizely.com/field-notes/articles/how-we-use-a...

        https://www.bleepingcomputer.com/news/security/optimizely-su...

        Is it possible that the false of productivity is leading to worse products and massive loss in valuable headcount? Instead of AIs directly writing bad buggy code? Or is "everything rage-bait, and AI is all sunshine"

        • cochne 46 minutes ago
          These kinds of things happened before AI. It's not clear whether the incidence has increased since AI, or maybe AI is leading to faster development and therefore higher rates of incidents may be due to shipping more code, faster, while perhaps maintaining a similar or better level of incident to code/progress ratio. Or, there could just be more attention as GP suggests. Pointing to one example of a company that's using AI and has has a major incident doesn't really tell us much, especially when most companies who haven't had major incidents are probably using AI in similar amounts.

          It would be interesting if someone could do a proper analysis, though!

        • ragequittah 29 minutes ago
          I believe it's also possible that a large portion of the world wants this to be true more than it actually is so attribution to things that have nothing to do with AI go directly to AI as a knee-jerk reaction.
          • GolfPopper 25 minutes ago
            I'm not a fan of where I see LLMs taking us, but they're just the latest notch in decades-long institutional decay.
          • AeroNotix 6 minutes ago
            If we're also just vibing it (like the majority of code these days) then I vibe that there are a lot of AI-apologists that desperately don't want their magic machine to be cast in a negative light, lest their easy i-win button gets taken away or devalued.
        • wolfinterval 5 minutes ago
          I would assume if anything it is that the open weight models are lagging by 6-8 months and 6 months ago Mythos was announced.

          It would hardly be shocking that hostile nations are able to cause some problems with Mythos level tools.

          If anything, it is more interesting how we can invade another country and the idea that the country or a collection of hostile countries could hit us back in any way, shape or form is just an impossibility. An insane conspiracy theory.

          Maybe it is not that but surely the probablity is not zero.

        • femboyvtuber 43 minutes ago
          [dead]
      • matt3210 45 minutes ago
        what was your system prompt again?
    • EvanAnderson 49 minutes ago
      Aside: I forget the name of the Baader–Meinhof phenomenon when it hasn't been discussed in awhile. When I'm reminded of it I begin seeing it happening frequently for the next few days. I wish I was joking. I assume it's the Baader–Meinhof phenomenon phenomenon.
      • smolder 20 minutes ago
        The term "frequency illusion" is a lot more memorable and intuitive. It's what I prefer.
        • ihumanable 10 minutes ago
          I've heard the term "frecency bias" a portmanteau of frequency and recency to refer to people focusing intently on a set of outlier events that happen to occur with a heightened frequency very recently.

          I'm sure most of us here have been on the receiving end of managers suddenly obsessed with a failure mode because it happened to occur twice in one month.

    • kesslern 35 minutes ago
      This situtation reminds me of the East Palestine train derailment in Ohio. A bad train accident caused a big chemical spill, made national news. For months afterward, any issue with any train in the state was reported as news. People thought the trains were breaking down. In reality, train derailments happen pretty often but it's not usually a big deal so nobody hears about it.
    • 220hertz 17 minutes ago
      I think it lies at the intersection of Occam's and Hanlon's razors. The simplest explanation is incompetence brought on by a lack of skill from cuts, and therefore stupidity not malice.
      • warkdarrior 10 minutes ago
        It's probably both stupidity and malice, especially on the government side. Project 2025 makes sure that malice is there as a starting point and then the current loyalty-driven hiring practices means that stupidity is a key factor in the execution.
    • dmix 1 hour ago
      The media will certainly try to give that impression either way. They serve the doomscrollers now.
      • causal 44 minutes ago
        Yeah "cyber thing went down" has certainly become a reporting trend. I'm not sure we would have even heard about NSA internal network issues a year ago. May or may not be significant.
    • lettergram 11 minutes ago
      Everyone is being pushed for rapid updates, the AI is doing the updating (error prone) and there's actually an increase in CVEs and potential risks.

      The combination is a cycle

      1. AI finds bugs

      2. Engineers us AI to try and fix

      3. AI testing approves, minimal engineering review

      4. Engineers push release rapidly, it's critical after all

      5. Bugs introduced

      6. Cycle starts again

    • stronglikedan 14 minutes ago
      This is all just to distract us from something much, much worse! /s
  • tgsovlerkhgsel 1 hour ago
    That's a LOT of things going down, and I haven't heard of some major infrastructure outage likely to affect many sites in this manner.

    Coincidence or is there some common explanation that I missed?

    • bhouston 1 hour ago
      There is a lot of things going down.

      I think the opportunity right now where we have powerful AI hacking systems but not enough protections is to establish long-term presence in all networks of interest. Not bring them down.

      • colinrand 1 hour ago
        Completely agree. Things are moving too fast and so are breaking.

        For example, we updated a GCP managed base image of Ubuntu that was released with 350 CVEs patched, some were as high as 9.8. In doing so, they (GCP) changed the DEFAULT setting for hardware GRO from off by default to on. That's a massive networking change packed into a security update.

        We only caught it AFTER it started causing issues in production. We didn't have the OS pinned because (decently valid reasons) or catch it in non-prod (decent reason + human mistake).

        This is on a modern highly automated tested resilient well architected platform.

        I can only imagine these other platforms are dramatically more exposed that we are, hence these common large scale outages.

      • cj9 14 minutes ago
        Also, anyone anywhere that horded/designed exploits will be wanting to use their backlog of these "assets" before systems are patched. From criminal networks to nation states.

        Outages could be grey-hat testers and emergency updates.

        A temporary period like we are experiencing is to be expected when moving to a time where a single engineer can perform the equivalent work of a team of excellent engineers working for weeks. Security has always been a game of who has the capabilities, means, and requirement for any given attack surface.

      • pianopatrick 29 minutes ago
        That assumes that people did not already have long-term presence in networks of interest.

        More plausible to me is that spy agencies had previously found and used some of these hundreds or thousands of vulnerabilities to hack into those networks just fine. But now other people like cyber criminals are using these vulnerabilities via AI. That is messing up both the attackers who had already gotten in quietly and do not want the attention and the defenders who now have to respond to a noisy intrusion they cannot ignore.

      • daheza 1 hour ago
        None of these have been due to hacking? It has all been just standard issues probably increased by cost cutting measures.
        • bhouston 1 hour ago
          Or the increased use of AI agent coding.
          • johnofthesea 11 minutes ago
            All of above and incompetent people in power. (Not implying in this specific case).
    • postsantum 22 minutes ago
      Appstore connect is also down
    • chabes 1 hour ago
      What do people expect when top government offices are occupied by former Fox News hosts?
  • gnabgib 1 hour ago
    CNN Source (27 points, 5 comments) https://news.ycombinator.com/item?id=50532849
  • ragle 28 minutes ago
    I have also been curious about frequency bias / Baader–Meinhof - but I am also wondering: what are the chances that a state actor is responsible for the last week of issues?

    Trump is headed to China for a meeting with Xi and Putin.

    We have 3 carriers off the coast of Iran and (despite Trump's pleas) NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil infra.

    I've seen a few takes from L3 Harris / FAA employees calling the official line about the ATC issue bunk and there are 3 other major issues in the last week, now this NSA outage - curious if these incidents could be the BRICS axis flexing their own cybermuscles (independently in multiple separate attacks with carefully placed assets) ahead of their meeting with Trump in China?

    An interesting geopolitical Mexican standoff, if true.

    I'm not claiming this is true. I'm literally pulling it out of my rear with little context beyond a bunch of random news stories and X posts, it's pure pedestrian speculation / curiosity.

    Would love someone steeped in national security / geopolitics (like: professionally as their job, not armchair analysts) to offer their take on this.

    Is this a dumb, naive theory?

    • GolfPopper 19 minutes ago
      >NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil infra.

      I know it's not the thrust of your post, but I see this or similar framing a lot, and I feel compelled to point out that Putin can end the strikes on his oil infrastructure at any time by ending the war he started by invading another country.

      • ragle 17 minutes ago
        Oh, I couldn't agree more. Long time supporter of Ukraine in many ways here.

        Historically he's chosen to go about things differently, which I guess is part of what is tickling the pattern matching part of my brain here.

    • ranger_danger 4 minutes ago
      I suppose rather than straight up 'cyberattack', that some kind of sabotage is also possible... but I don't see how that could ever really be proven without massive state-level investigations... at which point people will also question whether they are themselves involved somehow.
    • whynotmaybe 14 minutes ago
      > NATO is essentially pushing their boot into Putin's neck w/ continued Ukrainian strikes on Russian oil

      I'm not sure about naive or dumb but you might review your POV regarding this.

  • dist-epoch 6 minutes ago
    What if they were post training a cyber model and got hacked from inside by the model, like OpenAI got hacked by it's own HuggingFace swarm.
  • bayat 40 minutes ago
    Following the series of such events over the past couple of days, I have to say I’m a bit surprised by how quickly some in this community are brushing them off as AI-related pain or simply another example of the clickbait-driven media ecosystem.

    I understand the skepticism, and I certainly don’t think we should go down a rabbit hole of conspiracy theories. But shouldn’t we also be a little skeptical of dismissing these events so quickly? Is it unreasonable to at least consider the possibility that they’re not entirely coincidental, or to wonder why so many seem to be taking place in the US?

    • arrowleaf 25 minutes ago
      IMO it's unreasonable not to think they are all related in some way, even if that relation is just bad actors seeing what other bad actors have been capable of recently. This all started with the freezers in Europe and seems to be snowballing since.
  • ericcholis 1 hour ago
    There's a part of me that doesn't want to know this....
  • swalsh 35 minutes ago
    I have this distinct memory in I think Jan 2020 where I looked at the front page of CNN or NYT or something. Big picture of Trump impeachment, and proceedings. Small picture at the bottom of the page of China dealing with this virus. Worthy news, but not top page news. Nobody was paying attention. Feb, more people started paying attention, but it wasn't top page yet. By March, people were starting to realize that this is a serious crisis that's going to severely impact their lives.

    I'm getting Feb 2020 vibes. I don't know what's going on, but this feels like more than just some routine outages going on.

    • ssl-3 31 minutes ago
      I was sicker than I'd ever been for ~all of February of 2020.

      So far this fall, none of my systems have gone haywire (yet!), so I forecast that we'll be OK.

      • techmunky 3 minutes ago
        you probably had covid mate
    • GolfPopper 9 minutes ago
      The fantasy/cyberpunk TTRPG Shadowrun depicted a world in upheaval after a global computer crash in the late 2020s. In addition to the pandemics and the AIs.
    • sanex 26 minutes ago
      Well the good news is that even though 2020 and 2021 were weird years we're still here. Humanity is resilient, we always figure it out.
  • stevenhubertron 1 hour ago
    It’s been a hella few weeks. Hard to believe it’s coincidence even if you want to blame AI.
    • rglover 1 hour ago
      It's AI. Or more specifically: rushed, careless usage of AI under the pretense that it's already "solved" software and you can haphazardly push junk to the world and expect real systems to run on it.
      • londons_explore 37 minutes ago
        Traditional software development normally involves at least a few veterans on the team familiar with almost every line of code.

        However, AI works like a speedy intern who hasn't yet got to grips with the whole codebase, and sometimes makes assumptions. That's mostly because the codebase is usually bigger than the context window, so it is 'writing blind'.

      • ranger_danger 36 minutes ago
        Do you have any sources for this claim?
        • nope12123123 8 minutes ago
          This statement is self evidently true to anyone who's worked at any capacity in software teams
      • cokeandpepsi 21 minutes ago
        [dead]
  • howmayiannoyyou 37 minutes ago
    https://chinaselectcommittee.house.gov/media/reports/strange...

    Allowing our infrastructure to be manufactured elsewhere, maintained by companies who struggle to retain top talent, and overseen by folks out of touch with cutting edge threats, and funded on annual budget cycles by legislators even more out-of-touch ... is a seemingly insurmountable set of obstacles.

  • reedf1 2 hours ago
    Anyone have knowledge of the backend of this system?
    • cucumber3732842 6 minutes ago
    • aeonik 1 hour ago
      Are you implying that you DON'T have detailed docs and architecture of one of the most classified and secretive organizations in the world? ;-P
      • infinite_spin 1 hour ago
        php and spit
        • grep_it 48 minutes ago
          ... and Doug's perl script, which was never committed to source control. It lives on a single box and silently upholds a number of implicit system invariants. In someones inbox is an email from Doug, sent just before he retired, that the script needs to be copied over to the new box before the hardware migration is complete.
      • doublerabbit 1 hour ago
        You telling me you don't? Bro just download it off Limewire.
        • SketchySeaBeast 50 minutes ago
          We might have more luck checking the War Thunder Forums.
          • zmgsabst 24 minutes ago
            Wireporn on Discord can have some spicy server/rack labels.
    • Havoc 27 minutes ago
      My money is on something absolutely antique but wrapped in modern post quantum encryption.

      Not unlike Bloomberg terminals…dos era UI but does the job well

    • bix6 41 minutes ago
      Just join Hegseth’s signal chat and he’ll give you the deets. Ideally pick a username like WAREAGLE.
    • Zambyte 40 minutes ago
      Surely
  • joshka 1 hour ago
    > The NSA did not immediately return a message seeking comment about the reported outage.

    Did you try sending it via outlook.com?

    • Semiapies 29 minutes ago
      Just whisper the question to your phone.
  • chinathrow 11 minutes ago
    Well, on the other hand, a bridge is down as well.
  • classified 0 minutes ago
    Does that mean we can shitpost Trump jokes without getting arrested at the border?
  • redwood 6 minutes ago
    "The NSA did not immediately return a message seeking comment about the reported outage". There's something humorous about that framing
  • kinabalu76 30 minutes ago
    I see the shoe is now on the other foot NSA.
  • demo4567 11 minutes ago
    More fear mongering about day to day incidents: The network goes down every time I mess with DNS, VPC peering, TGWs, route tables.
  • throw7 46 minutes ago
    Memo to the NSA... pulling the plug won't work. ;)
  • swozey 1 hour ago
    Just in time for the FEMA national emergency test this week
  • vrganj 1 hour ago
    There's some speculation that all of these recent outages might be related to attempts at taking Genesis Mission online - the Trump / Oracle project to have AI tie into government and private services:

    https://en.wikipedia.org/wiki/Genesis_Mission

    https://substack.com/@thedreydossier/note/c-389611426