Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Categories

In this Discussion

› › CrowdStrike update caused BSOD on hospitals, banks, servers worldwide - Page 3
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

CrowdStrike update caused BSOD on hospitals, banks, servers worldwide

1356

Comments

  • MumblyMumbly Member
    edited July 2024

    @default said: And this was likely done automatically, because Windows does updates automatically.

    You don’t get it, honey, do you? It’s not a Windows update, it’s a CrowdStrike update.
    Oh, and the antivirus updates automatically? Well, isn’t that a surprise…

  • defaultdefault Veteran

    @dev_vps said:
    Handwritten boarding pass issued as system was down. Image taken from internet.

    It seems we can always return to pen and paper.

  • defaultdefault Veteran

    @Mumbly said:

    @default said: And this was likely done automatically, because Windows does updates automatically.

    Oh, the antivirus updates automatically? Well, isn’t that a surprise…

    Initially it was announced a Windows problem as in Windows update. Now it turns out to be an update from CrowdStrike which affected Windows machines.

    You were right to blame the software.

  • JerryHouJerryHou Member

    Biggest IT down day ever?

  • LeviLevi Member

    Crowdstrike stocks plumeting this month. But overall they are doing fine.

  • SplitIceSplitIce Member, Host Rep
    edited July 2024

    Thanked by 5_MS_ default Void sh97 Chuck
  • LeviLevi Member

    @JerryHou said:
    Biggest IT down day ever?

    Y2K was bigger afaik.

  • jfreak53jfreak53 Member, Patron Provider

    Maybe after this blunder the world might learn they need to run critical on linux not windows 🤦🏻‍♂️

  • COLBYLICIOUSCOLBYLICIOUS Member

    Just imagine if Cloudflare will go down how many websites will be dead.

    Thanked by 1sasslik
  • LeviLevi Member

    @COLBYLICIOUS said:
    Just imagine if Cloudflare will go down how many websites will be dead.

    And cf has good recipe for disaster - single homed domain. All their 400+ nameservers are *.cloudflare.com

    Thanked by 1t0m
  • bootboot
    edited July 2024

    @Levi said:

    @JerryHou said:
    Biggest IT down day ever?

    Y2K was bigger afaik.

    Were you alive for Y2K? Nothing happened.

    @COLBYLICIOUS said:
    Just imagine if Cloudflare will go down how many websites will be dead.

    Cloudflare could go away permanently and I wouldn't lose a rem of sleep.

    Thanked by 1kevinds
  • bootboot

    Blue screen of def.

    Thanked by 1Maounique
  • raindog308raindog308 Administrator, Veteran

    Yawn...yeah, been up since 3am working on this.

    @default said: Because they are idiots. They never heard of *BSD or Linux. Many people think that paying a license fee makes things easier and the problems magically go away.

    Tell me you have never worked in a large enterprise environment without telling me.

    Big orgs have thousands, maybe tens of thousands of interconnected applications. You don't always get to pick the platform.

    Every Fortune 1000 CIO has heard of Linux and every single one deploys a ton of Linux, other than maybe MSFT itself.

    When I was on the Linux Foundation advisory board in the early 2000s, it was filled with big household name company CIOs and they all were very eager to deploy Linux because it was commoditized and cheaper. They are not "idiots". They just work in the real world.

    Thanked by 5jsg Void Cabbage Voltrina mrTom
  • defaultdefault Veteran
    edited July 2024

    @COLBYLICIOUS said:
    Just imagine if Cloudflare will go down how many websites will be dead.

    That would bad for the moment, but it might prove to be good for the future of a more decentralized web. In my humble opinion too much stuff is reliant on a single brand.

  • yoursunnyyoursunny Member, IPv6 Advocate

    @COLBYLICIOUS said:
    Just imagine if Cloudflare will go down how many websites will be dead.

    How to ask for a Cloudflare refund for a website on the free plan?

  • COLBYLICIOUSCOLBYLICIOUS Member

    @yoursunny said:

    @COLBYLICIOUS said:
    Just imagine if Cloudflare will go down how many websites will be dead.

    How to ask for a Cloudflare refund for a website on the free plan?

    "I am losing millions of dollars/minute"

  • LeviLevi Member

    @boot said: Were you alive for Y2K? Nothing happened.

    I was in my late 50 at that moment. "Nothing happened"... It made to the movies! And whole world (only murica tbh) spoke about it! That's huge.

  • defaultdefault Veteran

    @raindog308 said:
    Yawn...yeah, been up since 3am working on this.

    @default said: Because they are idiots. They never heard of *BSD or Linux. Many people think that paying a license fee makes things easier and the problems magically go away.

    Tell me you have never worked in a large enterprise environment without telling me.

    Big orgs have thousands, maybe tens of thousands of interconnected applications. You don't always get to pick the platform.

    Every Fortune 1000 CIO has heard of Linux and every single one deploys a ton of Linux, other than maybe MSFT itself.

    When I was on the Linux Foundation advisory board in the early 2000s, it was filled with big household name company CIOs and they all were very eager to deploy Linux because it was commoditized and cheaper. They are not "idiots". They just work in the real world.

    I made an assumption of Windows fault based on the information made public through media as a problem coming from an Windows update. I was wrong. I humbly take my words back in light of new information.

    To answer your challenge: No. I never worked in large enterprise environment with thousands of applications.

    Thanked by 1Mumbly
  • MumblyMumbly Member

    @jfreak53 said:
    Maybe after this blunder the world might learn they need to run critical on linux not windows 🤦🏻‍♂️

    Really? If the people who say this only knew their Linux...

    Let me explain you. This is not Windows' fault and could happen to any OS.
    A properly written AV recognition pattern would probably crash any OS as AV operates at the kernel level. Even in Linux, a poorly written kernel module can crash the system.

  • LeviLevi Member

    We need something from IBM…

  • dev_vpsdev_vps Member

    None of our Windows Servers or client workstation affected at my work place.

    So this is definitely nothing to do with Windows Auto update.

  • dosaidosai Member

    https://news.ycombinator.com/item?id=41003390

    Thanked by 2Mumbly Chuck
  • hecataehecatae Member

    Crowdstrike have managed to shave 11% off their share price before the NASDAQ has even opened for trading:
    https://www.nasdaq.com/market-activity/stocks/crwd

    Crowdstrike CEO is calling it a Content Update, whatever that is, I do not use Microsoft for anything I use.

    Thanked by 1jsg
  • MumblyMumbly Member

    @dosai said:
    https://news.ycombinator.com/item?id=41003390

    Thanks!

    Throwaway account...
    CrowdStrike in this context is a NT kernel loadable module (a .sys file) which does syscall level interception and logs then to a separate process on the machine. It can also STOP syscalls from working if they are trying to connect out to other nodes and accessing files they shouldn't be (using some drunk ass heuristics).

    What happened here was they pushed a new kernel driver out to every client without authorization to fix an issue with slowness and latency that was in the previous Falcon sensor product. They have a staging system which is supposed to give clients control over this but they pissed over everyone's staging and rules and just pushed this to production.

    This has taken us out and we have 30 people currently doing recovery and DR. Most of our nodes are boot looping with blue screens which in the cloud is not something you can just hit F8 and remove the driver. We have to literally take each node down, attach the disk to a working node, delete the .sys file and bring it up. Either that or bring up a new node entirely from a snapshot.

    This is fine but EC2 is rammed with people doing this now so it's taking forever. Storage latency is through the roof.

    I fought for months to keep this shit out of production because of this reason. I am now busy but vindicated.

    Edit: to all the people moaning about windows, we've had no problems with Windows. This is not a windows issue. This is a third party security vendor shitting in the kernel.

  • DPDP Administrator, The Domain Guy

    Thanked by 7jsg MannDude host_c sasslik xvps sh97 mrTom
  • FAT32FAT32 Administrator, Deal Compiler Extraordinaire

    I have to explain to so many people it is NOT freaking Microsoft issues but Crowdstrike...

    Thanked by 1hecatae
  • MumblyMumbly Member

    @FAT32 said:
    I have to explain to so many people it is NOT freaking Microsoft issues but Crowdstrike...

    I am doing this since the beginning of this thread... :P

    Thanked by 1FAT32
  • dev_vpsdev_vps Member
    edited July 2024

    @FAT32 said:
    I have to explain to so many people it is NOT freaking Microsoft issues but Crowdstrike...

    Absolutely.
    Not a Microsoft Windows issue.
    Here is my @host_c VPS running Windows Server 2022 OS

    -- it would have been much more than 38 days, but I had to restart when upgraded to 10TB storage

    Thanked by 2host_c hecatae
  • MaouniqueMaounique Host Rep, Veteran

    @raindog308 said: other than maybe MSFT itself.

    MS runs on Linux :P

    "Native Azure services are often running on Linux. Microsoft is building more of these services. For example, Azure's Software Defined Network (SDN) is based on Linux."

    Of course, that is not an official policy and there are still tons of services running on Windows, but you get the picture.

    Thanked by 3raindog308 tentor mrTom
  • VoidVoid Member

    Meanwhile Microsoft presale teams:

    Reasons why you should pick MDE for your EDR needs.

    1) Not Crowdstrike

    Thanked by 3hecatae host_c mrTom
This discussion has been closed.