Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Backdoor in upstream xz/liblzma leading to SSH server compromise (openwall.com) (via Hacker News) - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Backdoor in upstream xz/liblzma leading to SSH server compromise (openwall.com) (via Hacker News)

2

Comments

  • SplitIceSplitIce Member, Host Rep

    Going through my servers, looks like Debian being a bit behind the latest saved me from being affected. Even the staging which runs a newer environment is golden.

    Really goes to show though how you are dammed if you do, dammed if you don't (keep your system entirely up to date).

  • daviddavid Member

    I'm not affected. But I'd think if someone has already been running an infected system, the solution isn't just to update xz, but you'd want to wipe the system clean and restore from a known good backup from before the infection. If it was exploited, who knows what damage was already done.

    Thanked by 1Carlin0
  • Not_OlesNot_Oles Moderator, Patron Provider

    Fedora Rawhide reverted xz.

    Subject: Fedora rawhide compose report: 20240329.n.1 changes
    From: Fedora Rawhide Report <rawhide@xxxxxxxxxxxxxxxxx>
    Date: Sat, 30 Mar 2024 03:03:03 +0000 
    
      [ . . . ]
    
    Package:      xz-1:5.4.6-3.fc41
    Old package:  xz-5.6.1-1.fc41
    Summary:      LZMA compression utilities
    RPMs:         xz xz-devel xz-libs xz-lzma-compat xz-static
    Size:         3.77 MiB
    Size change:  -189.70 KiB
    Changelog:
      * Thu Mar 28 2024 Richard W.M. Jones <rjones@xxxxxxxxxx> - 1:5.4.6-3
      - Revert to 5.4.6, bump epoch
    
    

    One of the MetalVPS servers is running an affected version of xz.

    [Not_Oles@polonium ~]$ xz --version
    xz (XZ Utils) 5.6.1
    liblzma 5.6.1
    [Not_Oles@polonium ~]$ 
    

    What I did while waiting for the Fedora fix was firewall off port 22 except for my main and backup IPs. As of a few minutes ago, it looks like the latest Fedora Rawhide update hasn't propagated to my mirror yet.

    [root@polonium ~]# # Update to revert xz
    [root@polonium ~]# # https://www.spinics.net/lists/fedora-devel/msg319328.html
    [root@polonium ~]# date
    Sat Mar 30 04:56:24 AM UTC 2024
    [root@polonium ~]# dnf update --refresh
    Fedora rawhide openh264 (From Cisco) - x86_64                     2.5 kB/s | 989  B     00:00    
    Fedora - Rawhide - Developmental packages for the next Fedora rel  26 kB/s |  12 kB     00:00    
    Fedora - Rawhide - Developmental packages for the next Fedora rel 667 kB/s | 442 kB     00:00    
    Dependencies resolved.
    Nothing to do.
    Complete!
    [root@polonium ~]# # Not on mirror yet, apparently.
    [root@polonium ~]# 
    

    The report on oss-security was amazing![1]

    [1]https://www.openwall.com/lists/oss-security/2024/03/29/4

  • drkdrk Member

    @pangkus said:
    Is it just priv/pub key that they steal?

    From the analysis in oss-security, RSA_public_decrypt function is being redirected to the malware code, so it affects public-key ssh authentication, i.e. priv keys are probably compromised. Maybe user-password authentication is not affected.

  • Could firewall prevent it?

  • MaouniqueMaounique Host Rep, Veteran

    @hiphiphip0 said: Could firewall prevent it?

    Of course not, unless allows access for only your IPs/ranges. Still, a bot would most likely check only port 22 attempting an exploit.

    Thanked by 1hiphiphip0
  • Not_OlesNot_Oles Moderator, Patron Provider

    @Not_Oles said: One of the MetalVPS servers is running an affected version of xz.

    [Not_Oles@polonium ~]$ xz --version
    xz (XZ Utils) 5.6.1
    liblzma 5.6.1
    [Not_Oles@polonium ~]$ 
    

    Fixed now. Got the xz revert! :)

    [Not_Oles@polonium ~]$ date -u
    Sat Mar 30 04:07:46 PM UTC 2024
    [Not_Oles@polonium ~]$ xz --version
    xz (XZ Utils) 5.4.6
    liblzma 5.4.6
    [Not_Oles@polonium ~]$ 
    
    Thanked by 1dogemeister
  • Update from Lasse Colin (the original/main maintainer of xz utils) https://tukaani.org/xz-backdoor/

    Arch Update: https://archlinux.org/news/the-xz-package-has-been-backdoored/

    Thanked by 1Not_Oles
  • matey0matey0 Member

    @nullnothere said:
    Update from Lasse Colin (the original/main maintainer of xz utils) https://tukaani.org/xz-backdoor/

    They probably targeted xz specifically because they knew the primary maintainer takes regular breaks and
    a) would probably be looking for another maintainer to cover for him and
    b) wouldn't be reviewing commits during that time.

    I think everything speaks for the attackers having joined the project with the long-term intention of doing this as opposed to using this opportunity spontaneously.

    Lasse regularly has internet breaks and is on one at the moment, started before this all kicked off. He has posted an update at https://tukaani.org/xz-backdoor/ and is working with the community.

  • matey0matey0 Member
    edited March 31

    An analysis of the obfuscated bash stage by one of my fav security researchers: https://gynvael.coldwind.pl/?id=782

    tl;dr:

    This whole thing basically looks like an "extension/patching" system that would allow adding future scripts to be run in the context of Stage 2, without having to modify the original payload-carrying test files. Which makes sense, as modyfing a "bad" and "good" test files over and over again is pretty suspicious. So the plan seemed to be to just add new test files instead, which would have been picked up, deciphered, and executed.

    and

    Someone put a lot of effort for this to be pretty innocent looking and decently hidden. From binary test files used to store payload, to file carving, substitution ciphers, and an RC4 variant implemented in AWK all done with just standard command line tools. And all this in 3 stages of execution, and with an "extension" system to future-proof things and not have to change the binary test files again. I can't help but wonder (as I'm sure is the rest of our security community) – if this was found by accident, how many things still remain undiscovered.

    And a nice summary of the events that took place for the attackers to become trusted in the project (and other suspicious things they did): https://boehs.org/node/everything-i-know-about-the-xz-backdoor
    It makes even more clear that this was a targeted operation, having noticed that the maintainer is vulnerable. They pressured/"bullied" him under multiple accounts which led to this: https://www.mail-archive.com/[email protected]/msg00567.html

    I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see.

  • drkdrk Member

    @Maounique said:

    @hiphiphip0 said: Could firewall prevent it?

    Of course not, unless allows access for only your IPs/ranges. Still, a bot would most likely check only port 22 attempting an exploit.

    While I'm not a security expert, I believe that utilizing iptables to block users/bots attempting to guess usernames could significantly hinder an attacker's success in activate the backdoor. The new analysis on payload shows that it is RCE attack and not bypass authentication. So, using firewall certainly will reduce the attack surface. The first step of the attacker is to send a public-key, but he also needs the right username of the system, here is an example of a possible implementation of the previous : https://gist.github.com/keeganryan/a6c22e1045e67c17e88a606dfdf95ae4 .

  • MaouniqueMaounique Host Rep, Veteran

    @drk said: I believe that utilizing iptables to block users/bots attempting to guess usernames could significantly hinder an attacker's success in activate the backdoor

    I am not sure what you mean. Of course it helps, but a firewall can't block that kind of exploit or even RCE. Once the connections is made, unless we run DPI, the firewall is off.
    Running DPI can help with many types of attacks, but it is a bitch to get right and not hinder more than help.

  • drkdrk Member
    edited March 31

    @Maounique
    From what I read in order the attacker to make a connection sends a public key, i.e. he uses the publickey method to get the backdoor activated through the lzma vulnerable code. Of course he needs a username (I suppose root login is not the case). Here the attacker probably will enumerate some common usernames I suppose. So, a firewall such as iptable can block such attempts.

  • matey0matey0 Member
    edited March 31

    @drk said: Of course he needs a username

    Do you have evidence on that? I've found nothing conclusive.
    It may be the case - but it may also be the case that sshd calls the hooked public-key decryption function even for invalid users (to log information about the login attempt for example).
    The RCE itself will happen as the user sshd is running as (usually root), before sshd has dropped privileges for the user shell session.
    Thus if the hooked function is executed for an invalid user the attacker still gains root access.

    Either way, unless a server explicitly disables root logins the exploit will certainly work - regardless of any firewall configuration.

    The main take-away from what has been posted about the payload today is: Random script kiddies will probably not be able to exploit your server. It only works with the attacker's private key.

    Thanked by 1drk
  • drkdrk Member

    @matey0 said:

    @drk said: Of course he needs a username

    Do you have evidence on that? I've found nothing conclusive.
    It may be the case - but it may also be the case that sshd calls the hooked public-key decryption function even for invalid users (to log information about the login attempt for example).
    The RCE itself will happen as the user sshd is running as (usually root), before sshd has dropped privileges for the user shell session.
    Thus if the hooked function is executed for an invalid user the attacker still gains root access.

    Either way, unless a server explicitly disables root logins the exploit will certainly work - regardless of any firewall configuration.

    The main take-away from what has been posted about the payload today is: Random script kiddies will probably not be able to exploit your server. It only works with the attacker's private key.

    Yes, maybe this is the case, i.e. the check of the username happens later in the process...
    So, from what you understood, "PermitRootLogin no" is enough to avoid the attack?

  • matey0matey0 Member
    edited March 31

    @drk said:

    @matey0 said:

    @drk said: Of course he needs a username

    Do you have evidence on that? I've found nothing conclusive.
    It may be the case - but it may also be the case that sshd calls the hooked public-key decryption function even for invalid users (to log information about the login attempt for example).
    The RCE itself will happen as the user sshd is running as (usually root), before sshd has dropped privileges for the user shell session.
    Thus if the hooked function is executed for an invalid user the attacker still gains root access.

    Either way, unless a server explicitly disables root logins the exploit will certainly work - regardless of any firewall configuration.

    The main take-away from what has been posted about the payload today is: Random script kiddies will probably not be able to exploit your server. It only works with the attacker's private key.

    Yes, maybe this is the case, i.e. the check of the username happens later in the process...
    So, from what you understood, "PermitRootLogin no" is enough to avoid the attack?

    I'm not totally sure of that either. What I meant is that if root logins are not disabled then the attacker definitely won't have to enumerate users. The best way to avoid the attack is to not run a vulnerable version of sshd - every OS that was vulnerable should have patches available by now.

    We will have to wait until there is substantial analysis of the payload. There hasn't been anything complete yet.
    I thought we would be flooded with writeups by multiple security firms and individual researchers given how much of a high-profile case this is - an easy claim to fame. But maybe the payload is just so complex that it takes longer than usual. We will know for sure in the next few days.

    Thanked by 1drk
  • raindog308raindog308 Administrator, Veteran

    @matey0 said: They pressured/"bullied" him under multiple accounts

    I didn't get that. I read the original maintainer stepped down and a new one stepped in. The "longterm mental health issues but also due to some other things" doesn't mean that someone was pressuring / "bullying" him. More like someone spotted an opportunity.

    Possible I missed something else you're referring to.

  • matey0matey0 Member
    edited March 31

    @raindog308 said:

    @matey0 said: They pressured/"bullied" him under multiple accounts

    I didn't get that. I read the original maintainer stepped down and a new one stepped in. The "longterm mental health issues but also due to some other things" doesn't mean that someone was pressuring / "bullying" him. More like someone spotted an opportunity.

    Possible I missed something else you're referring to.

    See https://boehs.org/node/everything-i-know-about-the-xz-backdoor under the heading "2022".
    They were pressuring him on the mailing list with multiple different (obvious sockpuppet) accounts, voicing complaints about slow development, calling the project abandoned, saying forks are necessary. The intention behind this was clearly to take advantage of the maintainer's mental health - and it worked: Shortly after this the maintainer had been approached by Jia Tan who had started to gain his trust.
    The maintainer said it himself (alluding to Jia Tan being the solution to the complaints):

    Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see.

  • raindog308raindog308 Administrator, Veteran

    @matey0 said: under the heading "2022".

    Ahhh...ok. That page has been updated many times and the earlier version I read didn't have that history. Thanks.

  • yusrayusra Member

    Discovered by Andres Freund, a PostgreSQL developer and not a security researcher just because he decided to go down the rabbit hole and find the cause of performance issues caused by a poorly tested backdoor!

    One can just imagine how many other well tested backdoors are out there, hiding in plain sight!

  • vovlervovler Member

    So does this only affect Debian or other distros as well?

  • matey0matey0 Member

    @vovler said:
    So does this only affect Debian or other distros as well?

    Not only debian. But you could've googled that just as quickly as you wrote that comment. Every distro has published a statement regarding it.

  • vovlervovler Member

    @matey0 said:

    @vovler said:
    So does this only affect Debian or other distros as well?

    Not only debian. But you could've googled that just as quickly as you wrote that comment. Every distro has published a statement regarding it.

    Are you ok? I've asked a question on a forum, yes I could have google'd it, by I tend to not google recent news or events, and instead just ask it here, as any of you guys that may spend a few hours on twitter may have much more up to date information than any news article google gives me.

    But yes, I've google'd it now.

  • matey0matey0 Member
    edited March 31

    @yusra said:
    Discovered by Andres Freund, a PostgreSQL developer and not a security researcher just because he decided to go down the rabbit hole and find the cause of performance issues caused by a poorly tested backdoor!

    One can just imagine how many other well tested backdoors are out there, hiding in plain sight!

    A trusted maintainer committing one among many binary test files, containing a heavily obfuscated, multi-stage loader which modifies the output source tree build scripts to include a backdoor, again heavily obfuscated, isn't exactly hiding in plain sight.

    Even legitimate build/test files are practically unreadable and look the same as the malicious ones at first and second glance even to a security researcher's eyes.

    But yes, this won't be the last time and likely wasn't the first time. Even under these same pseudonyms the attackers made suspicious commits to other repos like libarchive, which removed some security protections, likely to enable some sort of broken 0day they had.

    @vovler said:

    @matey0 said:

    @vovler said:
    So does this only affect Debian or other distros as well?

    Not only debian. But you could've googled that just as quickly as you wrote that comment. Every distro has published a statement regarding it.

    Are you ok? I've asked a question on a forum, yes I could have google'd it, by I tend to not google recent news or events, and instead just ask it here, as any of you guys that may spend a few hours on twitter may have much more up to date information than any news article google gives me.

    But yes, I've google'd it now.

    You didn't even have to google, it would've been enough to just read this thread (the full 2 pages of it!). Forums like these are a place for discussion, not a place for dumb questions better posed to google. Every affected distro will have published an advisory and be on the first page of search results.

  • @matey0 said:
    'Forums like these are a place for discussion, not a place for dumb questions better posed to google.

    Everyone should be able to ask anything here.

    Guys, I have a sack of Centos 7 servers, but I'm too lazy to look at them. Probably xz version 5.2
    Can someone please check?

    Thanked by 1rsk
  • dogemeisterdogemeister Member
    edited March 31

    @tra10000 said:

    @matey0 said:
    'Forums like these are a place for discussion, not a place for dumb questions better posed to google.

    Everyone should be able to ask anything here.

    Guys, I have a sack of Centos 7 servers, but I'm too lazy to look at them. Probably xz version 5.2
    Can someone please check?

    screen -S myCentOSsrv -X stuff "xz --version^M"

    Hmm, now what? I didn't see anything. Oh well.

    Anyway, Praise "Bob"!

    Thanked by 1tra10000
  • davidedavide Member
    edited April 1

    @davide said:
    Jia Tan, who submitted the backdoor to Github, is the fucking maintainer of xz.

    Flushing the whole thing right now.

    Turns out I cannot flush liblzma5 from Debian without ripping off three quarters of the system. What an un-interoperable clusterfuck.

    Given the lack of a maintainer and the general criticism toward xz, my crystal ball tells me the life of xz as a system component is soon to be over.

    Thanked by 1quicksilver03
  • raindog308raindog308 Administrator, Veteran

    @davide said: Turns out I cannot flush liblzma5 from Debian without ripping off three quarters of the system.

    Are you running unstable?

    My 12.5 systems show liblzma5 is at 5.4.1-0.2

  • omelasomelas Member

    Would this cause xz itself in linux space?> @raindog308 said:

    @davide said: Turns out I cannot flush liblzma5 from Debian without ripping off three quarters of the system.

    Are you running unstable?

    My 12.5 systems show liblzma5 is at 5.4.1-0.2

    Liblzma is dependency of dpkg, so it's expected

  • davidedavide Member
    edited April 1

    @raindog308 said:
    Are you running unstable?

    My 12.5 systems show liblzma5 is at 5.4.1-0.2

    No, I checked all VMs and OSes, I even have one Ubuntu instance at Oracle, none have the affected versions of liblzma5. But the xz project itself is a disaster, I personally looked into the configure/build scripts and the release tarballs, it is something I don't want on my machines, it would superficially appear as an incompetent dumpster of garbled code but in hindsight it is meant to hide defects. And I read indirectly that the xz archival format, its multiple layers of encapsulation and its plugin mechanism are excessively over-engineered and undocumented. The whole xz thing is condemned regardless of the version.

Sign In or Register to comment.