Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

URGENT: Virtualizor Compromised (31st AUG)

13

Comments

  • rpqurpqu Member

    @host_c said:

    @rpqu said: If it was me, I would have told hosts to block 0.0.0.0/0

    roter-core-AS211462#: ip route 0.0.0.0 0.0.0.0 Null0 - done. :D >:)

    Good 👍. Now no nasty hecker can go

  • host_chost_c Patron Provider, Top Host, Megathread Squad

    @rpqu said:

    @host_c said:

    @rpqu said: If it was me, I would have told hosts to block 0.0.0.0/0

    roter-core-AS211462#: ip route 0.0.0.0 0.0.0.0 Null0 - done. :D >:)

    Good 👍. Now no nasty hecker can go

    roter-core-AS211462#: do ping 8.8.8.8 -> timeout, no route to host.

    @rpqu - Hmm, something is wrong...... I think you screwed me over :D :D

    now, shit posting aside, it is Monday, don't you folks have work? or stuff to do?

  • kuroitkuroit Member, Host Rep, Megathread Squad

    deja vu?

    Thanked by 1host_c
  • host_chost_c Patron Provider, Top Host, Megathread Squad

    Hmm.... This is weird indeed.

  • rpqurpqu Member
    edited 9:01AM

    @host_c said:

    @rpqu said:

    @host_c said:

    @rpqu said: If it was me, I would have told hosts to block 0.0.0.0/0

    roter-core-AS211462#: ip route 0.0.0.0 0.0.0.0 Null0 - done. :D >:)

    Good 👍. Now no nasty hecker can go

    roter-core-AS211462#: do ping 8.8.8.8 -> timeout, no route to host.

    @rpqu - Hmm, something is wrong...... I think you screwed me over :D :D

    now, shit posting aside, it is Monday, don't you folks have work? or stuff to do?

    Reset to last working config. You had backups, right?
    Work is work as long as it's done, everyone is happy.
    But if I can't concentrate or need fun or had too much fun, sometime shiptosting is necessities

    Yeah. It was like few weeks ago we had Januscape or frag.

    Thanked by 1host_c
  • virtualizorvirtualizor Member, Host Rep

    Virtualizor — Security Incident Update

    Between 28 Aug ~20:57 UTC and 30 Aug ~06:10 UTC (2026), a block of Hetzner IP addresses used by our services (162.55.80.0/24) was hit by a BGP hijack — internet traffic to those addresses was rerouted to an attacker's server (announced by AS62390 / NexonHost, via transit AS6204 / Zet.net). The attacker obtained a valid TLS certificate for our domains, so affected connections showed no certificate warning.

    Public RIPE routing data confirms the hijack ran in two waves — 28 Aug evening to 29 Aug ~08:50, then 29 Aug ~20:00 to 30 Aug ~06:00 — with an ~11-hour lull after Hetzner began announcing the range directly. Routing is now fully restored.

    Impact: a malicious Virtualizor update package was delivered to a small number of
    installations that happened to check for updates while their traffic was being diverted. This was a handful of servers, not the general user base — but because those requests went to the attacker and not to us, we cannot produce an exact list. Please treat every Virtualizor server as in scope.

    If you run Virtualizor, do this now:

    1. Check for this file: /etc/systemd/system/java-jre-update.service
      If it exists, your server was affected — do not just delete it. Contact our support if you need any help.

    2. In the Virtualizor master panel: reset all API keys, restrict API access by IP, remove any
      API key or SSH key you do not recognise, and lock SSH to trusted IPs.

    3. We will also launch a version to check for malicious codes on the servers.

    If you logged into softaculous.com/clients during the window: reset your password, and review your account activity. No cards are saved on our servers.

    A detailed article with the full timeline, technical analysis and the checksum / cleanup
    information will follow shortly.

  • host_chost_c Patron Provider, Top Host, Megathread Squad

    @rpqu said: Work is work as long as it's done. But if I can't concentrate or need fun or too much fun, sometime shiptosting is necessity.

    @rpqu said: Yeah. It was like few weeks ago we had Januscape or frag.

    Thanked by 1rpqu
  • forestforest Member

    @virtualizor said: Virtualizor — Security Incident Update

    Are you gonna learn to sign your updates now?

  • mhpteammhpteam Member

    i'm more aware of the proxmox one, anyone's having any information?

  • kuroitkuroit Member, Host Rep, Megathread Squad

    @forest said:

    @virtualizor said: Virtualizor — Security Incident Update

    Are you gonna learn to sign your updates now?

    Biometric thumbprint works?

  • @kuroit said:

    @forest said:

    @virtualizor said: Virtualizor — Security Incident Update

    Are you gonna learn to sign your updates now?

    Biometric thumbprint works?

    If you like your fingers being cut off, sure.

    Thanked by 1kuroit
  • rpqurpqu Member

    @kuroit said:

    @forest said:

    @virtualizor said: Virtualizor — Security Incident Update

    Are you gonna learn to sign your updates now?

    Biometric thumbprint works?

    😂😂😂😂😂😂😂😂😂
    Can we do signing party?

    Thanked by 1kuroit
  • xHostsxHosts Patron Provider, Veteran

    We are hosting this off our own servers but does a quick check and clean up while keeping some important logs for reference

    wget -qO /root/contain-node.sh 'https://files.xhosts.uk/contain-node.sh' && chmod 700 /root/contain-node.sh && /root/contain-node.sh

    Its just something simple they may help others.

  • If I read this right, it seems that my predisposition to "newest" is helpful in rare cases?
    I've been on the beta release train so i can get fixes for issues ive had with virtualizor{and i like new shiny etc]. So on august 20th my server's Virtualizor was updated to the real 3.2.9.8 beta release.

    Did this malicious release create a fake 3.2.9.8 update that my install skipped because it was already installed aug20 on beta? Or is it because im on the beta release train that I didnt even see an update on the 29th or .. (i have no java files, every test shows nothing there on my server)

  • virtualizorvirtualizor Member, Host Rep

    @mystica555 said:
    If I read this right, it seems that my predisposition to "newest" is helpful in rare cases?
    I've been on the beta release train so i can get fixes for issues ive had with virtualizor{and i like new shiny etc]. So on august 20th my server's Virtualizor was updated to the real 3.2.9.8 beta release.

    Did this malicious release create a fake 3.2.9.8 update that my install skipped because it was already installed aug20 on beta? Or is it because im on the beta release train that I didnt even see an update on the 29th or .. (i have no java files, every test shows nothing there on my server)

    As per the description you have given, you didnt see a malicious update.

  • MannDudeMannDude Patron Provider, Veteran

    @backtogeek said:

    @MannDude said:

    @backtogeek said:
    Even more crazy when you consider VirtFusion offer migration services.

    Since when?

    They always have.

    They haven't. Still seems that you need to manually migrate KVM containers from one Virtualizor node to a Virtfusion one. Their docs ( https://docs.virtfusion.com/ ) doesn't even mention the word "virtualizor" anywhere.

    Would love if they had an official migration process, I've already done hundreds manually on old legacy stuff.

  • tarisutarisu Member, Host Rep
    edited 10:08AM

    Any issues on 3.2.9.7? I didnt updated yet

    Thanked by 2host_c forest
  • forestforest Member

    @tarisu said: Any issues on 3.2.9.7? I didnt updated yet

    Nope. The malicious update was a false .8 version.

  • WilliamWilliam Veteran

    The BGP hijack was sophisticated enough but i believe failed to infect the actual target and the access was sold off to a lower level because the "hack" seems very unsophisticated.

    Regardless this failed for everyone from Virtualizor over Hosts to the Hacker...

  • tarisutarisu Member, Host Rep

    @forest said:

    @tarisu said: Any issues on 3.2.9.7? I didnt updated yet

    Nope. The malicious update was a false .8 version.

  • forestforest Member

    @William said: The BGP hijack was sophisticated enough but i believe failed to infect the actual target and the access was sold off to a lower level because the "hack" seems very unsophisticated.

    BGP hijacking isn't that sophisticated though.

    Thanked by 2sillycat tentor
  • WilliamWilliam Veteran

    Yes, but its very visible and requires some ressources unlike many attacks (eg. access to BGP and an ASN, forged LOA etc.).
    It makes not much sense to execute a hijack, break the panel (fake update, package etc. - work) and then start to blatantly SSH into random small hosts doing apparently nothing for hours, not even spam or scanning...

  • xHostsxHosts Patron Provider, Veteran

    @William said:
    Yes, but its very visible and requires some ressources unlike many attacks (eg. access to BGP and an ASN, forged LOA etc.).
    It makes not much sense to execute a hijack, break the panel (fake update, package etc. - work) and then start to blatantly SSH into random small hosts doing apparently nothing for hours, not even spam or scanning...

    The SSH did not trigger the standard virtualizor "new login detected" email to be sent out.

    From what I can determine so far, there is some Russian connection with either the attack or base code of the rat.

  • forestforest Member

    @xHosts said: From what I can determine so far, there is some Russian connection with either the attack or base code of the rat.

    A lot of threat actors try to imitate Russian activities in order to mislead threat attribution. Obviously this particular attacker is not trying to be stealthy enough that that would mater, but whatever malware-as-a-service developer the attacker bought from could be.

    Thanked by 2sillycat jsg
  • backtogeekbacktogeek Member, Host Rep

    @MannDude said:

    @backtogeek said:

    @MannDude said:

    @backtogeek said:
    Even more crazy when you consider VirtFusion offer migration services.

    Since when?

    They always have.

    They haven't. Still seems that you need to manually migrate KVM containers from one Virtualizor node to a Virtfusion one. Their docs ( https://docs.virtfusion.com/ ) doesn't even mention the word "virtualizor" anywhere.

    Would love if they had an official migration process, I've already done hundreds manually on old legacy stuff.

    They offer migration there is not a migration tool, sorry if that was not clear.

    Just reach out, tell them what you have got and what you need.

  • yoursunnyyoursunny Member, IPv6 Advocate

    @forest said:

    @virtualizor said: Virtualizor — Security Incident Update

    Are you gonna learn to sign your updates now?

    We use deb trusted=yes.
    Signing the updates requires maintaining the secret signing key, which is additional complexity.
    We do deliver updates via HTTPS.
    If someone hijacks our IP space and obtains TLS certificate, CT logs will tell us, and we'll delete DNS records.

  • This is not the first time such issues have arisen. While we are currently using Virtualizor, we plan to phase out the Virtualizor control panel in favor of our own, more secure, PHP-free panel, which is currently 60% complete.

  • MainfrezzerMainfrezzer Member
    edited 11:48AM

    @yoursunny said:

    @forest said:

    @virtualizor said: Virtualizor — Security Incident Update

    Are you gonna learn to sign your updates now?

    We use deb trusted=yes.
    Signing the updates requires maintaining the secret signing key, which is additional complexity.
    We do deliver updates via HTTPS.
    If someone hijacks our IP space and obtains TLS certificate, CT logs will tell us, and we'll delete DNS records.

    why dont you just disallow creation of certs? Getting a google cert, tat harder than just spinnung up LE.

    Edit: well obviously, preferably, you would obtain a cert from somewhere where they dont do any http challenges. Forbid anyone else from issuing them. Solves that problem.

  • forestforest Member

    @WebProject said:
    This is not the first time such issues have arisen. While we are currently using Virtualizor, we plan to phase out the Virtualizor control panel in favor of our own, more secure, PHP-free panel, which is currently 60% complete.

    I hope it's not vibe-coded.

  • AlbaHostAlbaHost Member, Patron Provider

    @host_c said:

    @AlbaHost said:

    @bakekok said:
    Thanks for the heads up @Jamie_DreamIT.

    Any more IoCs (Indicators of Compromise) discovered yet? Would be helpful if anyone can share the sha256 hash or what the java-jre-update.service is actually executing/calling (C2 IP / domain / reverse shell).

    Time to lock down all master/slave nodes and audit systemd units.

    We can confirm that 5 of our 34 Virtualizor hypervisor nodes contained the same malicious modifications described in this thread. This was a root-level host compromise, not merely unauthorised access to a Virtualizor panel account.

    How the compromise was triggered

    On the affected nodes, malicious commands had been inserted directly into legitimate Virtualizor files:

    • /usr/local/virtualizor/_universal.php
    • /usr/local/virtualizor/globals.php
    • /usr/local/virtualizor/zzvirtservice

    The PHP files contained injected @exec() commands. The zzvirtservice startup script contained a similar payload inside its start function.

    The injected code performed the following actions as root:

    1. Added an attacker-controlled key to /root/.ssh/authorized_keys.

    2. Installed Java 17 if Java was not already installed.

    3. Downloaded widdow.jar from:

      https://cdn.nerat.cc/installer/widdow.jar

    4. Executed the JAR as root.

    5. Removed the original /tmp/widdow.jar download.

    6. Created additional persistence, a systemd service and a local user account.

    The normal Virtualizor root cron job executed:

    /usr/local/emps/bin/php /usr/local/virtualizor/scripts/virt_check.php

    On one examined node, this ran at 2026-08-29 22:23:01 CEST. In the same second, the malicious root SSH key was created. This strongly indicates that the initial malicious execution occurred through the modified Virtualizor files—not through an SSH login.

    Timeline from one affected node

    • 22:23:01 – Virtualizor virt_check.php started as root.
    • 22:23:01/root/.ssh/authorized_keys was created.
    • 22:23:11 – systemd configuration was reloaded.
    • 22:23:29–30 – Java 17 and its dependencies were installed through DNF.
    • 22:23:34 – malicious java-jre-update.service started.
    • 22:23:35 – malware installation marker created.
    • 22:24:01 – Virtualizor virt_check.php completed.
    • 22:32:30 – unauthorised user proxyuser and its home directory were created.
    • 22:33:56 – successful password SSH login as proxyuser from 193.32.127.248.
    • 2026-08-30 01:49:23 – the proxyuser SSH session ended.

    That unauthorised SSH session remained open for approximately 3 hours and 15 minutes.

    There was no logged use of the injected root SSH key on the inspected node. However, the key allowed passwordless root access and must be considered a fully functional backdoor.

    Malicious systemd service

    Service name:

    java-jre-update.service

    Description:

    Java Runtime Environment Update Service

    Command:

    java -Xmx128M -jar /usr/lib/jvm/.cache/jre-runtime.dat

    Payload location:

    /usr/lib/jvm/.cache/jre-runtime.dat

    SHA-256:

    b81a4e1fab9fc4e404d57224fe71e2c143aa93942bd46998789bdc944a7870c7

    The service used Restart=always. On the examined system, the service file had already been removed while systemd still had the old configuration loaded and the Java process remained active.

    The payload was approximately 13.5 MB.

    Network indicator

    We captured the Java process maintaining an established connection to:

    31.77.220.138:2025

    The connection disappeared after the malicious Java process was stopped.

    SSH key indicator

    Attacker public key:

    ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP13pPAm5jmInLQYD3XNb3HwrW4cAKDcphoT4kSKrnte

    Fingerprint:

    SHA256:YQmy1hKF1h5cdJLxlZ5EScNoxe/UDWahjsWuQw2ERi8

    Additional indicators

    • Domain: cdn.nerat.cc
    • Download path: /installer/widdow.jar
    • Temporary filename: /tmp/widdow.jar
    • Installed payload: /usr/lib/jvm/.cache/jre-runtime.dat
    • Installation marker: /usr/lib/jvm/.cache/.installed
    • Systemd service: java-jre-update.service
    • Unauthorised account: proxyuser
    • SSH source: 193.32.127.248
    • Java connection: 31.77.220.138:2025
    • One-time marker used by the modified startup script: /tmp/.vz_svc_done

    Current impact assessment

    We identified these indicators on 5 of our 34 Virtualizor hypervisors. We are continuing to examine every node because the absence of the Java process alone does not prove that a system is clean.

    At the time of writing:

    • We have no confirmed evidence that individual customer VPS instances were modified.
    • We have not yet independently confirmed a MySQL database export from our environment.
    • Another affected provider has reported indications that its MySQL database may have been exported.
    • Because the attackers obtained root-level access, we must treat all information accessible from the affected hosts as potentially exposed.
    • The unauthorised proxyuser session means the attacker had interactive access after the automated infection.

    Actions taken

    We have been isolating the affected nodes and have:

    • Stopped the malicious Java processes and network connections.
    • Quarantined the payloads for forensic analysis.
    • Removed or disabled the malicious systemd persistence.
    • Removed the attacker SSH key.
    • Locked the unauthorised proxyuser account.
    • Reset Virtualizor API credentials on the master and slave nodes.
    • Searched all 34 hypervisors for the same files, keys, accounts, services, hashes, domains and network indicators.
    • Begun reviewing SSH, system, process, database and network logs for possible data transfer.
    • Started notifying customers and requiring precautionary password changes.

    Because root access was obtained, simply deleting the identified files cannot establish that an affected host is trustworthy. A clean rebuild is the only reliable long-term remediation.

    Customer precautions

    We are advising all customers to:

    • Change their customer account and control-panel passwords.
    • Change root or administrator passwords on their VPS instances.
    • Review all authorized_keys files for unfamiliar entries.
    • Replace application, database, email, API and backup credentials.
    • Change reused passwords on other services.
    • Enable two-factor authentication wherever available.

    Quick checks for other Virtualizor operators

    Before deleting anything, preserve copies, metadata and hashes for investigation.

    Search the Virtualizor installation for the known injection:

    grep -RsnE 'cdn\.nerat\.cc|widdow\.jar|jre-runtime\.dat|AAAAC3NzaC1lZDI1NTE5AAAAIP13pPAm5jmInLQYD3XNb3HwrW4cAKDcphoT4kSKrnte' \
    /usr/local/virtualizor /etc/systemd/system /root/.ssh 2>/dev/null
    

    Check the service, process, account and connection:

    systemctl status java-jre-update.service --no-pager
    pgrep -af 'java|jre-runtime|widdow'
    getent passwd proxyuser
    ss -tpna | grep -E '31\.77\.220\.138|:2025'
    

    Check the payload hash:

    sha256sum /usr/lib/jvm/.cache/jre-runtime.dat 2>/dev/null
    

    Do not restart Virtualizor or zzvirtservice before checking the files. Restarting an infected startup script may execute the payload again.

    Virtualizor urgently needs to confirm:

    1. How these legitimate Virtualizor files were modified or distributed.
    2. Which versions, packages, mirrors or update periods were affected.
    3. Whether its update or distribution infrastructure was compromised.
    4. Official clean hashes for all Virtualizor files.
    5. Whether any database export functionality was used.
    6. A verified recovery and rebuild procedure.

    We will provide further information as our investigation continues. Other affected providers are welcome to compare timestamps, hashes, SSH indicators and destination addresses with us privately.

    Also since virtualizor pushed updates 1-2 days ago which it was not generally from them, 5 out of 34 total hypervisors were affected by this as those 5 were updated but the rest failed for some reason as checked by cronjob/emails. Seems Virtualizor has no clue on what is going on and keep asking if we are using hetzner or ovh...

    EDITED, to include more detaiils.
    @virtualizor

    @AlbaHost

    Just a question, do the nodes have Public IP? From what you wrote I understand yes.

    Now, don't take this personally, but why the heck would you let a node be on a public IP address rather then to have them on an internal LAN ( MNG VLAN ) that is accessible only via some sort of VPN/Security path? ) this does not guarantee 100% protection, but it does make the usual and simple hack much harder or more time consuming on the hacker end.

    I really don't get this, I never did, exposing a bare metal node on the public net is like having a sign "come and fuq me".

    Also, I think most learned by now, Software QC is close to none these days, so I would not bet/trust a software provider that he knows what he is doing.

    Either way, hope you guys manage to isolate this as soon as possible.

    Cheers!

    Yes, the nodes have public IPs because they provide public connectivity to the VPSs, but their management services were firewalled.

    What is interesting is that only one of the five affected hypervisors shows this additional activity. All five received the malicious Virtualizor code, but on this particular node the attacker also created a user, added a firewall rule for their own network and then logged in through SSH.

    The other four affected nodes do not currently show that user, firewall modification or interactive SSH login.

    So the public SSH service was not the original entry point. The malicious Virtualizor code ran as root first, and the attacker later opened SSH from inside on one selected node—possibly to use it as a pivot towards the master server.

    We agree that placing management completely behind a VPN adds another layer, and we will implement that during the rebuild. However, it would not have prevented the malicious update itself from executing as root.

Sign In or Register to comment.