Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

URGENT: Virtualizor Compromised (31st AUG)

2

Comments

  • AlbaHostAlbaHost Member, Patron Provider

    @Radi said:
    *8 update was released on 20th August?

    Only if you have ticked for beta release in your virtualizor panel, otherwise they publish always earlier on their blog but live updates take place late i.e a week or two.

  • RadiRadi Host Rep, Veteran

    @AlbaHost said:

    @Radi said:
    *8 update was released on 20th August?

    Only if you have ticked for beta release in your virtualizor panel, otherwise they publish always earlier on their blog but live updates take place late i.e a week or two.

    I am on *7 and ticked Stable, well now changed to "NEVER".

  • aphexaphex Member

    @forest said: Let me guess, none of the hosts whose nodes were compromised will actually fully reinstall from scratch?

    Let me guess #2, virtualizor doesn't know what PKI or signing things is?

    Thanked by 2forest yoursunny
  • AlbaHostAlbaHost Member, Patron Provider

    @Radi said:

    @AlbaHost said:

    @Radi said:
    *8 update was released on 20th August?

    Only if you have ticked for beta release in your virtualizor panel, otherwise they publish always earlier on their blog but live updates take place late i.e a week or two.

    I am on *7 and ticked Stable, well now changed to "NEVER".

    Well if you had on Stable which is by default you will be better of to check your hypervisors as the fake update that pushed malware 1-2 days ago from virtualizor servers/mirrors was automatic update.

  • JerryHouJerryHou Veteran

    with these AI tools, everyone can become a vibe hacker in minutes...

  • forestforest Member

    @aphex said:

    @forest said: Let me guess, none of the hosts whose nodes were compromised will actually fully reinstall from scratch?

    Let me guess #2, virtualizor doesn't know what PKI or signing things is?

    Oh for sure. I didn't even believe for a moment that Virtualizor would be professional enough to sign releases.

  • chatboxchatbox Member

    OMG ! Would OVH be one of those affected ? have few VPS with them :|

  • d2411d2411 Member

    @Chunkserve also use Virtualizor

    Thanked by 2JohnnySac Chunkserve
  • AndruAndru Member

    @naranjatech use Virtualizor to.

  • Thanked by 2host_c kuroit
  • forestforest Member

    And iHostArt. :D

    Thanked by 2Void host_c
  • MurvMurv Member, Megathread Squad

    Virtualizor makes slop panels look good

    Thanked by 1forest
  • TrKTrK Veteran

    virtualizor got virtualizord.

  • host_chost_c Patron Provider, Top Host, Megathread Squad

    @Jamie_DreamIT said:
    Virtualizor has been compromised, their BGP hijack a few days ago seems to have a deployed a malicious package.

    • RESET/LIMIT ALL API CREDENTIALS IN THE VIRT MASTER PANEL BY IP
    • CHECK FOR SUSPICIOUS KEYS AND ENSURE SSH IS LOCKED DOWN
    • CHECK IF THIS FILE EXISTS ( /etc/systemd/system/java-jre-update.service )

    If you're an affected host, NOC or provider and would want to work together, please DM me, we'll appreciate any assistance here.

    Comms are being sent for existing customers, at this stage, we don't see evidence of VPS's being compromised.

    Good luck to other providers out there.

    Thanked by 2COLBYLICIOUS kuroit
  • xHostsxHosts Patron Provider, Veteran
    edited 7:49AM

    We are aware of this.

    We had been waiting on a reply from Virtualizor until about 30 minutes ago.

    We had written a few quick bash scripts and these have been run on all servers.

    One is a quick check and clean up script, while its not a failsafe, it cuts down the current risk

    We had spent most of yesterday collecting and saving evidence.

    We had fired off a large number of abuse reports while waiting on Virtualizor and did notice certain parts of the rat’s network become unreachable.

    We are in talks with other companies to have the remaining parts at the very least suspended.

    One advise I would give to other provides is to cycle your DB passwords on each slave.

    We used a quick bash script to change the password and update the universal.php allowing it to only take a few seconds rather than it being a pain in the ass manual task.

    We did not want to make a post until it had been confirmed by Virtualizor or draw a lot of attention if the attackers lurk on the forum.

  • ChunkserveChunkserve Member, Patron Provider
    edited 7:53AM

    @d2411 said:
    @Chunkserve also use Virtualizor

    Thanks for the mention. Right after I did a quick ansible script for rest of our Virtualizor nodes, that we still need to move in the future. The playbook looks incidators shared by @AlbaHost and @Jamie_DreamIT . I did not find anything here indicating we were compromised or targetted. I checked also virtualizor update log, and last "Server Update" task happened at July 31, 2026, 4:22 pm. Proactively, I ensured that auto updates are disabled and restricted virtualizor access for nearest future.

    I also mailed virtualizor about more details of this incident.

    Thank you for the awesome work, I really appreciate sharing these publicly.

  • rpqurpqu Member

    image
    Feels like near miss

  • host_chost_c Patron Provider, Top Host, Megathread Squad

    @AlbaHost said:

    @bakekok said:
    Thanks for the heads up @Jamie_DreamIT.

    Any more IoCs (Indicators of Compromise) discovered yet? Would be helpful if anyone can share the sha256 hash or what the java-jre-update.service is actually executing/calling (C2 IP / domain / reverse shell).

    Time to lock down all master/slave nodes and audit systemd units.

    We can confirm that 5 of our 34 Virtualizor hypervisor nodes contained the same malicious modifications described in this thread. This was a root-level host compromise, not merely unauthorised access to a Virtualizor panel account.

    How the compromise was triggered

    On the affected nodes, malicious commands had been inserted directly into legitimate Virtualizor files:

    • /usr/local/virtualizor/_universal.php
    • /usr/local/virtualizor/globals.php
    • /usr/local/virtualizor/zzvirtservice

    The PHP files contained injected @exec() commands. The zzvirtservice startup script contained a similar payload inside its start function.

    The injected code performed the following actions as root:

    1. Added an attacker-controlled key to /root/.ssh/authorized_keys.

    2. Installed Java 17 if Java was not already installed.

    3. Downloaded widdow.jar from:

      https://cdn.nerat.cc/installer/widdow.jar

    4. Executed the JAR as root.

    5. Removed the original /tmp/widdow.jar download.

    6. Created additional persistence, a systemd service and a local user account.

    The normal Virtualizor root cron job executed:

    /usr/local/emps/bin/php /usr/local/virtualizor/scripts/virt_check.php

    On one examined node, this ran at 2026-08-29 22:23:01 CEST. In the same second, the malicious root SSH key was created. This strongly indicates that the initial malicious execution occurred through the modified Virtualizor files—not through an SSH login.

    Timeline from one affected node

    • 22:23:01 – Virtualizor virt_check.php started as root.
    • 22:23:01/root/.ssh/authorized_keys was created.
    • 22:23:11 – systemd configuration was reloaded.
    • 22:23:29–30 – Java 17 and its dependencies were installed through DNF.
    • 22:23:34 – malicious java-jre-update.service started.
    • 22:23:35 – malware installation marker created.
    • 22:24:01 – Virtualizor virt_check.php completed.
    • 22:32:30 – unauthorised user proxyuser and its home directory were created.
    • 22:33:56 – successful password SSH login as proxyuser from 193.32.127.248.
    • 2026-08-30 01:49:23 – the proxyuser SSH session ended.

    That unauthorised SSH session remained open for approximately 3 hours and 15 minutes.

    There was no logged use of the injected root SSH key on the inspected node. However, the key allowed passwordless root access and must be considered a fully functional backdoor.

    Malicious systemd service

    Service name:

    java-jre-update.service

    Description:

    Java Runtime Environment Update Service

    Command:

    java -Xmx128M -jar /usr/lib/jvm/.cache/jre-runtime.dat

    Payload location:

    /usr/lib/jvm/.cache/jre-runtime.dat

    SHA-256:

    b81a4e1fab9fc4e404d57224fe71e2c143aa93942bd46998789bdc944a7870c7

    The service used Restart=always. On the examined system, the service file had already been removed while systemd still had the old configuration loaded and the Java process remained active.

    The payload was approximately 13.5 MB.

    Network indicator

    We captured the Java process maintaining an established connection to:

    31.77.220.138:2025

    The connection disappeared after the malicious Java process was stopped.

    SSH key indicator

    Attacker public key:

    ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP13pPAm5jmInLQYD3XNb3HwrW4cAKDcphoT4kSKrnte

    Fingerprint:

    SHA256:YQmy1hKF1h5cdJLxlZ5EScNoxe/UDWahjsWuQw2ERi8

    Additional indicators

    • Domain: cdn.nerat.cc
    • Download path: /installer/widdow.jar
    • Temporary filename: /tmp/widdow.jar
    • Installed payload: /usr/lib/jvm/.cache/jre-runtime.dat
    • Installation marker: /usr/lib/jvm/.cache/.installed
    • Systemd service: java-jre-update.service
    • Unauthorised account: proxyuser
    • SSH source: 193.32.127.248
    • Java connection: 31.77.220.138:2025
    • One-time marker used by the modified startup script: /tmp/.vz_svc_done

    Current impact assessment

    We identified these indicators on 5 of our 34 Virtualizor hypervisors. We are continuing to examine every node because the absence of the Java process alone does not prove that a system is clean.

    At the time of writing:

    • We have no confirmed evidence that individual customer VPS instances were modified.
    • We have not yet independently confirmed a MySQL database export from our environment.
    • Another affected provider has reported indications that its MySQL database may have been exported.
    • Because the attackers obtained root-level access, we must treat all information accessible from the affected hosts as potentially exposed.
    • The unauthorised proxyuser session means the attacker had interactive access after the automated infection.

    Actions taken

    We have been isolating the affected nodes and have:

    • Stopped the malicious Java processes and network connections.
    • Quarantined the payloads for forensic analysis.
    • Removed or disabled the malicious systemd persistence.
    • Removed the attacker SSH key.
    • Locked the unauthorised proxyuser account.
    • Reset Virtualizor API credentials on the master and slave nodes.
    • Searched all 34 hypervisors for the same files, keys, accounts, services, hashes, domains and network indicators.
    • Begun reviewing SSH, system, process, database and network logs for possible data transfer.
    • Started notifying customers and requiring precautionary password changes.

    Because root access was obtained, simply deleting the identified files cannot establish that an affected host is trustworthy. A clean rebuild is the only reliable long-term remediation.

    Customer precautions

    We are advising all customers to:

    • Change their customer account and control-panel passwords.
    • Change root or administrator passwords on their VPS instances.
    • Review all authorized_keys files for unfamiliar entries.
    • Replace application, database, email, API and backup credentials.
    • Change reused passwords on other services.
    • Enable two-factor authentication wherever available.

    Quick checks for other Virtualizor operators

    Before deleting anything, preserve copies, metadata and hashes for investigation.

    Search the Virtualizor installation for the known injection:

    grep -RsnE 'cdn\.nerat\.cc|widdow\.jar|jre-runtime\.dat|AAAAC3NzaC1lZDI1NTE5AAAAIP13pPAm5jmInLQYD3XNb3HwrW4cAKDcphoT4kSKrnte' \
    /usr/local/virtualizor /etc/systemd/system /root/.ssh 2>/dev/null
    

    Check the service, process, account and connection:

    systemctl status java-jre-update.service --no-pager
    pgrep -af 'java|jre-runtime|widdow'
    getent passwd proxyuser
    ss -tpna | grep -E '31\.77\.220\.138|:2025'
    

    Check the payload hash:

    sha256sum /usr/lib/jvm/.cache/jre-runtime.dat 2>/dev/null
    

    Do not restart Virtualizor or zzvirtservice before checking the files. Restarting an infected startup script may execute the payload again.

    Virtualizor urgently needs to confirm:

    1. How these legitimate Virtualizor files were modified or distributed.
    2. Which versions, packages, mirrors or update periods were affected.
    3. Whether its update or distribution infrastructure was compromised.
    4. Official clean hashes for all Virtualizor files.
    5. Whether any database export functionality was used.
    6. A verified recovery and rebuild procedure.

    We will provide further information as our investigation continues. Other affected providers are welcome to compare timestamps, hashes, SSH indicators and destination addresses with us privately.

    Also since virtualizor pushed updates 1-2 days ago which it was not generally from them, 5 out of 34 total hypervisors were affected by this as those 5 were updated but the rest failed for some reason as checked by cronjob/emails. Seems Virtualizor has no clue on what is going on and keep asking if we are using hetzner or ovh...

    EDITED, to include more detaiils.
    @virtualizor

    @AlbaHost

    Just a question, do the nodes have Public IP? From what you wrote I understand yes.

    Now, don't take this personally, but why the heck would you let a node be on a public IP address rather then to have them on an internal LAN ( MNG VLAN ) that is accessible only via some sort of VPN/Security path? ) this does not guarantee 100% protection, but it does make the usual and simple hack much harder or more time consuming on the hacker end.

    I really don't get this, I never did, exposing a bare metal node on the public net is like having a sign "come and fuq me".

    Also, I think most learned by now, Software QC is close to none these days, so I would not bet/trust a software provider that he knows what he is doing.

    Either way, hope you guys manage to isolate this as soon as possible.

    Cheers!

  • rpqurpqu Member

    @host_c said:

    @AlbaHost said:

    @bakekok said:
    Thanks for the heads up @Jamie_DreamIT.

    Any more IoCs (Indicators of Compromise) discovered yet? Would be helpful if anyone can share the sha256 hash or what the java-jre-update.service is actually executing/calling (C2 IP / domain / reverse shell).

    Time to lock down all master/slave nodes and audit systemd units.

    We can confirm that 5 of our 34 Virtualizor hypervisor nodes contained the same malicious modifications described in this thread. This was a root-level host compromise, not merely unauthorised access to a Virtualizor panel account.

    How the compromise was triggered

    On the affected nodes, malicious commands had been inserted directly into legitimate Virtualizor files:

    • /usr/local/virtualizor/_universal.php
    • /usr/local/virtualizor/globals.php
    • /usr/local/virtualizor/zzvirtservice

    The PHP files contained injected @exec() commands. The zzvirtservice startup script contained a similar payload inside its start function.

    The injected code performed the following actions as root:

    1. Added an attacker-controlled key to /root/.ssh/authorized_keys.

    2. Installed Java 17 if Java was not already installed.

    3. Downloaded widdow.jar from:

      https://cdn.nerat.cc/installer/widdow.jar

    4. Executed the JAR as root.

    5. Removed the original /tmp/widdow.jar download.

    6. Created additional persistence, a systemd service and a local user account.

    The normal Virtualizor root cron job executed:

    /usr/local/emps/bin/php /usr/local/virtualizor/scripts/virt_check.php

    On one examined node, this ran at 2026-08-29 22:23:01 CEST. In the same second, the malicious root SSH key was created. This strongly indicates that the initial malicious execution occurred through the modified Virtualizor files—not through an SSH login.

    Timeline from one affected node

    • 22:23:01 – Virtualizor virt_check.php started as root.
    • 22:23:01/root/.ssh/authorized_keys was created.
    • 22:23:11 – systemd configuration was reloaded.
    • 22:23:29–30 – Java 17 and its dependencies were installed through DNF.
    • 22:23:34 – malicious java-jre-update.service started.
    • 22:23:35 – malware installation marker created.
    • 22:24:01 – Virtualizor virt_check.php completed.
    • 22:32:30 – unauthorised user proxyuser and its home directory were created.
    • 22:33:56 – successful password SSH login as proxyuser from 193.32.127.248.
    • 2026-08-30 01:49:23 – the proxyuser SSH session ended.

    That unauthorised SSH session remained open for approximately 3 hours and 15 minutes.

    There was no logged use of the injected root SSH key on the inspected node. However, the key allowed passwordless root access and must be considered a fully functional backdoor.

    Malicious systemd service

    Service name:

    java-jre-update.service

    Description:

    Java Runtime Environment Update Service

    Command:

    java -Xmx128M -jar /usr/lib/jvm/.cache/jre-runtime.dat

    Payload location:

    /usr/lib/jvm/.cache/jre-runtime.dat

    SHA-256:

    b81a4e1fab9fc4e404d57224fe71e2c143aa93942bd46998789bdc944a7870c7

    The service used Restart=always. On the examined system, the service file had already been removed while systemd still had the old configuration loaded and the Java process remained active.

    The payload was approximately 13.5 MB.

    Network indicator

    We captured the Java process maintaining an established connection to:

    31.77.220.138:2025

    The connection disappeared after the malicious Java process was stopped.

    SSH key indicator

    Attacker public key:

    ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP13pPAm5jmInLQYD3XNb3HwrW4cAKDcphoT4kSKrnte

    Fingerprint:

    SHA256:YQmy1hKF1h5cdJLxlZ5EScNoxe/UDWahjsWuQw2ERi8

    Additional indicators

    • Domain: cdn.nerat.cc
    • Download path: /installer/widdow.jar
    • Temporary filename: /tmp/widdow.jar
    • Installed payload: /usr/lib/jvm/.cache/jre-runtime.dat
    • Installation marker: /usr/lib/jvm/.cache/.installed
    • Systemd service: java-jre-update.service
    • Unauthorised account: proxyuser
    • SSH source: 193.32.127.248
    • Java connection: 31.77.220.138:2025
    • One-time marker used by the modified startup script: /tmp/.vz_svc_done

    Current impact assessment

    We identified these indicators on 5 of our 34 Virtualizor hypervisors. We are continuing to examine every node because the absence of the Java process alone does not prove that a system is clean.

    At the time of writing:

    • We have no confirmed evidence that individual customer VPS instances were modified.
    • We have not yet independently confirmed a MySQL database export from our environment.
    • Another affected provider has reported indications that its MySQL database may have been exported.
    • Because the attackers obtained root-level access, we must treat all information accessible from the affected hosts as potentially exposed.
    • The unauthorised proxyuser session means the attacker had interactive access after the automated infection.

    Actions taken

    We have been isolating the affected nodes and have:

    • Stopped the malicious Java processes and network connections.
    • Quarantined the payloads for forensic analysis.
    • Removed or disabled the malicious systemd persistence.
    • Removed the attacker SSH key.
    • Locked the unauthorised proxyuser account.
    • Reset Virtualizor API credentials on the master and slave nodes.
    • Searched all 34 hypervisors for the same files, keys, accounts, services, hashes, domains and network indicators.
    • Begun reviewing SSH, system, process, database and network logs for possible data transfer.
    • Started notifying customers and requiring precautionary password changes.

    Because root access was obtained, simply deleting the identified files cannot establish that an affected host is trustworthy. A clean rebuild is the only reliable long-term remediation.

    Customer precautions

    We are advising all customers to:

    • Change their customer account and control-panel passwords.
    • Change root or administrator passwords on their VPS instances.
    • Review all authorized_keys files for unfamiliar entries.
    • Replace application, database, email, API and backup credentials.
    • Change reused passwords on other services.
    • Enable two-factor authentication wherever available.

    Quick checks for other Virtualizor operators

    Before deleting anything, preserve copies, metadata and hashes for investigation.

    Search the Virtualizor installation for the known injection:

    grep -RsnE 'cdn\.nerat\.cc|widdow\.jar|jre-runtime\.dat|AAAAC3NzaC1lZDI1NTE5AAAAIP13pPAm5jmInLQYD3XNb3HwrW4cAKDcphoT4kSKrnte' \
    /usr/local/virtualizor /etc/systemd/system /root/.ssh 2>/dev/null
    

    Check the service, process, account and connection:

    systemctl status java-jre-update.service --no-pager
    pgrep -af 'java|jre-runtime|widdow'
    getent passwd proxyuser
    ss -tpna | grep -E '31\.77\.220\.138|:2025'
    

    Check the payload hash:

    sha256sum /usr/lib/jvm/.cache/jre-runtime.dat 2>/dev/null
    

    Do not restart Virtualizor or zzvirtservice before checking the files. Restarting an infected startup script may execute the payload again.

    Virtualizor urgently needs to confirm:

    1. How these legitimate Virtualizor files were modified or distributed.
    2. Which versions, packages, mirrors or update periods were affected.
    3. Whether its update or distribution infrastructure was compromised.
    4. Official clean hashes for all Virtualizor files.
    5. Whether any database export functionality was used.
    6. A verified recovery and rebuild procedure.

    We will provide further information as our investigation continues. Other affected providers are welcome to compare timestamps, hashes, SSH indicators and destination addresses with us privately.

    Also since virtualizor pushed updates 1-2 days ago which it was not generally from them, 5 out of 34 total hypervisors were affected by this as those 5 were updated but the rest failed for some reason as checked by cronjob/emails. Seems Virtualizor has no clue on what is going on and keep asking if we are using hetzner or ovh...

    EDITED, to include more detaiils.
    @virtualizor

    I really don't get this, I never did, exposing a bare metal node on the public net is like having a sign "come and fuq me".

    Many host are like that. Maybe until the site admin made it a rule

  • forestforest Member

    @rpqu said:

    @host_c said:

    @AlbaHost said:

    @bakekok said:
    Thanks for the heads up @Jamie_DreamIT.

    Any more IoCs (Indicators of Compromise) discovered yet? Would be helpful if anyone can share the sha256 hash or what the java-jre-update.service is actually executing/calling (C2 IP / domain / reverse shell).

    Time to lock down all master/slave nodes and audit systemd units.

    We can confirm that 5 of our 34 Virtualizor hypervisor nodes contained the same malicious modifications described in this thread. This was a root-level host compromise, not merely unauthorised access to a Virtualizor panel account.

    How the compromise was triggered

    On the affected nodes, malicious commands had been inserted directly into legitimate Virtualizor files:

    • /usr/local/virtualizor/_universal.php
    • /usr/local/virtualizor/globals.php
    • /usr/local/virtualizor/zzvirtservice

    The PHP files contained injected @exec() commands. The zzvirtservice startup script contained a similar payload inside its start function.

    The injected code performed the following actions as root:

    1. Added an attacker-controlled key to /root/.ssh/authorized_keys.

    2. Installed Java 17 if Java was not already installed.

    3. Downloaded widdow.jar from:

      https://cdn.nerat.cc/installer/widdow.jar

    4. Executed the JAR as root.

    5. Removed the original /tmp/widdow.jar download.

    6. Created additional persistence, a systemd service and a local user account.

    The normal Virtualizor root cron job executed:

    /usr/local/emps/bin/php /usr/local/virtualizor/scripts/virt_check.php

    On one examined node, this ran at 2026-08-29 22:23:01 CEST. In the same second, the malicious root SSH key was created. This strongly indicates that the initial malicious execution occurred through the modified Virtualizor files—not through an SSH login.

    Timeline from one affected node

    • 22:23:01 – Virtualizor virt_check.php started as root.
    • 22:23:01/root/.ssh/authorized_keys was created.
    • 22:23:11 – systemd configuration was reloaded.
    • 22:23:29–30 – Java 17 and its dependencies were installed through DNF.
    • 22:23:34 – malicious java-jre-update.service started.
    • 22:23:35 – malware installation marker created.
    • 22:24:01 – Virtualizor virt_check.php completed.
    • 22:32:30 – unauthorised user proxyuser and its home directory were created.
    • 22:33:56 – successful password SSH login as proxyuser from 193.32.127.248.
    • 2026-08-30 01:49:23 – the proxyuser SSH session ended.

    That unauthorised SSH session remained open for approximately 3 hours and 15 minutes.

    There was no logged use of the injected root SSH key on the inspected node. However, the key allowed passwordless root access and must be considered a fully functional backdoor.

    Malicious systemd service

    Service name:

    java-jre-update.service

    Description:

    Java Runtime Environment Update Service

    Command:

    java -Xmx128M -jar /usr/lib/jvm/.cache/jre-runtime.dat

    Payload location:

    /usr/lib/jvm/.cache/jre-runtime.dat

    SHA-256:

    b81a4e1fab9fc4e404d57224fe71e2c143aa93942bd46998789bdc944a7870c7

    The service used Restart=always. On the examined system, the service file had already been removed while systemd still had the old configuration loaded and the Java process remained active.

    The payload was approximately 13.5 MB.

    Network indicator

    We captured the Java process maintaining an established connection to:

    31.77.220.138:2025

    The connection disappeared after the malicious Java process was stopped.

    SSH key indicator

    Attacker public key:

    ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP13pPAm5jmInLQYD3XNb3HwrW4cAKDcphoT4kSKrnte

    Fingerprint:

    SHA256:YQmy1hKF1h5cdJLxlZ5EScNoxe/UDWahjsWuQw2ERi8

    Additional indicators

    • Domain: cdn.nerat.cc
    • Download path: /installer/widdow.jar
    • Temporary filename: /tmp/widdow.jar
    • Installed payload: /usr/lib/jvm/.cache/jre-runtime.dat
    • Installation marker: /usr/lib/jvm/.cache/.installed
    • Systemd service: java-jre-update.service
    • Unauthorised account: proxyuser
    • SSH source: 193.32.127.248
    • Java connection: 31.77.220.138:2025
    • One-time marker used by the modified startup script: /tmp/.vz_svc_done

    Current impact assessment

    We identified these indicators on 5 of our 34 Virtualizor hypervisors. We are continuing to examine every node because the absence of the Java process alone does not prove that a system is clean.

    At the time of writing:

    • We have no confirmed evidence that individual customer VPS instances were modified.
    • We have not yet independently confirmed a MySQL database export from our environment.
    • Another affected provider has reported indications that its MySQL database may have been exported.
    • Because the attackers obtained root-level access, we must treat all information accessible from the affected hosts as potentially exposed.
    • The unauthorised proxyuser session means the attacker had interactive access after the automated infection.

    Actions taken

    We have been isolating the affected nodes and have:

    • Stopped the malicious Java processes and network connections.
    • Quarantined the payloads for forensic analysis.
    • Removed or disabled the malicious systemd persistence.
    • Removed the attacker SSH key.
    • Locked the unauthorised proxyuser account.
    • Reset Virtualizor API credentials on the master and slave nodes.
    • Searched all 34 hypervisors for the same files, keys, accounts, services, hashes, domains and network indicators.
    • Begun reviewing SSH, system, process, database and network logs for possible data transfer.
    • Started notifying customers and requiring precautionary password changes.

    Because root access was obtained, simply deleting the identified files cannot establish that an affected host is trustworthy. A clean rebuild is the only reliable long-term remediation.

    Customer precautions

    We are advising all customers to:

    • Change their customer account and control-panel passwords.
    • Change root or administrator passwords on their VPS instances.
    • Review all authorized_keys files for unfamiliar entries.
    • Replace application, database, email, API and backup credentials.
    • Change reused passwords on other services.
    • Enable two-factor authentication wherever available.

    Quick checks for other Virtualizor operators

    Before deleting anything, preserve copies, metadata and hashes for investigation.

    Search the Virtualizor installation for the known injection:

    grep -RsnE 'cdn\.nerat\.cc|widdow\.jar|jre-runtime\.dat|AAAAC3NzaC1lZDI1NTE5AAAAIP13pPAm5jmInLQYD3XNb3HwrW4cAKDcphoT4kSKrnte' \
    /usr/local/virtualizor /etc/systemd/system /root/.ssh 2>/dev/null
    

    Check the service, process, account and connection:

    systemctl status java-jre-update.service --no-pager
    pgrep -af 'java|jre-runtime|widdow'
    getent passwd proxyuser
    ss -tpna | grep -E '31\.77\.220\.138|:2025'
    

    Check the payload hash:

    sha256sum /usr/lib/jvm/.cache/jre-runtime.dat 2>/dev/null
    

    Do not restart Virtualizor or zzvirtservice before checking the files. Restarting an infected startup script may execute the payload again.

    Virtualizor urgently needs to confirm:

    1. How these legitimate Virtualizor files were modified or distributed.
    2. Which versions, packages, mirrors or update periods were affected.
    3. Whether its update or distribution infrastructure was compromised.
    4. Official clean hashes for all Virtualizor files.
    5. Whether any database export functionality was used.
    6. A verified recovery and rebuild procedure.

    We will provide further information as our investigation continues. Other affected providers are welcome to compare timestamps, hashes, SSH indicators and destination addresses with us privately.

    Also since virtualizor pushed updates 1-2 days ago which it was not generally from them, 5 out of 34 total hypervisors were affected by this as those 5 were updated but the rest failed for some reason as checked by cronjob/emails. Seems Virtualizor has no clue on what is going on and keep asking if we are using hetzner or ovh...

    EDITED, to include more detaiils.
    @virtualizor

    I really don't get this, I never did, exposing a bare metal node on the public net is like having a sign "come and fuq me".

    Many host are like that. Maybe until the site admin made it a rule

    There should be a tag on LET for providers that have gone through a very, very, very basic security audit. Like... not leaking L2 traffic, not able to hijack other IPs from other tenants, not exposing the node directly to the internet, and using reasonably up-to-date software.

    Thanked by 1rpqu
  • host_chost_c Patron Provider, Top Host, Megathread Squad
    edited 8:16AM

    @rpqu said: Many host are like that. Maybe until the site admin made it a rule

    It is a bad idea, it always was a bad idea, and it will always be a bad idea.

    This is not the first time we see this. nor the last.

    I get the fact that it is much more simple and easy to configure your nodes with a public ip address, yet, this is precisely the case where simple is not a good idea. ( unless you have something like a Palo-Alto/Fortinet/Check Point/Cisco or other in front of it - even then, bad idea :D ).

    Back in 2013 when I took my MSFT training courses and exams, all of them started with "don't set a public IP address on a server unless it is absolutely necessary, and even then be sure to put 3 packs of condoms on it". :D

    A decade later, same problem ( well sort of ).

    I am not mocking anyone here, that was not the idea behind my reply, but some should really change the way they do internal traffic shaping, as nodes and their traffic should be LAN traffic not WAN/Public, VM traffic is Public traffic, but nodes defenetly not.

    @forest said: There should be a tag on LET for providers that have gone through a very, very, very basic security audit. Like... not leaking L2 traffic, not able to hijack other IPs from other tenants, not exposing the node directly to the internet, and using reasonably up-to-date software.

    Then LET would be a very lonely place. :D

    As much as I like your idea, regulations rarely have a good outcome, and in that case, I would want KYC on users here, deal ? so the effort of Providers compensates with the price. It would be a fair trade in my view. :D

    Failure usually leads to improvements, so chill and wait it out. :D

    Also, most of you guys here filter out BS pretty quickly, well, most of the times you even get it right ( I will underline Most Of the Times :D ).

    <3 HOST-C

    Thanked by 1rpqu
  • SaahibSaahib Host Rep, Veteran
    edited 8:17AM

    @K4Y5 said:
    Virtualizor is a POS PHP based cesspool. Condolences to those that use it commercially and were affected by this.

    What is wrong with PHP, remember, you can't call a CAR faulty if driver has no proper skills.

    Virtfusion and hypervisor.io are built using PHP, so by that law, both are flawed ?

    Thanked by 1BharatB
  • backtogeekbacktogeek Member, Host Rep

    @MannDude said:

    @backtogeek said:
    Even more crazy when you consider VirtFusion offer migration services.

    Since when?

    They always have.

    Thanked by 1rpqu
  • xHostsxHosts Patron Provider, Veteran

    If allowed I would be happy to share the basic bash script we have to do the major parts of the checks if this is also fine with site admins

  • @xHosts said:
    If allowed I would be happy to share the basic bash script we have to do the major parts of the checks if this is also fine with site admins

    Why wouldn't it? I reposted the script the Muppets used to "destroy" the cloudcone vms.

    Thanked by 1forest
  • ChunkserveChunkserve Member, Patron Provider

    Speaking about vulnerabilities...

    WHMCS 9.0.7 and WHMCS 8.13.6 are now available as important maintenance releases for the WHMCS 9.0 and 8.13 series.

    These releases address a security vulnerability which if exploited can result in unauthenticated remote code execution. There are currently no known exploits in the wild, but details that the vulnerability exists are shared.

    We strongly recommend that all self-managed WHMCS customers update to the latest available version for their release series without delay.

    This never ends🥲

    Thanked by 2host_c forest
  • rpqurpqu Member
    edited 8:34AM

    @host_c @forest Please make tutorial for these basic opsec.

    If it was me, I would have told hosts to block 0.0.0.0/0

    Thanked by 2host_c forest
  • host_chost_c Patron Provider, Top Host, Megathread Squad

    @Chunkserve said:
    Speaking about vulnerabilities...

    WHMCS 9.0.7 and WHMCS 8.13.6 are now available as important maintenance releases for the WHMCS 9.0 and 8.13 series.

    These releases address a security vulnerability which if exploited can result in unauthenticated remote code execution. There are currently no known exploits in the wild, but details that the vulnerability exists are shared.

    We strongly recommend that all self-managed WHMCS customers update to the latest available version for their release series without delay.

    This never ends🥲

    Nop, it is becoming a weekly update as it seems. :D , talk about QC control, I really have the feeling is is being vibe-coded by now.

    Thanked by 1Chunkserve
  • host_chost_c Patron Provider, Top Host, Megathread Squad

    @rpqu said: If it was me, I would have told hosts to block 0.0.0.0/0

    roter-core-AS211462#: ip route 0.0.0.0 0.0.0.0 Null0 - done. :D >:)

    Thanked by 1rpqu
Sign In or Register to comment.