New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
To anyone curious - this was one of the attack vectors: https://github.com/serghey-rodin/vesta/commit/19400663ec500b9aa5af6b9a718ecd99c677e8c3
I got some insight from another LET member who was able to reproduce it. Maybe they want to step forward now that a push has been fixed and explain it.
@MasonR Maybe want to update title again,
[May 19 Security Update]or similar.I just say I’m surprised. I was genuinely convienced that after all this time doing nothing they have dropped the project. I guess better late than never.
So it was via roundcube after all?
It would be great if someone who's followed this affair could give a short summary.
https://pastebin.com/hknhub09 2 - that’s how the malicious config file looked on infected machines before the latest update… watch out for lines 685 and 797
this enabled the attacker to send a request to the webmailer including malicious code as header var ‘accept-charset’, which in the end would be executed through the eval-lines in that config.
you can do that e.g. with a curl request to the /webmail url and adding something like
--header "Accept-Charset: fwrite(fopen('/tmp/update','w'),'Hello angstrom');"traces in the logfiles would be a single barely noticable HEAD request without further clues …
this config file including those eval commands is NOT part of any default roundcube install but gets pulled from the vesta config repos during the install process. given the original timestamps of the file it’s been there from some date in december 2017 and got replaced some time after the DDOS happened - but only in the repos and not on the VMs.
for me this also makes sense if you think about the missing pattern on which VMs got hacked - only those that have been installed after december 2017 … got pretty much nothing to do with port 8083 or not, though this of course was the first most common conclusion to be made.
and that’s why I am thinking, machines that only got desinfected or maybe even reinstalled directly after the DDOS but before the repo got cleaned, would have been vulnerable until this update now.
Thanks, @Falzo, very useful.
This is also good to know for those (like me) who have a stand-alone roundcube installation.
And also all 6 Vulnerabilities that Patrick @SecNinja sent have also been patched.
As for the vector of the attack, it looks like those eval lines were injected into the roundcube configs somehow. The eval code is only present in my only server that was hacked, it is not there on any other ones. Infact, on some of my servers (Debian) I only have config file of 80 lines only. Also noted with a few more friends that these were only servers after Dec-2017 onwards as @Falzo said, any servers built prior to that date were not having that malicious eval() function in the roundcube config.
If this is to be presumed as the "point of attack" I guess it was not a code vulnerability at the outstretch of this problem, rather something malicious that got into that config file which is being pulled from c.vestacp.com webserver from where other configs are also pulled at installation time. The theory that Vesta Repo was hacked might not be completely false, though it was not the GIT repo, may be some other middle-ware server from where a few resources/configs/static-files are being pulled from as @Falzo also pointed out. Won't exactly know before Serghey Rodin and other official devs crack down on it completely.
I'm optimistic on VestaCP... just to learn the fact that the breach was planted through a hack or something to the repos and not a vulnerability inside Vesta actually. It's been fixed and Serghey has listened to devs and other people contributing. The only problem point left in my heart is "bad slow communication" from his side, even with his internal team. If he gets more interactive, I think the project will shine again.
So anyone re-enabled VestaCP or we should still wait?
Presuming the above scenario, I don't think it would have made a difference because VestaCP WebUI service or its API were not found to be vulnerable, even in the tests to reproduce the hack.
Someone has to be first :-)
Presuming the above scenario, I don't think it would have made a difference because VestaCP WebUI service or its API were not found to be vulnerable, even in the tests to reproduce the hack.
Infact, just out of Paranoia, I renamed the roundcube config file so that it won't even run, as I don't run any incoming email service on my web servers. If you also don't run email service, you may very well disable it and reduce your attack surface.
I have it re-enabled on almost all my servers (100+) barring a few very critical ones, just to be extra cautious
no need to wait, most likely the vesta-service itself was not even related to it after all so did not make a difference, if stopped or not.
probably no one can ever tell, if there had to be some kind of API access or the likes to elevate access even further. with the initial hit through the normal webserver (see above) you most likely could have done anything that the vesta admin was able to do... especially running all kinds of vesta-cli-commands via sudo for instance, like bringing the firewall down. or writing enough stuff to a file in /tmp and execute it, to deploy whatever trojan or malware you needed too and which could have taken care of the rest.
again: most likely nothing to do with the vesta service itself being up or down, nor with the port 8083 available or not.
An easy way to tell if you had the compromise as @Falzo showed is to run:
I had to add extra spacing because let won't let me post it without spaces.
If the defaults.inc.php is updated recently, May 18th and on. You had the eval attribute.
If business dont use Vesta.
If there was some kind of vulnerability in the deployment chain between github and vestacp.com (assuming deployment runs in that direction) then it really still has to be investigated. It's probably best to wipe vestacp.com completely and rebuild it from scratch, if the config file propagated from there.
The attack vector itself might have been malicious or it might have been an error someone made. So it's important to figure out exactly where and preferably how it was introduced.
That script to delete the "eval" lines from the roundcube config also looks dubious. The config file should also be rebuilt rather than just edited.
There is no edits to that file: https://github.com/serghey-rodin/vesta/commits/master/install/debian/9/roundcube/main.inc.php
I am running Debian 9, and it patched my file for example. I checked Debian 8 too. Neither file had any eval in it, or any history of it being added. So it got in there a different way than being added to the github repo.
Can confirm I did not see this on my test install either.
Right, it got into the file in some people's deployments, through some kind of exploit. So what was the exploit? Unless that is known, we should assume it is still out there.
I've checked that it doesn't appear in any commits in the github repo.
>
cough hacked repo cough
Well, where? Is github the authoritative repo? Did some mirror of it get hacked? Has the hacked mirror been located? If the github repo was hacked and rebuilt, I'd hope that the devs would have said something. The current repo there doesn't have any commits with that attack.
Think the time frame was Vesta installs between Dec-ish and just prior to April DDoS'ing. Not github, c.vestacp.com was hacked and someone slipped in the malicious roundcube configs.
Edit: read @mehargags comment a few posts up, that's the working theory
As soon as the alert went out I installed a test version of VestaCP on a disconnected machine, this was before any patches were released. There is no sign of any of the malicious configs on that machine.
Right, which is why I said "just prior to April DDoS'ing" - it got fixed at some point in between. We don't actually know how long the hacked config sat in there for, but the working estimate was sometime between Dec '17 and Apr '18 there was a hacked roundcube config on vesta's servers.
A friend is running over 400 Vesta installs across dozens of providers (including DO/OVH who had their ranges scanned and had a ton of boxes on their network hacked), all were installed before December and none of those 400+ installs were pwned.
It also wasn't a file that got pushed during an update, so it seems only to have infected machines with new installs in that time frame. It also didn't get fixed during any subsequent updates because it doesn't get pushed, until this latest update was pushed to help clear out the malicious lines if present.
If this is the case, then some one should have a copy of it with the alterations. Either a full copy or back up. I would assume that who ever backs that server up might have it.
If there is no copy, or they can't find one from that period that was altered it might not be because of Vesta itself. It could be a roundcube problem.
I saw mehargags' post saying people got the hacked config from c.vestacp.com, which in turn must have gotten it from somewhere. The "somewhere" could mean an upstream repo (maybe github, maybe something else) or it could mean c.vestacp.com was hacked directly. Either way, a vulnerability got exploited. Unless the vulnerability has been (with reasonable certainty) identified and fixed, why should we think it's not still open?
That's a fair assumption to make as their devs haven't released anything, to my knowledge, about how they've secured their repos.
I think it must've been hacked directly as I don't think anyone has found signs of the exploit on their github.
Here: https://pastebin.com/hknhub09
Lines 685 & 797. Copy of the config on a pwned box via @Falzo
Plausible but I'd like to know how code gets to c.vestacp.com in the first place. It might come from github (e.g. through commit hooks) or it might get there some other way. If it's some other way, then that has to be checked.
Yep
. I saw that and it's what I looked for in the git repo.
Would also like to know, but the Vesta developer(s) don't seem the most open about admitting their faults.
I remember a comment made when first update post-hack was pushed that the fixes made it 'bulletproof'. lul
Right but that doesn't mean that it came from a Vesta server. I meant if it did then it should still be some where. There might even be a log of it happening. Just depends on if it happened on individual servers or from the common repo.
I have a full ISO backup from one machine that got infected which dates 6th of march. so clearly before the malicious config.php was used for putting the trojan onto the servers (the HEAD requests and gcc.sh creation dated early in april).
I am going to restore it and get that compromised file and it's original timestamp for you ;-)
afaik there is no automatic sync from the github repo to that c.vesta.com server, where in the end the installer wgets all those config files from.
supposedly that's all in the hands of the main dev Serghey alone and he is the only one who has access and is in charge for it, so if his access to that webserver/ftp-server or whatever got compromised... well.
of course there is no chance to reliable prove that without his admittance and disclosure.
but narrowing down by the timeframe of installations and version of roundcube install like @Harambe, @jarland and others already did, will bring it very close to that.
It's the same for me, I had only three machines carrying that config-file and the gcc.sh stuff. those three had been installed after dec '17, the rest of my VMs were not affected and installed earlier...
as said before from the comments in different boards one could not make out a pattern about the port and vesta-service or api being involved or not. now that makes much more sense, why esp. for people with a lot of installs even on the same providers regularly only a few where affected, and what they have in common is the timeframe of their vesta installation. enough circumstancial evidence may be evidence after all ;-)