Black Friday 2020 - NVMe and Storage deals - deploy worldwide

Daniel15

@thedp said: Or you could just start off with apt-get install --reinstall procps

That's assuming that apt-get hasn't been compromised, there's no compromised system library that intercepts downloads of Debian packages and the GPG calls to verify the repository, etc. If someone replaces core libraries (like glibc) with compromised versions, there's not a lot you can do other than replace every single system library with a known good one, at which point you're essentially reinstalling file by file and may as well just do a real reinstallation.

DP

@Daniel15 said:

@thedp said: Or you could just start off with apt-get install --reinstall procps

That's assuming that apt-get hasn't been compromised, there's no compromised system library that intercepts downloads of Debian packages and the GPG calls to verify the repository, etc. If someone replaces core libraries (like glibc) with compromised versions, there's not a lot you can do other than replace every single system library with a known good one, at which point you're essentially reinstalling file by file and may as well just do a real reinstallation.

Yep, agreed.

Falzo

@Daniel15 said:

@thedp said: Or you could just start off with apt-get install --reinstall procps

That's assuming that apt-get hasn't been compromised, there's no compromised system library that intercepts downloads of Debian packages and the GPG calls to verify the repository, etc. If someone replaces core libraries (like glibc) with compromised versions, there's not a lot you can do other than replace every single system library with a known good one, at which point you're essentially reinstalling file by file and may as well just do a real reinstallation.

name a malware/attack that goes through all those efforts for running a f*cking miner ;-)

not saying that reinstall is a bad idea. it never is in such cases. sometimes just deleting that user account might be enough though...

Daniel15

@Falzo said: name a malware/attack that goes through all those efforts for running a f*cking miner ;-)

Yeah, I agree that most attackers won't go to such lengths, but it's always possible a dedicated attacker will do so, particularly if they find an entire subnet full of vulnerable machines.

In the past when I was more involved with web hosting (used to help run a few small hosts), I occasionally saw sophisticated attackers that would leave a backdoor and cover (or attempt to cover) their tracks. For example, run a root shell at a high port, then overwrite the netstat binary with a hacked version that filters out that port. They'd leave something obvious (like a weirdly named executable), the idea being that you'd find that, clean it up, then assume that's the only thing they did and that everything was safe. Then a month or two later, they'd come back in through the backdoor.

That was over 10 years ago though, before cryptocurrency was really a thing, so it's possible these attacks are less sophisticated these days and only go for things that make them money relatively quickly (miners and things like that).

DP

@Daniel15 said:

@Falzo said: name a malware/attack that goes through all those efforts for running a f*cking miner ;-)

Yeah, I agree that most attackers won't go to such lengths, but it's always possible a dedicated attacker will do so, particularly if they find an entire subnet full of vulnerable machines.

In the past when I was more involved with web hosting (used to help run a few small hosts), I occasionally saw sophisticated attackers that would leave a backdoor and cover (or attempt to cover) their tracks. For example, run a root shell at a high port, then overwrite the netstat binary with a hacked version that filters out that port. They'd leave something obvious (like a weirdly named executable), the idea being that you'd find that, clean it up, then assume everything is safe. Then a month or two later, they'd come back in through the backdoor.

That was over 10 years ago though, before cryptocurrency was really a thing, so it's possible these attacks are less sophisticated these days and only go for things that make them money relatively quickly (miners and things like that).

10 years ago, rootkits would also replace most/commonly used binaries with trojan/infected ones.

Unbelievable

And you thought the ticket queue was long before .....

orion504

I do hope that the issue will be resolved soon, no more resource abuse. I got network issue for around half a month, probably due to the neighbor VPSes are being abused with these crypto and others. I have to move production box into another provider, my (production) box was idled since then, and will be until the issue is fixed. I think my case was even worse then most here: I uploaded ISO and installed from that ISO, then saw network issue, tried to restore the default OS (template one), but no luck. Then found the compromise, tried to reinstall with the uploaded ISO but cannot restart machine if choosing mounted ISO, and cannot delete current ISO as well. Have to use a trick with online boot to completely destroyed the VPS and reinstall using netinst. I am completely OK with zero support at normal level basis. I like the provider, not because of appealing pricing, but because of one feature I love. I hope everyone is staying calm as well and waiting for the things to be resolved/settled.

TimboJones

@Unbelievable said:
Has anyone noticed that practically every company that pushes big sales and does a big volume of orders on black friday and Christmas- usually winds up with a shitload of problems? All that grief to save a few dollars? is it really worth it as a customer?

Sounds like Reddit should be on the watch for shorts on Racknerd.

Unbelievable

@TimboJones said:

@Unbelievable said:
Has anyone noticed that practically every company that pushes big sales and does a big volume of orders on black friday and Christmas- usually winds up with a shitload of problems? All that grief to save a few dollars? is it really worth it as a customer?

Sounds like Reddit should be on the watch for shorts on Racknerd.

Nah, at the end of time it will be the 100th iteration of wootalparacknerd it will never die

Shot2

Jan 31 16:14:19 storage-uk sshd[21019]: Accepted password for debianuser from 205.185.125.189 port 39242 ssh2

whois 205.185.125.189?
... Frantech

xauser

Jan 31 22:14:19 : Invalid user debianuser from 205.185.125.189 port 49876
Jan 31 22:14:19 : Disconnected from invalid user debianuser 205.185.125.189 port 49876 [preauth]
Jan 31 22:24:19 : Invalid user debianuser from 205.185.125.189 port 44506
Jan 31 22:24:19 : Disconnected from invalid user debianuser 205.185.125.189 port 44506 [preauth]

I prepare for reinstall

Razza

@xauser said:
Jan 31 22:14:19 : Invalid user debianuser from 205.185.125.189 port 49876
Jan 31 22:14:19 : Disconnected from invalid user debianuser 205.185.125.189 port 49876 [preauth]
Jan 31 22:24:19 : Invalid user debianuser from 205.185.125.189 port 44506
Jan 31 22:24:19 : Disconnected from invalid user debianuser 205.185.125.189 port 44506 [preauth]

I prepare for reinstall

Why it says invalid user mean the user doesn't exist on your server

MagicalTrain

@hosthatch said:

@snt said: Lets just wait for their next excuse and promise

I am going to respond to these 3 people who are likely not even our customers and keep making posts like this. It's a pattern - you will see the same 3-4 people making these posts. I can tag them, but would be better if I just take screenshots of all the comments they have made so people can form more informed opinions.

---

Yes - there are delays on BGP provisioning, IPv6 reverse DNS, custom ISO being broken after using a bad URL etc - we do not consider these to be critical as we are already backlogged with more important issues and working as fast as we can. I am sorry you have not received a response to these issues, but you likely will once we have rolled out fixes for these issues. It has nothing to do with support as all of these are engineering issues and support cannot do anything other than to ask you to wait. I am sorry if IPv6 reverse DNS was the specific reason you signed up for the service. It is not working in some of the newer locations, and we do not have any ETA on resolution for now.

---

On the point of the Debian template that was compromised - this was an official SolusVM template. They confirmed that it was compromised with a secondary user, and we are in the process of reaching out to the affected customers now and fixing this.

To be clear - we are not the only provider that uses these templates. I am not shifting blame here - as it is our responsibility to provide a secured working OS template to our customers and not our suppliers', but again, this was an official SolusVM template. Any other provider using the same template (which is a lot of them) is compromised - and hopefully SolusVM will be emailing their customers about this.

I would appreciate if the support could tell me to wait and just keep me in the loop. Id rather not have to come here to know why my tickets havent been answered in 1+ months.

xauser

Yes, user doesn't exist on my boxes but I will reinstall anyway.

SirFoxy

@MagicalTrain said:

@hosthatch said:

@snt said: Lets just wait for their next excuse and promise

I am going to respond to these 3 people who are likely not even our customers and keep making posts like this. It's a pattern - you will see the same 3-4 people making these posts. I can tag them, but would be better if I just take screenshots of all the comments they have made so people can form more informed opinions.

---

Yes - there are delays on BGP provisioning, IPv6 reverse DNS, custom ISO being broken after using a bad URL etc - we do not consider these to be critical as we are already backlogged with more important issues and working as fast as we can. I am sorry you have not received a response to these issues, but you likely will once we have rolled out fixes for these issues. It has nothing to do with support as all of these are engineering issues and support cannot do anything other than to ask you to wait. I am sorry if IPv6 reverse DNS was the specific reason you signed up for the service. It is not working in some of the newer locations, and we do not have any ETA on resolution for now.

---

On the point of the Debian template that was compromised - this was an official SolusVM template. They confirmed that it was compromised with a secondary user, and we are in the process of reaching out to the affected customers now and fixing this.

To be clear - we are not the only provider that uses these templates. I am not shifting blame here - as it is our responsibility to provide a secured working OS template to our customers and not our suppliers', but again, this was an official SolusVM template. Any other provider using the same template (which is a lot of them) is compromised - and hopefully SolusVM will be emailing their customers about this.

I would appreciate if the support could tell me to wait and just keep me in the loop. Id rather not have to come here to know why my tickets havent been answered in 1+ months.

its been known hosthatch support is awful you can't even complain its the price of using them

Daniel15

@Shot2 said:
Jan 31 16:14:19 storage-uk sshd[21019]: Accepted password for debianuser from 205.185.125.189 port 39242 ssh2

whois 205.185.125.189?
... Frantech

My guess would be that it's either someone running brute force attacks from their BuyVM VPS, a compromised VPS, or a Tor node (and someone is running brute force attacks via Tor). @Francisco?

I imagine that since this vulnerability is known, attackers are going to be scanning the IP ranges of VPS providers to find vulnerable servers.

lentro

@xauser said: 205.185.125.189

@Francisco seems like your IP, and it doesn't seem to be a TOR relay either.

DP

@Daniel15 said: I imagine that since this vulnerability is known, attackers are going to be scanning the IP ranges of VPS providers to find vulnerable servers.

If their ssh is running on a non-standard port, with PasswordAuthentication no, key only and allowing/accepting selected IPs, they should be fine I suppose.

Shot2

@Daniel15 said:
My guess would be that it's either someone running brute force attacks from their BuyVM VPS, a compromised VPS, or a Tor node (and someone is running brute force attacks via Tor). @Francisco?

I imagine that since this vulnerability is known, attackers are going to be scanning the IP ranges of VPS providers to find vulnerable servers.

Yep, I did not mean to imply there's some evil Frantech customer out there - only some (compromised?) BuyVM jumphost/VPN/TOR exit

On HostHatch's Buster template, debianuser is not a sudoer (fortunately). Still, pretty scary sh*t.

lentro

@Shot2 said: Yep, I did not mean to imply there's some evil Frantech customer out there - only some (compromised?) BuyVM jumphost/VPN/TOR exit

For these, I believe they're typically P2P worms, so infected machines try to contact other infected machines to create a web of infected machines.

wimiu

I knew the vulnerable issue in this discussion chain yesterday and receive HostHatch urgent reinstallation action request today~~

I still trust the service HostHatch deliver.
Hope HostHatch can get through a chain of issues they're facing.

ThinVps

I been told by hosthatch to create a ticket for my routing and latency issues, and they will answer me asap, so i created a ticket... its been 1+ weeks since then. Slowest ASAP ever.
p.s. it was my 2nd ticket. My 1st ticket is unanswered for 2+weeks. Someone mentioned above that you get what you pay for, and yes i got 22 dolla promo (1.8ish/month), but what if i paid full price, something like 5-7/month? That would be a robbery in broad daylight, and worse provider ever, and i been in this sphere since late 90s (bouncers, slots, dedis etc), and used many providers. Never experienced such a poor customer service. They should be banned as provider, they dont even meet low end tag. More like shittyend. Service like that should be at a dollar store (if that).

plumberg

@ThinVps said:
I been told by hosthatch to create a ticket for my routing and latency issues, and they will answer me asap, so i created a ticket... its been 1+ weeks since then. Slowest ASAP ever.
p.s. it was my 2nd ticket. My 1st ticket is unanswered for 2+weeks. Someone mentioned above that you get what you pay for, and yes i got 22 dolla promo (1.8ish/month), but what if i paid full price, something like 5-7/month? That would be a robbery in broad daylight, and worse provider ever, and i been in this sphere since late 90s (bouncers, slots, dedis etc), and used many providers. Never experienced such a poor customer service. They should be banned as provider, they dont even meet low end tag. More like shittyend. Service like that should be at a dollar store (if that).

So why purchase and cry about it?

ThinVps

@plumberg said:
So why purchase and cry about it?

Because i did not know? Duh. I had bad feeling about it once started noticing latency and routing issues, and then saw some post about customer services. Right away in my thread i mentioned possibility to do charge back because their tos is straight up wack, and i was attacked. I was right all this time, should have done charge back and moved along. But whatever. Just wanted to chime and report issue as well, so others dont get scammed into this.

Unbelievable

@thinvps - my advice, choose a course of action and do it. Either stay with hosthatch (and all that entails), walk away and eat the cost (make sure to shut off paypal subscription etc), or chargeback. Pick a path and don't look back. Many companies on here that grow too fast have major issues. Too many to name. One of lifes lessons-unfortunately, pay a little more, research a little more, dont jump on most black friday deals; and avoid most of LET and find services elsewhere. Use LET for hobby stuff, dev stuff, and all that random stuff - because often it will let you down.

plumberg

@ThinVps said:

@plumberg said:
So why purchase and cry about it?

Because i did not know? Duh. I had bad feeling about it once started noticing latency and routing issues, and then saw some post about customer services. Right away in my thread i mentioned possibility to do charge back because their tos is straight up wack, and i was attacked. I was right all this time, should have done charge back and moved along. But whatever. Just wanted to chime and report issue as well, so others dont get scammed into this.

Yes. Chargebacks are the way to resolve problems.

TOS why did you agree if you find them wack?

No one is scammed. You got into something without due diligence. Find a provider offering similar setup services at same price and then speak.

Support issues were something provider defined in the second post of the thread and you still choose to ignore and cry b*tch about your mistakes

Unbelievable

@plumberg no matter what you think of @thinvps - the reality is the fact that hosthatch is still trying to cleanup from selling way too many services in relationship to his firms capacity to handle them. The support issues were defined over 2 months ago. Rational companies who are aware of issues (even before public acknowledgement 2 months ago) , can generally fix things in that time frame - unless they have a reason NOT TO. Easy to trash customers on LET; often harder to get Providers to offer support in a timely fashion, especially those with big sales on black friday

redgreenblue

I previously had VMs with Hosthatch which had various problems, eg: IPv6 drops that just appeared overnight in one case, frequent IPv4 + IPv6 drops in another case. I just let those VMs run till the end of their paid up period and did not renew them when Hosthatch did not fix the problems. From memory, which admittedly can be sparse, there were tickets opened for months.

After seeing Hosthatch's promise to improve support, I thought I would give them a try again. But this incident, and seeing the way @hosthatch has communicated publicly, I am rather disappointed. It is one thing being willing to help smaller firms. But it is another when folks from the company do not seem to communicate well on a forum, even though they have been polite and showed professionalism when responding on tickets (when they actually do respond). I should have been wiser!

default

notarobo

anybody on LA server have much lower cpu perform now?