All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
HXServers | Our apologies | RCA | What happened?
Hello all,
I'm sure by now you've heard about the recent issues with HXServers. I wanted to create a new discussion to better inform existing customers, and potential new customers with actual facts instead of conjecture and opinions of what happened.
What was the reason to "RETIRE" plans?
HXServers started on "Platform 1.0" - Using Virtualizor. We switched to VirtFusion, calling it "Platform 2.0". All customers hosted on Virtualizor had their VPS plan named changed to "RETIRED". Originally, we stated we had no intention on forcing customers to Platform 2.0 under VirtFusion. Unfortunately, we've had to change that decision, as part of Platform 1.0 was renting IPs at $2 / IP from ReliableSite, GTHost, etc. We don't mind operating in the negative, but this was severely in the negative, and would've reduced our ability to improve our platform.
What was the plan to retire Platform 1.0?
The plan was to notify customers on October 1st that migration needs to happen by October 21st. Our intention was to provide a new VPS to customers under Platform 2.0, with the same VPS specs and price. Any customer who paid for October would be refunded in exchange for the inconvenience of having to migrate.
What actually happened?
We did not notify customers. We had thought we did. This was gross negligence on our part. We relied on automation to notify customers and never did double check that automation had run successfully. The only piece of automation that worked, was automated refunds and automatic VPS termination. What makes things worse, is we didn't know automation failed for sending notifications until it was blatantly brought to our attention after VPS' had already been terminated. Upon investigation, the Announcement in our billing system wasn't even posted live, like we had initially claimed on LowEndTalk. The announcement, the email notification, every method we use to communicate to our customers, failed - without our knowledge. We could've checked this at any time, but we didn't.
What VPS' were terminated??
LA - RETIRED - All VPS have been terminated, without notice, with refund.
NY - RETIRED - No VPS have been terminated.
Miami - RETIRED - All VPS have been terminated and refunded where renewal was recent. Any VPS with a renewal date after October 21st was not terminated but will not renew and will be terminated at the next billing cycle.
My VPS is terminated, can I get my data??
This is the biggest problem in what we've done. Any VPS that has been terminated, has been fully removed from our servers. Data recovery is not possible. Unfortunately, when our system removes a VPS, it also removes any associated backup of that VPS at the same time.
What happened with Dallas??
Dallas was a new location. We rented servers from GTHost. We rented an IP Block from IPXO. We launched Dallas on October 13th, and by October 16th, we decided Dallas wasn't going to be a popular location. The 10 VPS that was hosted in Dallas, was refunded and terminated October 21st. Notifications were not sent for Dallas removal either.
What are you going to do for RETIRED NY?
Retired plans in New York will be properly notified and terminated. We will be triple checking all customers on retired plans in NY are aware of the termination before any termination happens.
Many mistakes were made before, during, and after our plan to decommission our retired services and platform. Although refunds were processed correctly, we completely failed in notifying customers to prepare for migration. As a result, retired platform plans were terminated with seemingly no notice to our customers, causing data loss, mass confusion and frustration. As a company, HXServers needs to do better in the future, and we take this as a hard lesson to learn what not to do when handling customer data. Although this only affected a very small portion of our customer base, we're handling this as a disaster, caused by gross negligence. Customers affected by our gross negligence will receive a new VPS, with the exact same pricing (monthly only), and hardware specification free for the first month, and future months will match the previous price the customer paid. Although we're taking ownership of the recent issues, we must express our worrisome of our operations. As a company, HXServers needs to reevaluate our workflows, our (now former) employee(s), and our policies on customer data and customer notification processes. HXServers is committed to constant improvement, and hopefully some day we can rebuild the trust we once had in this community. We're extremely saddened by our recent actions and owe each and every one of our affected customers a huge apology. To those who are affected, we're so sorry for how we handled this, and we will never make these mistakes again.
Number of VPS affected: 32
Number of existing VPS: 1,432
We understand you're frustrated. We're here to help as much as we can. Feel free to vent, interrogate us, belittle us - whatever you need to do. Let it out
- Was this explanation helpful to you?152 votes
- Yes29.61%
- No70.39%
Comments
xD
When is your company going to "retire"?
We have zero plans on "retiring" our company.
You had a poll asking if this thread was of help.
Why was it retired a few seconds later?
What do you think about a great discount ora very limited offer as apologies?
Fixed.
Looking forward to the results.
Our offer is a free VPS for a month to affected customers, with a price lock on what they were paying before the disaster.
I like how no one connected the dots and never asked the question: Why the fuck no client opened a support ticket asking for details/refund/additional questions/I_want_my_IP for like 3 weeks.
I think that you could make better than this.
One of the customers who canceled their plan due to what happened here.
I've had the opportunity to test plans with many providers here on LET, and I understand that every beginning has its ups and downs. However, trust goes far beyond low prices and good hardware.
Surprisingly, my experience despite short, was very positive. The VPS specifications were good, the price was attractive, and the support never failed to meet expectations, always being helpful and prompt in their resolutions. But considering everything that transpired, I wouldn't trust my data with HX Servers for another second. Trust is the key.
Here is a link to a much more helpful LET thread re the @hxservers clusterfuck ->
https://lowendtalk.com/discussion/189504/my-server-is-terminated-by-hxservers-for-no-reason
And here's one of my posts that probably provides some more realistic insight:
So, you meant to send emails out but some unnamed former employee - whom you selected and employed - forgot to click 'publish'.
Then you meant to have a serious discussion here, after all you wanted to be perceived as a new but professional provider but someone, so I have to assume, hacked your LET account and wrote BS that was so weird that people began to laugh you out of the thread.
Then you got caught red handed by my gofer and now you tell us, that you meant 'recurring' when you typed 'first'.
Could you possibly have meant "guys, I'm a serial liar and also love to cheat. Any willing victims here?" when you typed "We're a small company, founded in 2023 based out of Irvine, California. Our goal is to provide excellent service and exceptional customer service." in your thread?
(I'm jsg's gofer and I confess having typed this)
(No, I am the gofer, you are just a mean cat who impersonates innocent little gofers!)
(Do you want to become an ex-gofer in my belly? Damn confess at least that it was your ex-girl-gofer!!!)
(Here's the gofer again. I lied. Sorry. Actually my ex-girl-gofer typed all of this! Ouch! Ouch! Stop already, i did what you asked for!)
(Hehe, stupid gofer thought he's smart. Well at least he tastes good, hehe, cat, uhm I mean, ex-gofer-girl over and out)
When the Dallas servers came out, I instantly bought one. Since the Spooky coupon was done.
As soon as I got the Dallas server, he released news there's going to be coupons for everywhere else, Spooky30 coupon.
Like, wtf was even that?
A few minutes later, someone was upset that their server got migrated without notice.
This all happened on Oct. 13.
I canceled my autopayment for next month and figured this company isn't very good, so I left it at that. Didn't even bother asking for a refund. I got one last night randomly though.
Yesterday sucked but I do appreciate the full breakdown and the apology.
When are you going to take care of my ticket?
Ticket #XEF-930314
you terminated my annual payment plan without any notifacation and it will never be recreated.
I've responded to you.
And I have responded to you in DETAIL.
You terminated my two VPS with data without any notice email of even 8 hours.
We're willing to offer you a monthly payment plan. We no longer support yearly payment plans.
I'm not asking you to reopen the annual payment plan, but the original annual payment plan should be retained. Because your mistake caused the annual payment plan to be canceled, this was not caused by user's mistake. Of course, you can also refund and break the contract at any time. This is your right.
Your VPS' were made within seconds of you providing us the information we needed.
Unfortunately, we cannot retain an annual plan as we no longer support annual payment plans.
And how many HOURS you took to respond to my ticket and response was provided only after ticket number was posted here
still, Thank you.
Customer service is the true embodiment of karma in business; treat your customers well, and the universe will reward your company in kind.
11 hours. You submitted a ticket just after I passed out. Remember, we're a one man shop at the moment.
VirMach lost data: one year free extension.
OVH burned server and lost data: three months free extension.
Nexril had a day of downtime (no data loss): two months free extension.
Good to know, thanks @yoursunny. However, all of the businesses you've mentioned - have been around significantly longer than we have. We're still a start up. A free month is what we're offering at this moment.
And I was ok until you mentioned seconds part in the post
Can you elaborate on the technical side of this? If the data and any backups have not (yet) been overwritten you should be scrambling to perform forensic recovery rather than posting on LET
The data has been overwritten already. Hence our initial delay on LowEndTalk.