All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
[Review & Urgent Help] Chunkserve: Production VPS Dead, WHMCS 2FA Broken (502 Gateway)
Hello LET community and @Chunkserve rep,
I am writing this thread both as a review of my current ongoing experience and an urgent cry for help. To be completely fair and objective, I want to state upfront that I have been a loyal customer; this is my second year (second renewal) with this exact same VPS, and I have never experienced a single issue since day one—until today.
Currently, my production VPS hosted with Chunkserve is completely dead, and I am trapped in a frustrating "Catch-22" situation: I need to open a support ticket, but their billing portal is completely broken during the 2FA login process.
Here is the exact chronology of the nightmare:
Part 1: The VPS Failure (Target VPS IP: 141.11.62.177)
- Earlier today, my VPS suddenly hung. The KVM dashboard showed the status as "Online", but I was completely unable to SSH into the Linux OS.
- I initiated a forced restart from the panel. It took an unusually long time but finally completed.
- However, after the restart, all services inside the OS (aaPanel, OpenLiteSpeed, etc.) were hung and completely unresponsive. My websites remained down.
- I decided to perform a graceful shutdown via the KVM dashboard to attempt a fresh boot.
- Now, the VPS refuses to start at all. Every time I click "Start", Virtualizor throws a red toast message: Error 1) The operation could not be completed. The task logs are filled with "Started VPS - Fail" statuses.
- I even tried enabling Rescue Mode, but that failed to start as well, throwing the exact same Error 1) The operation could not be completed message. This confirms the issue is heavily on the host node/hypervisor side, not just a corrupted OS.
- VNC Console is also completely dead. When I try to open VNC to diagnose the boot sequence, it is just stuck in an infinite "Connecting..." loop, likely because the VM process isn't running on the host node at all.
(Note regarding the attached Virtualizor log screenshot: The IP 114.10.98.229 shown in the task log is simply my local ISP connection IP performing the action, while the actual dead VPS IP is 141.11.62.177).
Part 2: The Billing Portal / WHMCS Bug (The Catch-22)
Since the VPS won't boot, my immediate reaction was to log into billing.chunkserve.com to open an urgent support ticket. This is where it gets worse:
- I enter my email and password successfully.
- I am prompted with the Two-Factor Authentication screen (/login/challenge).
- I input my 6-digit 2FA code and hit Login.
- The page hangs, redirecting to /login/challenge/verify, and eventually throws a Cloudflare 502 Bad Gateway Error (Host Error).
- I thought my authenticator app was out of sync, so I tried using my 2FA Backup Code (/login/challenge/backup). It results in the exact same 502 Bad Gateway Error.
Note: I can log into the Virtualizor panel perfectly fine, but I am completely locked out of the WHMCS billing portal due to this 502 backend bug on their end, making it literally impossible for me to contact their support team.
Call to Action:
@Chunkserve - I know you are active here. Please look into VPS IP 141.11.62.177 as the node seems to be having severe issues, and please fix your WHMCS 2FA verification endpoint so your clients can actually reach you when things go down!
I have attached all the screenshots proving the 502 error and the Virtualizor failure. I will update this review thread once the issue is resolved. Has anyone else experienced this 2FA 502 bug with them today?
Thanks in advance.
Comments
LOL
Hi @triondres !
Thank you so much for reporting the issue with 2FA. I was really unaware about it, no one previously reported problems with that recently. I'll take a look later this day.
As soon as I saw the problem, I checked it. Your VPS IP was pinging&port 22 was open:
wczapkowicz@MacBook-Pro-Wojciech-2 ~ % ping 141.11.62.177
PING 141.11.62.177 (141.11.62.177): 56 data bytes
64 bytes from 141.11.62.177: icmp_seq=0 ttl=52 time=104.042 ms
^C
--- 141.11.62.177 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 104.042/104.042/104.042/nan ms
wczapkowicz@MacBook-Pro-Wojciech-2 ~ % tcping 141.11.62.177 22
TCPinging 141.11.62.177 on port 22
Reply from 141.11.62.177 (141.11.62.177) on port 22 TCP_conn=1 time=104.650 ms
However, when trying to initiate an SSH session, it was completly "dead", not sending the host fingerprint.
On the proxmox, the VPS was running, that's why you also saw such status in the panel.
I verified that SSH traffic initiated by my end reaches the VPS NIC (tcpdump on TAP interface). At this point, my ability for further troubleshooting is limited, as our end was checked, and we cannot login to your server (and we do not want to).
I restarted it right after, and saw that SSH is responsive. Fixing a VNC for you now.
I'll DM you regarding the 2FA later. Many thanks again!!
What a disaster of provider… holy crap
UPDATE & RESOLVED (Partially):
Hello everyone, just want to give a quick update.
First of all, thank you to @Chunkserve (Wojciech) for stepping in quickly today. Despite the billing portal being inaccessible, he investigated the node, force-restarted my instance from the backend, and fixed the VNC console so I could finally get in.
To be completely fair to the provider: after successfully booting up and checking the logs, it turns out the OS level freeze was actually triggered by a recent service configuration update on my end. It caused a severe lock up, which perfectly explains why the KVM saw it as running and pingable, but the sshd process was completely unresponsive. I have now optimized my internal setup to prevent this.
Regarding the Catch-22 (WHMCS 2FA / 502 Bad Gateway Bug):
The admin has openly acknowledged the issue. It seems it was an unknown bug on their end that no one had reported recently. He is looking into it today and will reach out to me via DM to resolve my login access.
@Chunkserve - If the 2FA endpoint bug takes too much time or is too complex to fix immediately, please feel free to just completely disable/remove the 2FA from my account for now. I just need to regain access to the billing client area.
Thanks again to @Chunkserve for resolving the node access today, and thanks to the LET community for the space. I will wait for the DM!
Same for my vps and I restarted it myself and it was working again
Amazing fancy taking such proactive measures on your own without referring LET support
UPDATE 2: The 502 Bad Gateway is NOT just a 2FA bug, it's a WHMCS Session/Cloudflare issue with Dynamic IPs!
Hello @Chunkserve, I have some very valuable troubleshooting data for you regarding the 502 Bad Gateway issue on your billing portal.
I managed to briefly get in and completely disabled my 2FA. However, when I logged out and tried to log back in (without 2FA), I hit the exact same 502 Bad Gateway error, this time stuck at /login.
The Breakthrough:
I realized this might be tied to how your WHMCS or Cloudflare handles clients with dynamic/rotating IPs. My home ISP and my 4G mobile data frequently rotate IPs (CGNAT). To test this theory, I routed my connection through a Static SOCKS Proxy to ensure my IP remained completely static.
The result? The login, logout, and navigation processes were buttery smooth. Zero 502 errors!
It appears your WHMCS session validation is breaking (or Cloudflare is throwing a 502) whenever a client's IP changes slightly during the login handshake/session. You might want to check your WHMCS settings (e.g., turning off "Disable Session IP Check" in General Settings -> Security) or review your Cloudflare caching/WAF rules for the billing portal.
Hopefully, this helps you fix the root cause for all users with dynamic IPs!
That's not overly surprising when they can't login, is it?
[URGENT UPDATE 3 - July 4th]: HOST NODE IS COMPLETELY DEAD
Hello @Chunkserve,
I am forced to update this thread again because my production VPS (141.11.62.177) has suddenly died completely today, and the situation is much worse than the previous incident. And it's been 8 hours since my VPS has been down and I can't access it at all, I can't even reboot it.
I am posting this publicly instead of opening a standard support ticket because of your extremely poor support track record. In my previous ticket (#QCA-095917, which was just a simple PTR record request), it took you 12 days to provide an initial reply. When I replied back because the issue wasn't fully resolved, my follow-up has now been completely ignored for over a week (8+ days). Furthermore, during our last interaction in this very thread, you explicitly promised: "I'll DM you regarding the 2FA later." I have never received that DM, and the WHMCS 2FA bug remains completely unresolved to this day.
Given this track record of extreme delays and broken promises, I simply refuse to wait silently in a ticket queue while my critical production infrastructure is completely down.
Here are the absolute facts of the current outage:
https://virt1-ams.chunkserve.com:4083/andhttps://191.96.11.2:4083/are throwing anERR_EMPTY_RESPONSE.This clearly indicates that your AMS host node (
virt1-ams) has crashed, locked up, or lost network connectivity entirely.My production environment is currently bleeding traffic. I urgently request that you check the AMS node, reboot the hypervisor if necessary, and bring it back online immediately.
Please fix this node right away!
You're using a novice LET host for production?
He knows CF doesn't want his stinky connection. Yet doesn't try to use proxy to login. Production, yet no disaster recovery
8 Hours Down and you still call it production ? You need to move on with some reputed and reliable provider for your production grade work else don't call it that .
@forest @chatbox
In the engineering and publishing world, "Production" simply means the environment serving live, active user traffic. Period. I have been with this exact VPS for almost 1.5 years (currently mid-way through my second renewal) with absolute zero significant downtime until this past week's incidents. When a host handles live traffic flawlessly for that long, it earns its place in a production workflow. Not every project requires an overpriced enterprise setup right out of the gate, especially when historical uptime has been solid.
@rpqu
Please read point #3 of my post again carefully. I did use a proxy to log in. That is exactly how I was able to access the WHMCS dashboard, bypass the 502 Cloudflare block, and take the screenshots of the API connection errors I attached above.
I am here to get a dead hypervisor fixed by the provider, not to debate basic semantics or defend my operational architecture to the peanut gallery.
@Chunkserve Any updates on the AMS node?
I know you are active here (LET) and reading this thread.
with the 2nd incident, could be a 3rd and 4th on the way,
i suggest you shift your 'production' to a more reputable provider.
yes 1st year was good, then now bad but this is LET, those here long enough has seen the highs and the lows of a provider... Virmach was a good example
if you value your website your traffic, start to think of migration plan now,
this server, if it gets fixed can repupose as backup for your website.
Just giving you a heads up, I'm on about hour 20 of an outage (and have had weekly outages for the past 2 months). @Chunkserve has barely acknowledged these issues, so they probably won't acknowledge yours either.
Thankfully I didn't lose anything important and I wasn't putting anything important on this server. I'm keeping important stuff on my home server.
Hey,
Most of the attack is mitigated and servers should be working. Still replying to some tickets about it. Could you let me know the ticket ID?
Most likely, some prefixes may have too strict filters (or leaked traffic) - that's whyat I'd check.
And losing millions ofcourse.
I'm well aware of what production means, and I stand by what I said.
Hi!
I just wanted to post a quick update - most of the attacks stopped, and rest of the traffic is being filtered.
I acknowledged multiple support tickets regarding issues with initiating HTTP/HTTPS connection from the VPS - this should be working as well (it was caused by too strict filtering rule). Other complain about issues with VPS management panel - currently working on it, but on most nodes should be fine.
If anything is not working for anyone, feel free to contact us!