Layer7 FRA1 Down, 5+ Hours with No Support Response

akinfem

Just a heads-up for anyone else hosted there: my Layer7 VPS (193.24.210.*) in Frankfurt (FRA1) has been down for roughly 5 hours, significantly disrupting my operations.

I have opened multiple tickets about this issue, but the support team has been completely unresponsive so far. Has anyone managed to get in touch with them today, or know if there is ongoing maintenance?

JohnFilch123

@layer7

Mumbly

Multiple tickets? Within a 5-hour time frame? Oh, boy...

MaxTakeba

I would have been very patient... This is not an outage... I can ping multiple hosts in that subnet... Making multiple tickets whilst it's in the middle of the night to an EU provider is impatient.

@angstrom can this be moved? This is not a true outage... It's an isolated case.

besthost1

Same here. I'm having a hard time even accessing their client portal to submit a ticket.

akinfem

@MaxTakeba said:
I would have been very patient... This is not an outage... I can ping multiple hosts in that subnet... Making multiple tickets whilst it's in the middle of the night to an EU provider is impatient.

@angstrom can this be moved? This is not a true outage... It's an isolated case.

8am in the EU is not middle of the night.
I live in the EU.

MaxTakeba

@besthost1 said:
Same here. I'm having a hard time even accessing their client portal to submit a ticket.

I accessed it just fine.. need similarities here... Need way more information.

MaxTakeba

@MaxTakeba said:

@besthost1 said:
Same here. I'm having a hard time even accessing their client portal to submit a ticket.

I accessed it just fine.. need similarities here... Need way more information.

Let me just quote you.

has been down for roughly 5 hours, significantly disrupting my operations.
I have opened multiple tickets about this issue, but the support

When did you first open the ticket?

maxxxxx

@akinfem said:
Just a heads-up for anyone else hosted there: my Layer7 VPS (193.24.210.*) in Frankfurt (FRA1) has been down for roughly 5 hours, significantly disrupting my operations.

It's time for you to reconsider the architecture of your operations.

@akinfem said:
I have opened multiple tickets about this issue, but the support team has been completely unresponsive so far. Has anyone managed to get in touch with them today, or know if there is ongoing maintenance?

Don't do that. The way some of those ticketing systems work is that with multiple tickets you'll be put at the end of the queue every time and you'll just wait longer.

forest

root@forest-maxko-1-za:~# ping -c 3 193.24.210.1
PING 193.24.210.1 (193.24.210.1) 56(84) bytes of data.
64 bytes from 193.24.210.1: icmp_seq=1 ttl=52 time=177 ms
64 bytes from 193.24.210.1: icmp_seq=2 ttl=52 time=177 ms
64 bytes from 193.24.210.1: icmp_seq=3 ttl=52 time=177 ms

--- 193.24.210.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 176.773/176.915/177.047/0.112 ms

Looks like it's a "you" problem, OP.

MaxTakeba

@forest said:

root@forest-maxko-1-za:~# ping -c 3 193.24.210.1
PING 193.24.210.1 (193.24.210.1) 56(84) bytes of data.
64 bytes from 193.24.210.1: icmp_seq=1 ttl=52 time=177 ms
64 bytes from 193.24.210.1: icmp_seq=2 ttl=52 time=177 ms
64 bytes from 193.24.210.1: icmp_seq=3 ttl=52 time=177 ms

--- 193.24.210.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 176.773/176.915/177.047/0.112 ms

Looks like it's a "you" problem, OP.

No longer a "you" problem.

It's now taking forever to load their WHCMS. It was very very fast before... Now quite slow.

It would be very helpful to get more information.

Edit 3, back to being very fast...

besthost1

@MaxTakeba said:

@besthost1 said:
Same here. I'm having a hard time even accessing their client portal to submit a ticket.

I accessed it just fine.. need similarities here... Need way more information.

To give you more info: I'm on the same FRA1 node as the OP, and the downtime timing is roughly the same. The portal was extremely slow for me, and when I finally squeezed in, my VPS status now says "Instance Not Found".

akinfem

@besthost1 said:

@MaxTakeba said:

@besthost1 said:
Same here. I'm having a hard time even accessing their client portal to submit a ticket.

I accessed it just fine.. need similarities here... Need way more information.

To give you more info: I'm on the same FRA1 node as the OP, and the downtime timing is roughly the same. The portal was extremely slow for me, and when I finally squeezed in, my VPS status now says "Instance Not Found".

Same here. Instance Not Found.
My other instance located in Paris is working fine though.

This is what I have been randomly getting from their WHCMS.

Service Temporarily Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Web Server at layer7.net

Hopefully they will fix this problem soon.

MaxTakeba

@besthost1 said:

@MaxTakeba said:

@besthost1 said:
Same here. I'm having a hard time even accessing their client portal to submit a ticket.

I accessed it just fine.. need similarities here... Need way more information.

To give you more info: I'm on the same FRA1 node as the OP, and the downtime timing is roughly the same. The portal was extremely slow for me, and when I finally squeezed in, my VPS status now says "Instance Not Found".

That information is helpful. I did validate the client panel issue, however it's inconsistent.

besthost1

Update: My server just came back online, and the client portal is running smoothly now.

fredo1664

@besthost1 said:
Update: My server just came back online, and the client portal is running smoothly now.

and all this without opening multiple tickets!

akinfem

Services back online.
Tech support reported "bladechassis hardware" issue.

Thank you all for your contributions.

zed

@MaxTakeba said: This is not a true outage... It's an isolated case.

oops

zejjnt

EDIT: Opened thread before two last posts, shit happens

traceroute to 193.24.210.1 (193.24.210.1), 64 hops max
1 88.214.24.1 0.269ms 0.250ms 0.303ms
2 193.178.0.168 1.615ms 1.439ms 1.324ms
3 193.178.0.56 2.039ms 2.069ms 2.124ms
4 87.245.239.148 0.579ms 0.722ms 0.636ms
5 87.245.232.78 9.482ms 9.497ms 9.488ms
6 193.24.210.1 10.033ms 9.934ms 9.982ms

From L7 France, most seem fine.

for x in $(seq 1 255); do ping -s1 -n -c1 -w1 193.24.210.$x | grep -v "0% packet loss" | grep "bytes from"; done
9 bytes from 193.24.210.1: icmp_seq=1 ttl=59
9 bytes from 193.24.210.2: icmp_seq=1 ttl=58
9 bytes from 193.24.210.4: icmp_seq=1 ttl=58
9 bytes from 193.24.210.5: icmp_seq=1 ttl=58
9 bytes from 193.24.210.6: icmp_seq=1 ttl=58
9 bytes from 193.24.210.9: icmp_seq=1 ttl=58
9 bytes from 193.24.210.10: icmp_seq=1 ttl=58
9 bytes from 193.24.210.12: icmp_seq=1 ttl=58
9 bytes from 193.24.210.14: icmp_seq=1 ttl=58
9 bytes from 193.24.210.17: icmp_seq=1 ttl=58
9 bytes from 193.24.210.18: icmp_seq=1 ttl=58
9 bytes from 193.24.210.19: icmp_seq=1 ttl=58
9 bytes from 193.24.210.21: icmp_seq=1 ttl=58
9 bytes from 193.24.210.25: icmp_seq=1 ttl=58
9 bytes from 193.24.210.29: icmp_seq=1 ttl=58
9 bytes from 193.24.210.30: icmp_seq=1 ttl=58
9 bytes from 193.24.210.31: icmp_seq=1 ttl=58
9 bytes from 193.24.210.32: icmp_seq=1 ttl=58
9 bytes from 193.24.210.33: icmp_seq=1 ttl=58
9 bytes from 193.24.210.35: icmp_seq=1 ttl=58
9 bytes from 193.24.210.36: icmp_seq=1 ttl=58
9 bytes from 193.24.210.38: icmp_seq=1 ttl=57
9 bytes from 193.24.210.41: icmp_seq=1 ttl=58
9 bytes from 193.24.210.42: icmp_seq=1 ttl=58
9 bytes from 193.24.210.43: icmp_seq=1 ttl=58
9 bytes from 193.24.210.49: icmp_seq=1 ttl=58
9 bytes from 193.24.210.50: icmp_seq=1 ttl=58
9 bytes from 193.24.210.53: icmp_seq=1 ttl=58
9 bytes from 193.24.210.54: icmp_seq=1 ttl=58
9 bytes from 193.24.210.55: icmp_seq=1 ttl=58
9 bytes from 193.24.210.56: icmp_seq=1 ttl=58
9 bytes from 193.24.210.57: icmp_seq=1 ttl=58
9 bytes from 193.24.210.58: icmp_seq=1 ttl=58
9 bytes from 193.24.210.59: icmp_seq=1 ttl=58
9 bytes from 193.24.210.60: icmp_seq=1 ttl=58
9 bytes from 193.24.210.61: icmp_seq=1 ttl=122
9 bytes from 193.24.210.62: icmp_seq=1 ttl=58
9 bytes from 193.24.210.63: icmp_seq=1 ttl=122
9 bytes from 193.24.210.64: icmp_seq=1 ttl=122
9 bytes from 193.24.210.65: icmp_seq=1 ttl=122
9 bytes from 193.24.210.66: icmp_seq=1 ttl=122
9 bytes from 193.24.210.67: icmp_seq=1 ttl=58
9 bytes from 193.24.210.68: icmp_seq=1 ttl=58
9 bytes from 193.24.210.69: icmp_seq=1 ttl=58
9 bytes from 193.24.210.70: icmp_seq=1 ttl=58
9 bytes from 193.24.210.71: icmp_seq=1 ttl=58
9 bytes from 193.24.210.72: icmp_seq=1 ttl=58
9 bytes from 193.24.210.73: icmp_seq=1 ttl=58
9 bytes from 193.24.210.75: icmp_seq=1 ttl=122
9 bytes from 193.24.210.76: icmp_seq=1 ttl=122
9 bytes from 193.24.210.77: icmp_seq=1 ttl=58
9 bytes from 193.24.210.78: icmp_seq=1 ttl=58
9 bytes from 193.24.210.79: icmp_seq=1 ttl=58
9 bytes from 193.24.210.81: icmp_seq=1 ttl=58
9 bytes from 193.24.210.82: icmp_seq=1 ttl=58
9 bytes from 193.24.210.84: icmp_seq=1 ttl=58
9 bytes from 193.24.210.88: icmp_seq=1 ttl=58
9 bytes from 193.24.210.89: icmp_seq=1 ttl=122
9 bytes from 193.24.210.93: icmp_seq=1 ttl=58
9 bytes from 193.24.210.96: icmp_seq=1 ttl=58
9 bytes from 193.24.210.98: icmp_seq=1 ttl=58
9 bytes from 193.24.210.99: icmp_seq=1 ttl=58
9 bytes from 193.24.210.100: icmp_seq=1 ttl=58
9 bytes from 193.24.210.101: icmp_seq=1 ttl=58
9 bytes from 193.24.210.103: icmp_seq=1 ttl=58
9 bytes from 193.24.210.108: icmp_seq=1 ttl=58
9 bytes from 193.24.210.109: icmp_seq=1 ttl=58
9 bytes from 193.24.210.111: icmp_seq=1 ttl=58
9 bytes from 193.24.210.112: icmp_seq=1 ttl=122
9 bytes from 193.24.210.114: icmp_seq=1 ttl=58
9 bytes from 193.24.210.115: icmp_seq=1 ttl=58
9 bytes from 193.24.210.116: icmp_seq=1 ttl=58
9 bytes from 193.24.210.117: icmp_seq=1 ttl=58
9 bytes from 193.24.210.118: icmp_seq=1 ttl=58
9 bytes from 193.24.210.120: icmp_seq=1 ttl=58
9 bytes from 193.24.210.121: icmp_seq=1 ttl=58
9 bytes from 193.24.210.124: icmp_seq=1 ttl=58
9 bytes from 193.24.210.125: icmp_seq=1 ttl=58
9 bytes from 193.24.210.127: icmp_seq=1 ttl=58
9 bytes from 193.24.210.129: icmp_seq=1 ttl=58
9 bytes from 193.24.210.130: icmp_seq=1 ttl=58
9 bytes from 193.24.210.132: icmp_seq=1 ttl=122
9 bytes from 193.24.210.134: icmp_seq=1 ttl=58
9 bytes from 193.24.210.135: icmp_seq=1 ttl=58
9 bytes from 193.24.210.136: icmp_seq=1 ttl=58
9 bytes from 193.24.210.137: icmp_seq=1 ttl=58
9 bytes from 193.24.210.138: icmp_seq=1 ttl=58
9 bytes from 193.24.210.140: icmp_seq=1 ttl=58
9 bytes from 193.24.210.141: icmp_seq=1 ttl=122
9 bytes from 193.24.210.143: icmp_seq=1 ttl=58
9 bytes from 193.24.210.144: icmp_seq=1 ttl=58
9 bytes from 193.24.210.148: icmp_seq=1 ttl=58
9 bytes from 193.24.210.149: icmp_seq=1 ttl=58
9 bytes from 193.24.210.150: icmp_seq=1 ttl=58
9 bytes from 193.24.210.151: icmp_seq=1 ttl=58
9 bytes from 193.24.210.155: icmp_seq=1 ttl=58
9 bytes from 193.24.210.158: icmp_seq=1 ttl=58
9 bytes from 193.24.210.159: icmp_seq=1 ttl=58
9 bytes from 193.24.210.160: icmp_seq=1 ttl=58
9 bytes from 193.24.210.161: icmp_seq=1 ttl=58
9 bytes from 193.24.210.163: icmp_seq=1 ttl=122
9 bytes from 193.24.210.164: icmp_seq=1 ttl=58
9 bytes from 193.24.210.166: icmp_seq=1 ttl=58
9 bytes from 193.24.210.167: icmp_seq=1 ttl=58
9 bytes from 193.24.210.168: icmp_seq=1 ttl=58
9 bytes from 193.24.210.170: icmp_seq=1 ttl=58
9 bytes from 193.24.210.171: icmp_seq=1 ttl=58
9 bytes from 193.24.210.173: icmp_seq=1 ttl=58
9 bytes from 193.24.210.175: icmp_seq=1 ttl=58
9 bytes from 193.24.210.177: icmp_seq=1 ttl=58
9 bytes from 193.24.210.178: icmp_seq=1 ttl=58
9 bytes from 193.24.210.180: icmp_seq=1 ttl=58
9 bytes from 193.24.210.181: icmp_seq=1 ttl=58
9 bytes from 193.24.210.183: icmp_seq=1 ttl=58
9 bytes from 193.24.210.185: icmp_seq=1 ttl=58
9 bytes from 193.24.210.186: icmp_seq=1 ttl=58
9 bytes from 193.24.210.188: icmp_seq=1 ttl=58
9 bytes from 193.24.210.189: icmp_seq=1 ttl=58
9 bytes from 193.24.210.192: icmp_seq=1 ttl=58
9 bytes from 193.24.210.193: icmp_seq=1 ttl=58
9 bytes from 193.24.210.194: icmp_seq=1 ttl=58
9 bytes from 193.24.210.195: icmp_seq=1 ttl=58
9 bytes from 193.24.210.198: icmp_seq=1 ttl=58
9 bytes from 193.24.210.199: icmp_seq=1 ttl=58
9 bytes from 193.24.210.200: icmp_seq=1 ttl=58
9 bytes from 193.24.210.201: icmp_seq=1 ttl=58
9 bytes from 193.24.210.202: icmp_seq=1 ttl=58
9 bytes from 193.24.210.203: icmp_seq=1 ttl=58
9 bytes from 193.24.210.205: icmp_seq=1 ttl=58
9 bytes from 193.24.210.207: icmp_seq=1 ttl=58
9 bytes from 193.24.210.208: icmp_seq=1 ttl=58
9 bytes from 193.24.210.210: icmp_seq=1 ttl=58
9 bytes from 193.24.210.213: icmp_seq=1 ttl=58
9 bytes from 193.24.210.215: icmp_seq=1 ttl=58
9 bytes from 193.24.210.217: icmp_seq=1 ttl=58
9 bytes from 193.24.210.220: icmp_seq=1 ttl=58
9 bytes from 193.24.210.221: icmp_seq=1 ttl=58
9 bytes from 193.24.210.222: icmp_seq=1 ttl=58
9 bytes from 193.24.210.223: icmp_seq=1 ttl=58
9 bytes from 193.24.210.224: icmp_seq=1 ttl=58
9 bytes from 193.24.210.226: icmp_seq=1 ttl=58
9 bytes from 193.24.210.227: icmp_seq=1 ttl=58
9 bytes from 193.24.210.228: icmp_seq=1 ttl=122
9 bytes from 193.24.210.229: icmp_seq=1 ttl=122
9 bytes from 193.24.210.231: icmp_seq=1 ttl=58
9 bytes from 193.24.210.232: icmp_seq=1 ttl=58
9 bytes from 193.24.210.233: icmp_seq=1 ttl=58
9 bytes from 193.24.210.237: icmp_seq=1 ttl=58
9 bytes from 193.24.210.238: icmp_seq=1 ttl=58
9 bytes from 193.24.210.239: icmp_seq=1 ttl=58
9 bytes from 193.24.210.240: icmp_seq=1 ttl=58
9 bytes from 193.24.210.241: icmp_seq=1 ttl=58
9 bytes from 193.24.210.243: icmp_seq=1 ttl=58
9 bytes from 193.24.210.244: icmp_seq=1 ttl=58
9 bytes from 193.24.210.245: icmp_seq=1 ttl=122
9 bytes from 193.24.210.246: icmp_seq=1 ttl=58
9 bytes from 193.24.210.249: icmp_seq=1 ttl=58
9 bytes from 193.24.210.251: icmp_seq=1 ttl=58
9 bytes from 193.24.210.252: icmp_seq=1 ttl=58
9 bytes from 193.24.210.253: icmp_seq=1 ttl=58
9 bytes from 193.24.210.254: icmp_seq=1 ttl=58

layer7

Hi,

yes, just like written in the network issue section and answered via tickets, there was a technical issue with the HP bladehardware leading to a situation where the cluster of our old Mid CPU virtual servers ( the Intel line that is not offered since ~ 1 year now ) shut down and did not like to start.

After spending ~ 1h different approaches it was decided that some poor soul needs to go to the datacenter and check / fix things. Took until ~ 11:30 AM CET that all servers were back online. Sorry for the hassle!

And just to clarify it: Of course not the whole FRA1 location was down. It was just an old cluster. Also not the whole network ( 193.24.210.X ) was down, as there were no network issue. Just coincidence that there might be some more from 193.24.210.0/24 used there.

All the other infrastructure was unharmed.

And yes, if the clientarea can not reach the cluster, the performance will suffer a lot. The beautiful WHMCS engine does its rest to make sure everyone will love using the system in that state. We are currently working to replace it.

And yes, opening multiple tickets is usually not helpful for everyone as it will just slow down the ticket processing.

But of course we are thankful for any feedback, the good and the bad one

---

Just like already mentioned in the ticket replies ( i think ) the migration of this old server to a new platform will be processed now with some more priority. It was anyway planned to remove this cluster ( and offer its RAM hrhr ) and move the virtual servers to the current Intel SP Gold platform and preparations for that were done. Of course murphy is always on the hunt and made sure its exploding before we moved the servers, and not after...

If there are questions, feel free to ask.

Otherwise, please accept our apology for this unplanned outage! Reached roughly 600 days uptime with that ;-;

MaxTakeba

@zed said:

@MaxTakeba said: This is not a true outage... It's an isolated case.

oops

Famous last words.

bjo

@layer7 said:
Hi,

yes, just like written in the network issue section and answered via tickets, there was a technical issue with the HP bladehardware leading to a situation where the cluster of our old Mid CPU virtual servers ( the Intel line that is not offered since ~ 1 year now ) shut down and did not like to start.

My box in fra ran fine, but do you have any public status page? Or are maintenances etc announced on https://login.layer7.net/serverstatus.php ?

layer7

@bjo said:
My box in fra ran fine, but do you have any public status page? Or are maintenances etc announced on https://login.layer7.net/serverstatus.php ?

Hi,

we use the network issue announcement for this that is offered in WHMCS.

But thats far away from perfect or how we want it to be.

The plan for 2026 is to take care of all this software infrastructure issues and finally removing WHCMS which cause performance wise and from the point of features too many issues.

We want to replace it in Q2 2026 and this should improve all kind of things -- including automatic customer communication in case of technical issue. Because thats for sure one of our weak points. Usually things run, thats why it had never priority to improve that. But if it happens, its of course annoying for the customers. So that will be improved.

Thank you for the feedback(s) so far!

Umair

@layer7 said:

The plan for 2026 is to take care of all this software infrastructure issues and finally removing WHCMS which cause performance wise and from the point of features too many issues.

We want to replace it in Q2 2026 and this should improve all kind of things -- including automatic customer communication in case of technical issue. Because thats for sure one of our weak points. Usually things run, thats why it had never priority to improve that. But if it happens, its of course annoying for the customers. So that will be improved.

Which billing/support system you are moving to? Something made in house or available as alternative to WHMCS ?

layer7

@Umair said:
Which billing/support system you are moving to? Something made in house or available as alternative to WHMCS ?

Hi,

will be something made in house, but its also an idea to make it available for purchase/rent.

Since we have this year more free time as we anyway dont buy any hardware for now ( hopefully ) we have some major progress with the software development.

So basically there will be available billing, VPS management system with API ( currently talking with proxmox, but later also with public cloud systems ) and a VPS provisioning module that will work with the billing and WHMCS. As soon as this all is released i guess we will also make something for other less broadly used billing systems like blesta and what ever else interesting is out there.

mans_xd

i hope oliver move from whmcs layer7 response is bad , i can see he already working in the main website layer7 but it's new css and still no policy in main page only when order and alot of common information not found , i already a customer that doesn't bother me at all just a normal opinine , i know layer7 i one person for all work , but consider in next 6 month make a an updates and for information contoral panel

services are fine even oliver not strict at all and flexable but dman that feel when you want to tell someone to make more effort in that nice things