New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
DelimiterVPS does not restart server after maintenance
A heads up to those who want to host with Delimiter.
Server was down this morning, ILO reported STANDBY. I raised a ticket and got "you should check our network issue page, there was scheduled maintenance posted 10 days ago".
For one, I don't have time for daily browsing of all my operators websites, if there is an event like this, I am expecting email alert.
Secondly, when you are done with mainenance, START MY BLOODY SERVER BACK UP !
Comments
Yep I did the same thing, reported a ticket after about ~30 minutes of downtime. Told me to go check the page and was posted 10 days before.
I told him an email or something would have been nice to know, no one checks the client area constantly.
My server was rebooted just fine -- I'll bet yours would have been too, but if there was a large number of affected servers then it might have taken awhile to get to it.
And ask about their incident list -- they subscribed me so I'll get notifications in the future. Kind of strange that it's opt-in instead of opt-out, but at least the option to receive notifications is there if you ask about it.
For sure, I told him it would have been nice to receive an email about it and he added me to the list.
My server went down on 2014/6/14 23:10
Learned about this on 2014/6/15 10:00
Incident was closed at 00:34, 9.5hrs before I learned about my server being down.
Reply to my ticket had no hint of apology, tech blamed me for not checking their network issues page (which actually promises the servers being restarted at the end of the maintenance) and closed the ticket, ignoring my subsequent comments about email alerts. I was not offered to be signed-up to the list.
I would expect something like this from GreenValueHost but not Delimiter, now my impression of Delimiter is also 14yr old kid with minimum professional integrity.
Good to know, time to look around.
Delimiter is a well known company, they're a smaller brand of what's bigger Yomura Holdings, Which has alot of datacenters
@loyd - what was the ticket #, I'd like to read what went on.
Obviously size and fame don't automatically mean professional conduct these days.
I cancelled my Delimiter X5150 earlier this month. The server itself was pretty good, once I got past the setup nightmare, but their rude and unresponsive support is a deal-breaker.
@loyd - I've asked for a copy of the ticket. Lets see what happened
Just sent in a ticket...
My server is still down and I can't attempt to power cycle it or anything as for some reason it's listed as suspended in WHMCS... Bill's paid up and it's simply used as a staging server for development of our software so no TOS issues (and I haven't received any emails indicating that).
@lucashayes - Please PM me the IP of your server. When was it suspended and do you know why it was suspended?
check your pm
@loyd - I know already received, as I said I am waiting for a copy of the ticket now from Delimiter
It has apparently been taken care of. It got suspended a few weeks ago because I forgot to pay the bill but apparently when I paid it they unsuspended the service but didn't mark it as unsuspended in WHMCS. Problem being that the power management button goes away when it's marked as suspended so I couldn't reboot it to get it back online.
@loyd, I've received the ticket and the maintenance notes showing that the results of a visual inspection of your blade showed it was already powered up and therefore did not require resetting. The engineers on-site walked the site three times over the course of the evening to make sure that all servers were powered, however it was noted that a few blades were reported down, despite showing a green health and green power light illuminated on the front of them.
It seems that in some cases (7 blades) the power management system on the blade had simply crashed and left the indicators in their last state ie powered and running error free. The engineers were unaware there was a problem with these servers until tickets were opened and they were requested to check them.
The blade management system showed that your blade was powered the whole time as well, so again there was no way for them to tell that there was an error or issue on your server.
Its impractical in this instances to manually connect a monitor/keyboard to all 1440 blades server and confirm that they have loaded to their OS. We used to monitor customer servers with ICMP but almost all customers disable ICMP response on their servers these days.
Unfortunately in this instance there was very little that could be done until this issue was reported. The physical indicators showed server was up, the management system confirmed the server was up. Their assumption therefore was the server was up.
@lucashayes - be aware that the Yomura companies don't use WHMCS for their real billing/support/status management, its just used a convenient front-end. All the suspension activity is handled on their internal billing system, which means sometimes the frontend (WHMCS) gets out of sync with the backend. So if you get suspended for non-payment and then unsuspended, make sure that WHMCS shows your server as unsuspended. If not just open a support ticket and have them do a manual update for you.
The issue did not require support since I was able to start my server from ILO.
Understandable, mistakes happens. Understandable, Delimiter does not use ILO API to verify actual server status. I am an engineer who knows things are not always ideal nor they go as planned. Simple answer "your blade's indicator incorrectly reported server as being up" to my ticket this morning would make a day and night difference, instead I was lead to believe it's my job to watch company website in order to receive service and my frustration exploded.
I appreciate your work here and thank you for taking the time to dig through and write-up an explanation. As I understand you are not Delimiter staffer and don't represent them here, so Delimiter did not take ownership of their mistake. Noted. Case closed. Thanks.
@loyd - Its actually our fault not theirs. Delimiter is a sales/support business thats sells Yomura's services. The datacentre your server is in is ours and staffed by our staff.
We have two ways to determine blade status - the obvious the LED on the front and the chassis management system which gives us live data about the status of the servers: http://imgur.com/KLBAprM
The servers were checked physically and virtually using the tool. More than likely in your case the watchdog on the ILO rebooted it after about an hour or two and so the ILO reported status of the blade reflected the fact it was down.
The wording wasn't good from their side, it was somewhat terse. I've already highlighted that point internally so it can be handled better the next time it happens.
Something you dont read often around here
Kudos.
Only suggestion as to this thread's topic, make email notifications for such announcements default.
@neroux - being advocating compulsory maintenance emails for a long time. The problem that kept happening is that customers received the notices, it got tagged as spam and then their invoice/ticket emails went into the spam bin.
I am going to bring it up again on Monday. Delimiter also posts to Twitter with their announcements both in realtime and for future maintenance events.
Just an idea, maybe a different email address/server for billing related emails?
@MarkTurner said: The wording wasn't good from their side, it was somewhat terse. I've already highlighted that point internally so it can be handled better the next time it happens.
Terse is a good way to describe my ticket responses as well. But to be honest I'm totally cool with terse if that's what allows them to respond so quickly. My tickets have been responded to in 3, 2, 3 and 9 minutes...for such a cheap service, this is awesome. And this is after marking them as low priority because...they were low priority!
Mine went into standby. It looks like they updated the iLO firmware on mine and something went wrong. Was fixed very quickly though! Props to Delimiter!
@Ree - There is terse and terse. Loyd's ticket was handled brashly.
All of Yomura's support team are told to read the customer's enquiry, if it can be dealt with inside 5 minutes then deal with it and respond that its done. If it will take more than 5 minutes or requires escalation then notify the customer that its in hand and then follow the escalation. They are told to keep responses clear and straight to the point, avoid waffling, be explicit.
Most of the time, I see their responses, no matter what the complexity of the issue as being words to the effect that 'its done' or 'this has been resolved'. If you need more data then just reply back to the ticket and ask them for an explanation.
I had the same problem with one of my three servers during the maintenance. Tech response was good and got me back up and running very shortly (~15 minutes) after I contacted them. Kudos.
There was a similar incident last night, this time email notice was sent and server was up and running in the morning. Is it early to state this thread was constructive and the service quality has improved? DelimiterVPS, thumbs up!
Yeah email was sent because 'As some customers did not see the notice last weekend that was posted in the Network Issues', more like no one seen it. But no issues at all last night, not even a bit of downtime.
I do not think so I am afraid. Only three days ago there was a failure of a network switch which made the iscsi mounted storage once more switch to read-only. No notification was sent in this case.
@Neroux - out of 400+ Atom servers, one rack was affected (44 Atoms). The switch for that rack failed. It was replaced within 30 minutes and servers rebooted.
@loyd - the irony is that we had four spam reports from that where live customers reported the notifications as spam.
@Catalystium - there are definitely quite a number of people who saw it because there was a flurry of tickets and PM's to me on here about which rack their server was located.