New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
There are three disks on this node, 2TB each. The first two are filled with customers. The last one has about 200-300GB of customer data and that disk is having problems, causing what you describe.
So to be very clear, "many VPS" is about 7% of them.
These are in queue for recreation and then of course we will also allow people to request their data if it is recovered. There are others waiting as well, and I apologize for the wait, but creating tickets at this point will not help. We have been aware of the issue and any updates will be via network status updates, email, and/or ticket. I do believe this was communicated to some degree on the offer thread as well as on a network status page before you created your thread. We need to schedule a maintenance window right now to determine if the cause is a kernel bug or if the disk failed, because we do not want to reboot the other 93% multiple times without notice, it will only create further grievances at an exponential level right now. Every time a node reboots, even for scheduled maintenance, we seem to get 20+ "work orders" which further puts a stress on everything.
Right now it is best to only have one ticket open at once. We are getting many duplicate tickets which is making it more difficult to help everyone, with some people creating as many as 47 tickets per account.
I understand in this case it did not make sense for the tickets to be merged but we unfortunately will have some false positives with the way we are quickly merging tickets via script.
Yes.
Thank you very much for coming forward to explain the current problems. Therefore, I can regard these things as not happening. But I have another question. We can't use the wasted time at present due to your problems. Will we make compensation if we recover in the future? There are five days in a year that cannot be used. Will you compensate for these five days?
A specific task at a certain time and place.
My earliest experience with "work orders" was when the Sears dishwasher repairman came to our house with a work order to fix it.
Typical work orders are for hooking up new services, like power or phone in a house.
A "work order" isn't used in a support communication context between customer and provider (hence why it sounds wrong to English people), it would be used internally between the provider and the worker (ie. Hosting provider requests the datacenter technician swap out hard drive, for example).
Tl;dr stop fucking saying "work orders" and say "support ticket".
LET just got freaky.
Low End Testicles
I'll admin, they do hang somewhat as I'm no longer a spring chicken.
Deeper than the stick so to speak?
Sadly, yes.
That's okay. I'm sure it dosen't bother you too much. We'll all be there one lovely day.
It's ok, but can occasionally cause discomfort when sitting or sleeping.
On the flip side, they really fly now when you're doing the nasty, for some reason ladies seem to enjoy having a pair of meaty bollocks slapping against them.
工單 is more accurate than 工作指示
Yeah, I can imagine. Mine move a lot after the act.
Most of MJJs is using machine translation, and they just copied and pasted,they don't care.
Yes. Open a ticket asking for credit for compensation and credit will be given after other priority issues are cleared.
For example, if you have a VPS with 768MB RAM, you will be given 0.2 USD (∽1.3CNY) credit.
30[USD/2yr] / 2[yr] / 365[day] * 5[day] = 0.2
Came for the drama, staying for @Nekki and @emgh's convo.
On their own?
Yeah, crazy.
That’s quite the party trick. Do they just sort of ‘twitch’ or actually move from one location to another? Do they move more or less depending on how hard you just nutted?
Twitch but stay in their new position, so I guess they move a little. More when I breath, less when I hold my breath. I don't know if hardness has anything to do with it but way more when one of the opposite sex has been involved compared to when my hand has been involved.
I opened only one ticket and the ticket was closed already. Should I reopen it again in case the unbootable vps would be forgotten？
Oh definitely, as entertaining as a hand-shandy is, it will never compare to the real thing, there would certainly be a significant difference in testicle hardness.
No RAID? I realize RAID is not back-up but I wouldn't use such a setup for a production something.
I have two longer posts about this explaining it, but you are correct, no RAID on these. All the disks that have been failing have been one brand and they all have problems when they hit 200-300GB usage. I'm trying to get in contact with the manufacturer but we already phased almost all of these out and not adding any of them to new servers. We probably have two of them out in the wild right now.
We are still in the process of setting this up, but all nodes will have another drive that they'll constantly be using for backups so in the future the process will be a lot smoother. And later down the line as we get closer to running into failure territory we'll increase the frequency with another and have another set of backups as well on independent storage nodes.
We'll of course also still consider RAID moving forward, it's just been difficult to find a good option for NVMe SSDs. Super short version of my previous explanations: [1] software RAID is pretty bad in multiple ways, [2] AMD RAID for NVMe isn't really supported anymore on Linux, [3] hardware RAID is pretty bad unless you go with an insanely expensive option and even then performance is degraded to the point where using NVMe SSDs is somewhat questionable, [4] other hardware RAID isn't really hardware RAID, it's just software RAID but offloading some of the work away from the CPU. And I'll add this one here as well: over the years, using hardware RAID, we've noticed some catastrophic cases caused by the controller itself, with datacenter hands also being prone to making mistakes. This will still happen but it's less likely we'll lose the entire array, instead the idea is that at any given time 70% of people can remain online as we work out the other 30% with recent backups, and we deployed servers with excess space so it's possible to quickly move people onto another disk that's immediately available. Also with SSDs, especially with high transfer speeds, we can watch for errors and migrate people off pre-emptively, and quickly, off a single drive. This gives us the option of doing this as hardware ages whereas with RAID, it would first of all not be an economical option, as in no budget for doing that, and secondly with DC hands human error and everything else that can go wrong with the array, it's just added risk trying to schedule this regularly. Finally, should something go wrong with the controller to where the whole thing has to be replaced that's a huge headache as well and long downtime. So we do have a plan in place, it's just being very poorly executed initially as we're swamped with work right now.
If we could go back in time and we also used more robust software, something like ZFS would make the most sense? But I'm not even going to pretend I'd be an expert with that.
Of course I'll end this with saying that we do actually have a chunk of servers with hardware RAID, and they're actually sitting in the datacenters right now ready to be configured. I'm just holding off on them. Oops, looks like this turned into another one of my long posts about this.
I feel you and I understand. Yes, this type of disks have a problem with RAID, also raid amplification writing and all that shit.
In the end it boils down to remote hands, I think, if you had bad experience with that it is better that you have a partial failure for longer than total failure for same amount of time AND bigger time to reload the whole backup.
That being said, it looks like NVMe is not there yet for servers or not tested enough, most failures I have heard of lately are NVMe, controller, raiser, onboard controller, the banks per se could be alright... I am afraid changing brand, unless it is some really-really strong Enterprise one, would not help.
My laptops: totally cool! RAID in a server... not so much.
Uncle is very conservative and resisted NVMe even as I have pushed, this time he appears to have been right.