FranTech (BuyVM.net) loses all data - "FRIED NODE"

deank

He pooped on the node and it's gone to the shit. Not hard to understand.

Why this thread has 200 posts is something I don't get it.

Anyway, let's make it 666.

user37489

@Francisco said:
This is how basically every VPS panel handles it. Solus, Virtualizor, Vultr, DO, Linode. Maybe DO & Linode can be a bit smarter about it if you have their Agent installed, but I'd guess if you don't it just goes by 'process status' like we do.

Francisco

I think if the CEO of Digital Ocean, took out my VPS's NVMe drives and took them home on Friday, they would have found a way to let me know that the VPS was in fact not ONLINE and wouldn't wait for me to open a support ticket on Sunday to discover it myself. However, I haven't brushed up on Digital Ocean's definition of online...

default

@Francisco said:

Are you going to message every provider and ask if they use a riser in their server for NVMEs? Tons do. Many are favorites around these parts.

desperand

Shit happens... Sad story...

default

@Francisco said:

@default said: Correct. At $60/month it is too low to have some cold-storage backup included for low-end market. That is way too low, there is no room for any profits.

Go signup with any of the big providers. DO, Linode, Vultr, whatever. Pick a plan. Any plan.

Already did. Ever heard of that low-end provider called Scaleway? Yes, they did it for me.

Edit: and I was paying 2 euro per month.

jar

@user37489 said: I think if the CEO of Digital Ocean, took out my VPS's NVMe drives and took them home on Friday, they would have found a way to let me know that the VPS was in fact not ONLINE and wouldn't wait for me to open a support ticket on Sunday to discover it myself. However, I haven't brushed up on Digital Ocean's definition of online...

Answering tickets at DO is why I'm interested in this topic, please beware that when they have a hypervisor failure all data is lost. I've personally had to send the notifications, and it's not fun when people tell you all about the hardship you've created for them when there's nothing you can do about it. Make sure you use their backup feature (but understand that it can fail too).

Hxxx

Unless it is a managed VPS service that explicitly includes backups, generally speaking.. VPS services never includes backups.

I once had a Vultr VPS and a in a random sunny day the node "kaboom", all I received was a ticket saying the host is dead, the data cannot be recovered, restore from your backups. I was paying for daily backups (an additional service) so I just restored from snapshots and moved on.

So I can't imagine bitching to providers about hardware failing if I'm not paying for backups in-site and externally. In the end data is the user's responsibility, is on the ToS. There is always a clause that says something like "X company shall not be responsible for user's data."

I recommend taking a few minutes and analyzing "how worthy is my data". "Can I live without it?" If the answer is no, you should really have like backups in 3 to 5 different places. The more is worthy, the more you backup.

Nekki

To summarise:

• BuyVM were 'testing' (their words, no takesies-backsies) a different configuration on a production node, and things went boom.
• The drives, with everyone's data (retrievable or not), are currently on @Francisco's coffee table
• Stallion shows/showed the VM as 'Online' even when it was dead. @user12345 is particularly upset about this, for reasons that aren't terribly obvious.
• @Maounique still doesn't like @Francisco, but by God does he respect him now.
• Tor nodes are only used for terrible, terrible purposes.
• Standard LET pile-ons, mainly in favour of BuyVM.
• There is a terrifying census amongst LET'ers that it is not necessary to have a back up of your data.

Did I miss anything?

deank

ToS or not, if it's something you care, you back it up.

Just like your body. Get regular checkups once you are past 30 if you care. If you catch a cancer early, you can defeat it.

comXyz

@default said: Edit: and I was paying 2 euro per month.

So you're saying Scaleway secretly keep backups of your VPS data without your permission? Go sue them, you would become millionaire easily.

default

@comXyz said:

@default said: Edit: and I was paying 2 euro per month.

So you're saying Scaleway secretly keep backups of your VPS data without your permission? Go sue them, you would become millionaire easily.

No. I want them to have a backup. They had a backup of a week before. I was happy and thanked them.

user37489

@Hxxx said:
Unless it is a managed VPS service that explicitly includes backups, generally speaking.. VPS services never includes backups.

I once had a Vultr VPS and a in a random sunny day the node "kaboom", all I received was a ticket saying the host is dead, the data cannot be recovered, restore from your backups. I was paying for daily backups (an additional service) so I just restored from snapshots and moved on.

So I can't imagine bitching to providers about hardware failing if I'm not paying for backups in-site and externally. In the end data is the user's responsibility, is on the ToS. There is always a clause that says something like "X company shall not be responsible for user's data."

I recommend taking a few minutes and analyzing "how worthy is my data". "Can I live without it?" If the answer is no, you should really have like backups in 3 to 5 different places. The more is worthy, the more you backup.

So Vultr sent you a ticket that the host is dead? How nice of them.

If BuyVM had sent me a notification on Friday when they unracked the node, I could have spent the weekend reinstalling all of my workspace applications. Instead, I received no notice at all. So when I needed to work on the VPS on Sunday, I couldn't because RDP would not connect.

Everyone is so hung up on backups, and I've never even complained about back ups.

My problem with BuyVM is that they hid the fact that the VPS was fried from me. Had I not needed to RDP this month, I'm sure they would have kept auto billing me into next month too.

And when I called out BuyVM about the fact that there was no notification, and even their VPS control panel still showed online, the "excuse is that mailgun is spotty"

No email?... maigun is spotty

VPS control panel says online?... that's just our definition of online

default

@user37489 said:
No email?... maigun is spotty

VPS control panel says online?... that's just our definition of online

Hxxx

@user37489 In general... if you care about uptime and knowing when things go down, you setup an alert using a third party service.

Francisco

@Nekki said: Stallion shows/showed the VM as 'Online' even when it was dead. @user12345 it particularly upset about this, for reasons that aren't terrible obvious.

This is incorrect, the OP's been BS'ing people. Since he refused to shared his email I decided to go log dive to find his email.

mysql> select userid, starttime, description from jobs where targetid = 115467 order by  starttime desc;
+--------+------------+----------------------------+
| userid | starttime  | description                |
+--------+------------+----------------------------+
|  41753 | 1636913292 | Started Virtual Server     |
|  41753 | 1636912706 | Restarted Virtual Server   |
|      0 | 1612794666 | Started Virtual Server     |
|  41753 | 1612794579 | Reinstalled Virtual Server |
|      0 | 1612794501 | Created Virtual Server     |
+--------+------------+----------------------------+

The only actions done to his VPS were done by him..or the system.

The VM was not started by us, it wasn't 'ONLINE' as he put it. He issued the Start commands himself.

And also in full colour:

Francisco

user37489

@Hxxx said:
@user37489 In general... if you care about uptime and knowing when things go down, you setup an alert using a third party service.

Thanks, I've recently signed up for Amazon Workspaces.

Do you have any recommendations for a third party service to follow Bezos around, in case he takes the NVMe drives my workspace is on home with him?

Anyone have the contact info for that NSO Group?

danielcardosopt

@Hxxx said:
@user37489 In general... if you care about uptime and knowing when things go down, you setup an alert using a third party service.

Exactly this, there are many free services that will monitor if your vps is up and notify you when the service is down.

Nekki

@Francisco said:

@Nekki said: Stallion shows/showed the VM as 'Online' even when it was dead. @user12345 it particularly upset about this, for reasons that aren't terrible obvious.

This is incorrect, the OP's been BS'ing people. Since he refused to shared his email I decided to go log dive to find his email.

> mysql> select userid, starttime, description from jobs where targetid = 115467 order by  starttime desc;
> +--------+------------+----------------------------+
> | userid | starttime  | description                |
> +--------+------------+----------------------------+
> |  41753 | 1636913292 | Started Virtual Server     |
> |  41753 | 1636912706 | Restarted Virtual Server   |
> |      0 | 1612794666 | Started Virtual Server     |
> |  41753 | 1612794579 | Reinstalled Virtual Server |
> |      0 | 1612794501 | Created Virtual Server     |
> +--------+------------+----------------------------+
> 

The only actions done to his VPS were done by him..or the system.

The VM was not started by us, it wasn't 'ONLINE' as he put it. He issued the Start commands himself.

And also in full colour:

Francisco

Dun-Dun-DUUUUUUUN!

user37489

@Francisco said:

@Nekki said: Stallion shows/showed the VM as 'Online' even when it was dead. @user12345 it particularly upset about this, for reasons that aren't terrible obvious.

This is incorrect, the OP's been BS'ing people. Since he refused to shared his email I decided to go log dive to find his email.

> mysql> select userid, starttime, description from jobs where targetid = 115467 order by  starttime desc;
> +--------+------------+----------------------------+
> | userid | starttime  | description                |
> +--------+------------+----------------------------+
> |  41753 | 1636913292 | Started Virtual Server     |
> |  41753 | 1636912706 | Restarted Virtual Server   |
> |      0 | 1612794666 | Started Virtual Server     |
> |  41753 | 1612794579 | Reinstalled Virtual Server |
> |      0 | 1612794501 | Created Virtual Server     |
> +--------+------------+----------------------------+
> 

The only actions done to his VPS were done by him..or the system.

The VM was not started by us, it wasn't 'ONLINE' as he put it. He issued the Start commands himself.

And also in full colour:

Francisco

I have NEVER said that the VPS was started by you. And I have been 100% honest with everything I wrote. I have not bs'ed anyone.

And yes, the VPS showed ONLINE when logging into Stallion. Your log literally proves nothing.

Again, here are the facts:

1. Log into VPS control center ariund 12:58 PM on Sunday
2. VPS shows status ONLINE
3. Cannot RDP, so I pressed the restart button
4. Still cannot RDP, so I open a support ticket.

I'm done with this forum

Hxxx

@user37489 Also if it makes a difference to you if the disk is in the DC or in a home, then you should make sure you follow best practices and encrypt your VM space. In my eyes it was so important to @Francisco to solve this issue that he took the job to his home to continue attempting to recover. I believe that's a good act.

serverian

@jar said:

@user37489 said:
Can a mod delete this entire thread. I've come to the conclusion that low end talk is basically just a marketing forum for BuyVM, and everyone here loves BuyVM.

Everyone stresses the importance of external backups, yet no one has ever actually backed up a Windows VPS. 6 pages of useless comments.

I've already received my refund, and my colleagues will likely request there's too.

I'm done, please delete. I'll learn how to externally back up a Windows VPS on my own, because no one here, besides my Francisco, has a clue.

This is not at all the case. I understand that some people want to make accusations like that, but it isn't the case. You cannot backup a full disk image of a Windows machine and carry it from provider to provider. This cannot happen. Take your hard drive out of your computer, shove it into an entirely different computer, and see if it boots. It will not. This isn't about marketing or not having a clue, it's about exactly what you're asking. You're just not happy with the answer.

When every single one of us mentions backups, we mean backup your data so that you can port it to a fresh OS install on a new system.

Not related but I think this works fine with Windows 10 now. I have pulled the OS disk from wife's old computer (AMD) to her new computer (Intel) and it booted up fine and she has been using it for the past few months without any errors. MS finally accomplished what Linux was doing for the past 15+ years.

deank

If Fran is correct, then I think we know what OP really is.

user37489

Does anyone here have a VPS with BuyVM?

If so, can you please log into Stallion and then confirm that the status of your VPS says ONLINE. Then, click on the actions to Start VPS. Lastly, what does the log show? Does it show a user issued start command?

I can't believe I have to explain such basic things to the VPS host himself on how his control panel works.

Francisco

@user37489 said: VPS shows status ONLINE

The VPS status didn't show as online. The VPS was never booted by us, or by me when the node was reracked.

• At ~2:00PM we got alerts of issues with the node.
• By ~2:45PM I was on site debugging the node.
• By ~5:30PM I declared things were a write off and provision a different server with brand new drives, but i'd still try to debug the drives at home where I have different hardware (different motherboard brand, things like that).
• By ~5:45PM the new node was racked online.
• By ~6:00PM Akash and team were sending out notice and helping people that had tickets.

At no point were user VM's mass started. The only person with that level of access is me, and why am I going to boot VM's that have no OS to boot from? Services do not auto boot when a node is powered back on, that's intentional in the off chance we're looking into some underlying issue and don't want to disrupt things (for instance if an array is acting up).

What are we going to hide from? We've been upfront that a failure happened. We've worked to help everyone. We've been more than fair with credits, and in your case, even a refund. Hell, 1 user even got some free time on his unaffected slab, just because he asked nicely.

My logs & the image show what went on. If you want to go ahead and argue about what 'ONLINE' means, that's fine, do that. Anyone thats developed a panel agrees with me on this one.

Francisco

serverian

@user37489 said:
Can a mod delete this entire thread. I've come to the conclusion that low end talk is basically just a marketing forum for BuyVM, and everyone here loves BuyVM.

Everyone stresses the importance of external backups, yet no one has ever actually backed up a Windows VPS. 6 pages of useless comments.

I've already received my refund, and my colleagues will likely request there's too.

I'm done, please delete. I'll learn how to externally back up a Windows VPS on my own, because no one here, besides my Francisco, has a clue.

If you are fine with taking it offline during the backups, boot it with a live Linux distro like Finnix, then dd the disk over network to the destination. Then you'll have the image of the full disk, which can be dd'd back when you need.

Also never trust anyone who claims that they are backing up your files for you and do your own backups. Take at least 2 backups on a cycle that you are comfortable with and put them in 2 diverse geographical locations. There is never too many backups.

user37489

@Francisco said:

@user37489 said: VPS shows status ONLINE

The VPS status didn't show as online. The VPS was never booted by us, or by me when the node was reracked.

• At ~2:00PM we got alerts of issues with the node.
• By ~2:45PM I was on site debugging the node.
• By ~5:30PM I declared things were a write off and provision a different server with brand new drives, but i'd still try to debug the drives at home where I have different hardware (different motherboard brand, things like that).
• By ~5:45PM the new node was racked online.
• By ~6:00PM Akash and team were sending out notice and helping people that had tickets.

At no point were user VM's mass started. The only person with that level of access is me, and why am I going to boot VM's that have no OS to boot from? Services do not auto boot when a node is powered back on, that's intentional in the off chance we're looking into some underlying issue and don't want to disrupt things (for instance if an array is acting up).

What are we going to hide from? We've been upfront that a failure happened. We've worked to help everyone. We've been more than fair with credits, and in your case, even a refund. Hell, 1 user even got some free time on his unaffected slab, just because he asked nicely.

My logs & the image show what went on. If you want to go ahead and argue about what 'ONLINE' means, that's fine, do that. Anyone thats developed a panel agrees with me on this one.

Francisco

Literally nothing I can say when it is apparent that you've never used your own control panel.

BuyVM is an amazing host, mods please delete this thread.

Queue more posts about back ups below

serverian

@user37489 said:
Does anyone here have a VPS with BuyVM?

If so, can you please log into Stallion and then confirm that the status of your VPS says ONLINE. Then, click on the actions to Start VPS. Lastly, what does the log show? Does it show a user issued start command?

I can't believe I have to explain such basic things to the VPS host himself on how his control panel works.

Stallion connects to the node's Qemu management API and checks for the KVM process running for the particular guest and if it's running, it's showing it as ONLINE. (As far as I understand)

The word ONLINE here should be RUNNING to avoid this kind of confusion really, @Francisco

Nekki

@user37489 said:

BuyVM is an amazing host, mods please delete this thread.

Entertaining threads will not be deleted ever. The only exception is if they include pornography.

You know what to do.

Francisco

@user37489 said: BuyVM is an amazing host, mods please delete this thread please

Sure, but I'm fine replying a bit more if it helps you understand.

@user37489 said: Does anyone here have a VPS with BuyVM?

If so, can you please log into Stallion and then confirm that the status of your VPS says ONLINE. Then, click on the actions to Start VPS. Lastly, what does the log show? Does it show a user issued start command?

I can't believe I have to explain such basic things to the VPS host himself on how his control panel works.

It will show the action, but your VM wasn't running. There was only 1 time your VPS was booted, here's the node side logs for you.

All message logs on the node for everything happening (not just your VM):

(13:25:12) KVM-72:/var/log/libvirt/qemu root: ls -alh /var/log/messages*
-rw-r----- 1 root adm 227K Nov 15 13:23 /var/log/messages
-rw-r----- 1 root adm 988K Nov 14 00:00 /var/log/messages.1
-rw-r----- 1 root adm  13K Nov  7 00:00 /var/log/messages.2.gz
-rw-r----- 1 root adm  12K Oct 31 00:00 /var/log/messages.3.gz
-rw-r----- 1 root adm  20K Oct 24 00:00 /var/log/messages.4.gz

Your VM's network interface coming up/going down:

(13:25:03) KVM-72:/var/log/libvirt/qemu root: zgrep kvm115467 /var/log/messages* | grep promisc
/var/log/messages:Nov 14 10:08:13 KVM-72 kernel: [146446.755364] device kvm115467.0 entered promiscuous mode
/var/log/messages:Nov 15 04:01:58 KVM-72 kernel: [210872.527465] device kvm115467.0 left promiscuous mode

On the 14th, yesterday, you booted it at 10AM PST. Today it went down because you had a pending immediate termination.

Francisco

default

@user37489 said:
BuyVM is an amazing host, mods please delete this thread.

No. I do not want this deleted. This is his precious involucration and care about customer's data.