New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Thanks for reaching out to Franta's Little Helpers Helpdesk!
SOMETIMES WE JUST HAVE TO LET THINGS GO.
SOMETIMES WE JUST HAVE TO LET THINGS GO.
By someone who can't let things go. Oh, the irony.
https://my.frantech.ca/submitticket.php
Hello,
There's a power outage in Luxembourg, causing the corruption of some storage blocks.
Sorry for the trouble.
Your bandwidth has been doubled.
Thanks, LowEndSupport
My BuyVM storage block is working just fine.
I monitor mine, haven't had an issue in a good while. I have a script that runs on cron and writes a small file, deletes it, and pings pushover if it fails.
I hadn't heard of that one. Seems super reasonable - $5 one-time per receiving platform (e.g., iOS) with a 7500/month limit.
Could you share that perhaps? I was thinking about creating something similar to monitor mine, but if it already exists and you are willing to share it then that would be great.
-Tim
Ok well that explains my problem. I'm assuming the data is gone (NBD).
Seems my memory failed and I apparently rewrote it to look for evidence in the Dovecot log. Either way it's a good bit of the work done. It goes with this cron job alongside it:
0 0 * * * /usr/bin/rm -f /root/read_only_alerted_today
https://clbin.com/8HPkv
I was about to send a ticket to ask about the many I/O errors in
dmesg, then I saw this. It's somewhat reassuring that I'm not the only one with issues.It's curious only some slabs are affected though. Only the one I rented recently is borked.
Mine just crashed as well
Mine seems to be fine now. Noticed I got kicked off SSH because of a VM reboot. Once I logged back in, slab was accessible again (and I didn't notice any data loss).
I've used Pushover for years, and have it used with a bunch of scripts across most of my systems. Super reliable, good developer.
My data is gone sadly, though luckily the data in question was only ~1 day old so all I lost is the time spent setting up a service yesterday.
Hello,
Shouldn't be any loss on the drives unless it was trying to write to it when the drive was out.
Have you tried unattaching and reattaching? You can ticket and we'll check it out.
Postmortem:
3 weeks ago or so we had a power strip fail and it knocked 3 of the cluster members offline. When we deployed these nodes we told ROOT that they must be double cabled but they ignored it, it seems.
Strip died, knocked the nodes offline. 2 of the cluster members came back happily enough, the last one wanted a fsck at the time. It got fsck'd twice (XFS being dumb) and things were fine.
Today's a repeat of that but on a different member. Nodes cleared up, all vm's rebooted that were attached to it.
It's possible you placed an order today for a slab and it didn't provision right. If you got a slab that won't show up just ticket and we'll remake it quickly.
The DC has fixed the double cabling at least....
Francisco
We hope
They took photos of such now. They've been very good in the past about following instructions (the original ryzen nodes & slab node), even the old e3 slices, so I wasn't worried about them botching that.
Anyway, lesson learned on that one.
Francisco
I bet @Francisco put the storage nodes to use for chia-ing, and disconnected everyone else that was using them 🤣
"Sorry guys, there was an 'outage'. Unrelated, you see that CHIA doubled in price?!"
Francisco
Dude, using trippelite again?
APC's over there.
I didn't provide the strips in LUX, they're included with the monster setup fee they charge.
Francisco
Doesn't surprise me given most "remote hands" staff are paid peanuts and either don't care or don't get clear instructions.
Had that before with supposed Highly available stuff where someone messed up and connected things to the same power feed instead of A+B like it was supposed to be done, it happens.
I was supposed to be the one that deployed the nodes into LUX but with rona that got held up. Still I didn’t want to delay ryzen and slabs there.
It’s ok, it’s resolved as of now. If anyone is having an issue, please open a ticket and I’ll check it out, just finishing some dinner.
Francisco
You can bet in LUX they get 2500-5000EUR/month. It's the most expensive country to hire people in the entire EU.
Far as I know many of their workers drive in from Germany every day.
Francisco
In EU they still get LUX salary (but pay higher DE taxes, but then also lower DE living costs), it's just that you can't find people to hire in LUX.
Yeah it broke my order for an 8TB slab pls DM me a personal link ( ͡~ ͜ʖ ͡°)
@codydoby why bother with block storage when you have unlimited access to unlimited dodgy google drive accounts each with unlimited storage?