New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
This is where the saying "Don't put all your eggs in one basket" comes into play.
If you spread your backups across multiple datacenters, in multiple countries, or different parts of the country -- odds go up that you don't lose everything.
Random VPS on its own, yes bad idea. Multiple random VPS's from different providers in different physical locations, with checks from time to time that the data is intact? Perfectly fine.
There's already a solution that takes that into account, the 3-2-1 rule.
Three separate backups, two on different physical types of physical media, one stored off-site.
Always got to worry about silent corruption, since like the name implies it's silent unless you look for it.
But, some modern filesystems do have integrity streaming and the ability to perform scrub operations (looking for silent corruption).
Yep, at least three copies or it's not a backup.
What I personally follow. It's also not a backup unless you VERIFY that it can be restored.
Well what I mean is that I won't need to worry about the filesystem trying to allocate something to a known bad-sector. The bad block list tells the filesystem never to allocate to one of those sectors, so there's no risk of silent corruption from that specifically. There's still a risk of silent corruption from plenty of other sources, of course (bit rot, uncorrected bit flips in the page cache or the storage bus, filesystem bugs, other types of hardware failure, etc.).
Personally if I need to detect (but not necessarily repair) corruption, I'll use ext4 with dm-integrity.
100% agree. A backup that isn't verified is not a backup.
I experienced the same. It sounds like we were on the same server. I was using it as a backup server.
The server was completely lost this week, and it was only 6 weeks ago that it also had a problem. The first time, the data was supposedly recovered, but when I checked the data integrity, it was corrupt. So I wiped it and backed my data up again.
This time, it was down for 3+ days, and they gave up on it. They provisioned service on a new server/location, and I restored the system and ran my backups again.
I figured with a RAID storage server, if there was a problem, it would just be a matter of swapping the failed drive out and letting it rebuild. But apparently this was a bigger issue. It's too bad, but it happens.
I'm fine with the way it was handled, though, since service is working again at the new location. The extended down time and effort involved to restore and backup again was a hassle but understandable.
Since it's a backup server, I guess you hope your systems don't have a problem at the same time the backup server is down. But I also have backups on a hard drive at home, if needed.
I think data recovery is always the customer's responsibility, unless the service you bought explicitly includes backups. Even then, you might be the one expected to restore from that backup.
Well, I heard you guys.
I get it; backing up is my responsibility.
So the conclusion is, never rely on a provider systems.
I need to get Oracle free-tier storage now
I got an important lesson in testing my backups the other day. My Navidrome DB got corrupted and the latest backups were also bad. The last good one was from May so I lost quite a bit of activity. The moral of the story is it's not much use replicating your backups with 3-2-1 if they're not good to restore when you need them.
Indeed, learned that lesson the hard way too.
Basically you need to test a restore at-least one time (an actual restore preferably), then have an archive hash that you routinely validate to detect corruption preemptively -- so it's not just sitting in storage corrupt on your backup servers.
Just a simple cron job checking daily or weekly with your next push is good enough.
Better than a single hash, use dm-integrity so you can figure out exactly where corruption occurs if it does occur.
I think they are aware atm as they mentioned somewhere about it
Just keep a hash of each file. Then reverify the hashes. But then, the hashes of the hash list has to be hashed to.
Almost like a hash tree, which is commonly used for ensuring integrity (e.g. dm-verity and other Merkle tree-based integrity checkers).
Usually I do ZFS on my backup nodes, both physical or VM's, and where there's a SAN involved (like with BuyVM's slabs) I just request two of them and put in a ticket to make sure they're physically separated on different pools. Rely on ZFS's integrity check to scrub out detected rot, and setup email alerts if the pool hits an error in the scrub.
But the idea is that anything is much better than nothing here. If you have a simple hash and a daily check, if the hash gets damaged you get an alert, and if the files get damaged, you get an alert. If the machine goes unstable (RAM / CPU), obviously get an alert there too. Gotta love automated scripts!
Lost count of how many storage setups that I've had fail over the years, heh. But yes, definitely don't leave em running blind anymore. That's the main point I'm trying to make to anyone reading through here.
Don't be like us, don't learn the hard-way.
Yes, those automated script helps. Then, it become overwhelming so you start automating the alert tracking.
I think most of us learned it the hard way before we become data hoarder
It's still just a single planet of failure though.
is it safer to locally back your data up onto a USB stick or USB HDD, or a $1/mth storage plan you bought on LET? discuss.
This is why you're supposed to transmit your backups (encrypted, of course) in radio signals into deep space. Try to aim right at the edge of a black hole's event horizon and it should orbit it until the whole thing evaporates.
Restoring the data is a bit pricey though, but at least it's cheaper than restoring from Amazon Glacier.
USB HDD. If that fails, you can at least send it to a lab that can (for a not-insignificant price) swap platters and fix it up. If a USB stick fails, it'll be nearly impossible to retrieve anything from the damaged NAND flash.
It's much better to buy an internal HDD and put it in a USB enclosure (or better, a dual drive enclosure so you can set up RAID1) than to buy a cheapo "USB HDD", because those use binned parts with absolutely awful reliability.
Glacier Deep Archive you mean?
Deep Archive is just one of their storage classes (and the most expensive to retrieve from), but Glacier has two others: Instant Retrieval and Flexible Retrieval.
ugh, its pain as a costumer though.
Like Offline.net or OneDevider, they do that.
They just like yea, its stuck in the rack yk, we to lazy to like fix it.
Come on guys, @DeluxHost isn't that bad. It seems they do care but are overwhelmed "temporarily". Clearly they are NOT a high quality+high reliability provider, that's just not possible with their prices, but they are not a crappy "we don't care" yolo providers.
If you need a dirt-cheap VPS they are among the better options, but they certainly are not a place I'd put my important "must not fail ever" data or services on.
Maybe a question of personal preference. My personal guide line is cathedral, not bazaar, well established and time proven over a long time, plus. and very important IMO, as low complexity as reasonably feasible, but not a micron less and definitely no high complexity behemoths like e.g. zfs, sorry.
FULL ACK
Plus/but doing backups sensibly requires a sensible analysis and adequate strategy. One criterion often ignored is how dynamic the data are. That's why I have a different stategy for e.g. DBs vs. "just some largely static website data".
Plus of course backup sets of different age reaching back long enough.
Yes - but: The goal, at least the realistic goal of doing backups is not "zero risk, no matter what" but rather minimizing risk as far as possible within one's means and needs.
Btw and FWIW my strongly preferred choice of storage provider is @host_c who has demonstrated solid understanding of engineering aspects and approach and who does use good technology.
link it ?
What is the local portable backup HDD/storage solution for these use cases?
Agreed. Honestly long as a provider tries to prevent their arrays from failing or to recover things, I'm personally happy with that effort -- even if they don't succeed. The big name players fail too, even the most expensive providers lose data pretty routinely. It happens-
Per the other comment. Yep, ZFS can (****and does****) fail, just like any other RAID bearing tech does, especially if you have a major system instability. Can also just get unlucky and have multiple disks fail at the same time or during resilvers. There's no such thing as a magic bullet. Not outstanding acts of god too, like a fire in the DC or a flood.
Like the saying goes, RAID is NOT a backup. Yet that also doesn't make RAIDs bad either. Everything is better than nothing, all methods of redundancy help reduce the odds of loss.
Nothing fancy. Just plugging in USB enclosures w/ spinning rust and doing a manual copy, reading it back, verifying, then unplugging the drive and putting it somewhere safe.
Since it's not network connected and it's offline, harder to accidentally wipe the copy.
No, and in my experience they're actually less reliable, probably because people are hammering the storage all the time.
But yeah, if it's a backup, by definition you have a copy elsewhere, so it's no great disaster if it fails. And a good data recovery strategy should have at least 3 redundant backups.
Data recovery is your problem, unless you are explicitly paying the provider to do it. And even then, it's still your problem if they can't restore it.
At the end of the day, it's your data. If it's important to you, make MANY redundant backups of it.
Is this from the OVH training manual?