All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Recovering data after server's disk crashed? HELP
Really looking forward to some good ideas and sincere advice..
I had 2 servers with a host. One we used as a main to host few websites, the other as a backup. The backup started having hardware issues a fortnight ago and new hardware meant all data was gone. That wasn't the end of it.
A week ago main live server stop delivering web pages. After I SSHed in the main I found that while I could browse through the file/folders the file system itself was in a readonly mode & even reboot cmd wudnt work. In the mysql client I cud see all the databases intact but since everything was readonly I cudnt get a dump or create a home tarball.
I ordered reboot from host's panel & the system went down. When it didn't come back online I created a ticket and the devs found the disk had numerous errors, they tried fsck but the disk still has errors so they gave up & installed a new drive in the server. So, it appears as things are now the data is all gone 
On my request the dev guys have plugged the old drive via USB to the server. Bcoz of block errors the system won't mount it. It is essential for me to retrieve the home directory and mysql dump of one website in particular.
Please guide me with any ideas regarding data recovery if it's possible. At this point I wud take even pieces.
p.s. The drive is a 1Tb magnetic disc. System is Debian.
Comments
ddrescue the disk to somewhere safe before you do anything else
If you are short on space to store the image, use btrfs transparent compression or mount remote storage
Basically you are saying that your backup strategy failed and also your main server disk died. Assuming there is no RAID here (hence you are asking question here).
Only sensible way to recover data is going for professional data recovery service. If you care about data, don't mess with drive anymore and send it for recovery.
I hope you get your data back safely.
Actually I was thinking about the same thing because new disk is the same size as the old one. I'm going to ask the host to temporarily replace the new drive with a bigger variant.
Ouch, tough situation here, I don't know too much about data recovery practices BUT I'm inclined to believe the odds of you getting that data back is very slim. One time I actually sent a drive over to the US for data recovery and that was unsuccessful (this wasn't my personal drive but something for an organization).
Anyways, as someone said, the best thing you can do is really not interfere with the messed up drive any further or you will risk further data loss. Serious condolences on what you have experienced - can imagine your customer is probably super emotional as well.
Good luck! Let us know if you do actually get through!
In recovery mode you can use ddrescue as recommended above. Then you can try using the testdisk utility. If it is only a filesystem problem, it might give a good result. If the source of the problem is a faulty disk, however, the only way to try to fix the problem is to send the faulty disk to a professional service. They have specialized equipment that can help.
Never reboot system when it is in read-only state. If no backups - gg. Suggestions here won't bring you a relief, only confusion.
Wheres the backup for your backup?
This exactly this.... I have a direct admin server (nothing terribly fatal if it died just my email and a neglected blog) I back that up to a storage server and then back that up to borgbase.... if the backup and the backup of the backup fail ... I was meant to lose that data ... and I will live with it
I hope you get your data back. I also hope you develop a proper backup strategy that involves multiple levels of backups, that are both on line and off line, that have versions, are stored in different locations and secured with encryption if necessary.
... some other thing I forgot to mention
Most people tend to become backup experienced after losing data at least once (and it usual hurts), maybe it is his first major experience. Then there's (some learning) and 2nd scenaria happens when the backup fails (more hurt) before they make a backup of the backup.
I know I've seen it many times where people (companies) ... don't want any downtime and 5 mins downtime costs 5k (in lost labour and billing etc) however you say to them you need a 10k device with cloud backup and it will cost 1k per month to keep it offsite cloud backup as well as the backup on the premises on this device (which offers a virtual spin up in 5 mins and a cloud recovery time of 30 mins) and they go...
NAH that's too extensive
Not on LET.
I've seen people losing data over and over again and still never think about having backups.
I'm pretty sure a few businesses folded due to not having proper backups.
3 2 1 Backup ??? I had probably same situation
After I was about to have a heart attack, I was able to reconstruct the raid and the data was back.
After this situation, I even save my dick photos in the cloud. Everything in the cloud !
As long as "the cloud" isn't your one and only primary backup?
I have had bad experiences with raid. Can take a day or days to rebuild. Sometimes the controller fails.
Yes raid is not a backup but I was to naive.
No I have one more Storage Group configured in Raid 1 and all the data on raid 6 are backed up to raid 1 and the data are mirrored as well to Google drive.
I have seen someone who talks about backups all day but never makes backups.
It has to really hurt for once and you have to really get hit in the face. But then you make backups and you feel better.
I have family photos in a hard drive, with daily rsync to remote SFTP drive.
I'm rather concerned that someday I would delete the files by mistake (or they get deleted by ransomware), and then rsync propagates the mistake to remote storage.
Same goes for Nextcloud.
Once I mistakenly deleted a file via mobile app.
I realized right away, but I didn't enable snapshot so that I cannot recover from Nextcloud server.
Luckily my laptop was powered off at the time, so I turned off the home router, turned on the laptop, copied the file elsewhere when the Nextcloud client cannot sync.
I guess I'm waiting for more hurt at this stage.
Even better: automate the process and use that to test regularly. Have a VM in s asimilar configuration and have it delete the data and restore from the last backup on a timer. Have the process message you in all ways possible if there are errors, and manually login regularly to make sure it is running and has recent data. I do this for my mail server and a web server. It doesn't need to be as powerful as the main boxes, just powerful enough that you can login to eyeball things. The VM doesn't need to be world visible (in fact should not be), just visible to you.
You can't do that with masses and masses of data, for that automated checksumming and verifying against old sums and (in the case of your latest backup) current live data (care being taken there to allow for data modified since the last backup run), but for your most important data it is a great safety net. You know your last backup restores without error and the data you've checked is OK, so if you need to restore when the main machine fucks up you probably can. If push comes to shove you can probably even make the backup tester VM visible to the world, bump up its resources, and update DNS entries, to use it as live until you fix the normal location.
What excuse do you have for not having an offline copy? Get an few external drives and back up to those daily/ weekly/monthly/whenever you make an important change.
Your favorite porn site gets hacked and then starts distributing ransomware which encrypts your photos and somehow it propagates to your backups before taking the entire drive with it.
If its really important get a safety deposit box or something and rotate drives in there as often as you feel is necessary.
I also forgot to mention that drives expire and its a good idea to make a note to change them well before the expiration date. A the point they are expired but have been powered off most of the time (if used as offline drives) then I might still use them as last resort drives. But obviously wouldn't use them for anything important.
I do have a secondary HDD that is usually offline.
I rsync the photos into this HDD every 3 months.
Worst case scenario: ransomware arrives when I'm doing the rsync into this HDD, and also steals the SSH private key so that remote backup is deleted too.
Mega sad scenario: the house burns down while I'm in the gym and I didn't bring a phone.
Both HDDs are gone.
All the computers and phones are gone.
SSH private key is gone.
No private key => cannot login to backup server.
No phone => cannot pass two factor authentication to login to email or Virtualizor => cannot VNC into server => cannot pay renewal so that the server expires and data deleted.
Impossible.
I only watch UMD on a tablet in Incognito mode.
The tablet doesn't mount the SAMBA share.
The most important photos are printed.
My mother keeps them in the roof.
I can always redo the push-ups or revisit Universal Studios, but I can't reanimate the dead grandma and retake her photos.
If try is semi readable, ddrescue -f /old/drive /new/drive will usually get most of your data. needs equal or larger drive though. boot to a rescue image and let it run. may take a day or more depending on how bad drive is. I use this for Windows/Linux/etc with 90%+ success rate
I am not shitting on OP because he actually had a backup unlike 99% of cases out there on LET.
It's too bad that both drives passed away in a quick succession, but perhaps it was meant to be. Life often throws a wrench into gears.
Soft-offline snapshots: have another server pickup a copy from the main backup. Have it take snapshot copies (cp -al ...) and/or have its rsync transfer not include any delete options. Make sure your main locations and the backup can not authenticate against this machine (keep the keys on USB sticks and printed as QR codes, for when you need to access the soft-offline setup for admin).
That way ransomware can't get to the soft-offline copy (unless it is planted on you by someone local who also steals the physical key record) and you can't accidentally drop files (in fact snapshots on your main backup would cover this latter type of accident too).
Thank you all. What seemed lost at first was found:)
99.9% data recovered because of your advice ie ddrescue followed by fsck. Granted all the data was in the lost+found but i ultimately found the entire home & mysql data directory. Restored all of the websites.
Now, I am looking for a storage deal. Anything from 1, 2 or 4 terabytes cheap storage. Please lemme know if you know any.
Always a fan of this great community.
Hetzner storage box
5TB for 10 euro / ex vat or 1 for 3 euro / month
Nextcloud has a Trash feature, btw. I didn't realize it wasn't default.