New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
The learning by 🔥 thread
Today, I decided it was a good time to back up my backup files, so I wrote a script.
#!/bin/bash
BACKUP_DIR="/var/backup"
GPG_RECIPIENT="[email protected]"
find "$BACKUP_DIR" -name "*.tar.gz" -exec gpg --encrypt --recipient "$GPG_RECIPIENT" --trust-model always {} \;
rsync -avz --remove-source-files /var/backup/ user@remote_server:/backup/
Then I ran the script, checked the destination server, and realized I had made a small mistake, because it synced everything, including the unencrypted files.
So I deleted the files on the destination server (rm -rf *) and fixed my script by adding --include='*.gpg' --exclude='*' to the rsync line.
Then I ran the script again.
sent 51 bytes received 12 bytes 42.00 bytes/sec
total size is 0 speedup is 0.00
Please share one of your fuckups, so I feel a little less stupid.
Comments
Sorry for your loss but this is a great idea for a thread
Happens to best of us.
Also, a small suggestion, the idea with gpg encrypting every tar.gz archive is neat, but there is also rclone with crypt remote. It may actually make your script less sophisticated and thus more error prone
Personally, I didn't have any data loss incidents (yet) but once I made a typo in the
/etc/network/interfacesfile (something silly like255.255.25.0) and lost access to a remote server, so I had to drive there just to fix it.This is really helpful
i may or may not have punched my laptop battery with a screwdriver in attempt to swap out the thermal paste
Well my biggest fuck up is still this:
vs
Needless to say I spend a lot of time trying to recover the files
I was once fucked by a bind mount
I will never forget that moment
Always check the file directory or shell you're in. Once I deleted a DB from production thinking I was on dev environment none the less, fuck ups before it taught me to have backups and nothing was lost.
That's really doing it wrong. Get Arcticlean for next time. Will last forever.
Care to explain the difference to n00bz?
You have learned nothing if you had backups and lost nothing
sudo rm -rf $typo/$typovariations...The greatest part is that we tend of forget passwords by saving them in browsers and other applications!
I wrong mount disk on /etc/fstab (/dev/sdb instead /dev/sdd) for prolong time, and format /dev/sdb cause i thinking backup on /dev/sdd
This is the benefit of a Trash/Recycling bin feature. When will systemd fix that, eh?
Looks like the latest GPT-5.6-Sol made yet another variation of this mistake:
https://news.ycombinator.com/item?id=48865230
I had a storage server which had hdd and ssd.
I used the template to reinstall.
Voosh!
5 TB data gone down the drain. No backup, nothing.
Used to work at a fairly large Minecraft server network. Was running on many all nighters working for an upcoming server release. Usually all servers will have a dev server for staging/QA that can then be pulled from for updates/releases, this server did not. This specific server would go down at end of the map, and be down for days/weeks until time for new map release (I did not decide this). When I started working on the release, I made a backup of the old map called
${Server}-Oldand then copied that into${Server}, in the same directory. And then I went to work in the new directory.Release coming up, 2 in the morning, 7 hours until release. I was finally done and started to go through release checklist before going to bed. I went up a directory, and after typing the mandatory ls after cd, noticed the backup and went to delete it.
I had to pull another all nighter.
rsync --delete with the source and target switched (deleted all the source files since the target was empty).
That's why I have
alias rsync="rsync -n"and type\rsyncto override the alias when I'm ready.When I'm remembering that, I'll sometimes remove the "--delete" on the first run, since the target is empty, anyway.
I have dumped backups to wrong server. And aknowledge that after 2 months. Client held backups without knowing that.
forgot to takeout
--dry-runon a rsync production monthly backup script. didn't get caught until i have to update the hypervisor (3 months later) and making a checklist the backup must be 1:1 before anything funny being doneI deleted once all my code accidentally after working 30 hours straight without any sleep. when you get tired your hands brain coordination turns into self destructing mode apparently.
OMG so many mistakes i made, its part of life. Trick is to minimize the exposure to critical tasks. Like have data backups, redundant hardware, a disaster recovery plan, transaction based processing, version control. Correctly dosed caffeine levels and last but not least Enough sleep. LOL
i remember something similar happened restoring a backup to production when it should be done to an acceptance server. Oops, a very common mistake.