New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
The learning by 🔥 thread
Today, I decided it was a good time to back up my backup files, so I wrote a script.
#!/bin/bash
BACKUP_DIR="/var/backup"
GPG_RECIPIENT="[email protected]"
find "$BACKUP_DIR" -name "*.tar.gz" -exec gpg --encrypt --recipient "$GPG_RECIPIENT" --trust-model always {} \;
rsync -avz --remove-source-files /var/backup/ user@remote_server:/backup/
Then I ran the script, checked the destination server, and realized I had made a small mistake, because it synced everything, including the unencrypted files.
So I deleted the files on the destination server (rm -rf *) and fixed my script by adding --include='*.gpg' --exclude='*' to the rsync line.
Then I ran the script again.
sent 51 bytes received 12 bytes 42.00 bytes/sec
total size is 0 speedup is 0.00
Please share one of your fuckups, so I feel a little less stupid.
Comments
Sorry for your loss but this is a great idea for a thread
Happens to best of us.
Also, a small suggestion, the idea with gpg encrypting every tar.gz archive is neat, but there is also rclone with crypt remote. It may actually make your script less sophisticated and thus more error prone
Personally, I didn't have any data loss incidents (yet) but once I made a typo in the
/etc/network/interfacesfile (something silly like255.255.25.0) and lost access to a remote server, so I had to drive there just to fix it.This is really helpful
i may or may not have punched my laptop battery with a screwdriver in attempt to swap out the thermal paste
Well my biggest fuck up is still this:
vs
Needless to say I spend a lot of time trying to recover the files
I was once fucked by a bind mount
I will never forget that moment
Always check the file directory or shell you're in. Once I deleted a DB from production thinking I was on dev environment none the less, fuck ups before it taught me to have backups and nothing was lost.
That's really doing it wrong. Get Arcticlean for next time. Will last forever.
Care to explain the difference to n00bz?
You have learned nothing if you had backups and lost nothing
sudo rm -rf $typo/$typovariations...The greatest part is that we tend of forget passwords by saving them in browsers and other applications!
I wrong mount disk on /etc/fstab (/dev/sdb instead /dev/sdd) for prolong time, and format /dev/sdb cause i thinking backup on /dev/sdd
This is the benefit of a Trash/Recycling bin feature. When will systemd fix that, eh?
Looks like the latest GPT-5.6-Sol made yet another variation of this mistake:
https://news.ycombinator.com/item?id=48865230
I had a storage server which had hdd and ssd.
I used the template to reinstall.
Voosh!
5 TB data gone down the drain. No backup, nothing.
Used to work at a fairly large Minecraft server network. Was running on many all nighters working for an upcoming server release. Usually all servers will have a dev server for staging/QA that can then be pulled from for updates/releases, this server did not. This specific server would go down at end of the map, and be down for days/weeks until time for new map release (I did not decide this). When I started working on the release, I made a backup of the old map called
${Server}-Oldand then copied that into${Server}, in the same directory. And then I went to work in the new directory.Release coming up, 2 in the morning, 7 hours until release. I was finally done and started to go through release checklist before going to bed. I went up a directory, and after typing the mandatory ls after cd, noticed the backup and went to delete it.
I had to pull another all nighter.
rsync --delete with the source and target switched (deleted all the source files since the target was empty).
That's why I have
alias rsync="rsync -n"and type\rsyncto override the alias when I'm ready.When I'm remembering that, I'll sometimes remove the "--delete" on the first run, since the target is empty, anyway.
I have dumped backups to wrong server. And aknowledge that after 2 months. Client held backups without knowing that.
forgot to takeout
--dry-runon a rsync production monthly backup script. didn't get caught until i have to update the hypervisor (3 months later) and making a checklist the backup must be 1:1 before anything funny being doneI deleted once all my code accidentally after working 30 hours straight without any sleep. when you get tired your hands brain coordination turns into self destructing mode apparently.
OMG so many mistakes i made, its part of life. Trick is to minimize the exposure to critical tasks. Like have data backups, redundant hardware, a disaster recovery plan, transaction based processing, version control. Correctly dosed caffeine levels and last but not least Enough sleep. LOL
i remember something similar happened restoring a backup to production when it should be done to an acceptance server. Oops, a very common mistake.
I was having Gemini organize my ISOs. It kept organizing them and then deleting it when it went to remove old files.
So you tell it of the mistake, it apologizes and says it will do things differently and verify beforehand next time.
Happens 3x more... it just finds all the ways to fuck up the same way.
Programmers are negative useful after 12 hours. Probably 8 hours, really.
So, soooo many times when working through a problem and eventually everyone is exhausted and break for the night. Come in the next morning, everyone has dead simple solutions ready.
I worked for a company and a sysadmin there swore to me this story was true.
In the late 90s, they had a project called the Transportation Management Project. An unsophisticated business analyst connected to a server and uploaded a lot of research data in the directory he assumed was for this project. It was aptly named, had a lot of available space, and he had no permissions to write to it: /tmp
After some weeks of working on these files, he became very sad when the server was rebooted...
I was never sure if I should believe that story or not. I have worked with some very dull-witted business analysts over the years, so maybe.
idling for a few hours is often part of the programming solution, the brain is a muscle that needs a resting phase before being used again. That is why good itcompanies have a lounge room and a bar
or some a-holes using not production databases for some kind of production data.