New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How do you clean up log files?
Hello, I was setting up a logging system cleanup tool and I realized there is probably a better way to do this. We're generating a ton of logs right now daily. 99.999% of it is really only in case we receive a bug report, the other .001% is for record keeping purposes. After a few weeks, this data is basically useless. So I was planning on removing anything marked debug whenever the logs get to a certain size. I was wondering if anyone has better log management ideas? I guess I could always store them somewhere that's dirt cheap for storage but is this really worth it? Would love thoughts or ideas on this!
Comments
logrotate - https://man7.org/linux/man-pages/man8/logrotate.8.html
You can always check if you can compress the logs instead of removing them. Compressing logs (text) can be very efficient storage-wise most of the time.
If they are useless after a few weeks there is no reason to store them. Use a cron job to delete log files older than 2 weeks..
100% my suggestion as you make a config file for the logs and you forget about it. No custom scripting or other management of them is needed.
I guess no magics here, you'll need to decide which data you want to keep after truncating the most important of your log files so they don't exceed a certain size (probably using tail command and a cronjob) then compress the data and move it to some paid cold storage for backup.
Whether it's worth it for you or not is your decision.
There may be additional approaches to handling logs but I don't think you'll find any brilliant strategy that will make this matter cheaper.
Rsyslog. I store all logs centrally.
Right, but how verbrose are your logs?
Having debug level logs for daemons gets excessive.
Error and above. No info or warnings. I have no time for that.
sudo rm -rf /
Right, sounds like OP is logging debug, so a little different
Are we actually talking about syslog or is this some kind of application log?
Why not switch logs daily, and then it's easy to identify which are > 14 days and delete them.
rsyslog, syslog-ng, etc. do this.
But I'm sure you've already thought of something like this, so is there something else to consider? And how many MB/GB/TB/PB/EB of logs are you generating per day?