Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


How do you clean up log files?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

How do you clean up log files?

Hello, I was setting up a logging system cleanup tool and I realized there is probably a better way to do this. We're generating a ton of logs right now daily. 99.999% of it is really only in case we receive a bug report, the other .001% is for record keeping purposes. After a few weeks, this data is basically useless. So I was planning on removing anything marked debug whenever the logs get to a certain size. I was wondering if anyone has better log management ideas? I guess I could always store them somewhere that's dirt cheap for storage but is this really worth it? Would love thoughts or ideas on this! :)

Comments

  • logrotate - https://man7.org/linux/man-pages/man8/logrotate.8.html

    You can always check if you can compress the logs instead of removing them. Compressing logs (text) can be very efficient storage-wise most of the time.

  • kevindskevinds Member, LIR

    @MeltedMembrane said:
    . I was wondering if anyone has better log management ideas? I guess I could always store them somewhere that's dirt cheap for storage but is this really worth it? Would love thoughts or ideas on this! :)

    If they are useless after a few weeks there is no reason to store them. Use a cron job to delete log files older than 2 weeks..

  • PuDLeZPuDLeZ Member

    @let_rocks said:
    logrotate - https://man7.org/linux/man-pages/man8/logrotate.8.html

    You can always check if you can compress the logs instead of removing them. Compressing logs (text) can be very efficient storage-wise most of the time.

    100% my suggestion as you make a config file for the logs and you forget about it. No custom scripting or other management of them is needed.

    Thanked by 1yoursunny
  • farsighterfarsighter Member
    edited January 3

    I guess no magics here, you'll need to decide which data you want to keep after truncating the most important of your log files so they don't exceed a certain size (probably using tail command and a cronjob) then compress the data and move it to some paid cold storage for backup.

    Whether it's worth it for you or not is your decision.
    There may be additional approaches to handling logs but I don't think you'll find any brilliant strategy that will make this matter cheaper.

  • LeviLevi Member

    Rsyslog. I store all logs centrally.

  • kevindskevinds Member, LIR

    @Levi said:
    Rsyslog. I store all logs centrally.

    Right, but how verbrose are your logs?

    Having debug level logs for daemons gets excessive.

  • LeviLevi Member

    @kevinds said:

    @Levi said:
    Rsyslog. I store all logs centrally.

    Right, but how verbrose are your logs?

    Having debug level logs for daemons gets excessive.

    Error and above. No info or warnings. I have no time for that.

  • RubbenRubben Member

    sudo rm -rf /

    Thanked by 2ralf yoursunny
  • kevindskevinds Member, LIR

    @Levi said:
    Error and above. No info or warnings. I have no time for that.

    Right, sounds like OP is logging debug, so a little different ;)

  • raindog308raindog308 Administrator, Veteran

    @MeltedMembrane said: So I was planning on removing anything marked debug whenever the logs get to a certain size. I was wondering if anyone has better log management ideas?

    Are we actually talking about syslog or is this some kind of application log?

    Why not switch logs daily, and then it's easy to identify which are > 14 days and delete them.

    rsyslog, syslog-ng, etc. do this.

    But I'm sure you've already thought of something like this, so is there something else to consider? And how many MB/GB/TB/PB/EB of logs are you generating per day?

Sign In or Register to comment.