Log parsing in real time

Levi

Hello industry experts,

I have interesting problem to solve:

There is 20 users on debian machine which has 20 pure-ftp accounts. How to make separate, dedicated log files for those users about their FTP activity?

pure-ftp has "AltLog" parameter which passes output to a file like this: ~/ftp.log . I'am thinking that with 'tail -f' to monitor new entries and filter incoming line by user into separate files: ~/mouser.ftp.log, ~/harambe.ftp.log etc.,

But I have gut feeling that this approach is less than efficient with larger volumes of incoming log entries. Any thoughts about the most optimal way to approach this problem?

SpeedBus

Perhaps using something like https://betterstack.com/logtail could be an option?

Their free tier allows 1 GB of logs/month, only catch I see is the retention time on the free plan is 3-days

Hxxx

I know my reply is silly, apologies. But you could use DA or cPanel for this. IIRC the logging is per account.

Disable all the extra services.

yoursunny

Did you remember to buy the log parsing machine from balenciaga logging department?
It's between the chainsaw and the diesel generator.

1. Set pure-ftp AltLog to a named pipe, rather than an ordinary file. Most programs would be able to append to a named pipe just like a file.
2. Open the reading side of the named pipe, continuously read lines from it, and write them to per-user files.

For the reading side, you can use any scripting language.
If the username is at the same column, Awk can do it easily.

The reading script should run as a systemd service that is Before= and RequiredBy= pure-ftp.

tjn

https://www.graylog.org/ is pretty good - open-source and can be selfhosted

Levi

@yoursunny said:
Did you remember to buy the log parsing machine from balenciaga logging department?
It's between the chainsaw and the diesel generator.

1. Set pure-ftp AltLog to a named pipe, rather than an ordinary file. Most programs would be able to append to a named pipe just like a file.
2. Open the reading side of the named pipe, continuously read lines from it, and write them to per-user files.

For the reading side, you can use any scripting language.
If the username is at the same column, Awk can do it easily.

The reading script should run as a systemd service that is Before= and RequiredBy= pure-ftp.

I will send you some balenciaga trousers. Insulated to withstand Arctic cold in that DC of yours. Thank you my friend.