New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
What do you use for log aggregation/ search?
What does everyone use, if you aggregate, store, and search syslog and nginx logs, in a central location?
I'm seeing a lot of action in this space for self-hosted.
Edit: More importantly, I'm trying to remind everyone of the cool things we can still do, with virtualized servers and storage.
Webpages and databases aren't the Beginning and the End.
Comments
At work, splunk. It's fantastic and insanely expensive.
graylog is what we use
I tried spinning that up on a 2 core dediserve SSD instance and saw constant Load of 1.6 just sitting there before I piped any data to it.
I killed it after 3 hrs because I got nervous about being a bad neighbour on KVM.
Will try again on online.net dedi.
This has about 20-30 servers feeding into it:
ELK stack, or Nagios Log Server if you need a bit more functionality / ease of use / support. NLS is free for up to 500M of data a day though which might work out for you.
ELK is perfect, I have some massive ElasticSearch clusters feeding in terrabytes of log data per day.
Flawless!
elastic logstash
ELK, I wouldnt go as far as to say its perfect though.
I have found Kibana 3+ to need alot of work.
Did you open a ticket at the time, we could have checked it out for you? Launches (like at the moment) tend to see hundreds of new VMs kicking the blades hard with 'benchmarks'
Nah, this was back in January on a trusty old Amsterdam kvm.
I just put it down to java being java. Couldn't have been iowait; 25k-40k iops tells me that much.
ELK!
Ahh, in that case, no issues - you're free to max / kill your cores all day on our clouds.
That's not been my experience with Dediserve though.. Suspended for using "suspicious" CPU.
Never had any issues with Graylog on RamNode though, faster clock speed as well.
Its been great, we were running Splunk and that was just getting more and more expensive as the days go past.
ELK (we don't use Kibana FYI) gives us the ability to add nodes, rebalance the ElasticSearch cluster out and just keep scaling.
We are throwing among other things: syslog, text cdr, telemetry and it just handles it. We have about 2 years of data in there right now and performance is excellent, despite the extreme data bloat.
We e never suspended anyone for core usage. Unless there was another issue with terms or AUP? Feel free to pm me a ticket Id?
Our E5-2660v2's out perform most things in real world usage.
For search, grep and zgrep. For central location, rsync.
It feels like I'm old school, really I just love basic things and bash scripting
I'm using filebeat, topbeat and packetbeat to log-forward directly to elasticsearch and visualize in Kibana 4. Easy to set up and easy on the resources of the log forwarding server.
Grep mostly
Custom logcheck in combination with a dedicated log server.
logRhythm is amazing
https://logrhythm.com/
Orchestrate
ELK, but soon having problem to scale.
Finally adding a messaging queue cluster to solve the issue.