Linux log file monitoring

[Sinbad]

Right, I know you're all thinking "this one is easy".

Watching for certain strings in a log file. Nice and easy.

Snag:
There are ~1500 log files across 3 machines that need to be monitored this way, and they're pretty damned chatty.

So, anyone out there know a nice elegant (or even inelegant, just effective) way of scaling the simple requirement out to this size?

Thanks!

 

[ozziej]

Yes, try have a look at the "ELK" stack.

Elastic search, Logstash and Kibana.

Has a SQL like interface to search logs and also aggregates them.
https://www.elastic.co

 

[biena]

www.nagios.com
[-]Free version 500mb/day restriction https://www.nagios.com/news/2016/01/7181/[/-] doesnt seem to be available anymore

 

[AlphaJohn]

5 or less Servers Hit up Nagios

https://www.datadoghq.com/nagios-monitoring/

Or use opensource track

 

[Sinbad]

Used Nagios before. The log_check plugin really didn't scale nicely... but that was a while ago. Can it handle that many log files? (It used tail and diff IIRC)

 

[SauRoNZA]

Used Nagios before. The log_check plugin really didn't scale nicely... but that was a while ago. Can it handle that many log files? (It used tail and diff IIRC)

Would https://my-netdata.io/ do the job for you?

 

[Sinbad]

Would https://my-netdata.io/ do the job for you?

What's its special trick for logfiles?

 

[SauRoNZA]

What's its special trick for logfiles?

You know now that I look at it again I think I’m remembering the wrong thing.

Although I’m sure it has some kind of support.

What about Zabbix?

Can’t you use the error output syntax of your process to just pump errors to another stream instead of monitoring the logs directly?

 

[Sinbad]

You know now that I look at it again I think I’m remembering the wrong thing.

Although I’m sure it has some kind of support.

What about Zabbix?

Can’t you use the error output syntax of your process to just pump errors to another stream instead of monitoring the logs directly?

Not something we have obvious control over.
I've logged a request with the vendor. It must be possible.
All the typical stuff will choke on this volume. 15gb a week of log output.

 

[SauRoNZA]

Not something we have obvious control over.
I've logged a request with the vendor. It must be possible.
All the typical stuff will choke on this volume. 15gb a week of log output.

Radius?

 

[Sinbad]

We have qradar. But getting the stuff in is a challenge.

 

[SauRoNZA]

We have qradar. But getting the stuff in is a challenge.

Yeah it sounds almost like you need an application for your ****ty application.

Do you really need all the log data? Surely can just trim it off based on severity?

 

[Sinbad]

Yeah it sounds almost like you need an application for your ****ty application.

Do you really need all the log data? Surely can just trim it off based on severity?

That's what I need the vendor to tell me how to do. It's far too freaking verbose. Need a specific class of error to go to a different location.

 

[Lupus]

Used Nagios before. The log_check plugin really didn't scale nicely... but that was a while ago. Can it handle that many log files? (It used tail and diff IIRC)

The thing with Nagios is, you can build your own plugins using a variety of scripting tools, you don't have to just use the ones built by someone else. But so many log files I'm really not sure on. I use nagios with a combo of bash/perl scripts that dig into the logs for me.

 

[IOPS]

What data do you want to see?
If it's simply counting/ visualisations I'd take a look at Telegraf (which has log monitoring), InfluxDB and Grafana.

 

[Sinbad]

Keyword alerting

 

[WAslayer]

Keyword alerting

have you looked at rsyslog..? They have Property-Based Filters which uses compare operations based on string contains, is equal to or starts with etc..

https://www.rsyslog.com/doc/v8-stable/configuration/filters.html