[lug] Modern log analysis?

[Quentin Hartman]

What you're probably looking for has become known as ITOA (Information
Technology Operations Analytics) and is in pretty early stages still,
mostly because of the cost. Splunk, Moogsoft, EMC, and others all have
products in this space. I don't know of any OSS options. Moogsoft is my
favorite, based on doing some fairly brief evaluations:
https://www.moogsoft.com/lp/product-itoa

Most require a "learning period", where you basically train a machine
learning algorithm. It reminds me a lot of this:
https://www.youtube.com/watch?v=cQXT3E8Anq8

Once trained though, they (supposedly) do a pretty good job of filtering
out the noise. I haven't lived with one in production though.

QH

On Tue, Jun 5, 2018 at 4:54 PM Rob Nagler <nagler at bivio.biz> wrote:

> What do people use for log analysis to detect bugs and threats in
> real-time?
>
> Once upon a time, I thought Bayesian log analysis would take over, but
> searches show up papers and a few v.2 packages.  Now I also hoped for
> machine learning approaches, but not much there either.
>
> It seems that Nagios, Splunk,  Loggly, etc, are about you doing the work
> of figuring out what's important and what's not by using pattern matching
> of some sort (e.g. if 10 of event X in 10 minutes send an alert). That's
> fine, but it requires A LOT of work. It even seems that you are expected to
> look at fancy dashboard and run reports looking for problems. I've got
> other things to do...
>
> Any recommendations?
>
> TIA,
> Rob
>
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: irc.hackingsociety.org port=6667 channel=#hackingsociety
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20180607/a4281739/attachment.html>