Profiling of persistent SSHD brute force attack
Proper setting up and regular monitoring of logs gives you the avenue to know what's really happening with your box sitting out there in the internets, and to anticipate when bad things are about to happen. One of the warning signs would be that someone has been poking around your box, looking for an (easy?) way in.
The natural thing that would jump out at you then, is that this someone has been accessing your box in far higher volumes/durations, especially on services that should not be accessed by others.
This is one example of such accesses on a linux box: SSHD brute forcing over long periods of time.
Troubleshooting Splunk
Have been fiddling around with Splunk lately. Splunk's a really good tool to use for log collection and analysis (and that's oversimplifying it, I believe it can even do event correlation...), which really made my love for data mining go crazy of late:P Best part is that it has a perpetual free license, nice!
One of the things I encountered when using Splunk was that it didn't seem to be indexing all the log files that it was set to monitor. After some reading up and experimenting the reason became clear: Splunk will not work properly if you set it to monitor too many files.
How many is too many? For example, setting it to monitor a logfile directory which only has one active log and 100+++ rotated logs, is too many. What should be done instead is to set it to monitor the active logfile only, and use oneshot adding of the other logfiles to the index you want.
Gonna do some more sharing/writeups about this crazily great tool. There's really a lot that this thing can do man.