Proper setting up and regular monitoring of logs gives you the avenue to know what's really happening with your box sitting out there in the internets, and to anticipate when bad things are about to happen.  One of the warning signs would be that someone has been poking around your box, looking for an (easy?) way in.

The natural thing that would jump out at you then, is that this someone has been accessing your box in far higher volumes/durations, especially on services that should not be accessed by others.

This is one example of such accesses on a linux box: SSHD brute forcing over long periods of time.

Note: This post is more to talk about the process of digging/profiling, rather than the actual setup processes/log sources involved.  Feel free to ping me/comment below if you wish to discuss though.

The first thing you may ask is: what is "persistent"?  This would be the opposite of the run-of-the-mill opportunistic attackers.  These guys tend to bang your machine for a bit, then leave you alone immediately after failing:

Opportunistic attack: Tries and gives up.

This contrasts greatly with the persistent buggers:

Whoa!

After digging around first on the IP and supposed country of origin, we want to find out what did the attacker try to do?  One of the logs (*cough*... p0f... *cough*...) feeds info on the ports that were attempted to connect to, this could be a starting point:

Mostly port 22 (SSH), only 1 for port 80 (HTTP)?

Searching for and viewing the port 80 access attempt, by itself and in relation to the other activities shows the following:

Pinpointing the port 80 connection

Viewing the logs in chronological order (Splunk defaults to reverse chronological)

Viewing the logs in chronological order (Splunk defaults to reverse chronological) shows that the port 80 connection preceeded the many many many port 22 connections by 2 minutes.  What's going on here?  If somebody wanted to get at the SSH accounts, why not go for them straight, rather than accessing the web service only once?  Checking the web access logs might give the answer we're looking for:

So in that TCP/80 connection....NOTHING was retrieved

Accessing nothing in that (only) one connection makes this look like a ping of sorts, but we can't be certain.

The next thing is to look at what this somebody was doing over the past two weeks!  First we get an idea of the kinds of things that were happening:

Mostly "Attempts to login using a non-existent user", ala our dear Mr Force, Brute Force

...and "SSH scan"

What do these SSH scans mean?

Just means that the SSH handshake was not properly done/completed.

Since we already know that this is a brute force attempt, judging by the frequency of the failed SSH handshakes per day we can assume for now that they're just resulting from either the connections being blocked, or just "normal" failures in the midst of thousands of attempts.  More can be done to confirm this by zooming into the times where these errors occur, but let's say we're not interested in confirming this fact for now.

Looking at the nature of the attack provides some clues on the tools being used too.  For that we extract some stats concerning the tool's attack:

Extracting and counting targeted SSH userids show that 473 userids are attempted in a range from 1 to 21 times each

First occurrence of each targeted userid is spread out fairly evenly throughout the time period...

...and last occurrences of each userid being fairly even throughout too.

More stats would be needed depending on the theory you're trying to prove/disprove, but you get the picture.

One of the things I usually would want to see is the list of userids used to brute force.  In this case, it looks like a predominantly Japanese/Chinese wordlist/namelist being used.  Interesting.

Am I Japanese? Am I Chinese?

Maybe I should start blogging in other languages to see what kind of brute force wordlists turn up

For now, in any case, 122.166.127.116 (abts-kk-static-116.127.166.122.airtelbroadband.in), I AM WATCHING YOU.

One of the things I encountered when using Splunk was that it didn't seem to be indexing all the log files that it was set to monitor.  After some reading up and experimenting the reason became clear: Splunk will not work properly if you set it to monitor too many files.

How many is too many?  For example, setting it to monitor a logfile directory which only has one active log and 100+++ rotated logs, is too many.  What should be done instead is to set it to monitor the active logfile only, and use oneshot adding of the other logfiles to the index you want.

Gonna do some more sharing/writeups about this crazily great tool.  There's really a lot that this thing can do man.