Proper setting up and regular monitoring of logs gives you the avenue to know what's really happening with your box sitting out there in the internets, and to anticipate when bad things are about to happen.  One of the warning signs would be that someone has been poking around your box, looking for an (easy?) way in.

The natural thing that would jump out at you then, is that this someone has been accessing your box in far higher volumes/durations, especially on services that should not be accessed by others.

This is one example of such accesses on a linux box: SSHD brute forcing over long periods of time.

Note: This post is more to talk about the process of digging/profiling, rather than the actual setup processes/log sources involved.  Feel free to ping me/comment below if you wish to discuss though.

The first thing you may ask is: what is "persistent"?  This would be the opposite of the run-of-the-mill opportunistic attackers.  These guys tend to bang your machine for a bit, then leave you alone immediately after failing:

Opportunistic attack: Tries and gives up.

This contrasts greatly with the persistent buggers:

Whoa!

After digging around first on the IP and supposed country of origin, we want to find out what did the attacker try to do?  One of the logs (*cough*... p0f... *cough*...) feeds info on the ports that were attempted to connect to, this could be a starting point:

Mostly port 22 (SSH), only 1 for port 80 (HTTP)?

Searching for and viewing the port 80 access attempt, by itself and in relation to the other activities shows the following:

Pinpointing the port 80 connection

Viewing the logs in chronological order (Splunk defaults to reverse chronological)

Viewing the logs in chronological order (Splunk defaults to reverse chronological) shows that the port 80 connection preceeded the many many many port 22 connections by 2 minutes.  What's going on here?  If somebody wanted to get at the SSH accounts, why not go for them straight, rather than accessing the web service only once?  Checking the web access logs might give the answer we're looking for:

So in that TCP/80 connection....NOTHING was retrieved

Accessing nothing in that (only) one connection makes this look like a ping of sorts, but we can't be certain.

The next thing is to look at what this somebody was doing over the past two weeks!  First we get an idea of the kinds of things that were happening:

Mostly "Attempts to login using a non-existent user", ala our dear Mr Force, Brute Force

...and "SSH scan"

What do these SSH scans mean?

Just means that the SSH handshake was not properly done/completed.

Since we already know that this is a brute force attempt, judging by the frequency of the failed SSH handshakes per day we can assume for now that they're just resulting from either the connections being blocked, or just "normal" failures in the midst of thousands of attempts.  More can be done to confirm this by zooming into the times where these errors occur, but let's say we're not interested in confirming this fact for now.

Looking at the nature of the attack provides some clues on the tools being used too.  For that we extract some stats concerning the tool's attack:

Extracting and counting targeted SSH userids show that 473 userids are attempted in a range from 1 to 21 times each

First occurrence of each targeted userid is spread out fairly evenly throughout the time period...

...and last occurrences of each userid being fairly even throughout too.

More stats would be needed depending on the theory you're trying to prove/disprove, but you get the picture.

One of the things I usually would want to see is the list of userids used to brute force.  In this case, it looks like a predominantly Japanese/Chinese wordlist/namelist being used.  Interesting.

Am I Japanese? Am I Chinese?

Maybe I should start blogging in other languages to see what kind of brute force wordlists turn up

For now, in any case, 122.166.127.116 (abts-kk-static-116.127.166.122.airtelbroadband.in), I AM WATCHING YOU.

Amongst the new features that Splunk's advertising, a quick glance through the new version reveals that the revamped management interface might seem to make administering it/clusters easier. Also that the search and reporting features seem to have been beefed up too!

More to come after I poke around some more, and if I have the time to write something

Fired up Splunk to do a quick search over the past 7 days:

index=myblogindex | dedup useragent | fields useragent | sort useragent | format

The resulting string can be easily copied and massaged further in a text editor (replacing the "in between" strings like " ) OR ( useragent=" with \n)

I'm pretty interested still (as always) to see how easy it is to profile/"follow" an individual user due to uniqueness of each OS-browser's useragent (UA) strings, but that's another story for another exercise, another day...

Here're some of the more interesting UA strings and analyses. And these were harvested only over a span of 7 days!

BlackBerry9530/5.0.0.732 Profile/MIDP-2.1 Configuration/CLDC-1.1 VendorID/105

SonyEricssonC905/R1FA Browser/NetFront/3.4 Profile/MIDP-2.1 Configuration/CLDC-1.1 JavaPlatform/JP-8.4.3

T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240;) MSNBOT-MOBILE/1.1 (+http://search.msn.com/msnbot.htm)

Love it when I see mobile browsers' UA strings, wonder how much further could I dig into them in the future...

Flight Deck Bot 1.3 beta (http://www.flightdeckreports.com/bot)

Flight Deck's a game that I recently restarted my tactics experiments with, wonder how exactly did they hit my site? No referrers sent with the requests, but I suspect they came via Twitter.  Or was it even the same Flight Deck site?  Too lazy to dig further for now

Mozilla/4.0 (PSP (PlayStation Portable); 2.00)

PSP...?

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; sbcydsl 3.12; YComp 5.0.0.0; YPC 3.2.0; FunWebProducts; .NET CLR 1.1.4322; ZangoToolbar 4.8.2; yplus 5.1.04b)

Interesting to see how many people have installed adware/spyware like FunWebProducts. There're other examples in my logs too of such malware that modify the UA string, which makes it possible to do detection and statistics in perimeter devices like IDSes...

Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_1_2 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Mobile/7D11

Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_1_3 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Mobile/7E18

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Mobile/8A306

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A306 Safari/6531.22.7

Mozilla/5.0 (iPod; U; CPU iPhone OS 3_1_3 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7E18 Safari/528.16

Mozilla/5.0 (iPod; U; CPU iPhone OS 3_1_3 like Mac OS X; nl-nl) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7E18 Safari/528.16

iPhones/iPods/iWhatNot. OS AND browser versions all revealed! Now, how about some "automatic" "jailbreaking"? Heh heh heh...not!

SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

Googlebot using SAMSUNG phones?! Either Google has some wicked architecture to incorporate mobile phones as crawlers, or that this is a very confused bot

Wget/1.12 (linux-gnu)

Wget/1.9+cvs-stable (Red Hat modified)

curl/7.18.2 (i386-pc-win32) libcurl/7.18.2 zlib/1.2.3

curl/7.19.6 (i386-pc-win32) libcurl/7.19.6 OpenSSL/0.9.8k zlib/1.2.3

When you see your site being accessed by programs like wget and curl, and it's not Amazon's AWS (use Splunk's lookup dnslookup clientip to find out the clienthost name), it's a very safe bet that they're zombies/compromised user computers as part of a botnet. The clienthost names and many different IP addresses would confirm that they're zombies.

Well, that's all for today folks! Feel free to comment/discuss below

Setup of the p0f and logging is the same as in the OS Profiling post.

The Splunk search string has been extended to extract the source's internet link as a field too (go for the portion in bold for the field extracting commands):

| file /home/path/to/p0f.log | rex field=_raw "> (?<srcip>[^:]+):(?<srcport>[^ ]+) – (?<srcos>.+?) \(" | rex field=_raw "-> (?<dstip>[^:]+):(?<dstport>[^ ]+) " | rex field=_raw "link: (?<srclink>.*)\)$" |  regex srclink!="(unspecified|unknown)" | top limit=0 srclink

The fields that I extract with this:

• srcip -> source IP
• srcport -> source TCP port
• srcos -> source's OS (woot!)
• dstip -> destination IP (which is my machine's)
• dstport -> the destination port which the TCP connection was initiated to
• srclink -> the internet link of the source machine

After filtering out the "unspecified" and "unknown" links, the list of the detected links are as follows:

"ethernet/modem" points to mostly cable connections.  There're some interesting entries in the list though, like vtun, pppoe, Google/AOL, IPv6/IPIP (early adopters? haha).  Don't have any idea on what's IPSec/GRE, or vLAN here in this context though.

Just for the heck of it, here's the chart for this table, generated from the reports link in Splunk.

I like the charts, because they allow some interaction with the charts for simple datasets, but I digress

p0f allows you to determine the OS of the remote machine based on the TCP fields characteristics.  It can also tell whether the machine is behind a firewall, what kind of internet connection it is running from...pretty useful for information junkies like me

Here's what I did:

./p0f -t -u MyUseridHere -i eth0 'src not MyIPAddressHere' | tee -a p0f.log

Runs p0f, logging with actual timestamps (-t), chroot and setuid to MyUserIdHere (-u), listening on eth0 (-i), and filtering out packets for connections initiated from my machine itself (since I'm not interested in profiling my own machine).

tee is a (really nifty!) linux command.  What it does is to "split" the input (stdin) to two parts: stdout and the file specified.  The -a option tells it to append to the file instead of overwriting it.

Using this, p0f outputs logs like this one:

<Sat Jul  3 07:03:56 2010> 175.40.12.47:1095 - Windows 2000 SP2+, XP SP1+ (seldom 98)
-> 74.207.229.183:80 (distance 12, link: sometimes DSL (2))

One of the Splunk queries that I poked around with:

| file /path/to/p0f.log | rex field=_raw "> (?<srcip>[^:]+):(?<srcport>[^ ]+) - (?<srcos>.+?) \(" | rex field=_raw "-> (?<dstip>[^:]+):(?<dstport>[^ ]+) " | regex srcos!="UNKNOWN" | top limit=0 srcos

This query extracts out the source and destination IP and port, and the source OS.  Then after filtering out the OS tagged with UNKNOWN, the remaining entries are ranked...

The resulting chart, of not much real interest by itself, just shows that other than that the connections are predominantly from linux machines (hurhur), and there's a connection from a really old Netware machine (5 was released in Oct 1998!).

(Note to self: get the raw data with fields in the order that you want where possible/faster, rather than pumping it through sed.  Makes for good practice though.)

Using the csv file containing userids (visualized in yellow) and IPs (visualized in green) over the past few months from Splunk, here're the results of some of the experiments.

Oh, for the Windows users, you can use type instead of cat

First test using GraphViz's neato to layout:

perl afterglow.pl -b 1 -i <infile> -c color.properties -t | neato -Tgif -o output.gif

Huge, but better visualized with -e 5 option (Resulting image for that is too huge to upload though ). Note the single IP in the middle (the yellow explosion) that had been trying a LOT of userids to date.

Second test using fdp:

perl afterglow.pl -b 1 -i <infile> -c color.properties -t | fdp -Tgif -o output.gif

fdp doesn't seem to be well suited for this

Third test using sfdp:

No command here, you should have noticed the pattern from the first two...

_even_ less suited for this type of data...

Last test using twopi:

According to the GraphViz site, twopi's more suited for visualizing stuff like telecommunications flows.

twopi

1.---

This one is a Splunk query, run over the span of the last 7 days:

sourcetype="ossec_alerts" rule_number="5710"|

rex field=_raw "Invalid user (?<userid>[^ ]+) from"|

fields + src_ip,userid|fields - _*|

dedup src_ip userid|

outputcsv ssh-atk-attempts-userid-ip

2.---

Then some data massaging on the csv file...

[edit: this is not needed...just output the csv file with the fields in the order you want...and read the next post for better options with 2-column csv inputs]

cat ssh-atk-attempts-userid-ip.csv | \

sed 's/^.*$/&,server/' > ssh-atk-attempts-userid-ip2.csv

3.---

Then running it thru Afterglow and GraphViz's neato...

cat ssh-atk-attempts-userid-ip2.csv | \

./afterglow.pl | neato -Tgif -o ssh-atk-ip-userid.gif

Seems like very little overlap in the userids that were attempted (with the exception of the few favourites like root, guest, test).  A coordinated/distributed attack perhaps?  Haven't dug more into the IPs in question, but I'm pretty sure that they'd be broadband addresses, meaning that they are bots.

Of course we could try with a larger timespan, but the result isn't really readable... The resulting 1MB file (1813 x 1704 px) for over all time in Splunk only looks pretty, and not readable.

[edit: there're better results in the next post!]

Some (or probably most/all) of your searches might involve public IP addresses, and more often than not we would want to have additional info along with the IP address to work with.

Three of the things that we could do in Splunk automatically would be to get IP-location info, or to reverse lookup an IP to a domain, or to lookup a domain to an IP.

1. Geolocation

There're two ways to do geolocating of IPs: using the iplocation command, or to use the MAXMIND app.

1a. iplocation

The command iplocation is described as:

Finds ips in _raw and looks up the IP location using the hostip.info database. IPs are extracted as ip1, ip2, etc. Cities and Countries are likewise extracted.

What we only need to do is to pipe the search to iplocation and let it do the rest!  The lookups are done from the server on the fly, so make sure that the server is able to do whois/ns lookups on the network.

index=myindex | iplocation

1b. MAXMIND app

Like previously mentioned before: install the MAXMIND app, then pipe the field containing IPs to the lookup (the field name must be clientip, if not this will not work duh)

This can work with the server not having any internet connectivity, but the accuracy is entirely dependant on the cached MAXMIND database.

index=myindex | lookup geoip clientip

or

index=myindex2 | lookup geoip clientip as fieldwithip

2, 3. IP-hostname or hostname-IP

These two items are pretty similar.  Spunk 4 comes with a lookup script called external_lookup.py, and the config is already in the default transforms.conf.  So we only need to use it!

Resolving IPs to hostnames:

index=myindex | lookup dnslookup clientip

Resolving hostnames to IPs:

index=myindex | lookup dnslookup clienthost

(no screenshot, sorry )

Leave a comment if this helped, or if you want to ask anything!

(This post is pretty unpolished, partly because I can't be bothered to fiddle around with fitting the search strings into the width of the post, etc.  Nonetheless,  comments/discussions are always welcome heh)

One of my favourite tasks with log analysis is to get information on those people/bots which are brute forcing SSHD, so let's start with SSH attacks as an example.

Prerequisites

Before we start off, we'll need Splunk setup to be monitoring the appropriate logfiles.  I configured and run the OSSEC and Linux apps for Splunk, so that the data inputs are taken care of for me.  If you don't want to run these apps, just make sure you index the /var/log and OSSEC alert logs locations.  If you want to do the geolocation stuff the the MaxMind app for Splunk would be needed too.

List of SSH attacks

Let's start off with a simple query to see the list of previous SSH attacks:

source=*auth* sshd invalid user from

Using this search string with the needed time range set shows a pretty graph of how many attacks we've got over time, along with the list of log entries for the attack.

Click to enlarge

Seems that the attacks everyday are few, probably due to OSSEC's active responses.  A quick search would confirm that OSSEC is blocking the offending hosts.

sourcetype="ossec_alerts" 

action="SSHD brute force trying to get access to the system."

Click to enlarge

Drilling Down

Now we know that the attacks were especially active on the 22nd Feb, and OSSEC was responding correctly by blocking them off.  Why the large numbers then?  Was it because the attacks were from different IP addresses, or that that IP address was particularly persistent that day?  We could find out by getting more information on the src_ips for the time range in question.  First we click on the bar for the 22nd Feb, then the src_ip field in the sidebar.

Click to enlarge

With the time range fixed onto what we're interested in looking at, and the src_ip field showing the unique source IPs that were blocked, the results show that it was most likely a persistent attack by these two IPs.  A quick check with the auth logs tell the same story:

Click to enlarge

GeoIP Lookups

Now that we know which two IPs were actively poking around, let's map them to a location.  The MaxMind app for Splunk helps nicely for this task.

source=*auth* sshd invalid user from | 

lookup geoip clientip as src_ip

Click to enlarge

The app and local geoip database does the lookups for us nicely, mapping to geolocation information like country, city, latitude, longtidue and region.  The country information is available for most/all at least, the rest would be put in if available it seems.

List/Count of attacked userids for SSH

The strings for searching for this depends on your SSHD config, but for me searching for the invalid users is enough.

source=*auth* sshd invalid user from | 

rex field=_raw "Invalid user (?<atk_user_id>\S+) from "

Searching/sorting by the atk_user_id field would show us the attacked userids.  Click on the "Events Table" button to show the table of results with only the fields that you've selected.

Click to enlarge

If we want a sorted list of the top attacked userids, pipe the search string to a top command.

source=*auth* sshd invalid user from | rex field=_raw 

"Invalid user (?<atk_user_id>\S+) from "

 | top atk_user_id limit=1000

The Results Table should show automatically for this search.

Click to enlarge

Maybe we'd like an alphabetical list instead, so we just pipe the search to a sort command:

source=*auth* sshd invalid user from | rex field=_raw 

"Invalid user (?<atk_user_id>\S+) from "

 | top atk_user_id limit=1000 | sort atk_user_id

Click to enlarge

Alright, that's all for now

One of the things I encountered when using Splunk was that it didn't seem to be indexing all the log files that it was set to monitor.  After some reading up and experimenting the reason became clear: Splunk will not work properly if you set it to monitor too many files.

How many is too many?  For example, setting it to monitor a logfile directory which only has one active log and 100+++ rotated logs, is too many.  What should be done instead is to set it to monitor the active logfile only, and use oneshot adding of the other logfiles to the index you want.

Gonna do some more sharing/writeups about this crazily great tool.  There's really a lot that this thing can do man.