Proper setting up and regular monitoring of logs gives you the avenue to know what's really happening with your box sitting out there in the internets, and to anticipate when bad things are about to happen.  One of the warning signs would be that someone has been poking around your box, looking for an (easy?) way in.

The natural thing that would jump out at you then, is that this someone has been accessing your box in far higher volumes/durations, especially on services that should not be accessed by others.

This is one example of such accesses on a linux box: SSHD brute forcing over long periods of time.

Note: This post is more to talk about the process of digging/profiling, rather than the actual setup processes/log sources involved.  Feel free to ping me/comment below if you wish to discuss though.

The first thing you may ask is: what is "persistent"?  This would be the opposite of the run-of-the-mill opportunistic attackers.  These guys tend to bang your machine for a bit, then leave you alone immediately after failing:

Opportunistic attack: Tries and gives up.

This contrasts greatly with the persistent buggers:

Whoa!

After digging around first on the IP and supposed country of origin, we want to find out what did the attacker try to do?  One of the logs (*cough*... p0f... *cough*...) feeds info on the ports that were attempted to connect to, this could be a starting point:

Mostly port 22 (SSH), only 1 for port 80 (HTTP)?

Searching for and viewing the port 80 access attempt, by itself and in relation to the other activities shows the following:

Pinpointing the port 80 connection

Viewing the logs in chronological order (Splunk defaults to reverse chronological)

Viewing the logs in chronological order (Splunk defaults to reverse chronological) shows that the port 80 connection preceeded the many many many port 22 connections by 2 minutes.  What's going on here?  If somebody wanted to get at the SSH accounts, why not go for them straight, rather than accessing the web service only once?  Checking the web access logs might give the answer we're looking for:

So in that TCP/80 connection....NOTHING was retrieved

Accessing nothing in that (only) one connection makes this look like a ping of sorts, but we can't be certain.

The next thing is to look at what this somebody was doing over the past two weeks!  First we get an idea of the kinds of things that were happening:

Mostly "Attempts to login using a non-existent user", ala our dear Mr Force, Brute Force

...and "SSH scan"

What do these SSH scans mean?

Just means that the SSH handshake was not properly done/completed.

Since we already know that this is a brute force attempt, judging by the frequency of the failed SSH handshakes per day we can assume for now that they're just resulting from either the connections being blocked, or just "normal" failures in the midst of thousands of attempts.  More can be done to confirm this by zooming into the times where these errors occur, but let's say we're not interested in confirming this fact for now.

Looking at the nature of the attack provides some clues on the tools being used too.  For that we extract some stats concerning the tool's attack:

Extracting and counting targeted SSH userids show that 473 userids are attempted in a range from 1 to 21 times each

First occurrence of each targeted userid is spread out fairly evenly throughout the time period...

...and last occurrences of each userid being fairly even throughout too.

More stats would be needed depending on the theory you're trying to prove/disprove, but you get the picture.

One of the things I usually would want to see is the list of userids used to brute force.  In this case, it looks like a predominantly Japanese/Chinese wordlist/namelist being used.  Interesting.

Am I Japanese? Am I Chinese?

Maybe I should start blogging in other languages to see what kind of brute force wordlists turn up

For now, in any case, 122.166.127.116 (abts-kk-static-116.127.166.122.airtelbroadband.in), I AM WATCHING YOU.

Obligatory blog post graphic, to make this more "interesting" Meanwhile, check out the really nice Tonido Plug at http://www.tonidoplug.com/

My internet connection goes down periodically, and I used to have to power cycle the router in order to fix that.  When it started to become too frequent it posed a problem, since I'm too lazy to keep going to the room (my wife too) to restart it.  There's also the option of restarting the router via the web admin interface, but it required me to login, click to the page for restarting, and click "restart"!  Very complicated indeed for lazy people.

Inspired by this hack (Hack a Day) where the guy automated the physical power cycling process, I decided to automate mine too.  Since I have a Tonido plug which is almost always on, and I've just learnt Python too, I decided to go the scripting method.  As they say: to a man with a hammer, everything looks like a nail

A couple of lessons learnt

I was caught by surprise by when reproducing the login and restart sequence exactly didn't work, and I went so far as to reproduce ALL the requests made by a "normal human".  It turned out (after 2 hours and a shower break) that things worked just fine when I simply converted the minimally needed POST parameters to GET parameters.  Nice classic web application hacking trick learnt from my old job as a web application ethical hacker I'd say.

Also, the restart sequence for my router turned out to not only need the form "POST" to request a restart, but also a subsequent request for the "restarting now" status page, interesting...

Download

Note that before you use this, some reverse engineering of the web application calls is needed, and some Python coding too.  You have been forewarned!  Also, I'm not responsible for this script causing you direct/indirect damage in any way, so don't come crying when your lawnmower starts to act crazy because you installed this script.  The script is released under the GPL, and can be downloaded here.

How to install/use

• Edit 'router_host': '10.0.0.1', in line 8
• Reverse engineer the web admin login and restart sequence, see what you need.  I used tools like a transparent proxy (Burp Suite), notepad and some brain grease.
• Hack the restart_router() (lines 43-73) function in the python script according to your needs (you're on your own here...  Alternatively you could offer me a good amount of Coke/chips for me to help you with the reverse engineering/coding somehow )
• Copy into the Tonido plug's /root directory (assume running as root, for simplicity's sake)
• SSH into the Tonido plug as root
• # chmod 400 /root/internet_connection_monitor.py
• # crontab -e
• Add in this line: (makes the script run in the background, 4 minutes after every tonido plug reboot to give the router time to start up)
  @reboot sleep 4m && /usr/bin/python /root/internet_connection_monitor.py &
• Press Alt-X, then "y" to save the new crontab
• Reboot the Tonido plug
• Profit!

What are the risks to note

The script basically is a hardcoded piece of info revealing the password and sequence to your login/router's workings! Make sure the script is chmod'ed properly, and isn't accessible via Tonido's interfaces.  For me I don't have this problem, since I don't allow connecting to my Tonido from outside anyway, and people will have to brute force ssh public keys to get in...

Have fun!

(Note to self: get the raw data with fields in the order that you want where possible/faster, rather than pumping it through sed.  Makes for good practice though.)

Using the csv file containing userids (visualized in yellow) and IPs (visualized in green) over the past few months from Splunk, here're the results of some of the experiments.

Oh, for the Windows users, you can use type instead of cat

First test using GraphViz's neato to layout:

perl afterglow.pl -b 1 -i <infile> -c color.properties -t | neato -Tgif -o output.gif

Huge, but better visualized with -e 5 option (Resulting image for that is too huge to upload though ). Note the single IP in the middle (the yellow explosion) that had been trying a LOT of userids to date.

Second test using fdp:

perl afterglow.pl -b 1 -i <infile> -c color.properties -t | fdp -Tgif -o output.gif

fdp doesn't seem to be well suited for this

Third test using sfdp:

No command here, you should have noticed the pattern from the first two...

_even_ less suited for this type of data...

Last test using twopi:

According to the GraphViz site, twopi's more suited for visualizing stuff like telecommunications flows.

twopi

(This post is pretty unpolished, partly because I can't be bothered to fiddle around with fitting the search strings into the width of the post, etc.  Nonetheless,  comments/discussions are always welcome heh)

One of my favourite tasks with log analysis is to get information on those people/bots which are brute forcing SSHD, so let's start with SSH attacks as an example.

Prerequisites

Before we start off, we'll need Splunk setup to be monitoring the appropriate logfiles.  I configured and run the OSSEC and Linux apps for Splunk, so that the data inputs are taken care of for me.  If you don't want to run these apps, just make sure you index the /var/log and OSSEC alert logs locations.  If you want to do the geolocation stuff the the MaxMind app for Splunk would be needed too.

List of SSH attacks

Let's start off with a simple query to see the list of previous SSH attacks:

source=*auth* sshd invalid user from

Using this search string with the needed time range set shows a pretty graph of how many attacks we've got over time, along with the list of log entries for the attack.

Click to enlarge

Seems that the attacks everyday are few, probably due to OSSEC's active responses.  A quick search would confirm that OSSEC is blocking the offending hosts.

sourcetype="ossec_alerts" 

action="SSHD brute force trying to get access to the system."

Click to enlarge

Drilling Down

Now we know that the attacks were especially active on the 22nd Feb, and OSSEC was responding correctly by blocking them off.  Why the large numbers then?  Was it because the attacks were from different IP addresses, or that that IP address was particularly persistent that day?  We could find out by getting more information on the src_ips for the time range in question.  First we click on the bar for the 22nd Feb, then the src_ip field in the sidebar.

Click to enlarge

With the time range fixed onto what we're interested in looking at, and the src_ip field showing the unique source IPs that were blocked, the results show that it was most likely a persistent attack by these two IPs.  A quick check with the auth logs tell the same story:

Click to enlarge

GeoIP Lookups

Now that we know which two IPs were actively poking around, let's map them to a location.  The MaxMind app for Splunk helps nicely for this task.

source=*auth* sshd invalid user from | 

lookup geoip clientip as src_ip

Click to enlarge

The app and local geoip database does the lookups for us nicely, mapping to geolocation information like country, city, latitude, longtidue and region.  The country information is available for most/all at least, the rest would be put in if available it seems.

List/Count of attacked userids for SSH

The strings for searching for this depends on your SSHD config, but for me searching for the invalid users is enough.

source=*auth* sshd invalid user from | 

rex field=_raw "Invalid user (?<atk_user_id>\S+) from "

Searching/sorting by the atk_user_id field would show us the attacked userids.  Click on the "Events Table" button to show the table of results with only the fields that you've selected.

Click to enlarge

If we want a sorted list of the top attacked userids, pipe the search string to a top command.

source=*auth* sshd invalid user from | rex field=_raw 

"Invalid user (?<atk_user_id>\S+) from "

 | top atk_user_id limit=1000

The Results Table should show automatically for this search.

Click to enlarge

Maybe we'd like an alphabetical list instead, so we just pipe the search to a sort command:

source=*auth* sshd invalid user from | rex field=_raw 

"Invalid user (?<atk_user_id>\S+) from "

 | top atk_user_id limit=1000 | sort atk_user_id

Click to enlarge

Alright, that's all for now

Today's feature: the list of user IDs that has been used to attempt brute forcing on ssh till date! *drum roll*

From the looks of this list, some of these people/botnet operators think I'm German/Spanish/Japanese.  Really weird, or these botnets are just whacking away without using the correct wordlist.

00089 0123456789 a aa aaa aaaa aaaaa aaaaaa aaaaaaa aaaaaaaa Aadolf Aaliyah Aamu Aapeli aaron Aaron aarti abby abcs abel admin administrator adolfo agata alberto alexandre alexis alias amministratore ana andrew angel anthony anti art arthur backuppc bang bb benjie bind bob bond brian caja cameron candie candy carey cargan carina carissa carl carla carlo carrie cgarcia cgi-bin cis42 cisco clement conter coo cristi cristian cristina cristinel cs cvsroot da damian dasusr1 dati dave db2fenc1 db2inst1 desiree director djeli dk dke dl dle dm dmaac dme dmitra documenti domin Doo doris dragon droguri ebony ecampaig echo ed enzo fax fedora felipe fido finance foc francois ftp ftpuser gary ghost goncalo grant gt05 guest haiduc haitac hammer happiness hugo iasiasur ibiza information informix ionita ipbx jay jd jean joan johan joomla joseluis julius julius123 jun jurca kato kidskhan li71-183 li71-183.members.linode library lord ls lschmidt lscsymbiosis lsnoxell lucas lucia m magnos marian mark marketing marta mathis medina mercedes miguel mike miranda mireya mlmb monica montrelle myky mythtv nagios nana natalia natasha nathan nelson nicoara nlopez no nrg nu office offsite operatore oracle owen pamela pgsl plcmspip PlcmSpIp porno pos post postgres power powered prchal prueba pubblico public q1 r00t raimundo ram reboot recepcion recruit rene ricardo roby rocio Root root123 roto ruut sales samba sami scan se sebastian services sims sims2 sistema skbae skin skipe skype skywalker slayer spam sshadmin sshdu sss staff stan std015 stephanie stone stud student student1 sue swadok sybille teamspeak TeamSpeak teapa tech ted telegest temp test test1 theo thomas thx1138 tom tomcat tony trash ts tss upload user user1 utente ven vh vic vicky victor violet vn volume vova webadmin wen william WinD3str0y work xwang xwp yamazaki yes zoro

Some might say that a SSH blacklist daemon might help, but it only increases the time taken for a brute force attempt, and is of no use against a botnet trying to brute force the ssh login.

There are plenty of things that can be done to lock down the ssh server, and restricting it to only publickey is by far one of the most effective, counting that the resource (the server) you're protecting is pretty important.

Plenty of interesting IPs/hosts in this list, take a look if you're really interested, heh.

reverse mapping checking getaddrinfo for 93.184.69.3.vnet.sk [93.184.69.3] failed - POSSIBLE BREAK-IN ATTEMPT! : 237 time(s)
reverse mapping checking getaddrinfo for 95-128-245-59.wiseweb.ru [95.128.245.59] failed - POSSIBLE BREAK-IN ATTEMPT! : 567 time(s)
reverse mapping checking getaddrinfo for h-69-3-215-11-static.lsanca54.covad.net [69.3.215.11] failed - POSSIBLE BREAK-IN ATTEMPT! : 543 time(s)
reverse mapping checking getaddrinfo for iodc-74-206-96-142.ioconnect.net [74.206.96.142] failed - POSSIBLE BREAK-IN ATTEMPT! : 6 time(s)
reverse mapping checking getaddrinfo for 202-153-191-246-static.unigate.net.tw [202.153.191.246] failed - POSSIBLE BREAK-IN ATTEMPT! : 5 time(s)
reverse mapping checking getaddrinfo for corporat065-167059038.sta.etb.net.co [65.167.59.38] failed - POSSIBLE BREAK-IN ATTEMPT! : 19 time(s)
reverse mapping checking getaddrinfo for ev1s-75-125-43-50.theplanet.com [75.125.43.50] failed - POSSIBLE BREAK-IN ATTEMPT! : 46 time(s)
reverse mapping checking getaddrinfo for hst13.migrateplans.com [72.46.131.181] failed - POSSIBLE BREAK-IN ATTEMPT! : 68 time(s)
reverse mapping checking getaddrinfo for bzq-179-135-183.static.bezeqint.net [212.179.135.183] failed - POSSIBLE BREAK-IN ATTEMPT! : 298 time(s)
reverse mapping checking getaddrinfo for host112163.metrored.net.mx [200.77.249.163] failed - POSSIBLE BREAK-IN ATTEMPT! : 8 time(s)
Address 98.126.208.50 maps to customer.krypt.com, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! : 50 time(s)
reverse mapping checking getaddrinfo for corporat200-7543230.sta.etb.net.co [200.75.43.230] failed - POSSIBLE BREAK-IN ATTEMPT! : 97 time(s)
Address 61.168.44.5 maps to pc5.zz.ha.cn, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! : 50 time(s)
reverse mapping checking getaddrinfo for ip36.70.inetmar.com [92.42.36.70] failed - POSSIBLE BREAK-IN ATTEMPT! : 50 time(s)
Address 218.28.20.135 maps to pc0.zz.ha.cn, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! : 168 time(s)
reverse mapping checking getaddrinfo for 187-5-142-129.bnut3700.e.brasiltelecom.net.br [187.5.142.129] failed - POSSIBLE BREAK-IN ATTEMPT! : 478 time(s)
reverse mapping checking getaddrinfo for cliente-13108.iberbanda.es [82.198.115.50] failed - POSSIBLE BREAK-IN ATTEMPT! : 324 time(s)
reverse mapping checking getaddrinfo for host-203-92-76-19.lga.net.sg [203.92.76.19] failed - POSSIBLE BREAK-IN ATTEMPT! : 5 time(s)
reverse mapping checking getaddrinfo for 229.1.163.220.broad.km.yn.dynamic.163data.com.cn [220.163.1.229] failed - POSSIBLE BREAK-IN ATTEMPT! : 240 time(s)
reverse mapping checking getaddrinfo for 56h29.xjtu.edu.cn [202.117.56.29] failed - POSSIBLE BREAK-IN ATTEMPT! : 54 time(s)
reverse mapping checking getaddrinfo for 202.53.76.24.nettlinx.com [202.53.76.24] failed - POSSIBLE BREAK-IN ATTEMPT! : 45 time(s)
Address 218.28.103.202 maps to pc0.zz.ha.cn, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! : 373 time(s)
Address 72.9.228.73 maps to marisil.org, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! : 6 time(s)
Address 72.9.228.73 maps to marisil.org, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! : 6 time(s)
reverse mapping checking getaddrinfo for 74.126.30.110.static.a2webhosting.com [74.126.30.110] failed - POSSIBLE BREAK-IN ATTEMPT! : 15 time(s)
reverse mapping checking getaddrinfo for 74.126.30.110.static.a2webhosting.com [74.126.30.110] failed - POSSIBLE BREAK-IN ATTEMPT! : 15 time(s)

But hey, stop wasting your time and giving me entertainment like this already, you MUST have the correct private key in order to connect via SSH. Slightly more inconvenient than using passwords, but it takes a load off my mind knowing that brute forcing passwords is simply impossible, and it would take the SSH keys to be compromised in order to get in (which won't be worth the cost/effort anyway).

Maybe I will start listing these jokers in a wall of shame or something. Maybe.