Highly Predictive Blacklists
SANS Internet Storm Center has a service for DShield log contributors called HPBs (Highly Predictive Blacklists). Since their summary is succinct enough I will just quote it here:
DShield.org in collaboration with SRI International has established a new experimental custom source address blacklist generation service available to all DShield.org contributors. This new service utilizes a radically different approach to blacklist formulation called Highly Predictive Blacklisting. Each DShield contributor can now access a unique HPB (instructions below) that reflects the most probable set of source addresses that will connect to that contributor's network over a prediction window that may last several days into the future.
Highly predictive blacklists employ a link analysis algorithm similar to Google's PageRank scheme used to find the most relevant web pages given a user's query. Similar to a web query, DShield contributor's firewall logs are cross-compared in search of overlaps among the attackers they report. Each attacker address that is included in an HPB is selected by favoring those ad-dresses that are encountered by other contributors that share degrees of overlap with the HPB owner.
How does it work (for non math geeks
): We compare your firewall logs to firewall logs submitted by others. If you and other submitters are hit on similar ports, then your are more likely to be attacked by the same IPs. Your personal "HPB" is created from the IP addresses that target submitters with similar reports as you.
While this is directly useful to firewall administrators, the concept could potentially be extended to other domains/uses too. Filing this under "bag of tricks" for now
Got Spam?

I want spam!
Looking for samples of a particular type of spam, please help!
The kind of spam I'm looking for is a specific type, so please read on first!
The spam mail has to fulfil these criteria:
- Sent from a free webmail provider (Gmail, Yahoo, Hotmail/MSN, etc).
- Sent from an email address/friend/person that you know (i.e. they likely have your email address in their address book).
- Has a "blank" email subject (this shows up as "(no subject)" in Gmail for example).
- The body/content of the email appears to contain only one link, or very little content.
If you have received such emails, you can help by sending the headers and "raw" content to me. Both usually can be obtained when viewing the email headers (Gmail example: click on the down arrow at the top right corner of the email, select "Show original"). Just copy and paste the entire headers and content into a text file and send to spamcollector@rayfoo.info as an attachment.
If you're afraid that I might misuse the email addresses (don't worry, I'm not a spammer...
), feel free to obfuscate the email addresses before sending it out to me. I only request that you at least indicate the webmail domain portion of the email address that sent you that spam mail.
Any clarifications, please post in the comments section below. Thanks a lot!
Dynamic conversion of epoch timestamps in logs
In the course of your logs or text processing, you may come across certain timestamps in epoch format. Whilst there's always online resources to assist with the conversion of such timestamps, it may not be the best way if you need to keep the timestamp "secret" during then, or if you have many timestamps to convert going by the thousands, millions, etc.
Whilst there's always free tools like Splunk which is available for free to the masses (and yes, it does automatically convert epoch timestamps for you), there's always our "humble" awk.

