PulledPork 0.7.0 Released! #include <IP.Reputation>

PulledPork 0.7.0 - Swine Flu has been released and can be found at the PulledPork site.  There are numerous feature changes and enhancements that dramatically affect the functionality and capabilities of PulledPork since the last major 0.6.x release.  An excerpt of the changelog is at the bottom of this post and lists all of the changes/features/enhancements.   

The most significant change that you are likely to notice deals with how PulledPork now processes the rules tarball.  In the previous versions of PulledPork when you would run the application it would process the rules tarball as designated in your configuration, regardless of whether or not the source tarball had changed (no new rules tarball for example).  With the new changes the source rules tarball is ONLY processed if it is new/changed OR if you specify the -P runtime flag.  So for tuning exercises or out of band runs when the source tarball is unchanged, you MUST specify the -P flag for any processing to occur.

Inline with the new IP Reputation preprocessor that was introduced in Snort 2.9.1 we have included full support for this feature.  This support includes a couple of new configuration options that are located in the pulledpork.conf and allow for retrieval of multiple ip reputation lists (PulledPork will automatically de-dupe these lists).  If you are running Snort on Linux you are also able to specify at ./configure time an option to allow for in-memory reloading of IP Reputation lists, thus you do not have to SIGHUP or completely reload Snort.  This in-memory reload is accomplished by using a control socket that this version of PulledPork is capable of utilizing.

Working closely with the Barnyard2 team we have developed a new version of the data in the sid-msg.map.  This allows for more information to be included in intrusion events such as the revision of the rule (currently not included in alerts).  The default version is still version 1 of the sid-msg.map file, it is CRITICAL to note that only Barnyard 2.2+ supports this new version of the sid-msg.map file and as such ONLY when using this version or newer of Barnyard 2.2+ should you change this value in your pulledpork.conf.

When utilizing the default configuration that creates two single unified rules files (one for so_rules and one for text rules) the so_rules stub files are now included in the single rules tarball.  This means that you no longer need to include the so_rules.rules file.  This single rules file is now internally separated by category and rule type, or generator to allow for rapid rule location and more logical perusing of the file.

As per the usual, thank you for your continued support and usage of PulledPork and Snort.  Should you have any questions or concerns please feel free to file a bug report or new feature request at http://pulledpork.googlecode.com and also to participate in the community mailing list that can be found at http://groups.google.com/group/pulledpork-users

Bug Fixes:
- Bug #79 - Fixed race condition that did not allow for disabled rules to be modified using modifysid
These rules would then be enabled by flowbit dependency check and be unmodified
- Bug #77 - Adjusted chown property of archive::tar
- Bug #78 - Adjusted per bug report to allow for proper ignoring of preproc.rules
- Bug #102 - Only Enabled rules are written to sid-msg.map now when -E flag is specified
- Bug #99 - Doc Bug, updated docs associated with snort_version variable
- Bug #96 - Modified code to allow for same-line traling comments: "1:10011 #can haz disable!"
Also updated the rulestate files (enable,disable,drop)
- Bug #82 - Modified run order to force modifysid to run before all other sid state modification routines
This allows for sid changes to be made prior to automatic state determination ala automatic
flowbit resolution.  NOTE that this DOES NOT AND WILL NOT disable automatic flowbit
resolution, this is a critical piece.
- Bug #81 - Updated valid SO distro pre-compiled list
- Bug #114 - Update Regex to allow for null search/replace in modify_sid sub
- Unlisted Bug - Allow for escaped ; "\;" in references
- Bug #121 - Update to allow for new etpro.com url and cert!
- Bug #119 - Fixed regex [^\\], should have been negative look behind (?<!\\)
- Bug #120 - Updated proxy code for better support and proper runtime load order
- Unlisted Bug - Account for multiple flowbits that are separated using &| operators
(flowbits:isset,flowbit1&flowbit2;)(flowbits:isset,flowbit1|flowbit2;)
- Bug #126 - Removed Switch usage
- Bug #129 - Fixed to allow for -n usage (in conjunction with -P) when an ip list is used also
- Unlisted Bug - Fixed to allow for proper -P usage

New Features / changes:
- Bug #105 - Removed Switch function as it is deprecated in > 5.12 perl
- NEW - Added IP Reputation Preprocessor support
- NEW - Capability to use control socket for IP List reload
- NEW - -P runtime flag to (process even if there is no new rules tarball)**
- Bug #68 - Added basic surricata support
- Bug #115 - Single rules file now has category (and GID) separators
Correlating to this we have also removed the separate so_rules.rules file
All rules are now in a single snort.rules file unless the keep flag is
specified at runtime.
- NEW - Numerous sub rewrites to allow for better performance
- NEW - New sid-msg.map format for barnyard 2.2+ gid || sid || rev || class || pri || msg || @refs
- NEW - SO rule categories are now prepended with VRT-SO
- NEW - More advanced sid-msg.map structure (for use with by2.2+) and backward compatibility
This allows for better mapping of gid:sid:rev in the database!
- NEW - Rewrote the way that extraction is handled, to properly support a single rules tarball being
updated.  This includes how md5 validation is done and in what order.  If a single
file is updated then they are all extracted and processed.

The importance of PulledPork

Bottom line up front:  If you aren't using PulledPork, you are going to have a gigantic depreciation in functionality.

You've heard it said on the Snort lists, you've heard it on this blog, you've heard it on Twitter,  you've heard it from CNN...  okay, well, not CNN..

The importance of PulledPork.

The reason that we are very heavily insist that you are using pulledpork is primarily for two very big reasons:

1. Flowbit auto-resolution
2. Default Policy usage

• The first reason: Flowbit auto-resolution

I've written in great length about the need for flowbits, you can find a couple of my blog posts here
http://blog.snort.org/2011/05/resolving-flowbit-dependancies.html
and
http://blog.snort.org/2011/12/if-you-are-having-problems-with-your.html

The reason that I'm writing about it again, is that to refresh your memory back when we created the file-identify rule category:
http://blog.talosintel.com/2011/11/say-hello-to-file-identify-category.html

We outlined that part of this conversion was to move all flowbit names from their old names (such as http.gif) to a new format (now file.gif).  This project has now been completed and all flowbits across all files have been moved to the new format, and all the rules that "set" flowbits have been commented to be "off" by default and in no policies.

What we are relying on here is either if you are using PulledPork or the Sourcefire product, you either select your base policy (which I'll talk about in section two), or, even if you don't, PulledPork will auto-resolve the flowbit "set" names that you'll need and go through and turn those on for you.  That way you are only running rules that you need to have on based upon either the policy you are running, or the state of the rules.  We really insist that you use the disablesid and enablesid functionality in PulledPork to be able to turn the rules that you WANT for your environment on and let PulledPork auto-resolve all the dependancies you need for you.

• The second reason: Policy usage

There are four states that we place rules in when we create them, three of the states are assigned to policies.

1. Connectivity
2. Balanced
3. Security

The last state is "in no policies".

1. The first, connectivity, means "Connectivity over Security".  Meaning this is a speedy policy for people that insist on blocking only the really known bad with no false positives.  
2. The second, balanced, means "Balanced between Connectivity and Security".  Meaning that this is a good starter policy for everyone.  It's quick, has a good base coverage level, and covers the latest threats of the day.  The policy contains everything that is in Connectivity.
3. The third, security, means "Security over Connectivity".  Meaning that this is a stringent policy that everyone should strive to get to through tuning.  It's coverage is much more exhaustive, and has some policy-type rules in it.  Rules that will alert on Flash contained within an Excel file and things like that.  This policy contains everything that is in the first two.

The last state is "in no policies".  This means that we insist that you look through these by product name or CVE in order to turn them on.  These may not have a fast content match, could be false positive prone, or the vulnerability it is covering is not in a very prevalent piece of software.  

The way we make the decision about what is "on" or "off" by default when you aren't using the policies, is, if it's in balanced, it's on by default, it's it not in balanced, it's off by default.  There are a ton of exceptions to this rule, but this is the general rule of thumb.

Nick Moore's Webcast Slides are posted

This past week's webcast was not recorded, however, I have posted the slides for the presentation, and they are available at the Snort Webcast Series Link.

I am going to try and get future webcasts recorded for easy playback, sorry for any inconvenience.

Nick's webcast was on the basics of Snort tuning, a "101" class.  Nick talks about the importance of variables, rule tuning, and lots of other goodies.

Working with unified and unified2 files

First you might ask "why do I care about unified files?" (unified2 these days, more on that later)

There is no perfect answer to that, just good reasons you should. Some of those reasons are:

• It is a unified format allowing for easy archive management
• It is the fastest output method available for snort
• It is the preferred method used by Sourcefire and the most tested.
• It is not easily modified. (It is still possible however)

8 years ago if you needed to get data from a snort unified file and do something with it you had few options, there was barnyard and that was it. This is fine and for good reason people were using unified output spooling to disk, reading those files with barnyard, and storing them in a mysql database. In the typical use case this was a perfect solution that worked for many years for many people. This solution had one major drawback though, unified files could not be processed by wetware without barnyard.

Normally this wouldn't be a problem as barnyard supported csv, fast alerts, database, pcap, etc as output methods. While a pain to process a unified file and then post process the results from several different file formats, it was sufficient to get the job done and not too painful.

...Unless you don't have barnyard available. This is exactly the situation that forced me to write the first iteration of SnortUnified.pm. I was working on a system that had unified files but no barnyard, no database, no compiler. Normally this isn't a problem either, you just install the tools you need but that was not allowed. As a matter of fact it was forbidden as these tools were not "certified" for use in production environments and the data was not allowed to leave this environment. The absurdity of that situation, having the data but no way to use it, by policy, was maddening. I thought to myself, who was the person that decided to use unified logging and nothing else? What was the value of having the data if you never intended to be able to use it? blag blah blah you can think the rest yourself, and just like I did, please keep those thoughts to yourself. Clearly their answer to not being able to use the data was to find a way to use the data when they had a need for it. Back to the point of the post.

In looking around on the system I quickly determined that there was xxd, perl, bash, and a smattering of other useful utilities. This was good as I was not looking forward to manual processing of these files from hex dumps. ( A side note is that I'm a fan of perl, it is a wonderful tool, so capable, so simple, so complex, it is what you need when you need something and at the same time the problem to your solution if done wrong.) The Snort source and perl was all I needed to get to the data they wanted and to save me from death by paper and pen. The first version of SnortUnified.pm was in fact process.pl and quite possibly the ugliest thing you have ever seen.

So here we are. What is this SnortUnified.pm thing?

SnortUnified.pm is a perl module for processing unified files (a set of perl modules actually). It isn't well documented outside the code itself, has few examples, and has saved me at least once. I know it is used in production environments and assume it is stable as I never get bug reports, just feature requests, though not since I added the processing hooks I'll be discussing below.

If you would like to know more you can browse the source in google code, get the latest source, or just continue reading, it is up to you. Working with SnortUnified is easy and depending on what you are trying to do could be as simple as a few lines of perl.

use SnortUnified(qw(:ALL));
use Data::Dumper;

$UF_Data = openSnortUnified(shift);
while ( $record = readSnortUnifiedRecord() ) {
print Dumper($record);
}
closeSnortUnified();

In the source tarball there is a samples directory with various samples available from converting a unified file to XML to signing the unified files or records within them; ensuring they cannot be modified without detection.

Two samples that I think are worth discussing and illustrate core functionality are uf_csv_handler.pl and uf_csv_qualifier.pl

The comments in the beginning of the handler script are sufficient for us to discuss here. First up are handlers, these get called for each record type / action defined and allow you to work with the records without having to change your functional tools. I initially envisioned this as a way to handle transformation of data within the records without having to rework the module. EG: convert an IP address integer to a dotted quad without having to change anything else. I also find it is quite useful for other tasks like converting payloads, debugging, adding output methods, etc. The result of a handler is irrelevant to the processing path, qualifiers are there for that.

# Handlers come before qualifiers come before pcre

# handlers will be run and regardless of the result processing will continue
# The available handlers for SnortUnified.pm are
# ("unified_opened", $UF);
# ("unified2_event", $UF_Record);
# ("unified2_packet", $UF_Record);
# ("unified2_unhandled", $UF_Record);
# ("unified2_record", $UF_Record);
# ("unified_record", $UF_Record);
# ("read_data", ($readsize, $buffer));
# ("read_header", $h);

register_handler('unified2_packet', \&make_ascii_pkt);
register_handler('unified_record', \&make_ascii_pkt);
# register_handler does not care about return values so the following will continue
# register_handler('unified_record', \&make_noise_fail);
# register_handler('unified_record', \&make_noise);

register_handler('unified2_record', \&printrec);
register_handler('unified_record', \&printrec);

# show_handlers();

# Qualifiers will be run, if any return a value less than 1
# then the record will be discarded and processing will continue
# with the next record in the file
# Only one option for unified types

# Skip all but sid 402
# register_qualifier(0,0,0, sub{return 0;});
# By having something specific for 402
# register_qualifier(0,1,402, \&printrec);
# register_qualifier(0,1,402, sub{return 1;});
# register_qualifier(0,1,402, \&make_noise);
# register_qualifier(0,1,402, \&make_noise);
# register_qualifier(0,1,402, \&make_noise_fail);
# register_qualifier(0,1,402, \&make_noise_never);

# But you can be granular with unified2 types
# register_qualifier($UNIFIED2_IDS_EVENT,1,402, \&make_noise);
# register_qualifier($UNIFIED2_PACKET,1,402, \&make_noise);


# register_pcre(1,402, "test");
# register_pcre(1,402, "*");
# register_pcre(1,402, ".*ads.clickagents.com.*");

# show_qualifiers();



Breaking out the comments above:

• unified_opened gets called every time a new unified file is opened. This allows you to do things like hook in verification routines, mark progress in a waldo file, create verbose informed logging, etc.

• unified2_event gets called every time a unified2 event record type is encountered in the unified file.

• unified2_packet gets called every time a unified2 packet record type is encountered in the unified file.

• unified2_unhandled gets called if a record type that is unknown / not handled by default is encountered in the unified2 file. This is useful as a bridge for new types, like the recently added 104 and 105 event types. I suggest that minimally you register a handler to log the unhandled types to a file so you can further investigate.

• unified_record gets called for earlier types of records only found in "unified" files, not "unified2" files

• read_data gets called every time we read data from the unified file.

• read_header gets called every time we read a header from the file.

• register_handler('unified2_packet', \&make_ascii_pkt); registers the "make_ascii_pkt" subroutine as a handler for all unified2_packet records. It gets called as follows make_ascii_packet($UF_Data) and line 105 of the uf_csv_handler is the beginning of that routine.

• show_handlers() will print the handlers that are registered.

Qualifiers are intended to allow you to "tune after the fact" and I think this is the right way to handle a lot of tuning tasks. We should not be artificially limited in our detection scope simply because that detection potentially creates a lot of "noise". I think that we should have the freedom to cast the widest net necessary and possible when it comes to these things, especially when investigating evolving threats and potentially targeted and ongoing attacks.

Qualifiers also allow you to get very specific in your actions. You can limit a qualifier to a specific sid or cast a super wide net on all events. Qualifiers are all executed so you can do things like easily suppress output, except for specific events. You can also get granular within those specific events. In the comments above all events are suppressed except gen:1 sid:402 and then only for sid:402 events that contain the PCRE ".*ads.clickagents.com.*". You could make the qualifier look at the IP address, check an archived (as in comprehensive) blacklist and change the context of the event, etc. I used a PCRE earlier, this is done with a special type of qualifier (purpose built?) that makes it trivial to use a regular expression as a qualifier though it has one caveat, a PCRE will not override the decision of other qualifiers, it only has a vote but in the case of conflict, the proper qualifier wins.

Qualifiers are called in the same way as handlers but the return code is checked and any return with a value that is less than one is discarded, processing continues on the next record, and your code never knows that an event was skipped. If you want or need tracking of these things you could use the qualifier to populate some stats and a handler to keep global stats.

If you use the code please drop me a note letting me know, it helps me gauge interest and any impact of possible changes. I've not modified the code in some time, largely because I've not needed to. But, if you have patches, requests, criticisms, commentary, or want to catch a pint at the next event we are both at feel free to let me know.

That is all for now, if you would like to know more or need assistance don't hesitate to ask. I can be reached at jason...at...snort.org or jason...at...sourcefire.com. My details are also in the README contained in the release tarball if you don't want to come back to the post.

Happy Snorting!

Can You Hold Your Data?

There is one mistake I see IPS analysts routinely make, one which considerably harms their organization’s ability to use the system and accurately respond to threats. And that is the failure to honestly consider how much time they actually have to devote to their IPS events.

How Much Time Do You Have For IPS?

If you want to get the maximum (or, really, any) value from your IPS (or IDS, if you prefer) then you have to stop and consider how much time you really have to spend with it. And be honest with yourself. How much time per day or week can you set aside? And get used to thinking about it, because you’re going to have to revisit this question regularly. After all, your schedule changes, and the value of your IPS alerts will always depend on the time you can give them.

The Value of Security Alerts

The value you gain from your IPS is not a database full of alert data, but the enhanced ability to respond to intruders. The data is useless without somebody to use it. So the value of your data will always be a function of how it is utilized. That is, how it helps you detect and respond to threats to your environment.

Raw data are often discussed as a commodity. And in many ways it is. I fear, however, that thinking about data this way encourages people to focus on collection and storage. Simply possessing data does not mean you are benefiting from it. A better analogy is that data are grown like, for example, corn. You can try to grow as much as you wish, but if you don’t have the manpower to harvest it, it will rot there.

Consuming Your Data

The value in farming corn is not simply having grown it, but that it can be transformed into something people want to consume. To continue to belabor this contrived analogy, we don’t just want to grow corn, we want to harvest it and turn it into a nice sour mash whiskey. Drinking whiskey is a lot like analyzing IPS alerts. The first sip you ever have is strange and burns. But once you develop a taste for it, it’s delicious. Then the first couple drinks are good, and sometimes the next couple are even better. But eventually having more makes you sick, and in obscene amounts it can even kill you.

It is the same with your IPS alerts (essentially, but I haven’t heard of any IPS related fatalities... yet). First time you plug your IPS in you may not get much value out of it. But once it’s tuned then the alerting you receive is beneficial. And more data may even give you a better picture of your network and who’s attacking it. But eventually there’s just too much data to handle. And not only are you receiving no value from this pile of data, but maintaining it and attempting to sort through it are costs to you. If you’ve reached this point then you either need to reduce the amount of data you have or hire more people to analyze it. Otherwise why even have the IPS? Collecting information nobody ever looks at is pointless.

Beginners’ Tolerance

While your new IPS system is being deployed and configured it is usually done by somebody focusing a lot of time on an IPS project. What you need to realize is that after this project is another project. And when your time is no longer focused primarily on IPS that data will still be rolling in. When focused on their IPS system people have a tendency to start collecting more data than they will be able to handle. The most common ways this happen are 1) turning on too many rules and 2) sensor placement.

First, turning on too many rules is a temptation some just can not refuse. But not only do all those extra rules place a performance burden on Snort, but they create alerts you need to analyze. So are those alerts really something useful? If not, then get rid of them. They’re creating useless data. And while those data continue to pour in, clogging the system with alerts, this only works to obscure the data you actually need to respond to.

Second, do you really need a sensor outside of the firewall? A lot of security analysts are by their very nature very curious. And the opportunity to see what kind of malware is floating around outside on the Internet is intriguing. But a month from now, when you’ve moved on to another project, are you going to have the time to wade through the flood of alerts that detected attacks that were subsequently stopped by your firewall? How useful is that? How will you be using and responding to these? Once you consider this, there’s a good chance you can find a better location for that sensor

Last Call

Your time and attention is limited and valuable. You need to understand this in order to get the full value from your IPS. You need to tune the system until you are no longer burdened with useless data (Or at least set that as the goal, it is typically a moving target.). And that means considering what traffic you need to monitor, and what rules will you actually respond to.


Think about it. Be honest. And when you can use your IPS without getting drunk on data you’ll be in good shape.