PulledPork 3 — Rule updating for Snort 3

We are incredibly excited to release PulledPork 3 — the next evolution for PulledPork, a companion piece of software for SNORTⓇ that is specifically designed for Snort 3. 

PulledPork 3 is built to use the LightSPD package. It allows a single ruleset package to adapt the rules it can run to the version of the engine running on the system and allows users to select a default policy for the ruleset.

Noah Dietrich, an extremely helpful and generous member of our community, re-wrote PulledPork from the ground up in Python (Pulled Pork for Snort 2.X is written in Perl). Not all PulledPork functionality carries over, but the tool is at a point now where it's ready for users to start testing it. We are considering PulledPork 3 to be in alpha.

Please check out the tool here. As always, we are looking for contributors to the project as well. If you are well-versed in Python, would love to have a hand in documentation, or simply want to help "QA" the tool, all issues and pull requests against the tool are welcome.

We also created a special PulledPork channel on the newly created Snort Discord server, so feel free to contribute there as well!

New version of PulledPork available on GitHub

The Snort community welcomes a new version of PulledPork on GitHub today.

Version 0.7.4 now supports Snort 3 and points to the new, correct, location of the IP blocklist. PulledPork is a Perl script that allows users to download new rules as soon as new vulnerabilities or exploits are discovered.

Here are some of the other changes in this version:

• Supports updating of Snort 3.0 signatures (0.8 will be released when Snort 3.0 moves out of BETA).
• Fixed some of the logic to allow updating with Perl on Windows
• ability to modify rules via regex in modifysid.conf
• Removal of opensource.gz processing (will speed up signature updating)
• Updated OS Distro list to match so_rules
• Added error checking around writing to directories that do not exist (i.e., block_list)
• Updated for new location of block list

PulledPork 0.7.3 release!

Released last night, PulledPork 0.7.3 has hit the streets and is downloadable from the pulledpork Github page.

The release notes say the following:

This release includes bug fixes related to some versioning code in the latest version of Snort and other outstanding issues.

The next version of PulledPork will begin work on Snort 3 as we are looking forward to the first beta and compatible ruleset with the engine.

PulledPork 0.7.2 has been released!

The newest version of PulledPork has been released and is available for download from the PulledPork Github repository!

This release fixes several bugs.  For those of you that haven't updated their version of PulledPork in awhile, this will fix many download issues you may have with the blacklist and official rulesets from Snort.org.

Everyone using PulledPork should grab it, and for the stragglers left that still use oinkmaster, you should start upgrading too.  For those of you that have oinkmaster configurations, you'll see in the contrib directory, a community member has submitted a small perl script that converts your oinkmaster configuration files to pulledpork configuration files.

Please start your upgrade engines, as Snort 2.9.9.0 should be released soon, and you'll want to be ready!

Are you getting 404 errors attempting to download the community ruleset?

Yesterday, it came to our attention during some routine cleanup and maintenance that there were about 15,000 people attempting to download a Community Rule file directly from an older S3 Bucket. (Which hadn't been updated in over a year.)

The link directly into the S3 bucket was apparently in use by the default pulledpork.conf, and many people had not updated it to the newest link now available on Snort.org.

I have submitted a pull request against the pulledpork.conf to correct that link, and that should be fixed shortly.

However, for those of you that need to change your installation, please find this line in your PulledPork.conf:

https://github.com/shirkdog/pulledpork/blob/master/etc/pulledpork.conf#L21

Which looks like this:

rule_url=https://s3.amazonaws.com/snort-org/www/rules/community/|community-rules.tar.gz|Community

and change it to:

rule_url=https://snort.org/downloads/community/|community-rules.tar.gz|Community

This will ensure that you are pulling the correct community rules file.

Sorry for any lack of notice, we figured these were old installations without any updates, and didn't realize that it was actually in the default pulledpork.conf.

Please update to the new rule file and join the hundreds of thousands of users that download that rule file on a daily basis!

As always, if you'd like to contribute to the community ruleset, please send your rules to either the Snort-sigs or directly to Talos.

IP Blacklist feed has moved locations!

For those of you using the IP Blacklist feed on labs.snort.org, we've had to move the URL to the new link.

You can find it at the following URL: http://talosintel.com/feeds/ip-filter.blf

The pulledpork.conf that is currently in Github has been updated to use the new URL, so a fresh download of pulledpork will help you.

Check it out!

PulledPork 0.7.0 Released! #include <IP.Reputation>

PulledPork 0.7.0 - Swine Flu has been released and can be found at the PulledPork site.  There are numerous feature changes and enhancements that dramatically affect the functionality and capabilities of PulledPork since the last major 0.6.x release.  An excerpt of the changelog is at the bottom of this post and lists all of the changes/features/enhancements.   

The most significant change that you are likely to notice deals with how PulledPork now processes the rules tarball.  In the previous versions of PulledPork when you would run the application it would process the rules tarball as designated in your configuration, regardless of whether or not the source tarball had changed (no new rules tarball for example).  With the new changes the source rules tarball is ONLY processed if it is new/changed OR if you specify the -P runtime flag.  So for tuning exercises or out of band runs when the source tarball is unchanged, you MUST specify the -P flag for any processing to occur.

Inline with the new IP Reputation preprocessor that was introduced in Snort 2.9.1 we have included full support for this feature.  This support includes a couple of new configuration options that are located in the pulledpork.conf and allow for retrieval of multiple ip reputation lists (PulledPork will automatically de-dupe these lists).  If you are running Snort on Linux you are also able to specify at ./configure time an option to allow for in-memory reloading of IP Reputation lists, thus you do not have to SIGHUP or completely reload Snort.  This in-memory reload is accomplished by using a control socket that this version of PulledPork is capable of utilizing.

Working closely with the Barnyard2 team we have developed a new version of the data in the sid-msg.map.  This allows for more information to be included in intrusion events such as the revision of the rule (currently not included in alerts).  The default version is still version 1 of the sid-msg.map file, it is CRITICAL to note that only Barnyard 2.2+ supports this new version of the sid-msg.map file and as such ONLY when using this version or newer of Barnyard 2.2+ should you change this value in your pulledpork.conf.

When utilizing the default configuration that creates two single unified rules files (one for so_rules and one for text rules) the so_rules stub files are now included in the single rules tarball.  This means that you no longer need to include the so_rules.rules file.  This single rules file is now internally separated by category and rule type, or generator to allow for rapid rule location and more logical perusing of the file.

As per the usual, thank you for your continued support and usage of PulledPork and Snort.  Should you have any questions or concerns please feel free to file a bug report or new feature request at http://pulledpork.googlecode.com and also to participate in the community mailing list that can be found at http://groups.google.com/group/pulledpork-users

Bug Fixes:
- Bug #79 - Fixed race condition that did not allow for disabled rules to be modified using modifysid
These rules would then be enabled by flowbit dependency check and be unmodified
- Bug #77 - Adjusted chown property of archive::tar
- Bug #78 - Adjusted per bug report to allow for proper ignoring of preproc.rules
- Bug #102 - Only Enabled rules are written to sid-msg.map now when -E flag is specified
- Bug #99 - Doc Bug, updated docs associated with snort_version variable
- Bug #96 - Modified code to allow for same-line traling comments: "1:10011 #can haz disable!"
Also updated the rulestate files (enable,disable,drop)
- Bug #82 - Modified run order to force modifysid to run before all other sid state modification routines
This allows for sid changes to be made prior to automatic state determination ala automatic
flowbit resolution.  NOTE that this DOES NOT AND WILL NOT disable automatic flowbit
resolution, this is a critical piece.
- Bug #81 - Updated valid SO distro pre-compiled list
- Bug #114 - Update Regex to allow for null search/replace in modify_sid sub
- Unlisted Bug - Allow for escaped ; "\;" in references
- Bug #121 - Update to allow for new etpro.com url and cert!
- Bug #119 - Fixed regex [^\\], should have been negative look behind (?<!\\)
- Bug #120 - Updated proxy code for better support and proper runtime load order
- Unlisted Bug - Account for multiple flowbits that are separated using &| operators
(flowbits:isset,flowbit1&flowbit2;)(flowbits:isset,flowbit1|flowbit2;)
- Bug #126 - Removed Switch usage
- Bug #129 - Fixed to allow for -n usage (in conjunction with -P) when an ip list is used also
- Unlisted Bug - Fixed to allow for proper -P usage

New Features / changes:
- Bug #105 - Removed Switch function as it is deprecated in > 5.12 perl
- NEW - Added IP Reputation Preprocessor support
- NEW - Capability to use control socket for IP List reload
- NEW - -P runtime flag to (process even if there is no new rules tarball)**
- Bug #68 - Added basic surricata support
- Bug #115 - Single rules file now has category (and GID) separators
Correlating to this we have also removed the separate so_rules.rules file
All rules are now in a single snort.rules file unless the keep flag is
specified at runtime.
- NEW - Numerous sub rewrites to allow for better performance
- NEW - New sid-msg.map format for barnyard 2.2+ gid || sid || rev || class || pri || msg || @refs
- NEW - SO rule categories are now prepended with VRT-SO
- NEW - More advanced sid-msg.map structure (for use with by2.2+) and backward compatibility
This allows for better mapping of gid:sid:rev in the database!
- NEW - Rewrote the way that extraction is handled, to properly support a single rules tarball being
updated.  This includes how md5 validation is done and in what order.  If a single
file is updated then they are all extracted and processed.

VRT Rule release for 02/09/2012

Join us as we welcome the introduction of the newest rule release for today from the VRT. In this release we introduced 10 new rules and made modifications to 4172 additional rules.

There were no changes made to the snort.conf in this release.

Today, we leveled the playing field between the various ways to get Snort rules. It has long been the case where Sourcefire products, by default, enabled rules in the balanced-ips policy.  

When you use PulledPork (http://code.google.com/p/pulledpork/), this is also the default behavior. But when you simply downloaded the rules from Snort.org, the rules were a hodge podge of rules that were enabled or disabled, denoted by whether or not the rule was commented out in the rules file.

In an effort to make the barrier to entry that much easier, the Open Source rule package downloaded on snort.org now exactly mirrors what you would get if you used PulledPork. All rules in balanced-ips are enabled and all rules not in balanced-ips are disabled. The exception to this is that rules that set flowbits that are used by rules that are in balanced-ips are also enabled. This means that the default Open Source ruleset will now provide a good balance between speed, performance, and detection and all rules should work as expected.  Those using Oinkmaster, or simply downloading the ruleset directly, will now be running the "balanced-ips" policy.  A rule's "on/off" state is now dictated by policy.

This change is in no way an indication that PulledPork is not the recommended way to manage your Open Source ruleset. PulledPork also tracks your own custom policy tailored to your environment and provides other benefits. If you want to use the security-ips policy, you may go through and enable these rules by default, or choose the easy way and use PulledPork to manage this for you. So, use PulledPork if you aren't already!

 In VRT's rule release:

Synopsis: This release adds and modifies rules in several categories.

Details: The Sourcefire VRT has added and modified multiple rules in the attack-responses, backdoor, bad-traffic, blacklist, botnet-cnc, chat, dns, dos, exploit, file-identify, finger, icmp, icmp-info, imap, misc, multimedia, netbios, nntp, oracle, p2p, password, policy, pop3, rpc, rservices, scada, scan, shellcode, smtp, specific-threats, spyware-put, sql, username, voip, web-activex, web-cgi, web-client, web-iis, web-misc and x11 rule sets to provide coverage for emerging threats from these technologies.

In order to subscribe now to the VRT's newest rule detection functionality, you can subscribe for as low as $29 US dollars a year for personal users, be sure and see our business pricing as well at http://www.snort.org/store. Make sure and stay up to date to catch the most emerging threats!

The importance of PulledPork

Bottom line up front:  If you aren't using PulledPork, you are going to have a gigantic depreciation in functionality.

You've heard it said on the Snort lists, you've heard it on this blog, you've heard it on Twitter,  you've heard it from CNN...  okay, well, not CNN..

The importance of PulledPork.

The reason that we are very heavily insist that you are using pulledpork is primarily for two very big reasons:

1. Flowbit auto-resolution
2. Default Policy usage

• The first reason: Flowbit auto-resolution

I've written in great length about the need for flowbits, you can find a couple of my blog posts here
http://blog.snort.org/2011/05/resolving-flowbit-dependancies.html
and
http://blog.snort.org/2011/12/if-you-are-having-problems-with-your.html

The reason that I'm writing about it again, is that to refresh your memory back when we created the file-identify rule category:
http://blog.talosintel.com/2011/11/say-hello-to-file-identify-category.html

We outlined that part of this conversion was to move all flowbit names from their old names (such as http.gif) to a new format (now file.gif).  This project has now been completed and all flowbits across all files have been moved to the new format, and all the rules that "set" flowbits have been commented to be "off" by default and in no policies.

What we are relying on here is either if you are using PulledPork or the Sourcefire product, you either select your base policy (which I'll talk about in section two), or, even if you don't, PulledPork will auto-resolve the flowbit "set" names that you'll need and go through and turn those on for you.  That way you are only running rules that you need to have on based upon either the policy you are running, or the state of the rules.  We really insist that you use the disablesid and enablesid functionality in PulledPork to be able to turn the rules that you WANT for your environment on and let PulledPork auto-resolve all the dependancies you need for you.

• The second reason: Policy usage

There are four states that we place rules in when we create them, three of the states are assigned to policies.

1. Connectivity
2. Balanced
3. Security

The last state is "in no policies".

1. The first, connectivity, means "Connectivity over Security".  Meaning this is a speedy policy for people that insist on blocking only the really known bad with no false positives.  
2. The second, balanced, means "Balanced between Connectivity and Security".  Meaning that this is a good starter policy for everyone.  It's quick, has a good base coverage level, and covers the latest threats of the day.  The policy contains everything that is in Connectivity.
3. The third, security, means "Security over Connectivity".  Meaning that this is a stringent policy that everyone should strive to get to through tuning.  It's coverage is much more exhaustive, and has some policy-type rules in it.  Rules that will alert on Flash contained within an Excel file and things like that.  This policy contains everything that is in the first two.

The last state is "in no policies".  This means that we insist that you look through these by product name or CVE in order to turn them on.  These may not have a fast content match, could be false positive prone, or the vulnerability it is covering is not in a very prevalent piece of software.  

The way we make the decision about what is "on" or "off" by default when you aren't using the policies, is, if it's in balanced, it's on by default, it's it not in balanced, it's off by default.  There are a ton of exceptions to this rule, but this is the general rule of thumb.

PulledPork makes the cover of Linux Pro magazine!

Our own JJ Cummings of Sourcefire and fellow Snort.org blogger is the author of Snort's PulledPork tool, the best way for keeping your rules up to date, was recently featured in an article in Linux Pro's Magazine.

Here's a picture of the cover, with the PulledPork article circled.

The cover says that there are other Snort-related tools discussed as well in the article, unfortunately, I do not have a copy of the magazine so I don't know which ones they are talking about.

If you have a copy of this edition of Linux Pro Magazine, please feel free to leave a comment and let us know what other tools were discussed!

Congratulations to JJ for all of his hard work to give such a great tool away to help people maintain their rule updates!  Thanks JJ!

Awesome.  As noted in the comments, the article has been posted online at Linux Magazine:
http://www.linuxpromagazine.com/Issues/2011/125/Snort-Helpers