Friday, March 9, 2012

Rule Category Reorganization


Sometimes the new becomes old, sometimes the old becomes new.  But then sometimes, the old is just plain old.

Snort was started over 10 years ago.  Since the ability to write rules to Snort was added, its rules have been organized into categories in different files.  For years they have been added to and added to, but now there has come a time when the ruleset is a bit large that we want to and have to refocus it, add features, and clear the old cruft away.

That time is now.

What we are going to do over the next few months is slowly progress from our old categories (rule files) into a new set of categories.

Let me provide you a bit of background.  When we sat down and started discussing what a new ruleset would look like, we started thinking about what we wanted to accomplish with the ruleset.  What information we, as rule writers, wanted to convey to you, the end user.
How could we make your lives simpler and get more of value out of the rules.

What were our goals?
To communicate to the user the:
  • Intent of the rule
  • Impact of the rule
  • Target of the attack which the rule covers
  • Mechanism of attack that the rule covers
  • Technique of the attack or attacker that the rule covers

This is going to fan out over several parts, two of which have happened already:
  • File-identify

This rule category was created to be able to identify and track rules that deal with file (client) based attacks.  Centrally locating all rules that "set" flowbits based off of two things, the download method, and the detection of the file actually coming down through characteristics of the file itself.
We will continually expand in this category and have done so with every rule release.  All of these rules are enabled based on the "balanced" policy, which is now our standard default policy.  For other policies (for instance security) you should use a tool named PulledPork to fix everything for you.  The Sourcefire product also handles this "flowbit resolution" for you seamlessly, I say this to illustrate that Oinkmaster does not provide either of these functions unfortunately.

The second part of this project has been happening quietly in the background:
  • Cleanup

This is an on-going project to modernize our ruleset with newer keywords and functionality that did not exist before.  For instance, adding file_data to all the rules that need it for better decoding, (making the IDS harder to evade) and pointer placement.

Standardized naming across all rule message names.  For instance, not using "Microsoft IE" or "IE" or "Windows IE", but instead adopting a the naming sequence "Microsoft Internet Explorer", and "Microsoft Windows".  "Apple QuickTime" instead of just plain "quicktime".  This allows you to search the ruleset easily and turn on what is in your network through the use of easy pcre inside of the enablesid.conf file using PulledPork.   We even went so far as to get rid of things like "from_server" and move to "to_client" inside of the flow statement for easier readability.

We started this cleanup back in October and since then we've averaged over 600 changes per rule release.

The third portion of this project is our biggest.  We call it:

  • Rule re-categorization


Moving all rules from their old categories:

attack-responses.rules
backdoor.rules
bad-traffic.rules
blacklist.rules
botnet-cnc.rules
chat.rules
content-replace.rules
ddos.rules
deleted.rules
dns.rules
dos.rules
experimental.rules
exploit.rules
file-identify.rules
finger.rules
ftp.rules
icmp-info.rules
icmp.rules
imap.rules
info.rules
local.rules
misc.rules
multimedia.rules
mysql.rules
netbios.rules
nntp.rules
oracle.rules
other-ids.rules
p2p.rules
phishing-spam.rules
policy.rules
pop2.rules
pop3.rules
rpc.rules
rservices.rules
scada.rules
scan.rules
shellcode.rules
smtp.rules
snmp.rules
specific-threats.rules
spyware-put.rules
sql.rules
telnet.rules
tftp.rules
virus.rules
voip.rules
web-activex.rules
web-attacks.rules
web-cgi.rules
web-client.rules
web-coldfusion.rules
web-frontpage.rules
web-iis.rules
web-misc.rules
web-php.rules
x11.rules


Into a new set of categories.  The new set will not only make us more agile and able to add new categories as we expand in future years, more professional, but will also be simpler for the end user.

Obviously all the new categories are not created yet.  We have a base set of names that we are going to start with, but we know for a fact that once we get into moving these rules, new categories will need to be created.

For example, we are going to create a series of rules based off of protocols, so that you may enable or disable them based upon your network needs.  The rules contained in here will not be software related, but related to vulnerabilities in the protocol itself.

PROTOCOL-FTP (protocol-ftp.rules)
PROTOCOL-FINGER (protocol-finger.rules)
PROTOCOL-RSERVICES (protocol-rservices.rules)
PROTOCOL-ICMP (protocol-icmp.rules)
PROTOCOL-IMAP (protocol-imap.rules)
PROTOCOL-POP3 (protocol-pop3.rules)
PROTOCOL-RPC (protocol-rpc.rules)
PROTOCOL-SMTP (protocol-smtp.rules)
PROTOCOL-SNMP (protocol-snmp.rules)
PROTOCOL-OTHER (protocol-other.rules)

We'll create a series of categories based upon file type allowing us to consolidate all rules that cover attack vectors based on those file types from many different rule files into one:

FILE-PDF (file-pdf.rules)
FILE-OFFICE (file-office.rules)
FILE-IMAGE (file-image.rules)
FILE-MULTIMEDIA (file-multimedia.rules)
FILE-FLASH (file-flash.rules)
FILE-JAVA (file-java.rules)
FILE-EXECUTABLE (file-executable.rules)
FILE-OTHER (file-other.rules)

You notice that we always have an "OTHER" at the end of the series.  This is our category where rules that just don't fit in any of the other categories will go, until there is a sufficient amount of them to warrant their own category.
For example, PDF wasn't a gigantic threat 5-8 years ago.  Maybe 10 or so rules that covered PDF files.  Now it'll have it's own category as we have hundreds.

Along the way we'll be revisiting every single rule for syntax, cleanliness, and speed.  If there are some quick optimizations we can make, then we'll make them.  I made several just the planning process for the first category.

Speaking of the first category, the first one we'll be breaking out is POLICY.  When looking through the policy.rules category, it looks like it'll be broken out in the following:

POLICY-SOCIAL
POLICY-MULTIMEDIA
POLICY-OTHER
INDICATOR-OBFUSCATION
INDICATOR-COMPROMISE
PUA-TOOLBARS
PUA-P2P
SERVER-MAIL
FILE-PDF
FILE-OFFICE
FILE-FLASH
FILE-OTHER

The good news is, if you are using the Sourcefire product, PulledPork, or Oinkmaster, the vast majority of you should be unaffected.  These products will handle the transition just fine.  The only way you will be affected using PulledPork (or Oinkmaster's related tools) is if you use enablesid.conf or disablesid.conf to enable or disable entire categories of rules. We will also notify you of any changes that will be made to the snort.conf to incorporate these changes.  

With each category that we transition, we'll do a blog post about it so that you are well aware before, during, and after the transition.

We will not be deleting the old category names until they are completely empty and local.rules will stay as is.

There are two more parts to this project that will begin after this transition is complete, introducing new functionality and intelligence into the ruleset that exists absolutely no where else in any product.  These two projects involve a total revamp of the classification and priority system, and finally additions to the metadata system within the rules.  These modifications are in addition to the significant changes we have set for the Snort engine itself in 2012.

If there are any questions, please contribute to the thread on the Snort-users mailing list here:
https://lists.sourceforge.net/lists/listinfo/snort-users

An exciting year is here!  Stay tuned!