Showing posts with label database. Show all posts
Showing posts with label database. Show all posts

Wednesday, July 18, 2012

Database output is dead. R.I.P.

Last June (2011) we gave you a heads up (and several reminders since) that in Snort 2.9.3.0, we were going to remove the spo_database output module as well as Aruba and Prelude outputs.

For those of you that originally compiled Snort like: ./configure --enable-mysql

Or, if you look in your snort.conf and your "output" lines look like this:
output database: alert
AND/OR
output database: log


this will affect YOU.

Our recommendation is that after you upgrade to Snort 2.9.3.0, you move to full unified2 logging and use barnyard2 to read those unified2 files and input them into your mysql database.

You can find more information about barnyard2 here:
https://github.com/firnsy/barnyard2
http://www.securixlive.com/barnyard2/

As always questions can be asked on the Snort Mailing Lists!  Thank you!

Monday, November 28, 2011

Snort 2.9.2 RC's output warnings

Beginning in Snort 2.9.2, if you are using an output method that is being depreciated in a future version of Snort, we are going to warn you on startup.

Examples of these depreciated output methods that you will be warned about are:
spo_database (Direct to database output method, or commonly referred to as the "database output method")
spo_aruba (Aruba output plugin)
spo_prelude (Prelude output plugin)

These output plugins will be totally removed in Snort version 2.9.3.

We are not depreciating "unified1" as an output method in 2.9.3, but we do have plans for it's EOL as well.

We suggest moving to unified2 as an output method, and also to barnyard2 (if you are still using the original barnyard)

Friday, September 16, 2011

Snort 2.9.1 HTTP and SMTP logging features

To provide better context to the alerts generated by Snort, Snort version 2.9.1 introduced some new logging features for HTTP and SMTP which will help the user to better analyze the alerts. This logging of extra data by HTTP Inspect and SMTP preprocessors is similar to the X-Forwarded-For/True-Client-IP logging introduced prior to Snort 2.9.1.

HTTP Logging:


Let's talk about HTTP URI and Hostname logging.

How to enable logging of HTTP URI and Hostname?

To turn on the logging of the HTTP Request URI, the config option to use is "log_uri" and to enable the logging of hostnames, use the config option "log_hostname" as shown in the example below:

preprocessor http_inspect: global memcap
preprocessor http_inspect_server: log_uri \
log_hostname


The memcap here will determine the maximum amount of memory the HTTP Inspect preprocessor will use for logging the URI and Hostname data. You can refer to the Snort Manual for further details.

When these features are turned on in HTTP Inspect, the HTTP Request URI and HTTP Request hostname headers are extracted and logged to unified2 as an "extra data event" with each alert generated for that particular session. It is recommended to turn on stream5 reassembly with PAF on HTTP ports to ensure correctness and accuracy.

When a HTTP Request URI is greater than 2048 or when a HTTP hostname (specified in the "Host" Request header) is greater than 256, Snort will log the truncated the URI and/or hostname. A preprocessor alert with GID:119 and SID:25 is generated when hostname exceeds 256 bytes.

There is also a preprocessor alert with GID:119 and SID:24 for multiple "Host" headers in one HTTP request.

Please note that the URI and hostname are only logged in Unified2 mode and not logged with -A cmg.

So, how do you read the output?

Unified2 can be read using the Snort tool u2spewfoo. You can find it under the tools/u2spewfoo directory in the Snort tarball.

Example of the Output:


(Event)
sensor id: 0 event id: 2 event second: 1299776137 event microsecond: 355217
sig id: 1 gen id: 1 revision: 0 classification: 0
priority: 0 ip source: 10.1.2.3 ip destination: 10.9.8.7
src port: 60710 dest port: 80 protocol: 6 impact_flag: 0 blocked: 0

Packet
sensor id: 0 event id: 2 event second: 1299776137
packet second: 1299776137 packet microsecond: 355217
linktype: 1 packet_length: 214
[ 0] 02 09 08 07 06 05 02 01 02 03 04 05 08 00 45 00 ..............E.
[ 16] 00 C8 00 04 00 00 40 06 5C 19 0A 01 02 03 0A 09 ......@.\.......
[ 32] 08 07 ED 26 00 50 00 00 00 02 00 00 00 02 50 10 ...&.P........P.
[ 48] 01 00 0C 19 00 00 47 45 54 20 2F 20 48 54 54 50 ......GET / HTTP
[ 64] 2F 31 2E 30 0D 0A 48 6F 73 74 3A 20 77 77 77 2E /1.0..Host: www.
[ 80] 77 33 2E 0D 0A 20 6F 72 67 20 0D 0A 55 73 65 72 w3... org ..User
[ 96] 2D 41 67 65 6E 74 3A 20 4D 6F 7A 69 6C 6C 61 2F -Agent: Mozilla/
[ 112] 35 2E 30 20 28 58 31 31 3B 20 55 3B 20 4C 69 6E 5.0 (X11; U; Lin
[ 128] 75 78 20 78 38 36 5F 36 34 3B 20 65 6E 2D 55 53 ux x86_64; en-US
[ 144] 3B 20 72 76 3A 31 2E 39 2E 32 2E 31 35 29 20 47 ; rv:1.9.2.15) G
[ 160] 65 63 6B 6F 2F 32 30 31 31 30 33 30 33 20 55 62 ecko/20110303 Ub
[ 176] 75 6E 74 75 2F 31 30 2E 31 30 20 28 6D 61 76 65 untu/10.10 (mave
[ 192] 72 69 63 6B 29 20 46 69 72 65 66 6F 78 2F 33 2E rick) Firefox/3.
[ 208] 36 2E 31 35 0D 0A 6.15..



(Event)
sensor id: 0 event id: 3 event second: 1299776137 event microsecond: 355254
sig id: 1 gen id: 1 revision: 0 classification: 0
priority: 0 ip source: 10.1.2.3 ip destination: 10.9.8.7
src port: 60710 dest port: 80 protocol: 6 impact_flag: 0 blocked: 0

Packet
sensor id: 0 event id: 3 event second: 1299776137
packet second: 1299776137 packet microsecond: 355254
linktype: 1 packet_length: 176
[ 0] 02 09 08 07 06 05 02 01 02 03 04 05 08 00 45 00 ..............E.
[ 16] 00 A2 00 05 00 00 40 06 5C 3E 0A 01 02 03 0A 09 ......@.\>......
[ 32] 08 07 ED 26 00 50 00 00 00 A2 00 00 00 02 50 10 ...&.P........P.
[ 48] 01 00 8F E5 00 00 41 63 63 65 70 74 2D 45 6E 63 ......Accept-Enc
[ 64] 6F 64 69 6E 67 3A 20 67 7A 69 70 2C 64 65 66 6C oding: gzip,defl
[ 80] 61 74 65 0D 0A 41 63 63 65 70 74 2D 43 68 61 72 ate..Accept-Char
[ 96] 73 65 74 3A 20 49 53 4F 2D 38 38 35 39 2D 31 2C set: ISO-8859-1,
[ 112] 75 74 66 2D 38 3B 71 3D 30 2E 37 2C 2A 3B 71 3D utf-8;q=0.7,*;q=
[ 128] 30 2E 37 0D 0A 4B 65 65 70 2D 41 6C 69 76 65 3A 0.7..Keep-Alive:
[ 144] 20 31 31 35 0D 0A 43 6F 6E 6E 65 63 74 69 6F 6E 115..Connection
[ 160] 3A 20 6B 65 65 70 2D 61 6C 69 76 65 0D 0A 0D 0A : keep-alive....


(ExtraDataHdr)
event type: 4 event length: 33


(ExtraData)
sensor id: 0 event id: 2 event second: 1299776137
type: 9 datatype: 1 bloblength: 9 HTTP URI: /


(ExtraDataHdr)
event type: 4 event length: 43


(ExtraData)
sensor id: 0 event id: 2 event second: 1299776137
type: 10 datatype: 1 bloblength: 19 HTTP Hostname: www.w3.org


In the output above, two extra data events are generated for the event with GID:1 and SID: 1. The first extra data event displays the URI and the second displays the hostname associated with that session.

It is important to note that the unique "event id" and "event second" are used to correlate the alerts with their extra data. The "event type" indicates the presence of an "extra data" record and the unique "type" will determine the type of the extra data event logged. Here is a list of extra data types used in Snort.

Type 1: True-Client-IP/XFF IPv4 address
Type 2: True-Client-IP/XFF IPv6 address
Type 4: HTTP Gzip decompressed data
Type 5: SMTP filename
Type 6: SMTP MAIL FROM addresses
Type 7: SMTP RCPT TO addresses
Type 8: SMTP Email headers
Type 9: HTTP Request URI
Type 10: HTTP Request Hostname
Type 11: Packet's IPv6 Source IP Address
Type 12: Packet's IPv6 Destination IP Address

SMTP Logging:


The SMTP preprocessor, similar to HTTP Inspect preprocessor, can log extra data associated with each alert. The data that SMTP logs are as follows:

1. The email addresses in the MAIL FROM command.
2. The email addresses in the RCPT TO command. When there are multiple RCPT TO headers, the email addresses are concatenated using commas.
3. The filename of the MIME attachment extracted from the "Content-Disposition" header. Multiple filenames are appended with commas.
4. SMTP email headers.

How to enable logging of SMTP "extra data"?

SMTP preprocessor can log the extra data mentioned above using the following config options.

preprocessor smtp: memcap 838860 \
log_mailfrom \
log_rcptto \
log_filename \
log_email_hdrs \
email_hdrs_log_depth 56


The config options "log_mailfrom", "log_rcptto", "log_filename" and "log_email_hdrs" do not take any arguments. However, email_hdrs_log_depth requires the user to pass a number between 0-20480, which will determine in bytes the amount of email headers logged. The config option "memcap" is used to determine the maximum memory used for logging the above mentioned data.

Again, stream5 reassembly needs to be turned on for SMTP ports for this feature to work correctly. Without stream5 reassembly, the SMTP extra data won't be logged.

How to read the Output?

Like HTTP extra data, SMTP extra data can be read using the u2spewfoo and "event id" and "event second" are used to correlate the alerts with their extra data events.

Example of the Output:


(ExtraData)
sensor id: 0 event id: 2 event second: 1273194706
type: 6 datatype: 1 bloblength: 33 SMTP MAIL FROM Addresses:

(ExtraData)
sensor id: 0 event id: 2 event second: 1273194706
type: 7 datatype: 1 bloblength: 57 SMTP RCPT TO Addresses: ,

(ExtraData)
sensor id: 0 event id: 2 event second: 1273194706
type: 5 datatype: 1 bloblength: 26 SMTP Attachment Filename: sample.txt,foo.txt

(ExtraData)
sensor id: 0 event id: 2 event second: 1273194706
type: 8 datatype: 1 bloblength: 323 SMTP EMAIL HEADERS:
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: multipart/mixed; boundary="_----------=_12731947069000"
Date: Thu, 6 May 2010 21:11:46 -0400
From: bbantwal@sourcefire.com
To: bbantwal@sourcefire.com
Subject: Email Sent via Perl
X-Mailer: MIME::Lite 3.027 (F2.74; T1.29; A2.06; B3.07; Q3.07
<bbantwal@sourcefire.com><bbantwal@sourcefire.com><foo-barfoo@gmail.com>


We will not be updating the spo_database output method to insert this information into the database.  As a reminder, this output method will be removed in a future Snort release in 2012.  Sourcefire has turned over the maintenance of the "ACID" database schema to the Barnyard2 group, and will also, no longer be maintaing it either.

Wednesday, July 27, 2011

Barnyard2 sets up a Google Group!

Greetings everyone,

The barnyard2 team want to announce the creation of two Google groups that will be used ease the way for users to report issue or find answers and discuss about barnyard2 related topics.

barnyard2-users and barnyard2-devel.

barnyard2-users@googlegroups.com (for users problems and issues)
barnyard2-devel@googlegroups.com (for development updates, fixes, patches, comments, and more)

We strongly encourage you to join if you have any issues/commenst/questions related to barnyard2.

We would also like to launch a special invitation to UI developers that are willing to improve the future of the database schema and the handling of unified2 EXTRA_DATA event type.

Any comments or question can also be directed to

Ian Firns firnsy@securixlive.com
Eric Lauzon beenph@gmail.com

Eager to see you arround in the barn *wink wink*

Update:  Here is the direct link to the barnyard2 group: http://groups.google.com/groups/dir?lnk=nhpsfg&q=barnyard2.  Thanks Jason!

-The barnyard2 team.

Tuesday, July 26, 2011

VRT Rule Update for 7/26/2011

Welcome the introduction of the newest rule release for today from the VRT. In this release we introduce 31 new rules and make modifications to 6 more.

There were no changes made to the snort.conf in this release.

In VRT's rule release:
Oracle Database Server (CVE-2011-0799):
Oracle Database Server contains a programming error that may allow a remote, unauthenticated attacker to access data residing on an affected system. The attack vector is an SQL injection vulnerability using the Oracle Warehouse Builder User account.

Rules to detect attacks targeting this vulnerability are included in this release and are identified with GID 1, SIDs 19599 and 19600.

Oracle Sun Products Suite (CVE-2011-2260):
The Oracle GlassFish Server component in the Oracle Sun Products Suite contains a programming error that may allow a remote attacker to execute a cross-site scripting attack.

A rule to detect attacks targeting this vulnerability is included in this release and is identified with GID 1, SID 19605.

Additionally, the Sourcefire VRT has added and modified multiple rules in the backdoor, blacklist, botnet-cnc, exploit, netbios, specific-threats, spyware-put and web-client rule sets to provide coverage for emerging threats from these technologies.

In order to subscribe now to the VRT's newest rule detection functionality, you can subscribe for as low as $29 US dollars a year for personal users, be sure and see our business pricing as well at http://www.snort.org/store. Make sure and stay up to date to catch the most emerging threats!

Monday, June 27, 2011

Snort's output methods

Ever since the beginning of Snort, one of the main concerns was "how do I get data out of Snort".  Some of the options that are available have their advantages and disadvantages:


  1. There's some that aren't used.
  2. There's some that cause Snort to be slow.
  3. There's some that we don't maintain and don't frequently test.
  4. There's some that we want to get rid of.


One of those output methods is the "spo_database" module.  The module within Snort that directly inputs data from Snort into a mysql, postgres, or an Oracle database.  This logging method was written back in the late 90's by a college student (along with the db schema and the interface ACID) as a project for his thesis.

It hasn't been very well maintained since then.  In fact, we don't test against it, and we don't recommend it for use.  It makes Snort, which is a high-speed data processor, have to stop doing what it's doing (being an IPS), and insert data into the database.  While Snort is inserting into the database, this stops inspection waiting for the database connection.

So we are going to remove it.


If you look in your snort.conf and your "output" lines look like this:
output database: alert, <db_type>, user=<username> password=<password> test dbname=<name> host=<hostname>
output database: log, <db_type>, user=<username> password=<password> test dbname=<name> host=<hostname>


This change will affect you.

In order to provide the type of functionality we'd like to provide with Snort in the next few releases (more data for you!), we needed someone to take over the maintenance of the db schema that is shipped with Snort as well.   As a result of the discussion on the Snort-devel list, the team members over at the barnyard2 project have agreed to take over the maintenance of these schemas.

It is our intention to distribute the unified2 format as our official output method, provide our documentation for it, and the u2spewfoo tool within Snort so that anyone is able to read it.  We are going to keep some other output methods as well, but...

At this point I'd like to hear from the community as well.  So please leave comments.

What output plugins do you use?
Will you be affected by this change (we hope a lot of you aren't using the spo_database method)?
What other output plugins do you think we can "show the door"?

Monday, March 28, 2011

Snort IDS Sensor with Sguil Framework ISO

Guy Bruneau of the Internet Storm Center  has released a CD that he calls a:

"a hardened OS that includes Snort IDS sensor (version 2.9.0.4) with all the Sguil components ready to use."
It's available in both 32 bit and 64 bit versions and each version has three options:

"sensor only, database only or all components on the same system"
Be sure and check it out, if this will help you get starting using Snort and Sguil faster.

Link to the Internet Storm Center Article is here.
 

Thursday, February 10, 2011

SQueRT 0.8 has been released

SQueRT 0.8, a web GUI interface for SGuiL's database format (SGuiL uses Snort) has been released. Check out the below note from Paul Halliday:

-- country mappings can now be done in the background via cron
-- an input box has been added that will accept country names and
codes to filter queries
-- a country 'tag cloud' that can be primed prior to event queries has bee added
-- an exclude input box has been added so that you can pick away at
your query results
-- different property files can be used when creating link graphs
-- different split modes can be used when creating link graphs
-- canvas colour can be changed when creating link graphs
-- fixed protocol function to acknowledge unknown entries
-- fixed sorting problem with 'existing files' drop down
-- sensor selection now fully enumerates the sensor table and
(supported) agent types

## Pictures are here:

http://www.pintumbler.org/sq8

## More information is available here:

http://www.pintumbler.org/Code/squert

## You can download it here:

http://sourceforge.net/projects/squert/files/

Sourcefire doesn't make any official endorsement to projects surrounding Snort.

Thursday, January 13, 2011

GUIs for Snort

I asked for people to send me topics that they'd like to learn more about in Snort, and I received a good amount of responses.  So I thought I'd get started on one of them.  (BTW - if you'd like to get our input on something Snort related for the blog, please feel free to email me at joel [at] snort.org)

Every so often (probably twice a year) there seems to be an uptick in the amount of people emailing the mailing lists asking about GUIs for Snort.  Many of them repeat offenders.  So I am guessing that either people don't know about the GUI options for Snort or people don't like the ones they have.  So let's start off with a few in alphabetical order:

BASE
BASE, the Basic Analysis and Security Engine was based off of the old ACID code codebase.  The ACID GUI interface (which is now dead, and has been for about five or six years) was a college project written by an attendee of Carnegie Mellon.  It hasn't been actively developed since about 2003.  BASE, a fork of the ACID code, picked up where the original author left off, added a bunch of new features, and made it easy to use, multi-language, and a  highly functional GUI.  There were plans for a redesign of BASE, including the database format that it reads from, but Kevin Johnson, the original BASE project manager has since left the project and turned the project over to new management.  However, it remains the most popular Snort GUI interface with over 215,000 downloads.  BASE is written in PHP, and has several dependencies.  BASE has it's own IRC channel #secureideas, although there is rarely anyone there, so most people come to the default #snort for help.

OSSIM
OSSIM, made by AlienVault stands for "Open Source Security Information Management".  Not only can it take the logs from Snort and display them in a great looking interface, but it also integrates with many other tools (p0f, arpwatch, pads, nessus, ntop, nagios, etc) for a consistant user interface.  I've personally never used this tool, but I've heard from the people that use do use it, and find it really a joy to use.

PLACID
Standing for "Phil Loathes ACID", it was originally made as a super stripped down way of simply looking at Snort Events in the Snort DB.  It has stayed that way.  There is a certain demographic of Snort users that like simple, text based interfaces, and PLACID serves that need.

SGUIL
(Pronounced "Squeel")  SGUIL started off as the "Snort GUI for Lamers".  The project, maintained by Bamm Vischer, is a multi part system consisting of a "Sensor", "Server", and "Client".  Not only is SGUIL a GUI for Snort, but it also integrates other technologies into the recording of data for use by the analyst as well (including fulltime, full packet capture).  This is a heavy weight technology, is written in TCL, and is a very well performing engine.  Most people start off with a GUI like BASE and move into SGUIL.  SGUIL also has it's own IRC channel #snort-gui.

Snorby
A relative newcomer to the Snort GUI area, Snorby uses a lot of "Web 2.0" effects and rendering providing the user with a very sharp and beautifully functioning tool.   This seems to be the current "go-to" web interface for Snort.  While it has many of the features of BASE (and a lot more, hotkeys, classifications, an iOS interface, and actual pdf reporting), and not as featured as SGUIL (in terms of architecture), it's extremely easy to deploy, looks fantastic, and functions as an alert browser very well.  Snorby's code is hosted on Github, here.  Another advantage of Snorby is that it integrates with the OpenFPC project.  Functioning similar to how SGUIL collects all information on the network using Full Packet Capture (FPC), Snorby gives you the ability to not only view the Snort alert, but also to view the alerts in context with the rest of the packet flow on the network.  Snorby's IRC channel can be found at #snorby.

SQueRT
Paul wrote in about SQueRT.  SQueRT uses the SGuil database format and is also web based.  You can see the screenshots and download it at the link above.

This is by no means complete, these are just the most common that I see people using.  If I have missed a free Snort GUI that you enjoy, please feel free to respond in the comments.  The more complete your post, the better.  Give people links to your favorite tool.

Update:  http://blog.snort.org/2011/10/comparison-of-3-popular-snort-guis.html

FirePOWER
While not free by any means, the FirePOWER system is the commercial system that we develop here at Cisco.  Not only making the administration and analysis of events from Snort (the engine embedded into FirePOWER) extremely simple, it couples hundreds of more features into an extremely complex system with a simple to understand and navigate GUI.  Made to keep large deployments simple, and small deployments even easier, this is by far, the best system made. (We're biased)  But, is not free.

Polman, another rule manager for Snort

Most people know about rule management tools for Snort, Oinkmaster and even PulledPork, however, one of the members of the Snort community, Edward FjellskÃ¥l, decided that neither one of those tools was for him, so he wrote a rule manager, with kind of a different twist to it.

It's called Polman, it uses a "database" type format, and allows for mass enable, disable, search, and display of rules.  I think the concept has a lot of merit.

Check out the example here, and check out the blog post about the tool here.  A link for the download for the tool is on the blog post.

Wednesday, December 29, 2010

Working with unified and unified2 files

First you might ask "why do I care about unified files?" (unified2 these days, more on that later)

There is no perfect answer to that, just good reasons you should. Some of those reasons are:

  • It is a unified format allowing for easy archive management
  • It is the fastest output method available for snort
  • It is the preferred method used by Sourcefire and the most tested.
  • It is not easily modified. (It is still possible however)

8 years ago if you needed to get data from a snort unified file and do something with it you had few options, there was barnyard and that was it. This is fine and for good reason people were using unified output spooling to disk, reading those files with barnyard, and storing them in a mysql database. In the typical use case this was a perfect solution that worked for many years for many people. This solution had one major drawback though, unified files could not be processed by wetware without barnyard.

Normally this wouldn't be a problem as barnyard supported csv, fast alerts, database, pcap, etc as output methods. While a pain to process a unified file and then post process the results from several different file formats, it was sufficient to get the job done and not too painful.

...Unless you don't have barnyard available. This is exactly the situation that forced me to write the first iteration of SnortUnified.pm. I was working on a system that had unified files but no barnyard, no database, no compiler. Normally this isn't a problem either, you just install the tools you need but that was not allowed. As a matter of fact it was forbidden as these tools were not "certified" for use in production environments and the data was not allowed to leave this environment. The absurdity of that situation, having the data but no way to use it, by policy, was maddening. I thought to myself, who was the person that decided to use unified logging and nothing else? What was the value of having the data if you never intended to be able to use it? blag blah blah you can think the rest yourself, and just like I did, please keep those thoughts to yourself. Clearly their answer to not being able to use the data was to find a way to use the data when they had a need for it. Back to the point of the post.

In looking around on the system I quickly determined that there was xxd, perl, bash, and a smattering of other useful utilities. This was good as I was not looking forward to manual processing of these files from hex dumps. ( A side note is that I'm a fan of perl, it is a wonderful tool, so capable, so simple, so complex, it is what you need when you need something and at the same time the problem to your solution if done wrong.) The Snort source and perl was all I needed to get to the data they wanted and to save me from death by paper and pen. The first version of SnortUnified.pm was in fact process.pl and quite possibly the ugliest thing you have ever seen.

So here we are. What is this SnortUnified.pm thing?

SnortUnified.pm is a perl module for processing unified files (a set of perl modules actually). It isn't well documented outside the code itself, has few examples, and has saved me at least once. I know it is used in production environments and assume it is stable as I never get bug reports, just feature requests, though not since I added the processing hooks I'll be discussing below.

If you would like to know more you can browse the source in google code, get the latest source, or just continue reading, it is up to you. Working with SnortUnified is easy and depending on what you are trying to do could be as simple as a few lines of perl.
use SnortUnified(qw(:ALL));
use Data::Dumper;

$UF_Data
= openSnortUnified(shift);
while ( $record = readSnortUnifiedRecord() ) {
print Dumper($record);
}
closeSnortUnified
();

In the source tarball there is a samples directory with various samples available from converting a unified file to XML to signing the unified files or records within them; ensuring they cannot be modified without detection.

Two samples that I think are worth discussing and illustrate core functionality are uf_csv_handler.pl and uf_csv_qualifier.pl

The comments in the beginning of the handler script are sufficient for us to discuss here. First up are handlers, these get called for each record type / action defined and allow you to work with the records without having to change your functional tools. I initially envisioned this as a way to handle transformation of data within the records without having to rework the module. EG: convert an IP address integer to a dotted quad without having to change anything else. I also find it is quite useful for other tasks like converting payloads, debugging, adding output methods, etc. The result of a handler is irrelevant to the processing path, qualifiers are there for that.
# Handlers come before qualifiers come before pcre

# handlers will be run and regardless of the result processing will continue
# The available handlers for SnortUnified.pm are
# ("unified_opened", $UF);
# ("unified2_event", $UF_Record);
# ("unified2_packet", $UF_Record);
# ("unified2_unhandled", $UF_Record);
# ("unified2_record", $UF_Record);
# ("unified_record", $UF_Record);
# ("read_data", ($readsize, $buffer));
# ("read_header", $h);

register_handler('unified2_packet', \&make_ascii_pkt);
register_handler('unified_record', \&make_ascii_pkt);
# register_handler does not care about return values so the following will continue
# register_handler('unified_record', \&make_noise_fail);
# register_handler('unified_record', \&make_noise);

register_handler('unified2_record', \&printrec);
register_handler('unified_record', \&printrec);

# show_handlers();

# Qualifiers will be run, if any return a value less than 1
# then the record will be discarded and processing will continue
# with the next record in the file
# Only one option for unified types

# Skip all but sid 402
# register_qualifier(0,0,0, sub{return 0;});
# By having something specific for 402
# register_qualifier(0,1,402, \&printrec);
# register_qualifier(0,1,402, sub{return 1;});
# register_qualifier(0,1,402, \&make_noise);
# register_qualifier(0,1,402, \&make_noise);
# register_qualifier(0,1,402, \&make_noise_fail);
# register_qualifier(0,1,402, \&make_noise_never);

# But you can be granular with unified2 types
# register_qualifier($UNIFIED2_IDS_EVENT,1,402, \&make_noise);
# register_qualifier($UNIFIED2_PACKET,1,402, \&make_noise);


# register_pcre(1,402, "test");
# register_pcre(1,402, "*");
# register_pcre(1,402, ".*ads.clickagents.com.*");

# show_qualifiers();


Breaking out the comments above:

  • unified_opened gets called every time a new unified file is opened. This allows you to do things like hook in verification routines, mark progress in a waldo file, create verbose informed logging, etc.
  • unified2_event gets called every time a unified2 event record type is encountered in the unified file.
  • unified2_packet gets called every time a unified2 packet record type is encountered in the unified file.
  • unified2_unhandled gets called if a record type that is unknown / not handled by default is encountered in the unified2 file. This is useful as a bridge for new types, like the recently added 104 and 105 event types. I suggest that minimally you register a handler to log the unhandled types to a file so you can further investigate.
  • unified_record gets called for earlier types of records only found in "unified" files, not "unified2" files
  • read_data gets called every time we read data from the unified file.
  • read_header gets called every time we read a header from the file.
  • register_handler('unified2_packet', \&make_ascii_pkt); registers the "make_ascii_pkt" subroutine as a handler for all unified2_packet records. It gets called as follows make_ascii_packet($UF_Data) and line 105 of the uf_csv_handler is the beginning of that routine.
  • show_handlers() will print the handlers that are registered.
Qualifiers are intended to allow you to "tune after the fact" and I think this is the right way to handle a lot of tuning tasks. We should not be artificially limited in our detection scope simply because that detection potentially creates a lot of "noise". I think that we should have the freedom to cast the widest net necessary and possible when it comes to these things, especially when investigating evolving threats and potentially targeted and ongoing attacks.

Qualifiers also allow you to get very specific in your actions. You can limit a qualifier to a specific sid or cast a super wide net on all events. Qualifiers are all executed so you can do things like easily suppress output, except for specific events. You can also get granular within those specific events. In the comments above all events are suppressed except gen:1 sid:402 and then only for sid:402 events that contain the PCRE ".*ads.clickagents.com.*". You could make the qualifier look at the IP address, check an archived (as in comprehensive) blacklist and change the context of the event, etc. I used a PCRE earlier, this is done with a special type of qualifier (purpose built?) that makes it trivial to use a regular expression as a qualifier though it has one caveat, a PCRE will not override the decision of other qualifiers, it only has a vote but in the case of conflict, the proper qualifier wins.

Qualifiers are called in the same way as handlers but the return code is checked and any return with a value that is less than one is discarded, processing continues on the next record, and
your code never knows that an event was skipped. If you want or need tracking of these things you could use the qualifier to populate some stats and a handler to keep global stats.

If you use the code please drop me a note letting me know, it helps me gauge interest and any impact of possible changes. I've not modified the code in some time, largely because I've not needed to. But, if you have patches, requests, criticisms, commentary, or want to catch a pint at the next event we are both at feel free to let me know.

That is all for now, if you would like to know more or need assistance don't hesitate to ask. I can be reached at jason...at...snort.org or jason...at...sourcefire.com. My details are also in the README contained in the release tarball if you don't want to come back to the post.

Happy Snorting!

Thursday, December 23, 2010

New Proposed Classification.config file setup

I know we are all in the Holiday Season, so one last item for you all to look over while you are roasting in front of the fire..

Recently on the Emerging-Threats Mailing list, Matt Jonkman proposed a new classification system to replace the aging Snort classification system that's been in use for years. We saw this as a good idea and after some internal discussions, decided to head the same route.

So we propose the following classification.config system to the community for comment, and we want to hear the feedback! Especially on descriptions and priorities. I'll assemble all the comments on January 12th (a date suggested by Matt Jonkman) and create a new classification.config file which we will then include in the official Snort tarball and in the VRT rules tarball.

We've made two major changes to the classification system as proposed by the Emerging-Threats list:
  1. We've converted all Underscores to Hyphens
  2. We've made everything lowercase.

This was done to ensure compatibility with existing output modules (barnyard2, unified, unified2, barnyard, SnortUnified.pm, etc), GUI's (BASE, Snorby, Placid, etc), and internal (to Snort) parsers.

The proposed classification.config configuration parameters are available for download here:
http://www.snort.org/assets/157/classifications.txt, and are pasted below. Please leave comments on the blog, and we'll assemble them into a final product:


config classification: exploit-shellcode, A known shellcode payload was detected,1
config classification: exploit-sql-injection, A known SQL injection attack was detected,1
config classification: exploit-browser, A known client-side browser exploit was detected,1
config classification: exploit-activex, A known client-side ActiveX exploit was detected,1
config classification: exploit-command-execution, A known command execution exploit was detected,1
config classification: exploit-cross-site-Scripting, A known cross site scripting XSS attack was detected,2
config classification: exploit-ftp, A known exploit targeting ftp servers or clients was detected,1
config classification: exploit-file-inclusion, A known file inclusion attack was detected,2
config classification: exploit-windows, A known attack targeting Windows systems was detected,1
config classification: exploit-directory-traversal, A directory traversal attack was detected,2
config classification: exploit-attack-response, A known string indicating a host has been comprised was detected,1
config classification: exploit-denial-of-service, A known DoS or DDoS packet payload was detected,2
config classification: exploit-pdf, A known exploit targeting PDF files was detected, 1
config classification: exploit-buffer-overflow, A known exploit using a buffer overflow was detected,1
config classification: exploit-spoofing, A known spoofing attacker was detected,3
config classification: exploit-format-string, A known exploit utilizating a format string overflow was detected,1
config classification: exploit-misc, A known exploit targeting an unclassificated system was detected,2
config classification: exploit-dns, A known exploit targeting DNS systemes was detected,1
config classification: exploit-mail, A known exploit targeting Mail servers was detected,1
config classification: exploit-samba, A known exploit targeting Samba servers or clients was detected,1
config classification: exploit-linux, A known exploit targeting Linux based systems was detected,1
config classification: authentication-bruteforce, An attempt to bruteforce usernames and passwords was detected,2
config classification: authentication-bypass, An attempt to bypass login authentication was detected,2
config classification: authentication-login, A login attempt to any service or system was detected,4
config classification: authentication-Failed, A failed login attempt was detected,4
config classification: authentication-cleartext, A authentication request was detected in plain text,4
config classification: authentication-logout, A logout request was detected,4
config classification: authentication-disclosure, During an authentication request the username or password was disclosed,4
config classification: authentication-default-credentials, An attempt to login with publicly known default usernames or passwords was detected,4
config classification: access-web-application-access, A known web application was accessed,4
config classification: access-file-Access, A known default file was accessed,4
config classification: access-misc, What is an Access-Misc,4
config classification: malware-spyware, A known Spyware application was detected,2
config classification: malware-adware, A known Adware application was detected,2
config classification: malware-fake-Antivirus, A known Fake Anti-virus application was detected,1
config classification: malware-keylogger, A known KeyLogger application was detected,1
config classification: malware-trojan, A known Trojan was detected,1
config classification: malware-virus, A kown Virus was detected,1
config classification: malware-worm, A known Worm was detected,1
config classification: malware-generic, A known unclassified malware application was detected,2
config classification: malware-backdoor, A known backdoor was detected,1
config classification: policy-adult, A known Adult website or other system was accessed,4
config classification: policy-p2p, A known P2P application was detected,4
config classification: policy-instant-messaging-chat, A known Instant Messaging application was detected,4
config classification: policy-anonymity, A known privacy application was detected,4
config classification: policy-games, A known online game was detected,4
config classification: policy-other, A generic policy violation has occurred,4
config classification: denial-of-service-web-application, A known Denial of Service attack was detected against a web application,3
config classification: denial-of-service-application, A known Denial of Service attack was detected against an application,4
config classification: denial-of-service-flood, A known traffic flooding tool was detected,4
config classification: denial-of-service-ddos, A known DDoS tool was detected,4
config classification: suspicious-blacklist-address, A known malicious host was detected,2
config classification: suspicious-web-attack-or-scan, A known scanning tool was detected,2
config classification: suspicious-bad-traffic, Malformed or incorrectly formatted network traffic was detected,4
config classification: suspicious-network-activity, Strange or suspicious network traffic was detected,4
config classification: suspicious-scada-activity, SCADA traffic was detected,4
config classification: suspicious-dns-activity, Suspicious DNS traffic was detected,4
config classification: suspicious-ssh-activity, Suspicious SSH traffic was detected,4
config classification: suspicious-nfs-activity, Suspicious NFS traffic was detected,4
config classification: suspicious-database-activity, Suspicious database activity was detected,4
config classification: suspicious-netbios-activity, Suspicious netbios activity was detected,4
config classification: suspicious-rpc-Activity, Suspicious RPC activity was detected,4
config classification: suspicious-mail-activity, Suspicious Mail activity was detected,4
config classification: network-tftp-activity, TFTP traffic was detected,4
config classification: network-ftp-Activity, FTP traffic was detected,4
config classification: network-snmp-Activity, SNMP traffic was detected,4
config classification: network-smtp-Activity, SMTP traffic was detected,4
config classification: network-telnet-activity, Telnet activity was detected,4
config classification: recon-misc, A network probe was detected,4
config classification: recon-scanner, A network scanner was detected,4
config classification: network-ntp-activity, NTP traffic was detected,4
config classification: network-sip-activity, SIP traffic was detected,4
config classification: network-dhcp-activity, DHCP traffic was detected,4
config classification: access-firewall-permit, A firewall permit rule triggered,4
config classification: access-firewall-deny, A firewall deny rule triggered,4
config classification: access-acl-permit, A ACL permit rule was triggered,4
config classification: access-acl-deny, A ACL deny rule was triggered,4
config classification: authentication-policy-added, A policy addition occured,4
config classification: authentication-policy-changed, A policy change occured,4
config classification: authentication-policy-deleted, A policy delete occured,4
config classification: authentication-ftp-login-succeeded, A successful FTP login occured,4
config classification: authentication-ftp-login-failed, A failed ftp login occured,4
config classification: authentication-password-change-failed, A password change failure occured,4
config classification: authentication-password-change-succeeded, A password change occured,4
config classification: authentication-user-created, A new user was created,4
config classification: authentication-user-deleted, A user was deleted,4
config classification: authentication-user-changed, A user was changed,4
config classification: authentication-admin-access, An admin accessed the system,4
config classification: authentication-group-added, A new group was added to the system,4
config classification: authentication-group-deleted, A new group was deleted from the system,4
config classification: authentication-group-changed, A group was changed on the system,4
config classification: authentication-auth-required, Authentication is required for access,4
config classification: authentication-account-lockout, An account was locked,4
config classification: authentication-account-unlocked, An account was unlocked,4
config classification: antivirus-virus-detected, An Antivirus system detected a virus,2
config classification: antivirus-virus-quarantine, An Antivirus system quarantined a virus,2
config classification: antivirus-virus-quarantine-failed, An Antivirus system filed to quarantine a virus,1
config classification: system-configuration-error, A system has indicated it has a configuration error,2
config classification: antivirus-definitions-updated, A system updated its Antivirus definition,4
config classification: antivirus-definitions-updated-failed, A system failed to update its Antivirus definitions,2
config classification: antivirus-unknown-event, A unknown event occured,4
config classification: antivirus-started, A antivirus agent came online,4
config classification: antivirus-disabled, An Antivirus agent was disabled,2
config classification: antivirus-scan-started, An Antivirus scan was started,2
config classification: antivirus-scan-finished, An antivirus scan has completed,2
config classification: antivirus-error, A unclassified error occured on an Antivirus system,3
config classification: application-web-opened, A web browser was opened, 4
config classification: application-web-closed, A web browser was closed, 4
config classification: application-web-reset, A web site sent a reset to a client, 4
config classification: application-web-terminated, A web site was terminated with extreme predujice, 4
config classification: application-web-denied, Packet come in packet deny, 4
config classification: application-web-redirected, A web client was redirected to a new page,4
config classification: application-web-proxy, A web proxy was detected,4
config classification: application-web-error, A misc error was detected,4
config classification: application-web-misc, A Web misc was detected,4
config classification: application-web-not-found, A web application generated a not found error,4
config classification: access-traffic-inbound, Inbound traffic was detected,4
config classification: access-traffic-outbound, Outbound traffic was detected,4
config classification: access-firewall-misc-event, A unclassified event occured on the firewall,4
config classification: suspicious-network-anomaly, Something strange happened I don't know what,4
config classification: suspicious-dns-protocol-anomaly, A suspicious DNS sessions or packet was detected,3
config classification: suspicious-ssh-protocol-anomaly, A suspicious ssh session or packet was detected,3
config classification: suspicious-telnet-protocol-anomaly, A suspicious telnet session or packet was detected,3
config classification: suspicious-http-protocol-anomaly, A suspicious HTTP session or packet was detected,3
config classification: suspicious-mail-protocol-anomaly, A suspicious Mail session or packet was detected,3
config classification: suspicious-ftp-protocol-anomaly, A suspicious FTP session or packet was detected,4
config classification: suspicious-threshold-exceeded, A suspicious threshold was triggered,4
config classification: denial-of-service-other, A new type of Denial of Service was detected,4
config classification: access-file-blocked, Access to a file was blocked,4
config classification: access-tunnel-connection, Access to a tunnel was identified,4
config classification: access-tunnel-closed, Access to a tunnel was closed,4
config classification: aystem-warning, A system Warning message was detected,4
config classification: system-emergency, A system Emergency message was detected,4
config classification: system-critical, A system Critical message was detected,4
config classification: system-error, A system Error message was detected,4
config classification: system-notification, A system Notification message was detected,4
config classification: system-information, A system Information message was detected,4
config classification: system-debug, A system Debug message was detected,4
config classification: system-alert, A system Alert message was detected,4
config classification: access-connection-opened, A connection was opened,4
config classification: access-connection-closed, A connection was closed,4
config classification: access-timeout, A timeout occurred,4
config classification: system-service-started, A service started,4
config classification: system-service-stopped, A service stopped,4
config classification: system-process-started, A process started,4
config classification: system-process-stopped, A process stopped,4
config classification: application-spam-detected, Some dirty spammer was detected,4
config classification: application-mail-dropped, The mail system dropped or refused mail,4
config classification: system-restart, A system restart was detected,4
config classification: system-started, A system startup was detected,4
config classification: system-stopped, A system stop was detected,4
config classification: system-locked, A system being locked was detected,4
config classification: system-unlocked, A system be unlocked was detected,4
config classification: network-ike-activity, IKE traffic was identified,4
config classification: network-h.323-activity, H.323 traffic was identified,4
config classification: network-ppp-activity, PPP traffic was identified,4
config classification: network-ocsp-activity, OCSP traffic was identified,4
config classification: network-l2tp-activity, L2TP traffic was identified,4
config classification: network-rip-activity, RIP traffic was identified,4
config classification: network-pptp-activity, PPTP traffic was identified,4
config classification: network-ssl-activity, SSL traffic was identified,4
config classification: network-igmp-activity, IGMP traffic was identified,4
config classification: network-ipsec-activity, IPSEC traffic was identified,4
config classification: network-pki-activity, PKI traffic was identified,4
config classification: voip-call-started, A VOIP call was started,4
config classification: voip-call-ended, A VOIP call was completed,4
config classification: voip-misc, A VOIP event occurred,4
config classification: network-bootp-activity, BOOTP traffic was identified,4
config classification: alert-ids-alert, The IDS did something,4
config classification: alert-ips-alert, The IPS did something,4
config classification: alert-hids-alert, The HIDS did something,4
config classification: application-mail-sent, An email was sent,4
config classification: application-mail-server-misc, A Mail server did something,4
config classification: application-mail-received, An email was recieved,4
config classification: availability-state-up, A system or service is now up,4
config classification: availability-state-down, A system or service is now down,4
config classification: availability-state-critical, A system or service is not in a critical state,1
config classification: availability-state-warning, A system or service has issued a warning,3
config classification: availability-state-unknown, A system or service is in an unknown state,3
config classification: availability-state-unreachable, A system or service is unreachable,1
config classification: application-vpn-opened, A VPN session was opened,4
config classification: application-vpn-closed, A VPN session was closed,4
config classification: application-vpn-denied, A VPN session was denied,2
config classification: application-vpn-misc, Something happened on a VPN session,2
config classification: system-configuration-changed, A system changed its configuration,4
config classification: network-misc, Something happened on the network,4
config classification: policy-phishing, A phishing attempt was detected,4
config classification: wireless-new-network, A new wireless network has been detected,4
config classification: wireless-client-associated, A new client has connected to the wireless network,4
config classification: wireless-flood, The wireless network is currently being flooded,2
config classification: wireless-disassociation, A wireless client has been disassociated from the network,4
config classification: wireless-deauthentication, A wireless client has been deauthenticated,4
config classification: wireless-anomaly, Something strange occurred on the wireless network,4
config classification: wireless-spoofing, Spoofing has been detected on the wireless network,2
config classification: wireless-scanner-detected, A scanner was detected on the wireless network,2
config classification: wireless-misc, Something happened on the wireless network,2
config classification: wireless-probe, A probe attempt was identified on the wireless network,4
config classification: inventory-service-detected, A new service has been identified,4
config classification: inventory-service-change, A service has changed,4
config classification: inventory-service-misc, A Misc service was detected,4
config classification: inventory-operating-system-detected, A new operating system was detected,4
config classification: inventory-operating-system-change, A system changed,4
config classification: inventory-operating-system-misc, A system met a Misc,4
config classification: inventory-mac-detected, A unhackable computer was detected,1
config classification: inventory-mac-change, A MAC address changed,4
config classification: policy-check-failed, A Policy check has failed,1
config classification: policy-check-passed, A Policy check has passed,1
config classification: network-high-load, The network currently has a high load,1
config classification: authentication-error, An authentication error was detected,4
config classification: application-web-modified, A content modified proxy request was identified,4
config classification: application-dhcp-release, A DHCP lease was released,4
config classification: application-dhcp-request, A DHCP request was detected,4
config classification: application-dhcp-lease, A DHCP lease was allocated,4
config classification: application-dhcp-pool-exhausted, All DHCP addresses have been allocated,4
config classification: application-dhcp-error, A DHCP error was detected,4
config classification: system-software-installed, A software package was installed,4
config classification: honeypot-connection-opened, Something connected to the honeypot sweet new warez,4
config classification: honeypot-attack-detected, A known attack was detected on the honeypot,4
config classification: honeypot-connection-closed, A connection to the honeypot was closed,4
config classification: application-dns-succesful-zone-tranfer, A succesful DNS zone transfer was detected,4
config classification: application-dns-zone-transfer-failed, A failed DNS zone transfer was detected,4
config classification: application-ftp-command-executed, An FTP command was executed,4
config classification: application-ftp-error, An FTP error was detected,4
config classification: application-ftp-connection-opened, An ftp connection was opened,4
config classification: application-ftp-connection-closed, An ftp connection was closed,4
config classification: database-login, A database login was detected,4
config classification: database-login-failed, A failed database login was detected,4
config classification: database-query, A database query was executed,4
config classification: database-logout, A database logout was detected,4
config classification: database-stop, A database was stopped,4
config classification: database-start, A database was started,4
config classification: database-error, A database error occurred,4

Thursday, December 16, 2010

Can You Hold Your Data?

There is one mistake I see IPS analysts routinely make, one which considerably harms their organization’s ability to use the system and accurately respond to threats. And that is the failure to honestly consider how much time they actually have to devote to their IPS events.

How Much Time Do You Have For IPS?

If you want to get the maximum (or, really, any) value from your IPS (or IDS, if you prefer) then you have to stop and consider how much time you really have to spend with it. And be honest with yourself. How much time per day or week can you set aside? And get used to thinking about it, because you’re going to have to revisit this question regularly. After all, your schedule changes, and the value of your IPS alerts will always depend on the time you can give them.

The Value of Security Alerts

The value you gain from your IPS is not a database full of alert data, but the enhanced ability to respond to intruders. The data is useless without somebody to use it. So the value of your data will always be a function of how it is utilized. That is, how it helps you detect and respond to threats to your environment.

Raw data are often discussed as a commodity. And in many ways it is. I fear, however, that thinking about data this way encourages people to focus on collection and storage. Simply possessing data does not mean you are benefiting from it. A better analogy is that data are grown like, for example, corn. You can try to grow as much as you wish, but if you don’t have the manpower to harvest it, it will rot there.

Consuming Your Data

The value in farming corn is not simply having grown it, but that it can be transformed into something people want to consume. To continue to belabor this contrived analogy, we don’t just want to grow corn, we want to harvest it and turn it into a nice sour mash whiskey. Drinking whiskey is a lot like analyzing IPS alerts. The first sip you ever have is strange and burns. But once you develop a taste for it, it’s delicious. Then the first couple drinks are good, and sometimes the next couple are even better. But eventually having more makes you sick, and in obscene amounts it can even kill you.

It is the same with your IPS alerts (essentially, but I haven’t heard of any IPS related fatalities... yet). First time you plug your IPS in you may not get much value out of it. But once it’s tuned then the alerting you receive is beneficial. And more data may even give you a better picture of your network and who’s attacking it. But eventually there’s just too much data to handle. And not only are you receiving no value from this pile of data, but maintaining it and attempting to sort through it are costs to you. If you’ve reached this point then you either need to reduce the amount of data you have or hire more people to analyze it. Otherwise why even have the IPS? Collecting information nobody ever looks at is pointless.

Beginners’ Tolerance

While your new IPS system is being deployed and configured it is usually done by somebody focusing a lot of time on an IPS project. What you need to realize is that after this project is another project. And when your time is no longer focused primarily on IPS that data will still be rolling in. When focused on their IPS system people have a tendency to start collecting more data than they will be able to handle. The most common ways this happen are 1) turning on too many rules and 2) sensor placement.

First, turning on too many rules is a temptation some just can not refuse. But not only do all those extra rules place a performance burden on Snort, but they create alerts you need to analyze. So are those alerts really something useful? If not, then get rid of them. They’re creating useless data. And while those data continue to pour in, clogging the system with alerts, this only works to obscure the data you actually need to respond to.

Second, do you really need a sensor outside of the firewall? A lot of security analysts are by their very nature very curious. And the opportunity to see what kind of malware is floating around outside on the Internet is intriguing. But a month from now, when you’ve moved on to another project, are you going to have the time to wade through the flood of alerts that detected attacks that were subsequently stopped by your firewall? How useful is that? How will you be using and responding to these? Once you consider this, there’s a good chance you can find a better location for that sensor

Last Call

Your time and attention is limited and valuable. You need to understand this in order to get the full value from your IPS. You need to tune the system until you are no longer burdened with useless data (Or at least set that as the goal, it is typically a moving target.). And that means considering what traffic you need to monitor, and what rules will you actually respond to.


Think about it. Be honest. And when you can use your IPS without getting drunk on data you’ll be in good shape.