Showing posts with label barnyard2. Show all posts
Showing posts with label barnyard2. Show all posts

Tuesday, May 14, 2013

Barnyard v.2.1-13 has been released!

We are happy to announce the latest STABLE release v2.1-13 which was tagged a few hours ago (https://github.com/firnsy/barnyard2/tags)

This release is a bug fix release that also introduce a few new features and enhancements.


UPGRADE REQUIREMENTS

If you are upgrading to barnyard2 2-1.13 (build 327) or above from a previous version and using output database.

You will need to delete every row in your sig_reference table. (DELETE FROM sig_reference;)

The table will be re-populated at startup, and has no impact on historical data.


FEATURE REQUESTS
Phil Daws - add interface and hostname field to spo_alert_csv if specified.
Jorge Pinto - spo_syslog_full support for ASCII,BASE64 payload
Jason Brvenik - variables ... (a long time ago, sorry :P)
Martin Olsson - remove some useless verbosity unless ./configure --enable-debug is specified and proper flag are used (spo_database and sid-msg.mapv2)
All other barnyard2 users who help and contribute.

BUG REPORTS
Martin Olsson - bug in sig_reference generation and good discussions. Rewrote the code & al
John Eure and others - autogen.sh could cause some issue on some system so [autoreconf -fv --install] is not set to autoreconf -fvi
John Naggets - spo_database: could stop barnyard2 from processing new event if some packets with ip option where processed and option_len was null.
Fäbu Hufi - spo_syslog_full: in complete mode was printing wrong ip version information and ip header length.
Jeremy Hoel - identified issue with suppression range in 2-1.13-BETA (fixed in release)
Bill Green - identified is with signature insertion mainly preprocessor in 2-1.13-BETA (fixed in release)
All other barnyard2 users who help and contribute.

NEW FEATURES
1. Support for sid-msg.map version 2 format.

A new sig-msg.map format can be generated by pulledpok (upcomming release, already in svn).

Detection of sid-msg.map version is done by a simple header in the file that shouldn't be altered if you want it to be processed correctly.

The sig-msg.map version 2 format extends the information already present in the sid-msg.map file created from rules.

This new format version allow signature pre-population if users are using output database method with barnyard2 2-1.13 and above.


sid-msg.map v1 format:

SID || MSG || REF 1 || REF N

sid := integer
msg := string
ref := string

sid-msg.map v2 format:

GID || SID || REV || CLASSIFICATION || PRIORITY || MSG || REF 1 || REF N

gid := integer
sid := integer
rev := integer
classification := string (if NULL set to NOCLASS)
priority := integer (if prio == 0, classification priority is used)
msg := string
ref := string

=====================
generator (GID, gen-msg.map) are defaulted to the following value
if their information is not overruled in sid-msg.map v2 file via processing of preprocessor.rules:

revision 1
classification 0
priority 3

If generator message is present in the sid-msg.map v2 file, and gen-msg.map message are longer
(more comprehensive by string length),
gen-msg.map messages are used instead of sid-msg.map v2 file generator messages.
=====================


2. Signature/event logging suppression at spooler level.

Read doc/README.sig_suppression


3. Configuration file variables.

You can now use [var VARNAME value] in the barnyard2 configuration file and every instance of $VARNAME will get replaced by value.

Note that variable declaration order is important only you include a variable with in a variable.

EX (is VALID):
var INTERFACE ethX
var PATH /var/log/IDS
var LOG $PATH/$INTERFACE/log
var ARCHIVE $PATH/$INTERFACE/archive

EX (is INVALID):
var LOG $PATH/$INTERFACE/log
var ARCHIVE $PATH/$INTERFACE/archive
var INTERFACE ethX
var PATH /var/log/IDS


4. New output database configuration keyword.

Keywords connection_limit and reconnect_sleep_time where added in 2-1.10 but where "undocumented" and shouldn't be modified unless you encounter an issue.

connection_limit : default 10
The maximum number of time that barnyard2 will tolerate a transaction faillure and or database connection failure.

reconnect_sleep_time : default 5
The number of seconds to sleep betwen connection retry.

disable_signature_reference_table
Tell the output plugin not to synchronize the sig_reference table in the schema.

Note: This option will speedup the process, especialy if you use sid-msg.mapv2 file or have alot of signature already in databases. (Make sure that you do not need that information before enabling this)


So we hope you enjoy the new release, as a side note the RELEASE.NOTES file has not been updated and will be removed in the next version. It's honestly the most laborious part of release time ;)

Regards,

The barnyard2 team.

Wednesday, April 10, 2013

Barnyard2 2-1.13-BETA is now available!

We are happy to announce the Availability of Barnyard2 2-1.13-BETA which can be downloaded from HERE: https://github.com/firnsy/barnyard2.git

This release is a bug fix release that also introduce a few new features and enhancements


=====================
UPGRADING REQUIREMENT
=====================
----------------------
If you are upgrading to barnyard2 2-1.13 Build 325 or above from a previous version  that is not 2-1.13 and using the output database.

***** We highly recommend ******
To delete every row in your sig_reference table. (DELETE FROM sig_reference;) The table will be re-populated at  process startup, and has no impact on historical data.
----------------------
=====================
UPGRADING REQUIREMENT
=====================





Feature request:
----------------
Phil Daws:        Add interface and hostname field to spo_alert_csv if specified.

Jorge Pinto:      spo_syslog_full support for ASCII,BASE64 payload

Jason Brvenik:  variables .....(a long time ago, sorry :P)

Martin Olsson:  Remove some useless verbosity unless ./configure --enable-debug is specified and proper flag are used (spo_database and sid-msg.mapv2)

*And all other barnyard2 users who help and contribute.

Bug report:
-----------
Martin Olsson:              - bug in sig_reference generation and good discussions.

John Eure and others   - autogen.sh could cause some issue on some system so [autoreconf -fv --install] is not set to autoreconf -fvi

John Naggets               - spo_database: could stop barnyard2 from processing new event if some packets with ip option where processed and option_len  was null.

Fäbu Hufi                     - spo_syslog_full: in complete mode was printing wrong ip version information and ip header length.

*And all other barnyard2 users who help and contribute.


New feature:
------------


Support for sid-msg.map Version 2 format.
-------
A new sig-msg.map format can be generated by pulledpok (upcoming release, already in svn). Detection of sid-msg.map version is done by a simple header in the  file that shouldn't be altered if you want it to be processed correctly.

sig-msg.map version 2 format extend the information already present in the sid-msg.map file created from rules.

This new format version allow signature  pre-population if users are using output database method with  barnyard2 2-1.13 and above.
______________________
sid-msg.map v1 format:
______________________
SID || MSG || REF 1 || REF N
sid := integer
msg := string
ref := string
______________________
sid-msg.map v2 format:
______________________
GID || SID || REV || CLASSIFICATION || PRIORITY || MSG || REF 1 || REF N
gid := integer
sid := integer
rev := integer
classification := string (if NULL set to NOCLASS)
priority := integer (if prio == 0, classification priority is used)
msg := string
ref := string
=====================
generator (GID, gen-msg.map) are defaulted to the following value if their information is not overruled in sid-msg.map v2 file via processing of preprocessor.rules:
revision 1
classification 0
priority 3
If generator message is present in the sid-msg.map v2 file, and gen-msg.map message are longer (more comprehensive by string length), gen-msg.map messages are used instead of sid-msg.map v2 file generator messages.
=====================
 -------


Signature/event logging suppression at spooler level
-------
Read doc/README.sig_suppression
configuration file Variables:
-------

Barnyard2 configuration Variables
 -------
You can now use [var VARNAME value] in the barnyard2 configuration file and every instance of $VARNAME will get replaced by value.
Note that variable declaration order is important only you include a variable in a variable.
EX (is VALID):
 var INTERFACE ethX
 var PATH /var/log/IDS
 var LOG $PATH/$INTERFACE/log
 var ARCHIVE $PATH/$INTERFACE/archive
 EX (is INVALID):
 var LOG $PATH/$INTERFACE/log
 var ARCHIVE $PATH/$INTERFACE/archive
 var INTERFACE ethX
 var PATH /var/log/IDS
 -------

new output database configuration keyword
-------

Keywords connection_limit and reconnect_sleep_time where added in 2-1.10 but where "undocumented" and shouldn't be modified unless you encounter connectivity issue.

connection_limit <integer>: default 10  - The maximum number of time that barnyard2 will
tolerate a transaction failure and or database connection failure.

reconnect_sleep_time <integer> : default 5 - The number of seconds to sleep between connection retry.

disable_signature_reference_table - Tell the output plugin not to synchronize the sig_reference table in the schema. This option will speedup the process, especially if you use sid-msg.mapv2 file or  have a lot of signature already in databases. (Make sure that you
do not need that information before enabling this)
 -------


Enjoy and do not hesitate to send feedback/suggestion/feature request.

The barnyard2 team.

Friday, November 30, 2012

Barnyard2 - v2-1.11 released


It appears that an early tag of 2-1.11 crept in a week or so ago before all the patches we wanted to merge were submitted. Nevertheless, we've now caught up with our queue and are formally tagging 2-1.11.

This is primarly a bug fix and usability improvement release. The salient points are as follows:

  * spo_database. Keep-alive (via ping) for postgresql databases.

  * Updated RPM spec file to support alternative pcap libraries and cleaned some existing cruft. Thanks to Brent Woodruff.

  * spo_alert_unixsock. Supports synchronisation, multiple connections and improved error reporting. Thanks to Martijn van Oosterhaut.

  * Many other general bug fixes and clean ups. Thanks to Jason Ish, Thorsten Fischer, Brad Voth and Bill Parker.

You can download the source in a number of ways:
  - https://github.com/firnsy/barnyard2/tags (as a zip/tarball)
  - git://github.com/firnsy/barnyard2.git (via a git clone)

- firnsy

Tuesday, September 25, 2012

Barnyard2 - v2-1.10 has been released

It's my great pleasure to finally announce the next stable release of barnyard2 v2-1.10 build(310).

After almost 20 months of development and continuous testing from the community we are happy to get this one out to the masses (without the beta tag).

This development cycle has seen a lot of changes, refinements and fixes. This will be the last version build arround the old database schema.

The next release of barnyard2 will come with new database output that only support the new schema, native IPv6 support and FULL unified2 support for all output plugin.

I could go on about the changes, but the wait has been long enough. Here's a summary of the more notable changes:
 * Additions
 - spo_database. Support of encrypted connections to postgresql is now available. See README.database for the appropriate options.
 - spo_sguil. Fixed issue with duplication of alerts.
 - Completely re-written database plugin for performance optimisation against the original DB schema. 
NOTE: If you have intentions of running this new version we highly recommended you to clean two databases table for better performance: reference and sig_reference, not doing so will not break anything but could slow the startup caching process).
 - New Bro output plugin (thanks to Seth Hall)
 - A new syslog plugin (syslog_full) that support local and remote TCP and UDP syslog. * Improvements
 - Improved support against the latest Unified 2 format. Extended headers are read, however no plugins use the information currently.
 - Improved core IPv6 support.
 - Compile under cygwin
 - And many, many bugfixes.

 You can download the source in a number of ways:
 - https://github.com/firnsy/barnyard2/tags (as a zip/tarball)
 - git://github.com/firnsy/barnyard2.git (via a git clone)

 I would like to pay a special thanks to Eric Lauzon (the newest member of the core development team) and the many people who have helped along the road: Russell Fulton, Tim Shelton, JJ Cummings. Michael Steele, Brett Edgar, Bill Parker, Miguel Alvarez, Martin Holste, Jason Haar and any others who I may have missed.

Regards,

firnsy

Friday, September 23, 2011

Snort 2.9.1 Installation Guide for RHEL 6.1 has been released

Thanks to Randal Rioux for producing his awesome installation guide for RHEL 6.1 Server x64 for Snort 2.9.1.

Not only does Randy display how to set up Snort 2.9.1, but also includes the instructions for Barnyard2.  Nice work.

Once again, thanks Randy for his document, it's posted on http://www.snort.org/docs as always!

Friday, September 16, 2011

Snort 2.9.1 HTTP and SMTP logging features

To provide better context to the alerts generated by Snort, Snort version 2.9.1 introduced some new logging features for HTTP and SMTP which will help the user to better analyze the alerts. This logging of extra data by HTTP Inspect and SMTP preprocessors is similar to the X-Forwarded-For/True-Client-IP logging introduced prior to Snort 2.9.1.

HTTP Logging:


Let's talk about HTTP URI and Hostname logging.

How to enable logging of HTTP URI and Hostname?

To turn on the logging of the HTTP Request URI, the config option to use is "log_uri" and to enable the logging of hostnames, use the config option "log_hostname" as shown in the example below:

preprocessor http_inspect: global memcap
preprocessor http_inspect_server: log_uri \
log_hostname


The memcap here will determine the maximum amount of memory the HTTP Inspect preprocessor will use for logging the URI and Hostname data. You can refer to the Snort Manual for further details.

When these features are turned on in HTTP Inspect, the HTTP Request URI and HTTP Request hostname headers are extracted and logged to unified2 as an "extra data event" with each alert generated for that particular session. It is recommended to turn on stream5 reassembly with PAF on HTTP ports to ensure correctness and accuracy.

When a HTTP Request URI is greater than 2048 or when a HTTP hostname (specified in the "Host" Request header) is greater than 256, Snort will log the truncated the URI and/or hostname. A preprocessor alert with GID:119 and SID:25 is generated when hostname exceeds 256 bytes.

There is also a preprocessor alert with GID:119 and SID:24 for multiple "Host" headers in one HTTP request.

Please note that the URI and hostname are only logged in Unified2 mode and not logged with -A cmg.

So, how do you read the output?

Unified2 can be read using the Snort tool u2spewfoo. You can find it under the tools/u2spewfoo directory in the Snort tarball.

Example of the Output:


(Event)
sensor id: 0 event id: 2 event second: 1299776137 event microsecond: 355217
sig id: 1 gen id: 1 revision: 0 classification: 0
priority: 0 ip source: 10.1.2.3 ip destination: 10.9.8.7
src port: 60710 dest port: 80 protocol: 6 impact_flag: 0 blocked: 0

Packet
sensor id: 0 event id: 2 event second: 1299776137
packet second: 1299776137 packet microsecond: 355217
linktype: 1 packet_length: 214
[ 0] 02 09 08 07 06 05 02 01 02 03 04 05 08 00 45 00 ..............E.
[ 16] 00 C8 00 04 00 00 40 06 5C 19 0A 01 02 03 0A 09 ......@.\.......
[ 32] 08 07 ED 26 00 50 00 00 00 02 00 00 00 02 50 10 ...&.P........P.
[ 48] 01 00 0C 19 00 00 47 45 54 20 2F 20 48 54 54 50 ......GET / HTTP
[ 64] 2F 31 2E 30 0D 0A 48 6F 73 74 3A 20 77 77 77 2E /1.0..Host: www.
[ 80] 77 33 2E 0D 0A 20 6F 72 67 20 0D 0A 55 73 65 72 w3... org ..User
[ 96] 2D 41 67 65 6E 74 3A 20 4D 6F 7A 69 6C 6C 61 2F -Agent: Mozilla/
[ 112] 35 2E 30 20 28 58 31 31 3B 20 55 3B 20 4C 69 6E 5.0 (X11; U; Lin
[ 128] 75 78 20 78 38 36 5F 36 34 3B 20 65 6E 2D 55 53 ux x86_64; en-US
[ 144] 3B 20 72 76 3A 31 2E 39 2E 32 2E 31 35 29 20 47 ; rv:1.9.2.15) G
[ 160] 65 63 6B 6F 2F 32 30 31 31 30 33 30 33 20 55 62 ecko/20110303 Ub
[ 176] 75 6E 74 75 2F 31 30 2E 31 30 20 28 6D 61 76 65 untu/10.10 (mave
[ 192] 72 69 63 6B 29 20 46 69 72 65 66 6F 78 2F 33 2E rick) Firefox/3.
[ 208] 36 2E 31 35 0D 0A 6.15..



(Event)
sensor id: 0 event id: 3 event second: 1299776137 event microsecond: 355254
sig id: 1 gen id: 1 revision: 0 classification: 0
priority: 0 ip source: 10.1.2.3 ip destination: 10.9.8.7
src port: 60710 dest port: 80 protocol: 6 impact_flag: 0 blocked: 0

Packet
sensor id: 0 event id: 3 event second: 1299776137
packet second: 1299776137 packet microsecond: 355254
linktype: 1 packet_length: 176
[ 0] 02 09 08 07 06 05 02 01 02 03 04 05 08 00 45 00 ..............E.
[ 16] 00 A2 00 05 00 00 40 06 5C 3E 0A 01 02 03 0A 09 ......@.\>......
[ 32] 08 07 ED 26 00 50 00 00 00 A2 00 00 00 02 50 10 ...&.P........P.
[ 48] 01 00 8F E5 00 00 41 63 63 65 70 74 2D 45 6E 63 ......Accept-Enc
[ 64] 6F 64 69 6E 67 3A 20 67 7A 69 70 2C 64 65 66 6C oding: gzip,defl
[ 80] 61 74 65 0D 0A 41 63 63 65 70 74 2D 43 68 61 72 ate..Accept-Char
[ 96] 73 65 74 3A 20 49 53 4F 2D 38 38 35 39 2D 31 2C set: ISO-8859-1,
[ 112] 75 74 66 2D 38 3B 71 3D 30 2E 37 2C 2A 3B 71 3D utf-8;q=0.7,*;q=
[ 128] 30 2E 37 0D 0A 4B 65 65 70 2D 41 6C 69 76 65 3A 0.7..Keep-Alive:
[ 144] 20 31 31 35 0D 0A 43 6F 6E 6E 65 63 74 69 6F 6E 115..Connection
[ 160] 3A 20 6B 65 65 70 2D 61 6C 69 76 65 0D 0A 0D 0A : keep-alive....


(ExtraDataHdr)
event type: 4 event length: 33


(ExtraData)
sensor id: 0 event id: 2 event second: 1299776137
type: 9 datatype: 1 bloblength: 9 HTTP URI: /


(ExtraDataHdr)
event type: 4 event length: 43


(ExtraData)
sensor id: 0 event id: 2 event second: 1299776137
type: 10 datatype: 1 bloblength: 19 HTTP Hostname: www.w3.org


In the output above, two extra data events are generated for the event with GID:1 and SID: 1. The first extra data event displays the URI and the second displays the hostname associated with that session.

It is important to note that the unique "event id" and "event second" are used to correlate the alerts with their extra data. The "event type" indicates the presence of an "extra data" record and the unique "type" will determine the type of the extra data event logged. Here is a list of extra data types used in Snort.

Type 1: True-Client-IP/XFF IPv4 address
Type 2: True-Client-IP/XFF IPv6 address
Type 4: HTTP Gzip decompressed data
Type 5: SMTP filename
Type 6: SMTP MAIL FROM addresses
Type 7: SMTP RCPT TO addresses
Type 8: SMTP Email headers
Type 9: HTTP Request URI
Type 10: HTTP Request Hostname
Type 11: Packet's IPv6 Source IP Address
Type 12: Packet's IPv6 Destination IP Address

SMTP Logging:


The SMTP preprocessor, similar to HTTP Inspect preprocessor, can log extra data associated with each alert. The data that SMTP logs are as follows:

1. The email addresses in the MAIL FROM command.
2. The email addresses in the RCPT TO command. When there are multiple RCPT TO headers, the email addresses are concatenated using commas.
3. The filename of the MIME attachment extracted from the "Content-Disposition" header. Multiple filenames are appended with commas.
4. SMTP email headers.

How to enable logging of SMTP "extra data"?

SMTP preprocessor can log the extra data mentioned above using the following config options.

preprocessor smtp: memcap 838860 \
log_mailfrom \
log_rcptto \
log_filename \
log_email_hdrs \
email_hdrs_log_depth 56


The config options "log_mailfrom", "log_rcptto", "log_filename" and "log_email_hdrs" do not take any arguments. However, email_hdrs_log_depth requires the user to pass a number between 0-20480, which will determine in bytes the amount of email headers logged. The config option "memcap" is used to determine the maximum memory used for logging the above mentioned data.

Again, stream5 reassembly needs to be turned on for SMTP ports for this feature to work correctly. Without stream5 reassembly, the SMTP extra data won't be logged.

How to read the Output?

Like HTTP extra data, SMTP extra data can be read using the u2spewfoo and "event id" and "event second" are used to correlate the alerts with their extra data events.

Example of the Output:


(ExtraData)
sensor id: 0 event id: 2 event second: 1273194706
type: 6 datatype: 1 bloblength: 33 SMTP MAIL FROM Addresses:

(ExtraData)
sensor id: 0 event id: 2 event second: 1273194706
type: 7 datatype: 1 bloblength: 57 SMTP RCPT TO Addresses: ,

(ExtraData)
sensor id: 0 event id: 2 event second: 1273194706
type: 5 datatype: 1 bloblength: 26 SMTP Attachment Filename: sample.txt,foo.txt

(ExtraData)
sensor id: 0 event id: 2 event second: 1273194706
type: 8 datatype: 1 bloblength: 323 SMTP EMAIL HEADERS:
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: multipart/mixed; boundary="_----------=_12731947069000"
Date: Thu, 6 May 2010 21:11:46 -0400
From: bbantwal@sourcefire.com
To: bbantwal@sourcefire.com
Subject: Email Sent via Perl
X-Mailer: MIME::Lite 3.027 (F2.74; T1.29; A2.06; B3.07; Q3.07
<bbantwal@sourcefire.com><bbantwal@sourcefire.com><foo-barfoo@gmail.com>


We will not be updating the spo_database output method to insert this information into the database.  As a reminder, this output method will be removed in a future Snort release in 2012.  Sourcefire has turned over the maintenance of the "ACID" database schema to the Barnyard2 group, and will also, no longer be maintaing it either.

Wednesday, July 27, 2011

Barnyard2 sets up a Google Group!

Greetings everyone,

The barnyard2 team want to announce the creation of two Google groups that will be used ease the way for users to report issue or find answers and discuss about barnyard2 related topics.

barnyard2-users and barnyard2-devel.

barnyard2-users@googlegroups.com (for users problems and issues)
barnyard2-devel@googlegroups.com (for development updates, fixes, patches, comments, and more)

We strongly encourage you to join if you have any issues/commenst/questions related to barnyard2.

We would also like to launch a special invitation to UI developers that are willing to improve the future of the database schema and the handling of unified2 EXTRA_DATA event type.

Any comments or question can also be directed to

Ian Firns firnsy@securixlive.com
Eric Lauzon beenph@gmail.com

Eager to see you arround in the barn *wink wink*

Update:  Here is the direct link to the barnyard2 group: http://groups.google.com/groups/dir?lnk=nhpsfg&q=barnyard2.  Thanks Jason!

-The barnyard2 team.

Monday, June 27, 2011

Snort's output methods

Ever since the beginning of Snort, one of the main concerns was "how do I get data out of Snort".  Some of the options that are available have their advantages and disadvantages:


  1. There's some that aren't used.
  2. There's some that cause Snort to be slow.
  3. There's some that we don't maintain and don't frequently test.
  4. There's some that we want to get rid of.


One of those output methods is the "spo_database" module.  The module within Snort that directly inputs data from Snort into a mysql, postgres, or an Oracle database.  This logging method was written back in the late 90's by a college student (along with the db schema and the interface ACID) as a project for his thesis.

It hasn't been very well maintained since then.  In fact, we don't test against it, and we don't recommend it for use.  It makes Snort, which is a high-speed data processor, have to stop doing what it's doing (being an IPS), and insert data into the database.  While Snort is inserting into the database, this stops inspection waiting for the database connection.

So we are going to remove it.


If you look in your snort.conf and your "output" lines look like this:
output database: alert, <db_type>, user=<username> password=<password> test dbname=<name> host=<hostname>
output database: log, <db_type>, user=<username> password=<password> test dbname=<name> host=<hostname>


This change will affect you.

In order to provide the type of functionality we'd like to provide with Snort in the next few releases (more data for you!), we needed someone to take over the maintenance of the db schema that is shipped with Snort as well.   As a result of the discussion on the Snort-devel list, the team members over at the barnyard2 project have agreed to take over the maintenance of these schemas.

It is our intention to distribute the unified2 format as our official output method, provide our documentation for it, and the u2spewfoo tool within Snort so that anyone is able to read it.  We are going to keep some other output methods as well, but...

At this point I'd like to hear from the community as well.  So please leave comments.

What output plugins do you use?
Will you be affected by this change (we hope a lot of you aren't using the spo_database method)?
What other output plugins do you think we can "show the door"?

Wednesday, December 29, 2010

Working with unified and unified2 files

First you might ask "why do I care about unified files?" (unified2 these days, more on that later)

There is no perfect answer to that, just good reasons you should. Some of those reasons are:

  • It is a unified format allowing for easy archive management
  • It is the fastest output method available for snort
  • It is the preferred method used by Sourcefire and the most tested.
  • It is not easily modified. (It is still possible however)

8 years ago if you needed to get data from a snort unified file and do something with it you had few options, there was barnyard and that was it. This is fine and for good reason people were using unified output spooling to disk, reading those files with barnyard, and storing them in a mysql database. In the typical use case this was a perfect solution that worked for many years for many people. This solution had one major drawback though, unified files could not be processed by wetware without barnyard.

Normally this wouldn't be a problem as barnyard supported csv, fast alerts, database, pcap, etc as output methods. While a pain to process a unified file and then post process the results from several different file formats, it was sufficient to get the job done and not too painful.

...Unless you don't have barnyard available. This is exactly the situation that forced me to write the first iteration of SnortUnified.pm. I was working on a system that had unified files but no barnyard, no database, no compiler. Normally this isn't a problem either, you just install the tools you need but that was not allowed. As a matter of fact it was forbidden as these tools were not "certified" for use in production environments and the data was not allowed to leave this environment. The absurdity of that situation, having the data but no way to use it, by policy, was maddening. I thought to myself, who was the person that decided to use unified logging and nothing else? What was the value of having the data if you never intended to be able to use it? blag blah blah you can think the rest yourself, and just like I did, please keep those thoughts to yourself. Clearly their answer to not being able to use the data was to find a way to use the data when they had a need for it. Back to the point of the post.

In looking around on the system I quickly determined that there was xxd, perl, bash, and a smattering of other useful utilities. This was good as I was not looking forward to manual processing of these files from hex dumps. ( A side note is that I'm a fan of perl, it is a wonderful tool, so capable, so simple, so complex, it is what you need when you need something and at the same time the problem to your solution if done wrong.) The Snort source and perl was all I needed to get to the data they wanted and to save me from death by paper and pen. The first version of SnortUnified.pm was in fact process.pl and quite possibly the ugliest thing you have ever seen.

So here we are. What is this SnortUnified.pm thing?

SnortUnified.pm is a perl module for processing unified files (a set of perl modules actually). It isn't well documented outside the code itself, has few examples, and has saved me at least once. I know it is used in production environments and assume it is stable as I never get bug reports, just feature requests, though not since I added the processing hooks I'll be discussing below.

If you would like to know more you can browse the source in google code, get the latest source, or just continue reading, it is up to you. Working with SnortUnified is easy and depending on what you are trying to do could be as simple as a few lines of perl.
use SnortUnified(qw(:ALL));
use Data::Dumper;

$UF_Data
= openSnortUnified(shift);
while ( $record = readSnortUnifiedRecord() ) {
print Dumper($record);
}
closeSnortUnified
();

In the source tarball there is a samples directory with various samples available from converting a unified file to XML to signing the unified files or records within them; ensuring they cannot be modified without detection.

Two samples that I think are worth discussing and illustrate core functionality are uf_csv_handler.pl and uf_csv_qualifier.pl

The comments in the beginning of the handler script are sufficient for us to discuss here. First up are handlers, these get called for each record type / action defined and allow you to work with the records without having to change your functional tools. I initially envisioned this as a way to handle transformation of data within the records without having to rework the module. EG: convert an IP address integer to a dotted quad without having to change anything else. I also find it is quite useful for other tasks like converting payloads, debugging, adding output methods, etc. The result of a handler is irrelevant to the processing path, qualifiers are there for that.
# Handlers come before qualifiers come before pcre

# handlers will be run and regardless of the result processing will continue
# The available handlers for SnortUnified.pm are
# ("unified_opened", $UF);
# ("unified2_event", $UF_Record);
# ("unified2_packet", $UF_Record);
# ("unified2_unhandled", $UF_Record);
# ("unified2_record", $UF_Record);
# ("unified_record", $UF_Record);
# ("read_data", ($readsize, $buffer));
# ("read_header", $h);

register_handler('unified2_packet', \&make_ascii_pkt);
register_handler('unified_record', \&make_ascii_pkt);
# register_handler does not care about return values so the following will continue
# register_handler('unified_record', \&make_noise_fail);
# register_handler('unified_record', \&make_noise);

register_handler('unified2_record', \&printrec);
register_handler('unified_record', \&printrec);

# show_handlers();

# Qualifiers will be run, if any return a value less than 1
# then the record will be discarded and processing will continue
# with the next record in the file
# Only one option for unified types

# Skip all but sid 402
# register_qualifier(0,0,0, sub{return 0;});
# By having something specific for 402
# register_qualifier(0,1,402, \&printrec);
# register_qualifier(0,1,402, sub{return 1;});
# register_qualifier(0,1,402, \&make_noise);
# register_qualifier(0,1,402, \&make_noise);
# register_qualifier(0,1,402, \&make_noise_fail);
# register_qualifier(0,1,402, \&make_noise_never);

# But you can be granular with unified2 types
# register_qualifier($UNIFIED2_IDS_EVENT,1,402, \&make_noise);
# register_qualifier($UNIFIED2_PACKET,1,402, \&make_noise);


# register_pcre(1,402, "test");
# register_pcre(1,402, "*");
# register_pcre(1,402, ".*ads.clickagents.com.*");

# show_qualifiers();


Breaking out the comments above:

  • unified_opened gets called every time a new unified file is opened. This allows you to do things like hook in verification routines, mark progress in a waldo file, create verbose informed logging, etc.
  • unified2_event gets called every time a unified2 event record type is encountered in the unified file.
  • unified2_packet gets called every time a unified2 packet record type is encountered in the unified file.
  • unified2_unhandled gets called if a record type that is unknown / not handled by default is encountered in the unified2 file. This is useful as a bridge for new types, like the recently added 104 and 105 event types. I suggest that minimally you register a handler to log the unhandled types to a file so you can further investigate.
  • unified_record gets called for earlier types of records only found in "unified" files, not "unified2" files
  • read_data gets called every time we read data from the unified file.
  • read_header gets called every time we read a header from the file.
  • register_handler('unified2_packet', \&make_ascii_pkt); registers the "make_ascii_pkt" subroutine as a handler for all unified2_packet records. It gets called as follows make_ascii_packet($UF_Data) and line 105 of the uf_csv_handler is the beginning of that routine.
  • show_handlers() will print the handlers that are registered.
Qualifiers are intended to allow you to "tune after the fact" and I think this is the right way to handle a lot of tuning tasks. We should not be artificially limited in our detection scope simply because that detection potentially creates a lot of "noise". I think that we should have the freedom to cast the widest net necessary and possible when it comes to these things, especially when investigating evolving threats and potentially targeted and ongoing attacks.

Qualifiers also allow you to get very specific in your actions. You can limit a qualifier to a specific sid or cast a super wide net on all events. Qualifiers are all executed so you can do things like easily suppress output, except for specific events. You can also get granular within those specific events. In the comments above all events are suppressed except gen:1 sid:402 and then only for sid:402 events that contain the PCRE ".*ads.clickagents.com.*". You could make the qualifier look at the IP address, check an archived (as in comprehensive) blacklist and change the context of the event, etc. I used a PCRE earlier, this is done with a special type of qualifier (purpose built?) that makes it trivial to use a regular expression as a qualifier though it has one caveat, a PCRE will not override the decision of other qualifiers, it only has a vote but in the case of conflict, the proper qualifier wins.

Qualifiers are called in the same way as handlers but the return code is checked and any return with a value that is less than one is discarded, processing continues on the next record, and
your code never knows that an event was skipped. If you want or need tracking of these things you could use the qualifier to populate some stats and a handler to keep global stats.

If you use the code please drop me a note letting me know, it helps me gauge interest and any impact of possible changes. I've not modified the code in some time, largely because I've not needed to. But, if you have patches, requests, criticisms, commentary, or want to catch a pint at the next event we are both at feel free to let me know.

That is all for now, if you would like to know more or need assistance don't hesitate to ask. I can be reached at jason...at...snort.org or jason...at...sourcefire.com. My details are also in the README contained in the release tarball if you don't want to come back to the post.

Happy Snorting!

Thursday, December 23, 2010

New Proposed Classification.config file setup

I know we are all in the Holiday Season, so one last item for you all to look over while you are roasting in front of the fire..

Recently on the Emerging-Threats Mailing list, Matt Jonkman proposed a new classification system to replace the aging Snort classification system that's been in use for years. We saw this as a good idea and after some internal discussions, decided to head the same route.

So we propose the following classification.config system to the community for comment, and we want to hear the feedback! Especially on descriptions and priorities. I'll assemble all the comments on January 12th (a date suggested by Matt Jonkman) and create a new classification.config file which we will then include in the official Snort tarball and in the VRT rules tarball.

We've made two major changes to the classification system as proposed by the Emerging-Threats list:
  1. We've converted all Underscores to Hyphens
  2. We've made everything lowercase.

This was done to ensure compatibility with existing output modules (barnyard2, unified, unified2, barnyard, SnortUnified.pm, etc), GUI's (BASE, Snorby, Placid, etc), and internal (to Snort) parsers.

The proposed classification.config configuration parameters are available for download here:
http://www.snort.org/assets/157/classifications.txt, and are pasted below. Please leave comments on the blog, and we'll assemble them into a final product:


config classification: exploit-shellcode, A known shellcode payload was detected,1
config classification: exploit-sql-injection, A known SQL injection attack was detected,1
config classification: exploit-browser, A known client-side browser exploit was detected,1
config classification: exploit-activex, A known client-side ActiveX exploit was detected,1
config classification: exploit-command-execution, A known command execution exploit was detected,1
config classification: exploit-cross-site-Scripting, A known cross site scripting XSS attack was detected,2
config classification: exploit-ftp, A known exploit targeting ftp servers or clients was detected,1
config classification: exploit-file-inclusion, A known file inclusion attack was detected,2
config classification: exploit-windows, A known attack targeting Windows systems was detected,1
config classification: exploit-directory-traversal, A directory traversal attack was detected,2
config classification: exploit-attack-response, A known string indicating a host has been comprised was detected,1
config classification: exploit-denial-of-service, A known DoS or DDoS packet payload was detected,2
config classification: exploit-pdf, A known exploit targeting PDF files was detected, 1
config classification: exploit-buffer-overflow, A known exploit using a buffer overflow was detected,1
config classification: exploit-spoofing, A known spoofing attacker was detected,3
config classification: exploit-format-string, A known exploit utilizating a format string overflow was detected,1
config classification: exploit-misc, A known exploit targeting an unclassificated system was detected,2
config classification: exploit-dns, A known exploit targeting DNS systemes was detected,1
config classification: exploit-mail, A known exploit targeting Mail servers was detected,1
config classification: exploit-samba, A known exploit targeting Samba servers or clients was detected,1
config classification: exploit-linux, A known exploit targeting Linux based systems was detected,1
config classification: authentication-bruteforce, An attempt to bruteforce usernames and passwords was detected,2
config classification: authentication-bypass, An attempt to bypass login authentication was detected,2
config classification: authentication-login, A login attempt to any service or system was detected,4
config classification: authentication-Failed, A failed login attempt was detected,4
config classification: authentication-cleartext, A authentication request was detected in plain text,4
config classification: authentication-logout, A logout request was detected,4
config classification: authentication-disclosure, During an authentication request the username or password was disclosed,4
config classification: authentication-default-credentials, An attempt to login with publicly known default usernames or passwords was detected,4
config classification: access-web-application-access, A known web application was accessed,4
config classification: access-file-Access, A known default file was accessed,4
config classification: access-misc, What is an Access-Misc,4
config classification: malware-spyware, A known Spyware application was detected,2
config classification: malware-adware, A known Adware application was detected,2
config classification: malware-fake-Antivirus, A known Fake Anti-virus application was detected,1
config classification: malware-keylogger, A known KeyLogger application was detected,1
config classification: malware-trojan, A known Trojan was detected,1
config classification: malware-virus, A kown Virus was detected,1
config classification: malware-worm, A known Worm was detected,1
config classification: malware-generic, A known unclassified malware application was detected,2
config classification: malware-backdoor, A known backdoor was detected,1
config classification: policy-adult, A known Adult website or other system was accessed,4
config classification: policy-p2p, A known P2P application was detected,4
config classification: policy-instant-messaging-chat, A known Instant Messaging application was detected,4
config classification: policy-anonymity, A known privacy application was detected,4
config classification: policy-games, A known online game was detected,4
config classification: policy-other, A generic policy violation has occurred,4
config classification: denial-of-service-web-application, A known Denial of Service attack was detected against a web application,3
config classification: denial-of-service-application, A known Denial of Service attack was detected against an application,4
config classification: denial-of-service-flood, A known traffic flooding tool was detected,4
config classification: denial-of-service-ddos, A known DDoS tool was detected,4
config classification: suspicious-blacklist-address, A known malicious host was detected,2
config classification: suspicious-web-attack-or-scan, A known scanning tool was detected,2
config classification: suspicious-bad-traffic, Malformed or incorrectly formatted network traffic was detected,4
config classification: suspicious-network-activity, Strange or suspicious network traffic was detected,4
config classification: suspicious-scada-activity, SCADA traffic was detected,4
config classification: suspicious-dns-activity, Suspicious DNS traffic was detected,4
config classification: suspicious-ssh-activity, Suspicious SSH traffic was detected,4
config classification: suspicious-nfs-activity, Suspicious NFS traffic was detected,4
config classification: suspicious-database-activity, Suspicious database activity was detected,4
config classification: suspicious-netbios-activity, Suspicious netbios activity was detected,4
config classification: suspicious-rpc-Activity, Suspicious RPC activity was detected,4
config classification: suspicious-mail-activity, Suspicious Mail activity was detected,4
config classification: network-tftp-activity, TFTP traffic was detected,4
config classification: network-ftp-Activity, FTP traffic was detected,4
config classification: network-snmp-Activity, SNMP traffic was detected,4
config classification: network-smtp-Activity, SMTP traffic was detected,4
config classification: network-telnet-activity, Telnet activity was detected,4
config classification: recon-misc, A network probe was detected,4
config classification: recon-scanner, A network scanner was detected,4
config classification: network-ntp-activity, NTP traffic was detected,4
config classification: network-sip-activity, SIP traffic was detected,4
config classification: network-dhcp-activity, DHCP traffic was detected,4
config classification: access-firewall-permit, A firewall permit rule triggered,4
config classification: access-firewall-deny, A firewall deny rule triggered,4
config classification: access-acl-permit, A ACL permit rule was triggered,4
config classification: access-acl-deny, A ACL deny rule was triggered,4
config classification: authentication-policy-added, A policy addition occured,4
config classification: authentication-policy-changed, A policy change occured,4
config classification: authentication-policy-deleted, A policy delete occured,4
config classification: authentication-ftp-login-succeeded, A successful FTP login occured,4
config classification: authentication-ftp-login-failed, A failed ftp login occured,4
config classification: authentication-password-change-failed, A password change failure occured,4
config classification: authentication-password-change-succeeded, A password change occured,4
config classification: authentication-user-created, A new user was created,4
config classification: authentication-user-deleted, A user was deleted,4
config classification: authentication-user-changed, A user was changed,4
config classification: authentication-admin-access, An admin accessed the system,4
config classification: authentication-group-added, A new group was added to the system,4
config classification: authentication-group-deleted, A new group was deleted from the system,4
config classification: authentication-group-changed, A group was changed on the system,4
config classification: authentication-auth-required, Authentication is required for access,4
config classification: authentication-account-lockout, An account was locked,4
config classification: authentication-account-unlocked, An account was unlocked,4
config classification: antivirus-virus-detected, An Antivirus system detected a virus,2
config classification: antivirus-virus-quarantine, An Antivirus system quarantined a virus,2
config classification: antivirus-virus-quarantine-failed, An Antivirus system filed to quarantine a virus,1
config classification: system-configuration-error, A system has indicated it has a configuration error,2
config classification: antivirus-definitions-updated, A system updated its Antivirus definition,4
config classification: antivirus-definitions-updated-failed, A system failed to update its Antivirus definitions,2
config classification: antivirus-unknown-event, A unknown event occured,4
config classification: antivirus-started, A antivirus agent came online,4
config classification: antivirus-disabled, An Antivirus agent was disabled,2
config classification: antivirus-scan-started, An Antivirus scan was started,2
config classification: antivirus-scan-finished, An antivirus scan has completed,2
config classification: antivirus-error, A unclassified error occured on an Antivirus system,3
config classification: application-web-opened, A web browser was opened, 4
config classification: application-web-closed, A web browser was closed, 4
config classification: application-web-reset, A web site sent a reset to a client, 4
config classification: application-web-terminated, A web site was terminated with extreme predujice, 4
config classification: application-web-denied, Packet come in packet deny, 4
config classification: application-web-redirected, A web client was redirected to a new page,4
config classification: application-web-proxy, A web proxy was detected,4
config classification: application-web-error, A misc error was detected,4
config classification: application-web-misc, A Web misc was detected,4
config classification: application-web-not-found, A web application generated a not found error,4
config classification: access-traffic-inbound, Inbound traffic was detected,4
config classification: access-traffic-outbound, Outbound traffic was detected,4
config classification: access-firewall-misc-event, A unclassified event occured on the firewall,4
config classification: suspicious-network-anomaly, Something strange happened I don't know what,4
config classification: suspicious-dns-protocol-anomaly, A suspicious DNS sessions or packet was detected,3
config classification: suspicious-ssh-protocol-anomaly, A suspicious ssh session or packet was detected,3
config classification: suspicious-telnet-protocol-anomaly, A suspicious telnet session or packet was detected,3
config classification: suspicious-http-protocol-anomaly, A suspicious HTTP session or packet was detected,3
config classification: suspicious-mail-protocol-anomaly, A suspicious Mail session or packet was detected,3
config classification: suspicious-ftp-protocol-anomaly, A suspicious FTP session or packet was detected,4
config classification: suspicious-threshold-exceeded, A suspicious threshold was triggered,4
config classification: denial-of-service-other, A new type of Denial of Service was detected,4
config classification: access-file-blocked, Access to a file was blocked,4
config classification: access-tunnel-connection, Access to a tunnel was identified,4
config classification: access-tunnel-closed, Access to a tunnel was closed,4
config classification: aystem-warning, A system Warning message was detected,4
config classification: system-emergency, A system Emergency message was detected,4
config classification: system-critical, A system Critical message was detected,4
config classification: system-error, A system Error message was detected,4
config classification: system-notification, A system Notification message was detected,4
config classification: system-information, A system Information message was detected,4
config classification: system-debug, A system Debug message was detected,4
config classification: system-alert, A system Alert message was detected,4
config classification: access-connection-opened, A connection was opened,4
config classification: access-connection-closed, A connection was closed,4
config classification: access-timeout, A timeout occurred,4
config classification: system-service-started, A service started,4
config classification: system-service-stopped, A service stopped,4
config classification: system-process-started, A process started,4
config classification: system-process-stopped, A process stopped,4
config classification: application-spam-detected, Some dirty spammer was detected,4
config classification: application-mail-dropped, The mail system dropped or refused mail,4
config classification: system-restart, A system restart was detected,4
config classification: system-started, A system startup was detected,4
config classification: system-stopped, A system stop was detected,4
config classification: system-locked, A system being locked was detected,4
config classification: system-unlocked, A system be unlocked was detected,4
config classification: network-ike-activity, IKE traffic was identified,4
config classification: network-h.323-activity, H.323 traffic was identified,4
config classification: network-ppp-activity, PPP traffic was identified,4
config classification: network-ocsp-activity, OCSP traffic was identified,4
config classification: network-l2tp-activity, L2TP traffic was identified,4
config classification: network-rip-activity, RIP traffic was identified,4
config classification: network-pptp-activity, PPTP traffic was identified,4
config classification: network-ssl-activity, SSL traffic was identified,4
config classification: network-igmp-activity, IGMP traffic was identified,4
config classification: network-ipsec-activity, IPSEC traffic was identified,4
config classification: network-pki-activity, PKI traffic was identified,4
config classification: voip-call-started, A VOIP call was started,4
config classification: voip-call-ended, A VOIP call was completed,4
config classification: voip-misc, A VOIP event occurred,4
config classification: network-bootp-activity, BOOTP traffic was identified,4
config classification: alert-ids-alert, The IDS did something,4
config classification: alert-ips-alert, The IPS did something,4
config classification: alert-hids-alert, The HIDS did something,4
config classification: application-mail-sent, An email was sent,4
config classification: application-mail-server-misc, A Mail server did something,4
config classification: application-mail-received, An email was recieved,4
config classification: availability-state-up, A system or service is now up,4
config classification: availability-state-down, A system or service is now down,4
config classification: availability-state-critical, A system or service is not in a critical state,1
config classification: availability-state-warning, A system or service has issued a warning,3
config classification: availability-state-unknown, A system or service is in an unknown state,3
config classification: availability-state-unreachable, A system or service is unreachable,1
config classification: application-vpn-opened, A VPN session was opened,4
config classification: application-vpn-closed, A VPN session was closed,4
config classification: application-vpn-denied, A VPN session was denied,2
config classification: application-vpn-misc, Something happened on a VPN session,2
config classification: system-configuration-changed, A system changed its configuration,4
config classification: network-misc, Something happened on the network,4
config classification: policy-phishing, A phishing attempt was detected,4
config classification: wireless-new-network, A new wireless network has been detected,4
config classification: wireless-client-associated, A new client has connected to the wireless network,4
config classification: wireless-flood, The wireless network is currently being flooded,2
config classification: wireless-disassociation, A wireless client has been disassociated from the network,4
config classification: wireless-deauthentication, A wireless client has been deauthenticated,4
config classification: wireless-anomaly, Something strange occurred on the wireless network,4
config classification: wireless-spoofing, Spoofing has been detected on the wireless network,2
config classification: wireless-scanner-detected, A scanner was detected on the wireless network,2
config classification: wireless-misc, Something happened on the wireless network,2
config classification: wireless-probe, A probe attempt was identified on the wireless network,4
config classification: inventory-service-detected, A new service has been identified,4
config classification: inventory-service-change, A service has changed,4
config classification: inventory-service-misc, A Misc service was detected,4
config classification: inventory-operating-system-detected, A new operating system was detected,4
config classification: inventory-operating-system-change, A system changed,4
config classification: inventory-operating-system-misc, A system met a Misc,4
config classification: inventory-mac-detected, A unhackable computer was detected,1
config classification: inventory-mac-change, A MAC address changed,4
config classification: policy-check-failed, A Policy check has failed,1
config classification: policy-check-passed, A Policy check has passed,1
config classification: network-high-load, The network currently has a high load,1
config classification: authentication-error, An authentication error was detected,4
config classification: application-web-modified, A content modified proxy request was identified,4
config classification: application-dhcp-release, A DHCP lease was released,4
config classification: application-dhcp-request, A DHCP request was detected,4
config classification: application-dhcp-lease, A DHCP lease was allocated,4
config classification: application-dhcp-pool-exhausted, All DHCP addresses have been allocated,4
config classification: application-dhcp-error, A DHCP error was detected,4
config classification: system-software-installed, A software package was installed,4
config classification: honeypot-connection-opened, Something connected to the honeypot sweet new warez,4
config classification: honeypot-attack-detected, A known attack was detected on the honeypot,4
config classification: honeypot-connection-closed, A connection to the honeypot was closed,4
config classification: application-dns-succesful-zone-tranfer, A succesful DNS zone transfer was detected,4
config classification: application-dns-zone-transfer-failed, A failed DNS zone transfer was detected,4
config classification: application-ftp-command-executed, An FTP command was executed,4
config classification: application-ftp-error, An FTP error was detected,4
config classification: application-ftp-connection-opened, An ftp connection was opened,4
config classification: application-ftp-connection-closed, An ftp connection was closed,4
config classification: database-login, A database login was detected,4
config classification: database-login-failed, A failed database login was detected,4
config classification: database-query, A database query was executed,4
config classification: database-logout, A database logout was detected,4
config classification: database-stop, A database was stopped,4
config classification: database-start, A database was started,4
config classification: database-error, A database error occurred,4