Monday, June 27, 2011

Snort's output methods

Ever since the beginning of Snort, one of the main concerns was "how do I get data out of Snort".  Some of the options that are available have their advantages and disadvantages:


  1. There's some that aren't used.
  2. There's some that cause Snort to be slow.
  3. There's some that we don't maintain and don't frequently test.
  4. There's some that we want to get rid of.


One of those output methods is the "spo_database" module.  The module within Snort that directly inputs data from Snort into a mysql, postgres, or an Oracle database.  This logging method was written back in the late 90's by a college student (along with the db schema and the interface ACID) as a project for his thesis.

It hasn't been very well maintained since then.  In fact, we don't test against it, and we don't recommend it for use.  It makes Snort, which is a high-speed data processor, have to stop doing what it's doing (being an IPS), and insert data into the database.  While Snort is inserting into the database, this stops inspection waiting for the database connection.

So we are going to remove it.


If you look in your snort.conf and your "output" lines look like this:
output database: alert, <db_type>, user=<username> password=<password> test dbname=<name> host=<hostname>
output database: log, <db_type>, user=<username> password=<password> test dbname=<name> host=<hostname>


This change will affect you.

In order to provide the type of functionality we'd like to provide with Snort in the next few releases (more data for you!), we needed someone to take over the maintenance of the db schema that is shipped with Snort as well.   As a result of the discussion on the Snort-devel list, the team members over at the barnyard2 project have agreed to take over the maintenance of these schemas.

It is our intention to distribute the unified2 format as our official output method, provide our documentation for it, and the u2spewfoo tool within Snort so that anyone is able to read it.  We are going to keep some other output methods as well, but...

At this point I'd like to hear from the community as well.  So please leave comments.

What output plugins do you use?
Will you be affected by this change (we hope a lot of you aren't using the spo_database method)?
What other output plugins do you think we can "show the door"?

15 comments:

  1. Get rid of the snort output options altogether (unified only) and make barnyard2 an official piece of snort.

    ReplyDelete
  2. As a snort package maintainer for a Linux distro, anything that reduces the number of --enable-/--disable- or --with-/--without- options to ./configure will reduce the delta between when Sourcefire releases a new version of snort and when I can release an updated package. The simpler dependency tree that would result from removing these options would also reduces the amount of time it takes for the new package to move from being marked as testing to being made stable.

    But, you can't get rid of everything. There are valid use cases for a number of the output options.

    Keep
    ----
    unified2
    - This is obvious. Listed here for the sake of completeness.

    log_tcpdump
    - There are many people who use Snort just to log network traffic. There are other tools that do this, but snort is one of the most actively developed tools that do this and can daemonize.

    alert_fast/alert_full
    - These are extremely useful to people who write rules, especially the sending of alerts to stdout. These let rule writers quickly test, tweak, and retest new rules. It could be argued that making these _only_ write to stdout and removing the option to write to a file might be a good idea. It could also be argued that if these are to be kept solely for rule testing purposes then alert_fast could be removed leaving just alert_full.

    Decommission
    ------------
    alert_syslog
    alert_unixsock
    database (all types)
    alert_csv
    alert_unified
    log_unified
    alert_prelude
    log_null
    alert_aruba_action
    Any other module I missed

    With that said, I think it is important that before an output module is decomm'ed from snort one of the following conditions needs to be met:

    1. Full, stable support in barnyard2, or
    2. A general consensus that the output module has out lived it's usefulness and is not being widely used.

    Also, time needs to be allotted to warn end users who might not follow the mailing list. It is probably not a good idea to remove anything from snort-2.9.1, but if we know before 2.9.1 is released that in the next release after 2.9.1 the database module will be removed, package maintainers can warn their user base (via distro forums, mailing lists, 2.9.1 post-install messages, etc) to start planing for that removal.

    My $.02

    ReplyDelete
  3. I vote remove it all and keep nothing.

    ReplyDelete
  4. Mephux --

    Well, we'll have to keep at least one. :)

    ReplyDelete
  5. Joel, outside of writing the unified2 file to disk that is. ;)

    ReplyDelete
  6. unified(1) should be kept on, barnyard2 has issues with logging tagged packets to db and also the csv output is not properly implemented on barnyard2
    and also for backwards competability with other custom applications built to read these files
    (siem parsers and the like)

    ReplyDelete
  7. A bit late to the discussion, but whatever.
    MySQL is a very valid way of handling data and output, especially when you have multiple servers and need a WEB based front end to handle things.

    There's no reason to penalize people who prefer mysql (or any database for that matter), just because you don't want to maintain the output for it. Binary output sucks, it's NOT human readable, and it requires further configuration of MORE applications to just understand it.

    I'll pass.

    ReplyDelete
  8. I'm running the new beta of by2 and it *does* log tagged packets -- they show up not as 'tagged packets' as in by1 but with the full original alert information.

    ReplyDelete
  9. Does this affect prelude?.

    I've been using prelude to log the alerts in multiple enterprise systems!

    Sorry to say but, if prelude support is removed I'm gonna have to trash snort as it is easier to replace it than replace all the prelude implementations already in place!

    ReplyDelete
    Replies
    1. I use barnyard2 instead. It reads the unified2 log from snort and talks to prelude.

      Delete
  10. Yes it does. The maintainer for the prelude output plugin is now unavailable and hasn't responded to our emails for years.

    ReplyDelete
    Replies
    1. He wasn't making money on the project. So he sold it. It reopened under new Management... but they are pushing the pro version.

      I had gotten pissed off and actually called the number in his whois information. He was very helpful and honest about the project.

      Delete
    2. We've made the output plugins dynamic. So even though we are removing the plugin from Snort, the developer is welcome to maintain and distribute the output plugin itself after the release of Snort 2.9.3.0.

      Delete
  11. has the output database after version 2.9.1 removed? I'm currently installing snort version 2.9.4, but there's no output database. If I want to do some logging into database, do I need to use barnyard2 so that I could get the database schema for snort? or is there any solution for my problem? I want to use multiple sensor on snort, but Barnyard2 cant config sensor more than 1.
    thanks.

    ReplyDelete
    Replies
    1. It was removed in 2.9.3. Yes, you need to use barnyard2.

      Delete