Tuesday, March 4, 2014

Firing up OpenAppID

The purpose of this post is to provide guidance to Snort users who would like to try out Snort 2.9.7.0 and the OpenAppID features that it comes with. It is not intended to guide the reader through setting up Snort from scratch, there are plenty of docs on how to set up Snort at http://www.snort.org/docs.
Snort
Before we can download the latest Snort source code and compile it, we have a new prerequisite to fulfill before we can compile Snort. You'll need to install Luajit, which is used to define application detectors with the new OpenAppID.
On my Ubuntu system this was as easy as the following command.
sudo apt-get install libluajit-5.1-2 libluajit-5.1-common libluajit-5.1-dev luajit
Now we’re all set to download and install the alpha version of Snort. At the time this was written that was 2.9.7.0 alpha (snort-2.9.7.0_alpha.tar.gz). Grab the latest source from snort.org’s Download Snort page. Then we can extract and build.
tar zxvf snort-2.9.7.0_alpha.tar.gz
cd snort-2.9.7.0.alpha
./configure --prefix=/usr/local/snort --enable-sourcefire --enable-open-appid
make
sudo make install
Note the new flag on the configure command. We are adding that switch to enable OpenAppID.
Application Detector Package
The next step is to download the Application Detector Package. This entirely new content contains what is at the heart of the OpenAppID feature. You can download it from the Download Snort page on snort.org (currently: snort-openappid-detectors.2014-02-22.187-0.tgz).
This package contains the Lua libraries for detecting applications, as well as the application detectors themselves.
sudo tar -xzf snort-openappid-detectors.2014-02-22.187-0.tgz -C /usr/local/snort
Once that is unpacked you will see your Snort directory now has an odp directory. Here you will see the appMapping.data file, which includes metadata about the application detectors. A simple way to explore what application detectors are included in this first release is to examine the second column of this file. If you’d like to see the list, try the following command.
cat appMapping.data | cut -f2
For example, if you would like to see what micro-applications that can be identified within Facebook, search for that data.
cat appMapping.data | cut -f2 | grep Facebook
Facebook Apps
Facebook
Facebook Chat
Facebook Comment
Facebook Read Email
Facebook Send Email
Facebook Status Update
Facebook search
Facebook event
Facebook post
Facebook video chat
Facebook message
Facebook video
You will also see the libs subdirectory, which contains the Lua libraries needed to run application detection. The port and lua subdirectories contain the applications detectors themselves. The port detectors are simple YAML files to patch port and application. The lua directory contains the bulk of the detectors. These lua scripts perform far more powerful detection.
Configure snort.conf
There are several changes to make in our snort.conf file. We must add the preprocessor command for OpenAppID and configure the output command.
Additionally, I suggest commenting out the following line.
#dynamicdetection directory /usr/local/snort/lib/snort_dynamicrules
At present time there there isn't a shared object rule release specifically for 2.9.7.0, so it is easier to skip this for now. 
Configure OpenAppID Preprocessor
Our next step is to add the configuration for the OpenAppID preprocessor to the snort.conf file. Find the lines for the reputation preprocessor. This should be followed by commented text for “Step 6.” Just after the reputation preprocessor and before Step 6 we will add another preprocessor setting.
preprocessor appid: app_stats_filename appstats-u2.log, \
   app_stats_period 60, \
   app_detector_dir /usr/local/snort
This will turn on the OpenAppID preprocessor. The first step simply names the configuration file to which application statistics will be logged. The second gives the time period used to sample this data. Finally, you must point out the directory which contains the odp directory we extracted from the Open App ID Detector package. 
Configure Output
The final step in configuring our snort.conf file will be to change the output command. Everyone should be using the Unified2 output module already, but there is a new command to add new the new event field for the application in use. 
Look into Step 6 find the lines explaining the unified2 output type. In that section add the following line.
output unified2: filename snort.log, limit 128, appid_event_types
Testing Snort
Now let’s fire up Snort.
/usr/local/snort/bin/snort -c /usr/local/snort/etc/snort.conf -i eth#
Where eth# is which ever interface you will be monitoring with (e.g. eth0).
If there aren't any errors you will see the final line output by Snort indicating a successful startup, as “Commencing packet processing.” If you examine the output above that you will see a new section for AppID Configuration. This will include the configuration given to the preprocessor for reference.
To test the new deployment, fire up a web browser to test out application detection. For my test I used Chrome to visit cnn.com.
If you are still examining the console output of the Snort process you will see something to the effect of “Opening /var/log/snort/appstats-u2.log.1393807981 for output.” Ending the Snort process will also write what data has not yet been output.
Examining Application Statistics
When you downloaded the Snort package, it included a tools subdirectory. In here you will find a couple of programs for changing the unified2 binary format to readable text. In this case we will use the u2openappid program to print the application statistics that Snort is collecting for us.
u2openappid /var/log/snort/appstats-u2.log.1393807981
statTime="1393807860",appName="chrome",txBytes="6043",rxBytes="111267"
statTime="1393807860",appName="dns",txBytes="8708",rxBytes="38103"
statTime="1393807860",appName="http",txBytes="200399",rxBytes="1444070"
statTime="1393807860",appName="cnn.com",txBytes="198478",rxBytes="1557970"
statTime="1393807860",appName="doubleclick",txBytes="5543",rxBytes="2598"
statTime="1393807860",appName="truste",txBytes="1829",rxBytes="12208"
statTime="1393807860",appName="washington_time",txBytes="2210",rxBytes="1401"
statTime="1393807860",appName="turner_broadcas",txBytes="1785",rxBytes="1316"
statTime="1393807860",appName="moat",txBytes="5707",rxBytes="3453"
From this you can see a number of entries. In this case the above were generated from my visit to cnn.com. You can see that they share the same statTime. OpenAppID identifies the client application (Chrome), DNS request, protocol (http), and web app (cnn.com). The rest of the entries are other web apps embedded within CNN’s web page.
To further illustrate the data we can get form OpenAppID, here is the output from when I instead use Firefox to visit Netflix.
statTime="1393809840",appName="google",txBytes="3500",rxBytes="5171"
statTime="1393809840",appName="firefox",txBytes="11695",rxBytes="20818"
statTime="1393809840",appName="http",txBytes="11695",rxBytes="20818"
statTime="1393809840",appName="netflix",txBytes="1072",rxBytes="1262"
statTime="1393809840",appName="https",txBytes="18330",rxBytes="33984"
statTime="1393809840",appName="verisign",txBytes="1450",rxBytes="2668"
Happy Snorting! Check out the new stuff. There are a few new features to explore here. In the next post we will take a look at using application identity information in Snort rules.

4 comments:

  1. Here are some preliminary results for Snort-2.9.7.0 Alpha on OpenSUSE 13.x:

    Joel,

    Here are my prelim findings on OpenSUSE 13.1 (which will
    probably also work for OpenSuSE 12.x as well:

    To build OpenAPPID under OpenSuSE 13.1 (x64) for Snort-2.9.7.0:

    You will need the following packages installed:

    libluajit-5_1-2-2.0.2-7.1.x86_64.rpm
    luajit-2.0.2-7.1.x86_64.rpm
    luajit-devel-2.0.2-7.1.x86_64.rpm

    Install the packages as 'root' in the following order below
    by using the following commands:

    rpm -i libluajit-5_1-2-2.0.2-7.1.x86_64.rpm
    rpm -i luajit-2.0.2-7.1.x86_64.rpm
    rpm -i luajit-devel-2.0.2-7.1.x86_64.rpm

    Note: libluajit MUST be installed first, then luajit, then luajit-devel.

    Next, download the Snort-2.9.7.0 Alpha tarball from the Snort Downloads
    page and use the commands below to configure and compile it:

    tar zxvf snort-2.9.7.0_alpha.tar.gz
    cd snort-2.9.7.0.alpha
    ./configure --enable-sourcefire --enable-control-socket --enable-open-appid

    checking for main in -lluajit-5.1... yes
    INFO! LuaJIT is installed <--- WOOHOO! :)
    checking for linuxthreads... no
    checking for yylex_destroy support... yes
    checking that generated files are newer than configure... done
    configure: creating ./config.status

    I haven't done the 'make/make install' as of yet, but if it
    got past the configure, it should be fine...more testing to
    follow :)

    Bill

    make
    sudo make install

    ReplyDelete
  2. I am unable to block TOR browser by using openappid please provide updates for same.
    Regard,
    Somesh

    ReplyDelete
    Replies
    1. Dear Somesh,

      I suggest that you address your question to the appropriate Snort mailing list found here:

      https://www.snort.org/community

      Delete
  3. I am unable to block TOR browser and other tor apps using openappid please provide update for same.
    Regards,
    Somesh

    ReplyDelete