Working with unified and unified2 files

First you might ask "why do I care about unified files?" (unified2 these days, more on that later)

There is no perfect answer to that, just good reasons you should. Some of those reasons are:

• It is a unified format allowing for easy archive management
• It is the fastest output method available for snort
• It is the preferred method used by Sourcefire and the most tested.
• It is not easily modified. (It is still possible however)

8 years ago if you needed to get data from a snort unified file and do something with it you had few options, there was barnyard and that was it. This is fine and for good reason people were using unified output spooling to disk, reading those files with barnyard, and storing them in a mysql database. In the typical use case this was a perfect solution that worked for many years for many people. This solution had one major drawback though, unified files could not be processed by wetware without barnyard.

Normally this wouldn't be a problem as barnyard supported csv, fast alerts, database, pcap, etc as output methods. While a pain to process a unified file and then post process the results from several different file formats, it was sufficient to get the job done and not too painful.

...Unless you don't have barnyard available. This is exactly the situation that forced me to write the first iteration of SnortUnified.pm. I was working on a system that had unified files but no barnyard, no database, no compiler. Normally this isn't a problem either, you just install the tools you need but that was not allowed. As a matter of fact it was forbidden as these tools were not "certified" for use in production environments and the data was not allowed to leave this environment. The absurdity of that situation, having the data but no way to use it, by policy, was maddening. I thought to myself, who was the person that decided to use unified logging and nothing else? What was the value of having the data if you never intended to be able to use it? blag blah blah you can think the rest yourself, and just like I did, please keep those thoughts to yourself. Clearly their answer to not being able to use the data was to find a way to use the data when they had a need for it. Back to the point of the post.

In looking around on the system I quickly determined that there was xxd, perl, bash, and a smattering of other useful utilities. This was good as I was not looking forward to manual processing of these files from hex dumps. ( A side note is that I'm a fan of perl, it is a wonderful tool, so capable, so simple, so complex, it is what you need when you need something and at the same time the problem to your solution if done wrong.) The Snort source and perl was all I needed to get to the data they wanted and to save me from death by paper and pen. The first version of SnortUnified.pm was in fact process.pl and quite possibly the ugliest thing you have ever seen.

So here we are. What is this SnortUnified.pm thing?

SnortUnified.pm is a perl module for processing unified files (a set of perl modules actually). It isn't well documented outside the code itself, has few examples, and has saved me at least once. I know it is used in production environments and assume it is stable as I never get bug reports, just feature requests, though not since I added the processing hooks I'll be discussing below.

If you would like to know more you can browse the source in google code, get the latest source, or just continue reading, it is up to you. Working with SnortUnified is easy and depending on what you are trying to do could be as simple as a few lines of perl.

use SnortUnified(qw(:ALL));
use Data::Dumper;

$UF_Data = openSnortUnified(shift);
while ( $record = readSnortUnifiedRecord() ) {
print Dumper($record);
}
closeSnortUnified();

In the source tarball there is a samples directory with various samples available from converting a unified file to XML to signing the unified files or records within them; ensuring they cannot be modified without detection.

Two samples that I think are worth discussing and illustrate core functionality are uf_csv_handler.pl and uf_csv_qualifier.pl

The comments in the beginning of the handler script are sufficient for us to discuss here. First up are handlers, these get called for each record type / action defined and allow you to work with the records without having to change your functional tools. I initially envisioned this as a way to handle transformation of data within the records without having to rework the module. EG: convert an IP address integer to a dotted quad without having to change anything else. I also find it is quite useful for other tasks like converting payloads, debugging, adding output methods, etc. The result of a handler is irrelevant to the processing path, qualifiers are there for that.

# Handlers come before qualifiers come before pcre

# handlers will be run and regardless of the result processing will continue
# The available handlers for SnortUnified.pm are
# ("unified_opened", $UF);
# ("unified2_event", $UF_Record);
# ("unified2_packet", $UF_Record);
# ("unified2_unhandled", $UF_Record);
# ("unified2_record", $UF_Record);
# ("unified_record", $UF_Record);
# ("read_data", ($readsize, $buffer));
# ("read_header", $h);

register_handler('unified2_packet', \&make_ascii_pkt);
register_handler('unified_record', \&make_ascii_pkt);
# register_handler does not care about return values so the following will continue
# register_handler('unified_record', \&make_noise_fail);
# register_handler('unified_record', \&make_noise);

register_handler('unified2_record', \&printrec);
register_handler('unified_record', \&printrec);

# show_handlers();

# Qualifiers will be run, if any return a value less than 1
# then the record will be discarded and processing will continue
# with the next record in the file
# Only one option for unified types

# Skip all but sid 402
# register_qualifier(0,0,0, sub{return 0;});
# By having something specific for 402
# register_qualifier(0,1,402, \&printrec);
# register_qualifier(0,1,402, sub{return 1;});
# register_qualifier(0,1,402, \&make_noise);
# register_qualifier(0,1,402, \&make_noise);
# register_qualifier(0,1,402, \&make_noise_fail);
# register_qualifier(0,1,402, \&make_noise_never);

# But you can be granular with unified2 types
# register_qualifier($UNIFIED2_IDS_EVENT,1,402, \&make_noise);
# register_qualifier($UNIFIED2_PACKET,1,402, \&make_noise);


# register_pcre(1,402, "test");
# register_pcre(1,402, "*");
# register_pcre(1,402, ".*ads.clickagents.com.*");

# show_qualifiers();



Breaking out the comments above:

• unified_opened gets called every time a new unified file is opened. This allows you to do things like hook in verification routines, mark progress in a waldo file, create verbose informed logging, etc.

• unified2_event gets called every time a unified2 event record type is encountered in the unified file.

• unified2_packet gets called every time a unified2 packet record type is encountered in the unified file.

• unified2_unhandled gets called if a record type that is unknown / not handled by default is encountered in the unified2 file. This is useful as a bridge for new types, like the recently added 104 and 105 event types. I suggest that minimally you register a handler to log the unhandled types to a file so you can further investigate.

• unified_record gets called for earlier types of records only found in "unified" files, not "unified2" files

• read_data gets called every time we read data from the unified file.

• read_header gets called every time we read a header from the file.

• register_handler('unified2_packet', \&make_ascii_pkt); registers the "make_ascii_pkt" subroutine as a handler for all unified2_packet records. It gets called as follows make_ascii_packet($UF_Data) and line 105 of the uf_csv_handler is the beginning of that routine.

• show_handlers() will print the handlers that are registered.

Qualifiers are intended to allow you to "tune after the fact" and I think this is the right way to handle a lot of tuning tasks. We should not be artificially limited in our detection scope simply because that detection potentially creates a lot of "noise". I think that we should have the freedom to cast the widest net necessary and possible when it comes to these things, especially when investigating evolving threats and potentially targeted and ongoing attacks.

Qualifiers also allow you to get very specific in your actions. You can limit a qualifier to a specific sid or cast a super wide net on all events. Qualifiers are all executed so you can do things like easily suppress output, except for specific events. You can also get granular within those specific events. In the comments above all events are suppressed except gen:1 sid:402 and then only for sid:402 events that contain the PCRE ".*ads.clickagents.com.*". You could make the qualifier look at the IP address, check an archived (as in comprehensive) blacklist and change the context of the event, etc. I used a PCRE earlier, this is done with a special type of qualifier (purpose built?) that makes it trivial to use a regular expression as a qualifier though it has one caveat, a PCRE will not override the decision of other qualifiers, it only has a vote but in the case of conflict, the proper qualifier wins.

Qualifiers are called in the same way as handlers but the return code is checked and any return with a value that is less than one is discarded, processing continues on the next record, and your code never knows that an event was skipped. If you want or need tracking of these things you could use the qualifier to populate some stats and a handler to keep global stats.

If you use the code please drop me a note letting me know, it helps me gauge interest and any impact of possible changes. I've not modified the code in some time, largely because I've not needed to. But, if you have patches, requests, criticisms, commentary, or want to catch a pint at the next event we are both at feel free to let me know.

That is all for now, if you would like to know more or need assistance don't hesitate to ask. I can be reached at jason...at...snort.org or jason...at...sourcefire.com. My details are also in the README contained in the release tarball if you don't want to come back to the post.

Happy Snorting!

New Proposed Classification.config file setup

I know we are all in the Holiday Season, so one last item for you all to look over while you are roasting in front of the fire..

Recently on the Emerging-Threats Mailing list, Matt Jonkman proposed a new classification system to replace the aging Snort classification system that's been in use for years. We saw this as a good idea and after some internal discussions, decided to head the same route.

So we propose the following classification.config system to the community for comment, and we want to hear the feedback! Especially on descriptions and priorities. I'll assemble all the comments on January 12th (a date suggested by Matt Jonkman) and create a new classification.config file which we will then include in the official Snort tarball and in the VRT rules tarball.

We've made two major changes to the classification system as proposed by the Emerging-Threats list:

1. We've converted all Underscores to Hyphens
2. We've made everything lowercase.

This was done to ensure compatibility with existing output modules (barnyard2, unified, unified2, barnyard, SnortUnified.pm, etc), GUI's (BASE, Snorby, Placid, etc), and internal (to Snort) parsers.

The proposed classification.config configuration parameters are available for download here:

http://www.snort.org/assets/157/classifications.txt, and are pasted below. Please leave comments on the blog, and we'll assemble them into a final product:

config classification: exploit-shellcode, A known shellcode payload was detected,1

config classification: exploit-sql-injection, A known SQL injection attack was detected,1

config classification: exploit-browser, A known client-side browser exploit was detected,1

config classification: exploit-activex, A known client-side ActiveX exploit was detected,1

config classification: exploit-command-execution, A known command execution exploit was detected,1

config classification: exploit-cross-site-Scripting, A known cross site scripting XSS attack was detected,2

config classification: exploit-ftp, A known exploit targeting ftp servers or clients was detected,1

config classification: exploit-file-inclusion, A known file inclusion attack was detected,2

config classification: exploit-windows, A known attack targeting Windows systems was detected,1

config classification: exploit-directory-traversal, A directory traversal attack was detected,2

config classification: exploit-attack-response, A known string indicating a host has been comprised was detected,1

config classification: exploit-denial-of-service, A known DoS or DDoS packet payload was detected,2

config classification: exploit-pdf, A known exploit targeting PDF files was detected, 1

config classification: exploit-buffer-overflow, A known exploit using a buffer overflow was detected,1

config classification: exploit-spoofing, A known spoofing attacker was detected,3

config classification: exploit-format-string, A known exploit utilizating a format string overflow was detected,1

config classification: exploit-misc, A known exploit targeting an unclassificated system was detected,2

config classification: exploit-dns, A known exploit targeting DNS systemes was detected,1

config classification: exploit-mail, A known exploit targeting Mail servers was detected,1

config classification: exploit-samba, A known exploit targeting Samba servers or clients was detected,1

config classification: exploit-linux, A known exploit targeting Linux based systems was detected,1

config classification: authentication-bruteforce, An attempt to bruteforce usernames and passwords was detected,2

config classification: authentication-bypass, An attempt to bypass login authentication was detected,2

config classification: authentication-login, A login attempt to any service or system was detected,4

config classification: authentication-Failed, A failed login attempt was detected,4

config classification: authentication-cleartext, A authentication request was detected in plain text,4

config classification: authentication-logout, A logout request was detected,4

config classification: authentication-disclosure, During an authentication request the username or password was disclosed,4

config classification: authentication-default-credentials, An attempt to login with publicly known default usernames or passwords was detected,4

config classification: access-web-application-access, A known web application was accessed,4

config classification: access-file-Access, A known default file was accessed,4

config classification: access-misc, What is an Access-Misc,4

config classification: malware-spyware, A known Spyware application was detected,2

config classification: malware-adware, A known Adware application was detected,2

config classification: malware-fake-Antivirus, A known Fake Anti-virus application was detected,1

config classification: malware-keylogger, A known KeyLogger application was detected,1

config classification: malware-trojan, A known Trojan was detected,1

config classification: malware-virus, A kown Virus was detected,1

config classification: malware-worm, A known Worm was detected,1

config classification: malware-generic, A known unclassified malware application was detected,2

config classification: malware-backdoor, A known backdoor was detected,1

config classification: policy-adult, A known Adult website or other system was accessed,4

config classification: policy-p2p, A known P2P application was detected,4

config classification: policy-instant-messaging-chat, A known Instant Messaging application was detected,4

config classification: policy-anonymity, A known privacy application was detected,4

config classification: policy-games, A known online game was detected,4

config classification: policy-other, A generic policy violation has occurred,4

config classification: denial-of-service-web-application, A known Denial of Service attack was detected against a web application,3

config classification: denial-of-service-application, A known Denial of Service attack was detected against an application,4

config classification: denial-of-service-flood, A known traffic flooding tool was detected,4

config classification: denial-of-service-ddos, A known DDoS tool was detected,4

config classification: suspicious-blacklist-address, A known malicious host was detected,2

config classification: suspicious-web-attack-or-scan, A known scanning tool was detected,2

config classification: suspicious-bad-traffic, Malformed or incorrectly formatted network traffic was detected,4

config classification: suspicious-network-activity, Strange or suspicious network traffic was detected,4

config classification: suspicious-scada-activity, SCADA traffic was detected,4

config classification: suspicious-dns-activity, Suspicious DNS traffic was detected,4

config classification: suspicious-ssh-activity, Suspicious SSH traffic was detected,4

config classification: suspicious-nfs-activity, Suspicious NFS traffic was detected,4

config classification: suspicious-database-activity, Suspicious database activity was detected,4

config classification: suspicious-netbios-activity, Suspicious netbios activity was detected,4

config classification: suspicious-rpc-Activity, Suspicious RPC activity was detected,4

config classification: suspicious-mail-activity, Suspicious Mail activity was detected,4

config classification: network-tftp-activity, TFTP traffic was detected,4

config classification: network-ftp-Activity, FTP traffic was detected,4

config classification: network-snmp-Activity, SNMP traffic was detected,4

config classification: network-smtp-Activity, SMTP traffic was detected,4

config classification: network-telnet-activity, Telnet activity was detected,4

config classification: recon-misc, A network probe was detected,4

config classification: recon-scanner, A network scanner was detected,4

config classification: network-ntp-activity, NTP traffic was detected,4

config classification: network-sip-activity, SIP traffic was detected,4

config classification: network-dhcp-activity, DHCP traffic was detected,4

config classification: access-firewall-permit, A firewall permit rule triggered,4

config classification: access-firewall-deny, A firewall deny rule triggered,4

config classification: access-acl-permit, A ACL permit rule was triggered,4

config classification: access-acl-deny, A ACL deny rule was triggered,4

config classification: authentication-policy-added, A policy addition occured,4

config classification: authentication-policy-changed, A policy change occured,4

config classification: authentication-policy-deleted, A policy delete occured,4

config classification: authentication-ftp-login-succeeded, A successful FTP login occured,4

config classification: authentication-ftp-login-failed, A failed ftp login occured,4

config classification: authentication-password-change-failed, A password change failure occured,4

config classification: authentication-password-change-succeeded, A password change occured,4

config classification: authentication-user-created, A new user was created,4

config classification: authentication-user-deleted, A user was deleted,4

config classification: authentication-user-changed, A user was changed,4

config classification: authentication-admin-access, An admin accessed the system,4

config classification: authentication-group-added, A new group was added to the system,4

config classification: authentication-group-deleted, A new group was deleted from the system,4

config classification: authentication-group-changed, A group was changed on the system,4

config classification: authentication-auth-required, Authentication is required for access,4

config classification: authentication-account-lockout, An account was locked,4

config classification: authentication-account-unlocked, An account was unlocked,4

config classification: antivirus-virus-detected, An Antivirus system detected a virus,2

config classification: antivirus-virus-quarantine, An Antivirus system quarantined a virus,2

config classification: antivirus-virus-quarantine-failed, An Antivirus system filed to quarantine a virus,1

config classification: system-configuration-error, A system has indicated it has a configuration error,2

config classification: antivirus-definitions-updated, A system updated its Antivirus definition,4

config classification: antivirus-definitions-updated-failed, A system failed to update its Antivirus definitions,2

config classification: antivirus-unknown-event, A unknown event occured,4

config classification: antivirus-started, A antivirus agent came online,4

config classification: antivirus-disabled, An Antivirus agent was disabled,2

config classification: antivirus-scan-started, An Antivirus scan was started,2

config classification: antivirus-scan-finished, An antivirus scan has completed,2

config classification: antivirus-error, A unclassified error occured on an Antivirus system,3

config classification: application-web-opened, A web browser was opened, 4

config classification: application-web-closed, A web browser was closed, 4

config classification: application-web-reset, A web site sent a reset to a client, 4

config classification: application-web-terminated, A web site was terminated with extreme predujice, 4

config classification: application-web-denied, Packet come in packet deny, 4

config classification: application-web-redirected, A web client was redirected to a new page,4

config classification: application-web-proxy, A web proxy was detected,4

config classification: application-web-error, A misc error was detected,4

config classification: application-web-misc, A Web misc was detected,4

config classification: application-web-not-found, A web application generated a not found error,4

config classification: access-traffic-inbound, Inbound traffic was detected,4

config classification: access-traffic-outbound, Outbound traffic was detected,4

config classification: access-firewall-misc-event, A unclassified event occured on the firewall,4

config classification: suspicious-network-anomaly, Something strange happened I don't know what,4

config classification: suspicious-dns-protocol-anomaly, A suspicious DNS sessions or packet was detected,3

config classification: suspicious-ssh-protocol-anomaly, A suspicious ssh session or packet was detected,3

config classification: suspicious-telnet-protocol-anomaly, A suspicious telnet session or packet was detected,3

config classification: suspicious-http-protocol-anomaly, A suspicious HTTP session or packet was detected,3

config classification: suspicious-mail-protocol-anomaly, A suspicious Mail session or packet was detected,3

config classification: suspicious-ftp-protocol-anomaly, A suspicious FTP session or packet was detected,4

config classification: suspicious-threshold-exceeded, A suspicious threshold was triggered,4

config classification: denial-of-service-other, A new type of Denial of Service was detected,4

config classification: access-file-blocked, Access to a file was blocked,4

config classification: access-tunnel-connection, Access to a tunnel was identified,4

config classification: access-tunnel-closed, Access to a tunnel was closed,4

config classification: aystem-warning, A system Warning message was detected,4

config classification: system-emergency, A system Emergency message was detected,4

config classification: system-critical, A system Critical message was detected,4

config classification: system-error, A system Error message was detected,4

config classification: system-notification, A system Notification message was detected,4

config classification: system-information, A system Information message was detected,4

config classification: system-debug, A system Debug message was detected,4

config classification: system-alert, A system Alert message was detected,4

config classification: access-connection-opened, A connection was opened,4

config classification: access-connection-closed, A connection was closed,4

config classification: access-timeout, A timeout occurred,4

config classification: system-service-started, A service started,4

config classification: system-service-stopped, A service stopped,4

config classification: system-process-started, A process started,4

config classification: system-process-stopped, A process stopped,4

config classification: application-spam-detected, Some dirty spammer was detected,4

config classification: application-mail-dropped, The mail system dropped or refused mail,4

config classification: system-restart, A system restart was detected,4

config classification: system-started, A system startup was detected,4

config classification: system-stopped, A system stop was detected,4

config classification: system-locked, A system being locked was detected,4

config classification: system-unlocked, A system be unlocked was detected,4

config classification: network-ike-activity, IKE traffic was identified,4

config classification: network-h.323-activity, H.323 traffic was identified,4

config classification: network-ppp-activity, PPP traffic was identified,4

config classification: network-ocsp-activity, OCSP traffic was identified,4

config classification: network-l2tp-activity, L2TP traffic was identified,4

config classification: network-rip-activity, RIP traffic was identified,4

config classification: network-pptp-activity, PPTP traffic was identified,4

config classification: network-ssl-activity, SSL traffic was identified,4

config classification: network-igmp-activity, IGMP traffic was identified,4

config classification: network-ipsec-activity, IPSEC traffic was identified,4

config classification: network-pki-activity, PKI traffic was identified,4

config classification: voip-call-started, A VOIP call was started,4

config classification: voip-call-ended, A VOIP call was completed,4

config classification: voip-misc, A VOIP event occurred,4

config classification: network-bootp-activity, BOOTP traffic was identified,4

config classification: alert-ids-alert, The IDS did something,4

config classification: alert-ips-alert, The IPS did something,4

config classification: alert-hids-alert, The HIDS did something,4

config classification: application-mail-sent, An email was sent,4

config classification: application-mail-server-misc, A Mail server did something,4

config classification: application-mail-received, An email was recieved,4

config classification: availability-state-up, A system or service is now up,4

config classification: availability-state-down, A system or service is now down,4

config classification: availability-state-critical, A system or service is not in a critical state,1

config classification: availability-state-warning, A system or service has issued a warning,3

config classification: availability-state-unknown, A system or service is in an unknown state,3

config classification: availability-state-unreachable, A system or service is unreachable,1

config classification: application-vpn-opened, A VPN session was opened,4

config classification: application-vpn-closed, A VPN session was closed,4

config classification: application-vpn-denied, A VPN session was denied,2

config classification: application-vpn-misc, Something happened on a VPN session,2

config classification: system-configuration-changed, A system changed its configuration,4

config classification: network-misc, Something happened on the network,4

config classification: policy-phishing, A phishing attempt was detected,4

config classification: wireless-new-network, A new wireless network has been detected,4

config classification: wireless-client-associated, A new client has connected to the wireless network,4

config classification: wireless-flood, The wireless network is currently being flooded,2

config classification: wireless-disassociation, A wireless client has been disassociated from the network,4

config classification: wireless-deauthentication, A wireless client has been deauthenticated,4

config classification: wireless-anomaly, Something strange occurred on the wireless network,4

config classification: wireless-spoofing, Spoofing has been detected on the wireless network,2

config classification: wireless-scanner-detected, A scanner was detected on the wireless network,2

config classification: wireless-misc, Something happened on the wireless network,2

config classification: wireless-probe, A probe attempt was identified on the wireless network,4

config classification: inventory-service-detected, A new service has been identified,4

config classification: inventory-service-change, A service has changed,4

config classification: inventory-service-misc, A Misc service was detected,4

config classification: inventory-operating-system-detected, A new operating system was detected,4

config classification: inventory-operating-system-change, A system changed,4

config classification: inventory-operating-system-misc, A system met a Misc,4

config classification: inventory-mac-detected, A unhackable computer was detected,1

config classification: inventory-mac-change, A MAC address changed,4

config classification: policy-check-failed, A Policy check has failed,1

config classification: policy-check-passed, A Policy check has passed,1

config classification: network-high-load, The network currently has a high load,1

config classification: authentication-error, An authentication error was detected,4

config classification: application-web-modified, A content modified proxy request was identified,4

config classification: application-dhcp-release, A DHCP lease was released,4

config classification: application-dhcp-request, A DHCP request was detected,4

config classification: application-dhcp-lease, A DHCP lease was allocated,4

config classification: application-dhcp-pool-exhausted, All DHCP addresses have been allocated,4

config classification: application-dhcp-error, A DHCP error was detected,4

config classification: system-software-installed, A software package was installed,4

config classification: honeypot-connection-opened, Something connected to the honeypot sweet new warez,4

config classification: honeypot-attack-detected, A known attack was detected on the honeypot,4

config classification: honeypot-connection-closed, A connection to the honeypot was closed,4

config classification: application-dns-succesful-zone-tranfer, A succesful DNS zone transfer was detected,4

config classification: application-dns-zone-transfer-failed, A failed DNS zone transfer was detected,4

config classification: application-ftp-command-executed, An FTP command was executed,4

config classification: application-ftp-error, An FTP error was detected,4

config classification: application-ftp-connection-opened, An ftp connection was opened,4

config classification: application-ftp-connection-closed, An ftp connection was closed,4

config classification: database-login, A database login was detected,4

config classification: database-login-failed, A failed database login was detected,4

config classification: database-query, A database query was executed,4

config classification: database-logout, A database logout was detected,4

config classification: database-stop, A database was stopped,4

config classification: database-start, A database was started,4

config classification: database-error, A database error occurred,4