Friday, August 26, 2016

Running Snort on Commodity Hardware - The pitfalls of large receive offload

While working on Snort integration for another project that does HTTP stream reassembly we came across some very strange behaviour:

During reassembly, fragments of the stream were missing or incorrectly reassembled.  Large chunks of the stream would be missing even though the packets that covered that piece of the stream's data had been received and processed.

At first we thought that this was a bug with the hardware checksum offload on the network card so we added '-k none' to the command line arguments. This seemed to resolve the issue, for the moment...

While testing the integration work we started noticing some very strange HTTP sessions: response codes that were very strange, invalid and missing, or truncated headers. This lead me to look into the issue again. Turning on full packet dumps showed me that the first packet in the reassembled stream coming from the Stream preprocessor was not the first packet in the stream, but instead a part of the response body.

The next step was to capture a pcap with tcpdump and use Snort in replay mode to reproduce the issue, this is where more strange things happened.  Using the pcap with Snort in replay mode, I could not reproduce the issue, but when watching the live stream it would fail ~3 out of 4 times.

This let me to look at what hardware acceleration features where enabled on the capture interface. It turned out that the card had large receive offload enabled (LRO) out of the box. This feature will automatically coalesce tcp frames in the same stream into larger frames rewriting all the headers to match the new larger frame.

Looking at the pcap showed that 2 frames in the stream had been coalesced into a larger 1900 byte frame, this frame was larger than Snort's default snaplen and being truncated. The truncation explained why '-k none' seemed to make it a little more reliable but not much. I tested both disabling LRO and raising the snaplen in Snort, both resolved the stream reassembly issues, and now your wondering which solution is the correct one. The answer is disabling LRO for a number of reasons, chief of which is:
  • LRO changes the stream that Snort sees on the wire, this means it can not do target based re-assembly and correctly detect common IDS avoidance techniques.

On Linux you can check the status of this feature using the following command, (replace "eth1" with the proper interface you are using as a sniffing interface):

ethtool -k eth1

And you can disable the feature as follows:

ethtool -K eth1 gro off
ethtool -L eth1 bro off

On FreeBSD you can see the interface flags in the output of ifconfig, to disable the features you can use the following command (replacing "em0" with the proper interface you are using as a sniffing interface):
ifconfig em0 -lro

Thursday, August 25, 2016

Snort Subscriber Rule Set Update for 08/25/2016

Just released:
Snort Subscriber Rule Set Update for 08/25/2016


We welcome the introduction of the newest rule release from Talos. In this release we introduced 25 new rules and made modifications to 10 additional rules.

There were no changes made to the snort.conf in this release.

Talos's rule release:
Talos has added and modified multiple rules in the blacklist, browser-plugins, file-office, file-pdf, malware-cnc, malware-other, os-linux, protocol-snmp and server-webapp rule sets to provide coverage for emerging threats from these technologies.


In order to subscribe now to Talos's newest rule detection functionality, you can subscribe for as low as $29 US dollars a year for personal users, be sure and see our business pricing as well at https://www.snort.org/products. Make sure and stay up to date to catch the most emerging threats!

Tuesday, August 23, 2016

Snort Subscriber Rule Set Update for 08/23/2016

Just released:
Snort Subscriber Rule Set Update for 08/23/2016


We welcome the introduction of the newest rule release from Talos. In this release we introduced 3 new rules and made modifications to 2 additional rules.

There were no changes made to the snort.conf in this release.

Talos's rule release:
+Talos has added and modified multiple rules in the blacklist, browser-plugins, +file-office, file-pdf, malware-cnc, malware-other, os-linux, protocol-snmp and +server-webapp rule sets to provide coverage for emerging threats from these +technologies.


In order to subscribe now to Talos's newest rule detection functionality, you can subscribe for as low as $29 US dollars a year for personal users, be sure and see our business pricing as well at https://www.snort.org/products. Make sure and stay up to date to catch the most emerging threats!

Thursday, August 18, 2016

Snort Subscriber Rule Set Update for 08/18/2016

Just released:
Snort Subscriber Rule Set Update for 08/18/2016


We welcome the introduction of the newest rule release from Talos. In this release we introduced 22 new rules and made modifications to 10 additional rules.

There were no changes made to the snort.conf in this release.

Talos's rule release:
Talos has added and modified multiple rules in the blacklist, browser-plugins, file-office, file-pdf, malware-cnc, malware-other, os-linux, protocol-snmp and server-webapp rule sets to provide coverage for emerging threats from these technologies.


In order to subscribe now to Talos's newest rule detection functionality, you can subscribe for as low as $29 US dollars a year for personal users, be sure and see our business pricing as well at https://www.snort.org/products. Make sure and stay up to date to catch the most emerging threats!

Wednesday, August 17, 2016

Snort Subscriber Rule Set Update for 08/16/2016

Just released:
Snort Subscriber Rule Set Update for 08/16/2016


We welcome the introduction of the newest rule release from Talos. In this release we introduced 8 new rules and made modifications to 3 additional rules.

There were no changes made to the snort.conf in this release.

Talos's rule release:
Talos has added and modified multiple rules in the browser-plugins, file-flash, file-image, indicator-compromise, indicator-obfuscation, malware-cnc and server-other rule sets to provide coverage for emerging threats from these technologies.


In order to subscribe now to Talos's newest rule detection functionality, you can subscribe for as low as $29 US dollars a year for personal users, be sure and see our business pricing as well at https://www.snort.org/products. Make sure and stay up to date to catch the most emerging threats!

Friday, August 12, 2016

Snort Subscriber Rule Set Update for 08/12/2016

Just released:
Snort Subscriber Rule Set Update for 08/12/2016


We welcome the introduction of the newest rule release from Talos. In this release we introduced 2 new rules and made modifications to 3 additional rules.

There were no changes made to the snort.conf in this release.



Talos's rule release:
Talos has added and modified multiple rules in the file-other and server-webapp rule sets to provide coverage for emerging threats from these technologies.


In order to subscribe now to Talos's newest rule detection functionality, you can subscribe for as low as $29 US dollars a year for personal users, be sure and see our business pricing as well at https://www.snort.org/products. Make sure and stay up to date to catch the most emerging threats!

Thursday, August 11, 2016

Snort Subscriber Rule Set Update for 08/11/2016

Just released:
Snort Subscriber Rule Set Update for 08/11/2016


We welcome the introduction of the newest rule release from Talos. In this release we introduced 31 new rules and made modifications to 6 additional rules.

There were no changes made to the snort.conf in this release.



Talos's rule release:
Talos has added and modified multiple rules in the blacklist, exploit-kit, file-office, file-other, file-pdf, indicator-compromise, malware-cnc, netbios and server-webapp rule sets to provide coverage for emerging threats from these technologies.


In order to subscribe now to Talos's newest rule detection functionality, you can subscribe for as low as $29 US dollars a year for personal users, be sure and see our business pricing as well at https://www.snort.org/products. Make sure and stay up to date to catch the most emerging threats!