Friday, August 26, 2016

Running Snort on Commodity Hardware - The pitfalls of large receive offload

While working on Snort integration for another project that does HTTP stream reassembly we came across some very strange behaviour:

During reassembly, fragments of the stream were missing or incorrectly reassembled.  Large chunks of the stream would be missing even though the packets that covered that piece of the stream's data had been received and processed.

At first we thought that this was a bug with the hardware checksum offload on the network card so we added '-k none' to the command line arguments. This seemed to resolve the issue, for the moment...

While testing the integration work we started noticing some very strange HTTP sessions: response codes that were very strange, invalid and missing, or truncated headers. This lead me to look into the issue again. Turning on full packet dumps showed me that the first packet in the reassembled stream coming from the Stream preprocessor was not the first packet in the stream, but instead a part of the response body.

The next step was to capture a pcap with tcpdump and use Snort in replay mode to reproduce the issue, this is where more strange things happened.  Using the pcap with Snort in replay mode, I could not reproduce the issue, but when watching the live stream it would fail ~3 out of 4 times.

This let me to look at what hardware acceleration features where enabled on the capture interface. It turned out that the card had large receive offload enabled (LRO) out of the box. This feature will automatically coalesce tcp frames in the same stream into larger frames rewriting all the headers to match the new larger frame.

Looking at the pcap showed that 2 frames in the stream had been coalesced into a larger 1900 byte frame, this frame was larger than Snort's default snaplen and being truncated. The truncation explained why '-k none' seemed to make it a little more reliable but not much. I tested both disabling LRO and raising the snaplen in Snort, both resolved the stream reassembly issues, and now your wondering which solution is the correct one. The answer is disabling LRO for a number of reasons, chief of which is:
  • LRO changes the stream that Snort sees on the wire, this means it can not do target based re-assembly and correctly detect common IDS avoidance techniques.

On Linux you can check the status of this feature using the following command, (replace "eth1" with the proper interface you are using as a sniffing interface):

ethtool -k eth1

And you can disable the feature as follows:

ethtool -K eth1 gro off
ethtool -L eth1 bro off

On FreeBSD you can see the interface flags in the output of ifconfig, to disable the features you can use the following command (replacing "em0" with the proper interface you are using as a sniffing interface):
ifconfig em0 -lro

1 comment:

  1. There is a document on the snort site which describes this and other pitfalls with network interface cards and system options under windows:

    Under the section 'Additional Resources' there is a link which reads:

    Possible Packet Loss During Reassembly for Snort IDS/IPS sensors

    which will give the snort user the above information and a lot more options to look for when using Snort in IDS or IPS modes...

    Disclosure: I did write the Packet Loss Document for the Snort Documents site :)