By Brandon Stultz.
Every day, new vulnerabilities are discovered in the software critical to the function of the modern world.
Security analysts take apart these new vulnerabilities, isolate what is necessary to trigger them and write signatures to block any exploits targeting them.
For Snort, these signatures are called Snort rules — and they’re extremely versatile. They can access specific network service fields, locate a vulnerable parameter and scan that parameter for the presence of an exploit.
They can also leverage numerous rule options to traverse protocols and file formats. Written well, these rules can have high efficacy and performance with few or no false positives.
This approach to defense is very good at protecting networks from known threats, but what if the threat is unknown? What if a vulnerability is discovered, an exploit for it is written, and the security community has no knowledge of it? We need another approach to defense that doesn’t require prior knowledge of the attack to function.
Over the past year at Cisco, we have been prototyping and building this new approach into a new detection engine for Snort. Today, I am proud to announce we are open-sourcing this engine to the community in the latest Snort 3 release (version 3.1.82.0). This new detection engine is called “SnortML.”
SnortML is a machine learning-based detection engine for the Snort intrusion prevention system. At a high level, there are two components to this new detection engine. The first component is the snort_ml_engine itself, which loads pre-trained machine learning models, instantiates classifiers based on these models and then makes the classifiers available for detection. The second is the snort_ml inspector, which subscribes to data provided by Snort service inspectors, passes the data to classifiers, and then acts on the output of the classifiers.
Currently, the snort_ml_engine module only has one model type, namely the http_param_model, but we plan on building other models in the future.
This http_param_model is used for classifying HTTP parameters as malicious or normal. Once the snort_ml_engine loads the http_param_model, it can be used in the snort_ml inspector to detect exploits.
The inspector subscribes to the HTTP request data provided by the HTTP inspector through the publish/subscribe interface. It then passes this data (HTTP URI query and optionally HTTP POST body) to a binary classifier based on the http_param_model. This classifier then returns the probability that it saw an exploit. Based on this probability, SnortML can generate an alert, similar to a Snort rule alert, which can be configured to block malicious traffic. Now that you know how the machine learning engine works, let’s get into how the models work.
SnortML models are designed to be extremely flexible, much like their Snort rule counterparts. To that end, we based our models and our inference engine on TensorFlow. The TensorFlow project is a free and open-source library for machine learning and artificial intelligence.
Any TensorFlow model can be a SnortML binary classifier model so long as it satisfies three conditions, namely, the model must have a single input tensor and a single output tensor, the input and output tensor types must be 32-bit floating point, and finally, the output tensor must have only a single element. We plan on adding other model types in the future (including multiclass classifiers), but right now, this is the only model type currently supported. The SnortML engine uses TensorFlow through a support library we call LibML. The LibML library handles loading, configuring and running machine learning models for Snort. It also includes the XNNPACK accelerator needed to run CPU-bound models at line rate. The easiest way to build a SnortML model is to use the TensorFlow Keras API. If you are new to machine learning, don’t worry, Keras is a simple but powerful deep-learning framework that allows you to build neural networks and train them in a few lines of Python. To get started, import the following:
We are going to train our example model on just two samples, but a real production model would use far more:
The next thing we need to do is prepare our data. SnortML models expect input data to be zero-padded which is what we are going to do here:
Now, we need to construct a neural network that can classify our data. This example uses a simple LSTM (Long Short-Term Memory) network, but other combinations of layers available in Keras work here as well. LSTM is a type of neural network that is keenly suited to identify patterns in sequences of data, such as the sequences of bytes in HTTP parameters.
To translate the bytes on the wire to tensors that the LSTM can accept, we can place an embedding layer in front of it. Embedding layers are a kind of association layer, they can learn relationships between input data (bytes in our case) and output those relationships as tensors that the LSTM neurons can accept.
Finally, we will converge the output of our LSTM neurons to a single output neuron with a Dense layer. This will serve as the output of the neural network.
Now for the fun part — let’s train this neural network:
Training output:
As you can see above, the accuracy of our network increased, and the loss dropped. These metrics show that the neural network learned to differentiate attack from normal in our example dataset.
Now, let’s save this model to a file so we can load it in Snort:
Now that we have a model file, we can run it against PCAPs with Snort 3:
If you have Snort 3 built with debug messages enabled, you can even trace the ML engine input and output.
Notice that even with variations in the SQL injection attack above, we still detected it. For years, we had dreamed about tackling the zero-day problem, providing coverage for attacks that were like those we had seen before, but targeting different applications or parameters. Now, with SnortML, this dream is becoming a reality. You can find the SnortML and LibML code here. Feel free to join the conversation on our Discord or on the Snort users mailing list if you have any questions or feedback.