Friday, March 15, 2024

Talos launching new machine learning-based exploit detection engine








By Brandon Stultz.

Every day, new vulnerabilities are discovered in the software critical to the function of the modern world. Security analysts take apart these new vulnerabilities, isolate what is necessary to trigger them and write signatures to block any exploits targeting them. For Snort, these signatures are called Snort rules — and they’re extremely versatile. They can access specific network service fields, locate a vulnerable parameter and scan that parameter for the presence of an exploit. They can also leverage numerous rule options to traverse protocols and file formats. Written well, these rules can have high efficacy and performance with few or no false positives. This approach to defense is very good at protecting networks from known threats, but what if the threat is unknown? What if a vulnerability is discovered, an exploit for it is written, and the security community has no knowledge of it? We need another approach to defense that doesn’t require prior knowledge of the attack to function. Over the past year at Cisco, we have been prototyping and building this new approach into a new detection engine for Snort. Today, I am proud to announce we are open-sourcing this engine to the community in the latest Snort 3 release (version 3.1.82.0). This new detection engine is called “SnortML.” SnortML is a machine learning-based detection engine for the Snort intrusion prevention system. At a high level, there are two components to this new detection engine. The first component is the snort_ml_engine itself, which loads pre-trained machine learning models, instantiates classifiers based on these models and then makes the classifiers available for detection. The second is the snort_ml inspector, which subscribes to data provided by Snort service inspectors, passes the data to classifiers, and then acts on the output of the classifiers. Currently, the snort_ml_engine module only has one model type, namely the http_param_model, but we plan on building other models in the future. This http_param_model is used for classifying HTTP parameters as malicious or normal. Once the snort_ml_engine loads the http_param_model, it can be used in the snort_ml inspector to detect exploits. The inspector subscribes to the HTTP request data provided by the HTTP inspector through the publish/subscribe interface. It then passes this data (HTTP URI query and optionally HTTP POST body) to a binary classifier based on the http_param_model. This classifier then returns the probability that it saw an exploit. Based on this probability, SnortML can generate an alert, similar to a Snort rule alert, which can be configured to block malicious traffic. Now that you know how the machine learning engine works, let’s get into how the models work. SnortML models are designed to be extremely flexible, much like their Snort rule counterparts. To that end, we based our models and our inference engine on TensorFlow. The TensorFlow project is a free and open-source library for machine learning and artificial intelligence. Any TensorFlow model can be a SnortML binary classifier model so long as it satisfies three conditions, namely, the model must have a single input tensor and a single output tensor, the input and output tensor types must be 32-bit floating point, and finally, the output tensor must have only a single element. We plan on adding other model types in the future (including multiclass classifiers), but right now, this is the only model type currently supported. The SnortML engine uses TensorFlow through a support library we call LibML. The LibML library handles loading, configuring and running machine learning models for Snort. It also includes the XNNPACK accelerator needed to run CPU-bound models at line rate. The easiest way to build a SnortML model is to use the TensorFlow Keras API. If you are new to machine learning, don’t worry, Keras is a simple but powerful deep-learning framework that allows you to build neural networks and train them in a few lines of Python. To get started, import the following:

import os

import numpy as np

import tensorflow as tf

from tensorflow.keras import layers

from urllib.parse import unquote_to_bytes

We are going to train our example model on just two samples, but a real production model would use far more:

# Example data

data = [

{ 'str':'foo=1', 'attack':0 },

{ 'str':'foo=1%27%20OR%201=1%2D%2D', 'attack':1 }

]

The next thing we need to do is prepare our data. SnortML models expect input data to be zero-padded which is what we are going to do here:

# Prepare Data

maxlen = 1024

X = []

Y = []

def decode_query(str):

return unquote_to_bytes(str.replace('+',' '))

for item in data:

arr = decode_query(item['str'])[:maxlen]

arrlen = len(arr)

seq = [0] * maxlen

for i in range(arrlen):

seq[maxlen - arrlen + i] = arr[i]

X.append(seq)

Y.append(item['attack'])

Now, we need to construct a neural network that can classify our data. This example uses a simple LSTM (Long Short-Term Memory) network, but other combinations of layers available in Keras work here as well. LSTM is a type of neural network that is keenly suited to identify patterns in sequences of data, such as the sequences of bytes in HTTP parameters. To translate the bytes on the wire to tensors that the LSTM can accept, we can place an embedding layer in front of it. Embedding layers are a kind of association layer, they can learn relationships between input data (bytes in our case) and output those relationships as tensors that the LSTM neurons can accept. Finally, we will converge the output of our LSTM neurons to a single output neuron with a Dense layer. This will serve as the output of the neural network.

#

# Build Model (Simple LSTM)

#

model = tf.keras.Sequential([

layers.Embedding(256, 32, input_length=maxlen, batch_size=1),

layers.LSTM(16),

layers.Dense(1, activation='sigmoid')])

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.summary()

Now for the fun part — let’s train this neural network: # # Train Model # model.fit(np.asarray(X).astype(np.float32), np.asarray(Y).astype(np.float32), epochs=100, batch_size=1) Training output: Model: "sequential" ----------------------------------------------------------------- Layer (type) Output Shape Param # ================================================================= embedding (Embedding) (1, 1024, 32) 8192 lstm (LSTM) (1, 16) 3136 dense (Dense) (1, 1) 17 ================================================================= Total params: 11,345 Trainable params: 11,345 Non-trainable params: 0 ----------------------------------------------------------------- Epoch 1/100 2/2 [==============================] - 1s 129ms/step - loss: 0.6910 - accuracy: 0.5000 ... Epoch 100/100 2/2 [==============================] - 0s 134ms/step - loss: 0.0208 - accuracy: 1.0000

As you can see above, the accuracy of our network increased, and the loss dropped. These metrics show that the neural network learned to differentiate attack from normal in our example dataset. Now, let’s save this model to a file so we can load it in Snort:

#

# Save Model

#

converter = tf.lite.TFLiteConverter.from_keras_model(model)

snort_model = converter.convert()

with open('snort.model', 'wb') as f:

f.write(snort_model)

Now that we have a model file, we can run it against PCAPs with Snort 3:

$ snort -q --talos \

--lua 'snort_ml_engine = { http_param_model = "snort.model" };' \

--lua 'snort_ml = {};' \

-r test.pcap

##### test.pcap #####

[411:1:0] (snort_ml) potential threat found in HTTP parameters via Neural Network Based Exploit Detection (alerts: 1)

#####

If you have Snort 3 built with debug messages enabled, you can even trace the ML engine input and output.

$ snort -q --talos \

--lua 'trace = { modules = { snort_ml = { all = 1 } } };' \

--lua 'snort_ml_engine = { http_param_model = "snort.model" };' \

--lua 'snort_ml = {};' \

-r test.pcap

P0:snort_ml:classifier:1: input (query): foo=1' OR 2=2-- P0:snort_ml:classifier:1: output: 0.971977 P0:snort_ml:classifier:1: <ALERT> ##### test.pcap ##### [411:1:0] (snort_ml) potential threat found in HTTP parameters via Neural Network Based Exploit Detection (alerts: 1) #####

P0:snort_ml:classifier:1: input (query): foo=1' OR 2=2--

P0:snort_ml:classifier:1: output: 0.971977

P0:snort_ml:classifier:1: <ALERT>

##### test.pcap ##### [411:1:0] (snort_ml) potential threat found in HTTP parameters via Neural Network Based Exploit Detection (alerts: 1) #####

##### test.pcap #####

[411:1:0] (snort_ml) potential threat found in HTTP parameters via Neural Network Based Exploit Detection (alerts: 1)

#####

Notice that even with variations in the SQL injection attack above, we still detected it. For years, we had dreamed about tackling the zero-day problem, providing coverage for attacks that were like those we had seen before, but targeting different applications or parameters. Now, with SnortML, this dream is becoming a reality. You can find the SnortML and LibML code here. Feel free to join the conversation on our Discord or on the Snort users mailing list if you have any questions or feedback. 

Thursday, March 7, 2024

Snort 2.9.8.3 and Snort 2.9.13.0 End of Life

We are announcing the end of life for Talos rules in the following versions of Snort 2:

  • Snort 2.9.8.3
  • Snort 2.9.13.0
  • Snort 2.9.8.3 Rules: This rule set is no longer available.
  • Snort 2.9.13.0 Rules: We will no longer produce Talos rules for these versions of Snort on or around July 1, 2024.

We encourage our open-source users to upgrade to the latest version of Snort 3 available here: Snort.org/downloads.

For users who would like to continue to use Snort 2, we recommend updating to Snort 2.9.20 as soon as possible, which can be found at Snort.org/downloads.

If you have any questions please feel free to reach out to us at: snort-sub@cisco.com or join our discord: Snort Discord Invite.

Tuesday, September 26, 2023

ICS protocol coverage using Snort 3 service inspectors










By Jared Rittle.

With more devices on operational technology (OT) networks now getting connected to wide-reaching IT networks, it is more important than ever to have effective detection capabilities for ICS protocols.

However, there are a few issues that usually arise when creating detection for ICS protocol traffic.

Oftentimes, the protocols connecting these devices on modern networks originate in older serial protocols. This transition resulted in protocols that use techniques like bitfields to reduce message size and multiple levels of encapsulation to avoid changes to the original protocol. These protocols often support combining multiple requests into one packet (pipelining) or splitting up a single request across multiple packets (fragmenting). Snort is fully capable of detecting traffic using any of these approaches, however, it requires a deeper understanding of the underlying protocol and more complicated plaintext rules, which is not always feasible.

The solution to these problems lies in the use of a Snort 3 service inspector for protocols requiring increased detection capabilities. Service inspectors are an evolution of Snort 2's preprocessors, providing access to additional built-in rules that look for protocol-level abnormalities, normalize pipelined and fragmented messages, and provide additional verification that the traffic being inspected is the expected protocol. Through the use of rule options exposed by existing service inspectors, plaintext rule writers can focus on the coverage of interest and let Snort handle protocol decoding and normalization.

Read the rest of this post over on the Talos blog.

Monday, April 3, 2023

Applications open now for 2023 Snort scholarship










Applications are now open for the $10,000 Snort scholarship. We encourage everyone eligible to apply here. We will be accepting applications through May 3. 

After that, our hand-picked panel will review the submissions and select two students to receive a $10,000 award each. 

For more detailed instructions on applying, check out the video below. 

To be eligible for the scholarship, you must have or be eligible to receive your high school diploma or an equivalent in 2023 as of the date Cisco receives your application. Each applicant must provide reasonable evidence to Cisco that you are seeking a degree in computer science, information technology, computer networking, cybersecurity or a similarly related field of study from a school located in the U.S. or a U.S. territory.   

To apply for the scholarship, you must answer a series of short essay questions, which will be the main basis for how we select the winners.  

The selection process is different from years past. Our panel will review all submissions and score the responses on the following 15-point scale:  

  • Originality (Score 1-5): Points will be assigned based on the assessment of original, fresh thoughts and concepts including anecdotes or examples of how security or a related field has shaped the personal and/or professional life of the applicant.  
  • Knowledge of Snort (Score 1-5): Points will be assigned on how well the applicant understands Snort and its use.  
  • Overall Submission Quality (Score 1-5): Points will be assigned on the overall quality of the submission. Factors include, but are not limited to, perceived effort and sincerity level.  

The panel of judges will score each submission, and then we will select a winner based on the top cumulative score. In the event of a tie, the judges will select the winner based on their responses’ originality.   

We hope these applications will introduce aspiring researchers and IT professionals to Cisco’s job pool and establish early communication between applicants and potential future job opportunities.

Monday, January 30, 2023

Snort v3.1.53.0 is now available!

The SNORTⓇ team recently released a new version of Snort 3 on Snort.org and the Snort 3 GitHub

Snort 3.1.53.0 contains several new features and bug fixes. Here's a complete rundown of what's new in this version. Users are encouraged to update as soon as possible, or upgrade to Snort 3 if they have not already done so.

Here's a rundown of all the changes and new features in this latest version of Snort 3:

  • appid: publish tls host set in eve process event handler only when appid discovery is complete
  • detection: show search algorithm configured
  • file_api: handling filedata in multithreading context
  • flow: add stream interface to get parent flow from child flow
  • memory: added memusage pegs
  • memory: fix unit test build w/o reg test

Snort 3 is the next generation of the Snort Intrusion Prevention System. The GitHub page will walk users through what Snort 3 has to offer and guide users through the steps of getting set up—from download to demo. Users unfamiliar with Snort should start with the Snort Resources page and the Snort 101 video series

You can subscribe to the newest rule detection functionality from Talos for as low as $29.99 a year with a personal account. See our business pricing as well here. Make sure and stay up to date to catch the most emerging threats. 

Tuesday, October 18, 2022

New Snort 3 rule writing guide available

Snort 3's new features, improvements and detection capabilities come with updates to the Snort rule language syntax and the rule-writing process.  

To help with that, direct from the Talos analyst team, comes the Snort 3 Rule Writing guide: Detailed documentation for all the different rule options available in Snort 3. 

The Snort 3 Rule Writing Guide is meant for new and experienced Snort rule-writers alike, focusing primarily on the rule-writing process. It is intended to supplement the documentation provided in the official Snort 3 repository (the official Snort User Manual). Each rule option has its own page to describe its functionality and syntax, along with examples to show how the option might be used in a Snort rule.  

The guide covers the essential information for new Snort users to get Snort 3 up and running. This includes installation and usage instructions, a brief look into Snort 3's internals, the basics of configuration files, and detailed information on writing effective Snort 3 rules. Despite the manual's broad scope, users will however still need to refer to the full user manual to find more comprehensive and advanced guidance on non-rule-writing-specific topics. 

Experienced Snort users who are already comfortable using Snort can skip the "Getting Started" section and instead jump right to the "Rule Options" section to get extensive documentation on the unchanged, updated and new rule options present in Snort 3. Watch out specifically for the now-sticky HTTP buffers, the new "alert file" and "alert http" rule types, as well as the new options like "http_param", "js_data", and "bufferlen".

As Snort 3 continues to evolve, this manual will too. The analyst team will provide updates to the manual to keep the greater Snort community abreast of any recent changes. 

Thursday, September 22, 2022

Snort OpenAppID Detectors have been updated

SNORTⓇ released a new update today for its OpenAppID Detector content.

This release — build 356 — includes:
  • 3,374 detectors. 
  • Additional detectors from the open-source community. For more details on which contributions were included — we have added them to the "Authors" file in this package.
The release is available now on our Downloads page. We look forward to users downloading and using the new features. If you have any feedback,  please share it with the OpenAppID mailing list.

The OpenAppID package is also compatible with our most recent Snort 3 releases.

For more information regarding the applications that are included in the open-source version of OpenAppID, feel free to visit our new application portal at appid.cisco.com.

Thursday, June 9, 2022

Changes to the community rule release schedule

By Jon Munshaw. 

As of this week, we are changing the cadence for releases for the Snort community rule set. 

Previously, the community rules were released every day at 11:40 a.m. ET, even if there are no rule changes. Now, the rule set will align with our normal open-source build and release schedule. This is usually every Tuesday and Thursday, though this may change based on public holidays and ad hoc releases for certain vulnerabilities or malware families. 

We apologize for any disruptions this may cause.  

Community rules are a set of rules that members of our open-source community or Snort integrators have submitted. These rules are freely available to all Snort users and are governed by the GPLv2. Anyone can submit a community rule using the Snort Rules mailer here

Community rules are available for anyone to download here without registration and are free of charge without any Rule Set License restrictions.  

Friday, April 1, 2022

Weekly Snort rule update for March 25 - April 1

 Cisco Talos released two new rule sets for SNORTⓇ this week, which you can view here and here.

There are multiple rules to protect against the exploitation of the highly publicized Spring4Shell vulnerabilities that could lead to remote code execution. Spring is a popular framework used to develop Java applications. Snort SIDs 30790 - 30793, 59388 and 59416 can detect this activity.

For more on these vulnerabilities, read the Talos blog here

All users can subscribe to Talos' newest rule detection functionality for as low as $29 a year with a personal account. Be sure and see our business pricing as well here. The Snort 3 release is also here after years of development and improvements, which you can upgrade to here.

Snort's rule blog posts are switching to a weekly recap format, rather than releasing every day a new rule update is released. If you have any feedback on this blog format, please reach out to us on Twitter @Snort

Friday, March 25, 2022

Weekly Snort rule update for March 21 - 25

Cisco Talos released two new rule sets for SNORTⓇ this week, which you can view here and here.

All users can subscribe to Talos' newest rule detection functionality for as low as $29 a year with a personal account. Be sure and see our business pricing as well here. The Snort 3 release is also here after years of development and improvements, which you can upgrade to here.

Snort's rule blog posts are switching to a weekly recap format, rather than releasing every day a new rule update is released. If you have any feedback on this blog format, please reach out to us on Twitter @Snort