US Army Analytics Group – Cybersecurity Anomaly Detection 1000X Faster With Less False Positives

The US Army Analytics Group (AAG) provides analytical services for various organizational operations and functions, including cybersecurity. AAG signed a Cooperative Research and Development Agreement (CRADA) with Entanglement, Inc., and strategic partner Groq, Inc., a US semiconductor company, to determine an optimal cybersecurity anomaly detection capability.

AAG has released a Validation Report confirming Entanglement AI’s solution that solves cybersecurity anomaly detection three orders of magnitude faster than traditional methods with fewer false positives. In this article, I will unpack the details behind these dramatic results.

GROQ

Methods for detecting cyber anomalies

All cyber-attacks, whether zero-day or ransomware, share a common thread: cyber anomalies. A cyber anomaly is something out of the ordinary, an outlier, such as excessive logins, spikes in traffic, or a considerable number of remote logins.

The three primary forms of anomaly detection are: unsupervised, supervised, and semi-supervised. Security analysts use each approach to varying degrees of effectiveness in cybersecurity applications.

Unsupervised anomaly detection uses an unlabeled test set of data. It involves training a machine learning (ML) model to identify normal behavior using an unlabeled dataset. The assumption is that most instances in the data set will be normal. The anomaly detection algorithm detects instances that appear not to fit with the data set. Unsupervised anomaly detection algorithms include Autoencoders, K-means, Gaussian Mixture Modelling (GMMs), hypothesis tests-based analysis, and Principal Component Analysis (PCAs).

Supervised anomaly detection uses data set with a set of “normal” and “abnormal” labels and a trained classification algorithm.

ML builds a predictive model from a labeled training set with normal and abnormal data. Supervised methods include Bayesian networks, k-nearest neighbors, decision trees, supervised neural networks, and support vector machines (SVMs).

Semi-supervised anomaly detection techniques use a combination of a small set of labeled data and large amount of unlabeled data for training. That model then detects anomalies by testing how likely the model is to generate any one instance encountered.

Executive order to adopt zero-trust security

In May of 2021, President Biden issued an Executive Order mandating all federal agencies to adopt zero-trust security. One month later, in June 2021, Entanglement, Inc., and strategic partner Groq, Inc., a US semiconductor company, made a no-cost offer of assistance to detect and resolve anomalies in support of a zero-trust environment.

The project’s goal was to continuously monitor a zero-trust security architecture, requiring an anomaly detection algorithm capable of constantly vetting all users on a network and actions. A similar algorithmic framework will be suitable for demonstrating Intrusion Detection Systems (IDS) and expanded threat awareness at network endpoints.

The project focused on three areas: improving auto-encoder functionality and performance over existing systems, accelerating generative adversarial network (GAN) functionality, and integrating a quantum-inspired optimization SVM algorithm using Quadratic Unconstrained Binary Optimization (QUBO).

Cybersecurity anomaly detection faster than traditional methods 

The work by Entanglement and Groq under the CRADA demonstrated cybersecurity anomaly detection faster than traditional methods and better performance measured by Key Performance Parameters (KPPs). The KPP’s covered metrics related to total inferences per second, percentage of threats detected, accuracy, recall, precision, other confusion matrix-based metrics, and Area Under the Curve (AUC).

Previous AAG efforts detected 120,000 inferences per second, the benchmark and standard achievable using a QUBO model.

Within six months, Entanglement achieved an anomaly detection rate of 72,000,000 inferences per second and demonstrated the potential of attaining 120,000,000 inferences per second across a broad domain of data processing systems.

Validation cases used the KDD Cup 1999 (KDD99) and CICIDS2017 data sets.

The calculated output demonstrated for the Autoencoder and GAN solution was highly effective in determining anomalies. The QUBO SVM was built in quantum-ready form and was also effective at anomaly detection.

Wrapping up

Entanglement has delivered a dramatically faster and more accurate cybersecurity anomaly detection capability – with far fewer false positives – than traditional technology. The Entanglement and Groq solution provided anomaly detection at 120 million inferences per second, three orders of magnitude faster than any other technology.

What is most surprising is that Entanglement used quantum-based algorithms, but there wasn’t a quantum computer that could perform as fast as GroqChip. The answer lies in the core Groq technology, a purpose-built digital circuit design with high degrees of parallelism, making it for solving a range of problems such as deep neural network models and Quadratic Unconstrained Binary Optimization (QUBO) problems.

We have known for a while that realizing the benefits of AI, innovative infrastructure, and predictive intelligence will require a much simpler and more scalable processing architecture than a legacy solution.

Groq designed a chip that delivers predictable and repeatable performance with low latency and high throughput across the system called the tensor streaming processor (TSP). The new, simpler processing architecture is designed specifically for the performance requirements of ML applications and other compute-intensive workloads.

Groq now has multiple customers across verticals who have used their accelerator solutions to achieve orders of magnitude performance improvements. I look forward to sharing those stories with you in the future.