Cybersecurity

Sophos AI Researchers Leverage Large Language Models to Solve the Cybersecurity Data Labeling Dilemma

In the high-stakes environment of modern cybersecurity, the efficacy of supervised machine learning models is inextricably linked to the quality and diversity of the data used to train them. Recent research presented by Sophos data scientists Ben Gelman, Tamas Nyiri, and Tibor Kristóf Lányi at BSides Las Vegas 2026 has introduced a novel architectural pipeline designed to overcome the persistent problem of the "long tail" in threat detection. By integrating traditional anomaly detection methods with Large Language Models (LLMs), the research team has developed a scalable, automated system that effectively labels unusual files—data previously ignored by standard training sets—to significantly bolster the accuracy of cybersecurity classifiers.

The presentation at BSides Las Vegas serves as a direct extension of the foundational research initially showcased at Black Hat USA 2025. This multi-year effort underscores a growing trend in the industry: the transition from viewing anomaly detection as a standalone alerting tool to utilizing it as a sophisticated pre-filtering mechanism for more robust supervised learning models.

The Core Problem: Training Data Bias and the Long Tail

Supervised classifiers are the workhorses of contemporary security operations centers (SOCs). However, they are inherently biased by the data used during their development. Benign training sets are typically curated from corporate environments, known-clean networks, and traffic patterns that are easily identified through regex-based rules. Consequently, these sets are dominated by common, "normal" patterns, while the vast, unusual, yet legitimate "long tail" of data is systematically excluded.

Conversely, malicious training sets are restricted to known attack vectors identified during past incident investigations. When production classifiers encounter files that fall into the gaps—the unusual but not strictly "known" malicious files—the result is a binary failure: false positives that generate alert fatigue for human analysts, or false negatives that allow sophisticated threats to bypass defenses.

For years, the industry has attempted to mitigate this by deploying anomaly detection systems, such as isolation forests or tree-based models, to surface unusual events. Technologies like SCADE, which utilizes isolation forests for command-line classification, or systems designed to discover anomalous parent-child process relationships, have consistently reported false positive rates between 2% and 3%. While these percentages appear negligible, in a production environment processing tens of millions of events daily, they produce an unsustainable volume of noise. The Sophos research team argues that these traditional approaches fail because they treat anomaly detection as an end-to-end alerting mechanism rather than a component of a larger, smarter classification pipeline.

The Chronology of Innovation

The Sophos team’s methodology has evolved through a structured, multi-phase research lifecycle:

  1. Phase I (2025): Initial research presented at Black Hat USA 2025 focused on the fundamental failures of traditional anomaly detection. The team demonstrated that relying on isolated detection methods led to inconsistent performance, establishing the need for a more nuanced approach to labeling.
  2. Phase II (Early 2026): Development of the multi-modal pipeline. The team moved away from single-method detection, opting for a combination of isolation forests, k-means clustering, and Principal Component Analysis (PCA) reconstruction error to ensure broader coverage across diverse feature representations.
  3. Phase III (Mid 2026): Refinement of the LLM-based labeling strategy. Recognizing that a single LLM prompt could not achieve high precision on both malicious and benign labels, the researchers implemented a "dual-prompt" system.
  4. Phase IV (BSides Las Vegas 2026): Formal presentation of the findings, demonstrating the efficacy of the pipeline across command-line, HTML, and RTF data types.

Architectural Precision through Dual-Prompting

The most critical innovation in the Sophos methodology is the use of two distinct LLM prompts, calibrated to be mutually exclusive in their bias. A single prompt trying to categorize both benign and malicious anomalies often fails in ambiguous scenarios, such as "dual-use" scripts—programs that are legitimate in a developer’s hands but dangerous in an attacker’s.

To solve this, the team deployed two parallel prompts. The first prompt is instructed to lean toward a "malicious" classification, meaning it is conservative about labeling anything as benign. This produces high-precision benign labels. The second prompt is instructed to lean toward a "benign" classification, maintaining high standards for what it considers malicious, thereby producing high-precision malicious labels.

The pipeline only accepts a label if both models reach a consensus. If the two prompts disagree, the example is discarded. This creates a "gold standard" dataset for retraining the supervised classifier. By discarding ambiguous cases, the researchers ensure that the training data is of the highest possible quality, effectively filtering out the "gray area" that plagues traditional automated labeling systems.

Empirical Evaluation and Data Performance

The Sophos AI team tested this method against three structurally distinct data types, each with varying levels of complexity and ingestion scales. The results demonstrate that the pipeline is not only scalable but also adaptable to different attack surfaces.

When testing against RTF (Rich Text Format) files, the team observed a significant performance jump. The addition of labeled anomalies from the isolation forest method increased the model’s performance by 2.41%—a substantial improvement in the context of high-performing ROC AUC (Receiver Operating Characteristic Area Under the Curve) models.

The evaluation of HTML files provided a more complex picture. With a higher baseline performance, the gains were more modest. Notably, while K-means clustering yielded a 0.68% increase, the PCA reconstruction error actually resulted in a slight performance decrease. This outcome validates the team’s decision to utilize multiple, complementary anomaly detection methods; by running several algorithms in parallel, the team was able to identify which detection method was most valuable for a specific data type without needing to know the result in advance.

Implications for the Cybersecurity Industry

The implications of this research are significant for both the AI development community and security practitioners. By successfully automating the "long tail" labeling process, Sophos has provided a blueprint for reducing the manual burden on data scientists.

1. Reduced Human Dependency: Historically, the most effective way to improve a classifier was to have human analysts manually review and label millions of anomalies—a task that is cost-prohibitive and slow. This pipeline shifts that burden to LLMs, which operate at machine speed.

2. Increased Model Resilience: By continuously feeding "hard" anomalies back into the training cycle, supervised classifiers can evolve alongside changing attacker tactics. The model essentially "learns" from its own previous failures.

3. Shift in Anomaly Detection Strategy: The research signals a move away from the "alerting" paradigm. Security vendors are increasingly likely to adopt this approach, where anomaly detection serves as the "eyes" that find the unusual, and the LLM acts as the "judge" that interprets the context, ultimately making the primary detection engine more precise.

Looking Ahead

For data scientists looking to replicate or build upon these findings, the Sophos team emphasizes the importance of diverse feature representations. The pipeline’s success is largely attributed to the low overlap between the detection methods used; because each algorithm (Isolation Forest, K-means, PCA) captures different structural characteristics of the data, the resulting dataset is far more diverse than one created by a single, monolithic detector.

As the industry faces an increasingly complex threat landscape, where attackers are themselves leveraging AI to obfuscate malicious payloads, the ability to rapidly identify and classify the "unusual" will become a cornerstone of effective defense. The work presented by the Sophos team represents a significant step toward a future where cybersecurity models are not just reactive, but self-improving, effectively closing the gap between the known threats of the past and the emerging, long-tail threats of the future.

Related Articles

Back to top button