From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry

The evolving landscape of artificial intelligence is challenging developers to rethink their interaction with sophisticated tools, prompting a crucial question: are we shaping our AI, or is it shaping us? This philosophical inquiry formed the bedrock of a recent presentation by Ben O’Mahony, Principal AI Engineer at Thoughtworks, who unveiled an innovative approach to building more personalized, efficient, and cost-effective AI tools. Speaking at a leading industry conference, O’Mahony detailed how developers can leverage OpenTelemetry (OTEL) to capture nuanced production telemetry, distilling the complex behaviors of frontier models into smaller, domain-specific language models (SLMs) through a continuous data flywheel. This methodology not only addresses the rising costs and inherent limitations of generic large language models (LLMs) but also empowers engineers to maintain agency over their AI-driven workflows.

The Challenge of AI Tooling: Shaping or Being Shaped?

O’Mahony opened his address with a resonant quote, "We shape our tools, and thereafter our tools shape us," attributed to John Culkin paraphrasing Marshall McLuhan. He argued that this sentiment has never been more pertinent than in the age of AI. Building an AI tool is akin to creating an entity that will fundamentally reshape workflows, habits, and even cognitive processes. The core question, therefore, transcends mere functionality; it delves into the nature of the person or team the tool creates. Developers, known for their meticulous customization of environments—from Neovim configurations to intricate plugin ecosystems—often find this level of control elusive when interacting with off-the-shelf AI APIs.

O’Mahony highlighted his personal experience with an AI-powered Language Server Protocol (LSP) he developed. While he could craft the agent’s harness, instructions, and interactions, he faced significant constraints with the underlying AI API. Customization options were limited primarily to model selection, offering little ability to permanently improve or adapt the AI’s "brain" over time. A major practical concern was the prohibitive cost of tokens. His LSP, designed to be proactive, would send data to an AI agent every time a file was saved, leading to escalating expenses for continuous coding. The need for semantic understanding from LLMs was undeniable, but the operational overhead was substantial. This predicament led to a pivotal realization: the very act of using the tool was generating the precise data needed for its improvement.

Foundations: Language Server Protocol and AI Integration

To contextualize his solution, O’Mahony first outlined the foundational role of the Language Server Protocol (LSP). The LSP is an industry-standard protocol that enables editor-agnostic language features, allowing integrated development environments (IDEs) like VS Code, Neovim, or Emacs to provide rich programming language support. An editor acts as a client, sending requests to a separate language server process that understands the programming language’s syntax and semantics. This server responds with invaluable information such as documentation, go-to-definition capabilities, and diagnostics (errors, warnings, suggestions) upon file saves.

Traditional LSPs, such as pyright for Python or rust-analyzer for Rust, are predominantly rule-based. They parse code into abstract syntax trees (ASTs) and check them against predefined language rules and type inference systems. While exceptionally fast, precise, and reliable for syntactic and basic semantic checks, they inherently lack the ability to grasp deeper semantic intent. They cannot, for instance, discern if a function’s name no longer accurately reflects its updated functionality, or if an error-handling pattern is fragile, or if a standard library function is being needlessly reimplemented.

This is precisely where the AI LSP developed by O’Mahony filled a critical gap. Designed for proactive, agentic AI suggestions within the developer’s immediate workflow, it bypassed the need for separate chat panels. A significant advantage of its AI-driven nature is its language agnosticism; it doesn’t rely on parsing specific ASTs but can analyze any code or even Markdown for potential improvements and suggest fixes. O’Mahony demonstrated its utility with a meta example: the AI LSP suggesting improvements to its own codebase, delivering diagnostics and actionable code fixes directly within the editor, akin to traditional linter errors. This integration minimizes context switching, keeping developers "in flow." He emphasized the importance of "context engineering" not just for the AI agent, but also for the human engineer consuming its outputs, advocating for digestible, actionable suggestions over large, unmanageable pull requests. The AI LSP also offered multiple fix options, the ability to dismiss unhelpful suggestions, and a "regenerate" feature for alternative solutions, providing crucial user control and feedback points.

OpenTelemetry: The Observability Backbone for AI Agents

The second foundational element of O’Mahony’s approach is OpenTelemetry (OTEL), the burgeoning industry standard for observability. OTEL provides distributed tracing, metrics, and logs, enabling engineers to follow a request’s entire journey across complex distributed systems, services, and now, AI agents. Each step in this journey is represented by a "span," containing timing information, metadata, and contextual details. For generative AI systems, these traces become exceptionally valuable.

O’Mahony detailed how OTEL’s specification is increasingly tailored for generative AI, incorporating specific events for inputs and outputs (e.g., system, user, assistant messages, tool calls), and critical metrics like token usage, operation duration, and time-to-first-token. It also defines model and agent spans for operations like agent creation, workflow execution, chat completion, multi-modal generation, and tool execution. He stressed that every AI agent should be instrumented with OTEL from day one, not merely for distillation but for fundamental debugging and understanding of system behavior. Auto-instrumentation, available across most modern AI frameworks and libraries (e.g., Logfire, Pydantic AI), simplifies this process, getting developers "99% of the way there."

Crucially, OTEL captures the "gold" for AI improvement: user actions. O’Mahony instrumented his AI LSP to record specific behavioral feedback:

• Accepted: A strong positive signal when a user applies a suggested fix, explicitly endorsing its helpfulness.
• Regenerated: A weak positive signal, indicating agreement with the diagnostic but a desire for an alternative solution.
• Dismissed: Strong negative feedback, invaluable for identifying unhelpful or erroneous suggestions.

This goes far beyond generic "thumbs up/down" feedback, capturing actual product usage patterns and providing high-fidelity, explicit signals that are typically expensive and difficult to acquire in traditional machine learning pipelines.

Building the Data Flywheel: From Production Telemetry to Personalized SLMs

The core innovation lies in transforming these OTEL traces into a continuous "data flywheel" for model training. O’Mahony illustrated a typical progression:

• Day 1: Initial configuration and natural coding yield a few interactions.
• Week 1: Interactions grow to tens or hundreds, revealing nascent patterns.
• Month 1: Hundreds of labeled interactions form a substantial dataset.

This dataset, generated passively through daily tool usage, becomes the basis for fine-tuning. The mechanics involve exporting OTEL traces (e.g., to Logfire, Honeycomb, Datadog, or locally to JSON files via an OTel Collector), querying full traces where user action spans exist, and filtering out irrelevant data. The output is a simple JSONL file, with each line serving as a training example, requiring minimal preprocessing.

With several thousand examples, O’Mahony used frontier models (like Gemini) to initiate the distillation process. The goal was not to train a model from scratch but to fine-tune a smaller, existing coding-aware language model (in the 7-13 billion parameter range) to his specific codebase, coding style, and eccentricities. Using tools like MLX, this local fine-tuning process took only a few hours. The result was a Small Language Model (SLM) that was local, cheaper to run, faster, and significantly more personalized. While not achieving 100% parity with frontier models on novel, complex evaluations (O’Mahony cited 80-85% quality), the SLM was profoundly more tailored to his workflow, avoiding suggestions he consistently dismissed. This continuous feedback loop ensures the SLM constantly learns and improves, solidifying the data flywheel.

Scaling AI Intelligence: Platform Capabilities and Organizational Impact

O’Mahony emphasized that this methodology transcends individual developer tools, offering a blueprint for robust AI platform capabilities within organizations. The core pattern remains consistent: instrument agents with OTEL, capture user actions as training signals, distill model behavior into SLMs, and deploy with continuous iteration and safety guardrails. This approach enables a shared instrumentation layer, extraction pipeline, fine-tuning/serving infrastructure, and evaluation tooling across a diverse ecosystem of AI agents.

This platform approach transforms AI development from a series of one-off projects into an extensible, scalable utility. Adding new agents to the distillation loop becomes incremental: instrument, define user actions and feedback scales (strong positive, weak positive, strong negative), and plug into existing infrastructure. O’Mahony also highlighted the economic angle: traditional machine learning with manual data labeling is expensive and infrastructure-heavy. The data flywheel inverts this, making product usage the labeling process, with users (or the developer themselves) labeling data as a natural byproduct, effectively "for free." This leads to compounding returns: better models drive more usage, which generates more data, leading to even better models.

The vision extends to 2025, which O’Mahony provocatively termed "The Year of Agents." He envisions organizations with dozens of internal AI agents—customer support bots, code review assistants, data analysis tools—each generating telemetry and capable of continuous improvement through distillation. Critically, these agents can learn from each other. A customer support bot’s nuanced understanding of a product could inform a documentation generator, with multi-agent workflows generating complex, interlinked traces for specialized training. This fosters a "learning organization" where AI tools organically improve from real-world production usage.

Economic and Strategic Implications

The implications of O’Mahony’s approach are far-reaching. Economically, it offers a tangible solution to the escalating costs associated with reliance on large, proprietary LLMs and their token-based pricing. By distilling behavior into smaller, local models, organizations can significantly reduce operational expenses while enhancing privacy, as sensitive data can remain within internal infrastructure, addressing a key concern raised during the Q&A about leaking intellectual property.

Strategically, this methodology empowers organizations to develop highly specialized, "fit-for-purpose" AI tools that outperform generic models in specific contexts. This personalization leads to higher developer satisfaction and productivity, as the tools adapt to individual or team workflows rather than forcing users to conform to a generic AI’s outputs. The continuous learning loop ensures that AI capabilities remain cutting-edge and relevant to evolving business needs. Furthermore, by simplifying the process of distillation, it democratizes advanced AI model development, enabling a broader base of engineers and data scientists to contribute without requiring massive compute budgets or specialized ML research teams.

Shaping the Future of AI Development

O’Mahony concluded by reflecting on the historical tension between humanity and its tools. Just as IDEs and Git reshaped coding paradigms, AI tools are now profoundly influencing how engineers work. His drive to build his AI LSP stemmed from a desire for control and the ability to personally configure and provide feedback, ensuring he remained "in the driving seat." This approach champions a future of "many different intelligences" rather than a singular AGI—specialized, personalized, and portable SLMs that run locally, granting users and organizations greater control over their workflows and data. It’s a vision where AI tools are continuously improving, but always in service to human agency.

Key Takeaways for Practitioners

O’Mahony offered several concrete, actionable takeaways for engineers:

1. Instrument all agents with OpenTelemetry from day one: Don’t delay; capture everything, especially user actions, as this provides immediate value for debugging and understanding, even before distillation. Auto-instrumentation simplifies this process.
2. Design user experiences for observable decisions: Embed feedback mechanisms directly into the product workflow. Explicit user actions like "accept," "regenerate," or "dismiss" are far more powerful signals than passive "thumbs up/down" buttons, creating a natural feedback loop.
3. Embrace distillation when sufficient data exists: The tooling for model distillation has matured significantly, making it cheaper and easier than many realize. Even small datasets (a few hundred interactions) can yield surprising improvements, particularly for personal workflows. Start small with one agent, collect a month’s worth of data, and experiment.

The future of AI tooling, O’Mahony asserted, is local, personalized, and continuously improving. The ultimate choice for developers and organizations remains whether to build tools that shape them or tools that they actively shape.