Does ‘rtk’ skill really cut agent tokens by 60–90%? We tested it.

Independent rigorous benchmarking by JetBrains AI has revealed that the much-touted ‘rtk’ (Rust Token Killer) skill for AI coding agents, despite its innovative compression techniques, failed to deliver its advertised 60–90% token savings when applied to real-world agentic coding tasks. In fact, comprehensive testing showed a marginal cost increase of 7.6% at low reasoning effort and no significant change at high effort, casting a critical light on how AI tool efficiency claims are often evaluated.

The Quest for LLM Efficiency: A Growing Imperative

In the rapidly evolving landscape of artificial intelligence, large language models (LLMs) are increasingly being deployed as autonomous agents capable of performing complex tasks, from code generation and debugging to system administration. A critical factor in the practical and economic viability of these agents is their token consumption. Every interaction with an LLM incurs a cost, measured in "tokens" – the fundamental units of text that LLMs process. High token usage translates directly to higher operational costs, increased latency, and limitations on the agent’s contextual understanding due to finite context windows. This makes any tool promising significant token savings immensely appealing to developers and organizations leveraging AI agents.

JetBrains, a leading developer of intelligent software development tools, has been at the forefront of integrating AI into its product ecosystem. Recognizing the importance of empirical validation in this space, the company has undertaken a methodical series of benchmarks to assess the real-world performance of various AI agent optimization tools. This particular study marks the second installment in this series, following a previous evaluation of a "caveman skill" which claimed 65% savings but measured a modest 8.5%. The consistent methodology underscores JetBrains’ commitment to providing data-driven insights for the AI development community.

The ‘rtk’ tool, an open-source command-line interface (CLI) proxy available on GitHub, presented a compelling value proposition. Its core function is to intercept standard shell commands executed by an AI agent (such as git status or python -m pytest), execute them, and then return a significantly compressed, human-readable summary of the output to the LLM. For instance, a verbose git status output with eleven lines of "porcelain" might be condensed to a concise * master / M a.txt / ?? b.txt. The tool’s documentation specifically highlighted a session where 118,000 tokens of command output were reduced to 24,000, promising a dramatic 60–90% reduction in token consumption. This mechanism, facilitated by a Claude Code PreToolUse hook, transparently rewrites eligible shell commands, theoretically allowing the agent to benefit from compression without needing explicit awareness of rtk‘s presence.

While the compression itself appeared technically sound and often elegant in its summarization, JetBrains’ researchers posed two fundamental questions that the public documentation did not adequately address: Firstly, what proportion of a typical AI agent’s session actually involves Bash output that rtk can intercept and compress? Secondly, does this compression, which inherently involves making "editorial judgments" about the output, compromise the correctness or quality of the agent’s work? Token savings at the expense of task accuracy would negate any perceived benefit.

A Rigorous Benchmark: Unpacking the Methodology

To answer these questions, JetBrains deployed a robust, paired A/B benchmarking framework designed to isolate the effects of rtk with high statistical confidence. The setup was meticulously controlled to ensure objective and reproducible results:

• Harness: Harbor 0.18, a specialized testing environment providing Docker sandboxes, comprehensive task verifiers, and the capability for paired runs. This ensured identical execution environments for both control and experimental arms.
• Agent: Claude Code 2.1.201, run in a headless configuration with bypassPermissions enabled, and strictly pinned in both arms to eliminate agent version variability. Claude Code is a sophisticated AI assistant designed for coding tasks, making it an ideal candidate for this evaluation.
• Model: claude-sonnet-5, a mid-tier LLM from Anthropic, chosen for its balance of performance and cost-effectiveness. The benchmark was run twice – once at "low reasoning effort" and again at "high reasoning effort" – to observe if the model’s cognitive load influenced its interaction with compressed output.
• Benchmark Suite: SkillsBench, a collection of 87 diverse coding tasks (86 were used in the final analysis), auto-graded with a 0–1 score and partial credit for granular evaluation of task completion quality.
• Arms:
  • Arm A (Control): Stock Claude Code, representing the baseline performance.
  • Arm B (Experimental): rtk v0.43.0, installed exactly as shipped by rtk init -g, including the binary, the PreToolUse hook, and the RTK.md documentation. This ensured that the test measured rtk‘s "as-shipped ceiling" without any user-induced configuration variations.
• Volume and Cost: The study involved four paired runs: a 10-task "smoke test," the same 10 tasks at k=3 (multiple attempts), and full 86-task runs at both low and high reasoning efforts. This amounted to 425 billed trials, incurring an estimated cost of approximately $320 (Harbor-recorded $317 plus reconstructed subagent spend), demonstrating a significant investment in empirical validation.

Crucially, every trial in the rtk arm persisted rtk‘s own audit log and analytics database, providing verifiable proof that the treatment (compression) was indeed applied and allowing for a direct comparison with rtk‘s self-reported metrics.

Finding 1: The Limited Addressable Surface Area

Before incurring the full cost of the benchmark, JetBrains’ researchers performed an initial analysis by replaying 83 existing baseline transcripts. This revealed a fundamental structural limitation of rtk‘s potential impact:

• Bypassed Tools: Claude Code, like many sophisticated agents, uses its built-in Read and Grep tools for file operations. These tools directly access the file system and completely bypass the Bash shell and, consequently, rtk‘s Bash hook. rtk‘s own documentation acknowledges this limitation in a footnote.
• Uncovered Commands: A significant portion of shell commands executed by agents consists of python3 ... calls and other commands that rtk does not currently cover. Additionally, a substantial fraction involves complex shell constructs like pipes-to-files, heredocs, and command substitutions, which rtk deliberately avoids rewriting to prevent potential correctness issues.

The cumulative effect of these limitations meant that only approximately 33% of Bash calls in the baseline transcripts were even eligible for rtk‘s intervention. More critically, these eligible calls carried just under 20% of the total tool-result characters. Furthermore, tool results themselves represent only a fraction of the total "input tokens" billed in a session, as the same contextual information (e.g., source code, previous interactions) is often re-read and billed on every turn. Factoring in these layers, even if rtk achieved its advertised 70% compression on its "whole share," the maximum potential token savings for a real agent session was projected to be less than 3% of total input tokens. This pre-benchmark analysis proved remarkably prescient in predicting the study’s outcome.

Finding 2: A Cost Increase, Not a Saving

The full benchmark results confirmed the initial structural analysis. Across 80 clean paired trials, the rtk arm exhibited a median +7.6% increase in cost per task compared to the baseline (with a statistical significance of p=0.004). This increase was accompanied by a median +13.8% more turns (p=0.03) and +14.3% more cache reads (p=0.008). In stark contrast, the "new input" token class – the only category rtk directly compresses – showed a negligible +3.2% change (p=0.23), effectively a flat null result where all the purported benefit was expected to reside.

Further analysis revealed a correlation between rtk‘s intervention and the cost penalty: tasks with heavy rtk exposure (i.e., where the hook rewrote many commands) were approximately 24% more expensive than baseline, compared to a 5% increase for tasks where rtk was barely touched. While harder tasks might naturally use more Bash, controlling for task difficulty did not explain this pattern. Transcript forensics, a detailed examination of agent interactions, uncovered a "thin, systematic tax" rather than a single catastrophic failure. This included one genuinely broken rewrite (a complex find command that generated usage errors and forced retries), a few instances of the agent re-reading files due to overly compressed or ambiguous output, and a general increase in "ordinary variance" as the model explored different solution paths. The core issue was not a dramatic breakdown but a subtle cumulative effect of increased interaction complexity.

Finding 3: High Effort, Neutral Impact

Anticipating the critique that the initial findings might be specific to "low reasoning effort," JetBrains conducted a full re-run of all 86 tasks at "high reasoning effort." This represented the most expensive single run of the entire series. Interestingly, at high effort, the observed cost penalty did not replicate. The median paired delta for cost was +0.1% (p=0.99), with turns showing +0.0 (p=0.74), and task quality remaining tied.

This suggests that at higher reasoning efforts, the claude-sonnet-5 model might be more robust to the nuances of compressed output, potentially requiring fewer exploratory turns or being more adept at inferring necessary information despite summarization. While a single run at high effort cannot definitively prove a difference between effort regimes, it strongly indicates that rtk did not provide any cost savings even under optimal conditions for agent performance.

Finding 4: Quality Preservation

A critical concern with any compression or summarization tool is the potential degradation of output quality or task correctness. The study found that rtk largely preserved task quality. A forensic review of transcripts, including those with the largest turn deltas, identified only one recurring broken rewrite (the compound find failure) and one instance of the agent deliberately bypassing the hook across approximately 150 Bash calls. Crucially, there were no widespread reports of agents recovering by reading raw files or misinterpreting compressed pipe output. The additional turns observed were primarily due to the model choosing alternative solution paths, not from rtk-induced confusion or errors.

Across the full runs, task scores (with partial credit) landed at 5 better / 4 worse / 71 tied at low effort, and 5 / 4 / 62 tied at high effort. A sign test yielded p=1.0 for both, indicating that the two arms were statistically indistinguishable on task quality.

It is important to note one specific compatibility issue: on the dialogue-parser task, rtk‘s binary failed to start within the task’s Docker image due to a glibc dependency. This caused the rtk trial to fail at setup in both full runs, while the plain arm successfully scored 0.667. While paired analysis typically excludes such tasks, even if this errored rtk trial were scored as zero, the overall arms would still remain tied on quality (sign test p=1.0), underscoring that this was a compatibility failure, not a compression-induced quality issue.

Finding 5: The Disconnect Between Self-Reported and Actual Savings

Perhaps the most illuminating finding concerned the significant discrepancy between rtk‘s internal scoreboard and the actual observed billable token usage. Across the low-effort full run, rtk‘s built-in analytics (rtk gain) reported a staggering 96.2 million tokens saved, claiming to compress 99.8% of everything it touched. Yet, the measured bill for these same trials went up. This stark divergence can be attributed to three primary mechanisms by which rtk‘s internal scoreboard inflates its reported savings:

1. Counterfactual Mismatch: rtk counts the full raw output of a command as its counterfactual "saved" amount. However, LLMs like Claude Code have internal truncation limits for tool results. For example, a cat command on a 1.2 MB CSV file might log 320,000 "saved" tokens by rtk, but the LLM would have only received a few thousand tokens of that output anyway due to its truncation. The benchmark observed 190 such large reads, averaging approximately 506,000 "saved" tokens each by rtk, which were never actually billable to the LLM in their full raw form.
2. Inaccurate Token Estimation: rtk estimates tokens simply as characters divided by four at the moment of execution. This is a heuristic that does not reflect the actual tokenization process of LLMs, which varies by model and can lead to significant discrepancies.
3. Ignoring Cached Re-reads: The majority of an agent session’s input cost often comes from cached re-reads of the prompt, context, and previous turns. These cached tokens are typically billed at a significantly lower rate (e.g., one-tenth the price) than new input tokens. rtk‘s hook, operating on fresh command output, simply never "sees" or accounts for this dominant cost component, nor can it influence its billing.

In essence, rtk‘s scoreboard was "grading its own homework" based on an idealized, raw counterfactual that did not align with how LLMs actually process and are billed for information.

Verdict: Honest Engineering, Misleading Counterfactual

The JetBrains AI team’s verdict on rtk is one of "honest engineering, wrong counterfactual." The tool itself is a well-crafted piece of software. Its filters are functional and often elegant in their summarization; it does not compromise task quality; and its hook mechanism operates precisely as designed. The problem lies not in rtk‘s technical execution, but in the fundamental assumptions underpinning its advertised savings.

On real agentic coding tasks with Claude Code, the promised 60–90% token savings simply had no viable pathway to materialize. The rtk hook only ever addresses a fraction (approximately one-fifth) of the total tool output, much of which is already truncated by the LLM itself. Furthermore, the dominant cost driver – cached re-reads of context – is entirely outside rtk‘s purview and billed at a much lower rate. What remains is a measured median +7.6% cost increase at low effort and a flat zero at high effort, representing a thin, systematic tax rather than a saving. This tax stems from minor inefficiencies like an occasional broken rewrite or the agent taking an extra turn to explore.

Broader Implications for AI Development

This study carries significant implications for the broader AI development community:

1. The Criticality of Independent Benchmarking: The stark contrast between advertised claims and empirical results underscores the absolute necessity of independent, rigorous benchmarking for any AI optimization tool. Developers cannot rely solely on a tool’s self-reported metrics, which can be based on idealized or incomplete counterfactuals.
2. Understanding LLM Billing Models: AI developers must gain a deep understanding of how LLMs are actually billed. Simple token counts are insufficient; nuances like truncation limits, different rates for input vs. output, and the cost of cached context are paramount. A tool that compresses raw output effectively might have minimal impact on the final bill if that raw output would have been truncated anyway or if the primary cost comes from other token categories.
3. The "Counterfactual Problem": This research highlights a common pitfall in evaluating optimization tools – the "counterfactual problem." The definition of "saved" tokens or resources can vary wildly between a tool’s internal logic and the real-world billing mechanisms. Developers should always measure the actual bill or the actual performance metric that matters, rather than a tool’s internal "diff."
4. Architectural Limitations: The study also points to inherent architectural limitations in how current AI agents interact with their environment and LLMs. As long as agents rely heavily on built-in file reading tools or if a significant portion of their context is repeatedly sent to the LLM, optimizations focused solely on shell command output will have a limited overall impact.
5. Future of AI Agent Optimization: The search for true efficiency in AI agents continues. Future innovations might need to focus on more holistic context management, smarter agent architectures that reduce redundant communication, or more tightly integrated tool systems that provide truly token-efficient interfaces to external environments.

JetBrains concludes with a firm piece of advice: "a tool’s self-reported savings are a claim about its counterfactual, not about your bill." For any context-compression or efficiency tool in the AI domain, the only reliable evaluation is to measure the paired bill directly, not the tool’s internal diff. The company remains committed to its rigorous testing methodology and invites suggestions for the next AI optimization tool to put on its evaluation ladder.