Cut Text2SQL token costs up to 81%, and stop paying for wrong answers: a Neo4j semantic layer

The rapid evolution of generative artificial intelligence and Large Language Models (LLMs) has revolutionized how enterprises interact with their data repositories. Yet, as organizations increasingly deploy Text-to-SQL agents—AI systems designed to translate natural-language questions directly into database queries—they encounter a persistent structural bottleneck: data context. While modern data platforms provide seamless access to massive data warehouses, AI agents frequently struggle to comprehend complex, sprawling business landscapes. Recent benchmark testing across large, deliberately hostile data catalogs on Google Cloud BigQuery and Databricks indicates that integrating a graph-backed semantic layer utilizing Neo4j and Neocarta can radically transform Text-to-SQL performance. By converting opaque physical schemas into searchable knowledge graphs, organizations have successfully slashed token consumption by up to 81 percent, reduced monthly operational expenses significantly, and dramatically elevated query accuracy across both economical and high-end models.

The Core Problem: Access Is Solved, But Context Is Missing

Historically, the primary challenge of enterprise analytics was mere data access—extracting information from siloed legacy systems and centralizing it into modern cloud warehouses like Snowflake, BigQuery, or Databricks Unity Catalog. Today, that operational barrier has largely vanished. Modern data infrastructure platforms are fully equipped for high-concurrency read and write operations. However, a profound contextual vacuum remains.

As Arun Srinivasan of ThoughtWorks noted in recent industry analyses, enterprise agents can readily reach data stores, but they cannot reliably interpret localized business definitions. For instance, a standard corporate data lakehouse might contain dozens of tables with "revenue" somewhere in their titles. When prompted to calculate quarterly financial performance, an unguided AI agent will arbitrarily select one of these tables. While the agent will invariably produce a formal answer, it cannot guarantee or explain the provenance of its underlying data selection.

When scaled to enterprise environments featuring hundreds of cryptically named tables, thousands of fragmented columns, and evolving database schemas, this ambiguity becomes catastrophic. Traditional Text-to-SQL architectures typically attempt to resolve this by injecting the entire database schema—often formatted as a massive text blob or search index—directly into the model’s prompt. This naive approach triggers three critical failure modes: exorbitant token expenditure, high contextual noise leading to hallucinated table joins, and rapid staleness as flat-file documentation rots.

For large-scale catalogs, these obstacles quickly become insurmountable. The United States Census Bureau dataset hosted on BigQuery, for example, represents approximately 475,000 tokens of schema metadata—a volume far exceeding the context window limits of standard 128K LLMs. Consequently, the agent cannot ingest the schema directly; instead, it must engage in a protracted process of trial-and-error discovery, repeatedly querying metadata and burning through compute budgets before ultimately timing out or failing.

The Mechanics of a Graph-Backed Semantic Layer

To overcome the limitations of flat schema dumps, data architects are increasingly turning to semantic layers. While traditional platforms like dbt Semantic Layer, Snowflake Semantic Views, and Databricks Unity Catalog Metric Views offer robust, metric-first governance, they are predominantly single-platform solutions designed to answer predefined questions over curated, high-level tables. They assume that the foundational challenge of data discovery has already been solved.

To bridge this discovery gap, developers are implementing metadata-only graph databases positioned underneath traditional metric layers. Utilizing Neo4j in conjunction with Neocarta—an open-source semantic modeling tool—organizations can ingest raw warehouse catalogs to construct a comprehensive metadata graph. This graph maps databases, schemas, tables, and columns, while automatically translating foreign keys into explicit join edges, indexing sample values, and embedding textual descriptions. Crucially, only structural metadata crosses into the graph database; the underlying enterprise rows remain securely within the native cloud warehouse.

This architectural shift replaces brute-force schema injection with targeted retrieval. When a user submits a natural-language query, the agent executes a two-step retrieval pattern: semantic similarity search instantly identifies an accurate "anchor" table, and a subsequent graph traversal fetches the exact related tables and foreign key join paths. Rather than guessing structural relationships, the LLM reads the verified join edges directly from the graph.

Empirical Evidence: BigQuery Census and Databricks Case Studies

To rigorously evaluate the efficacy of this architecture, recent benchmarking subjected two massive, messy catalogs to standardized Text-to-SQL workflows. The first study examined the public Google Cloud BigQuery Census dataset (bigquery-public-data.census_bureau_acs), which comprises 278 near-identical tables containing roughly 68,000 total columns. Tables within this catalog differ only by geographical granularity, temporal span, and survey type, featuring highly obscure column nomenclatures.

When tested using economical models such as gpt-4o-mini without a semantic layer, the agent repeatedly failed, entering infinite loops of metadata scanning and ultimately timing out on 50 percent of verified questions while burning over 66,000 tokens per failed run. When integrated with the Neocarta semantic layer, the exact same model successfully answered 100 percent of the benchmark queries correctly. Furthermore, utilizing compact-retrieval configurations—which extract solely the specific columns required for a given query—demonstrated token reductions ranging from 45 percent to 81 percent across five distinct LLMs, with every model achieving a perfect accuracy score.

The second benchmark utilized a synthetic legacy-enterprise lakehouse hosted on Databricks Unity Catalog, engineered deliberately to challenge blind exploration. Containing 264 opaque tables, 5,614 columns, and 12 distinct schemas, the repository masked true business concepts behind legacy alphanumeric identifiers (such as dm_fin_20.f_2001 for Annual Recurring Revenue). True business definitions resided exclusively in table and column comments, obscured by redundant year-and-region sharding and intentional distractor tables.

Evaluating eight distinct Anthropic models against this hostile catalog yielded definitive results. With the Neo4j semantic layer enabled, every tested model achieved an 11 out of 11 accuracy score. More importantly, the architectural integration generated substantial cost and efficiency savings. Across high-performance configurations like Opus 4.5, token usage dropped by up to 54 percent. For an enterprise processing 10,000 queries per month, this optimization translates to a projected cost reduction from approximately $1,527 down to $693 monthly—saving nearly $834 every month while guaranteeing absolute analytical correctness.

Broader Industry Implications and Future Outlook

The implications of graph-backed semantic layers extend far beyond immediate cost savings and error reduction. As enterprises accelerate their transition toward autonomous agentic workflows, the demand for deterministic, governed data navigation will only intensify.

By treating schema metadata as a dynamic, interconnected knowledge graph, organizations establish a foundational substrate that supports not just Text-to-SQL conversion, but a comprehensive enterprise knowledge layer. This architecture naturally accommodates business glossaries, access-control sensitivities, usage-inferred join paths derived from historical query logs, and persistent agent memory.

Industry analysts emphasize that this methodology represents a fundamental paradigm shift in how artificial intelligence interfaces with corporate data estates. Rather than attempting to expand LLM context windows indefinitely or forcing models to navigate raw, unstructured database schemas, organizations can leverage GraphRAG principles to provide precise, context-aware navigational maps.

Ultimately, the empirical data confirms that modern language models do not require infinite context windows; they require structured, reliable cartography. By deploying lightweight semantic layers over multi-cloud data estates, data engineering teams can eliminate runaway token expenditures, eradicate hallucinated query paths, and deliver scalable, enterprise-grade generative AI analytics capable of withstanding the complexities of real-world data environments.