Adaptive Parsing: The LLM as the Last Line of Defense in Enterprise RAG Systems

In the evolving landscape of enterprise document intelligence, ensuring the accuracy and reliability of information extracted from diverse document formats is paramount for Retrieval Augmented Generation (RAG) systems. While initial parsing stages often appear robust, critical failures can remain undetected until the final output, leading to confident yet incorrect answers. This challenge has spurred the development of adaptive parsing strategies, positioning large language models (LLMs) as a crucial, albeit expensive, final line of defense against subtle parsing errors.

The Challenge of Document Intelligence for RAG

The core promise of RAG systems is to provide accurate, grounded answers by retrieving relevant information from a knowledge base and using an LLM to synthesize it. However, this promise hinges entirely on the quality of the initial document parsing. Traditional Optical Character Recognition (OCR) tools, while effective at recovering text, frequently falter when confronted with complex document structures. A classic example is EasyOCR, which excels at extracting words but often discards critical structural information, such as tables. When a document’s table structure is lost, the extracted text might appear plausible at first glance, but it becomes "plausible-looking nonsense"—data that reads fine until cross-referenced with the original source. An LLM, fed such structurally compromised input, can confidently generate incorrect answers, with no immediate flag from upstream processes. This issue underscores the need for a sophisticated, multi-layered approach to document parsing that can adapt to varying document complexities and proactively identify and rectify parsing deficiencies.

This article is the second part of a two-part series on adaptive parsing, itself a component of Part III of "Enterprise Document Intelligence." This comprehensive series aims to construct a robust enterprise RAG system using four fundamental building blocks: document parsing, question parsing, retrieval, and generation. The preceding article, "Loop engineering with adaptive PDF parsing: start cheap, pay for a heavier parser only when the page needs it," laid the groundwork by establishing an escalation cascade and deterministic checks designed to flag parsing failures efficiently. This current installment delves into the integration of LLMs as the ultimate safeguard, detailing their role through two real-world escalation scenarios.

Adaptive Parsing: A Cost-Effective Escalation Strategy

The dilemma in document parsing lies in balancing cost and accuracy. Basic parsing tools like PyMuPDF can process a page in mere milliseconds, incurring virtually no cost. In contrast, a sophisticated vision LLM applied to the same page might cost tens of thousands of times more and take up to ten seconds. For questions whose answers reside in plain prose, the cheaper parser is perfectly adequate. However, when the crucial information is embedded within a table, a figure, a scanned region, or a flattened layout, the inexpensive parser will silently fail to provide useful content, leading the LLM to confidently generate erroneous responses.

Running the most powerful parser on every single page is economically unsustainable and wasteful. Conversely, relying solely on the cheapest parser is inherently unreliable. The strategic solution lies in adaptive parsing: commencing with the most economical parsing method and escalating to more resource-intensive alternatives only when the system detects that the initial parse is insufficient to answer the query. This detection signal must originate from within the RAG pipeline itself, driven by a chain of checks performed against the parse output, the question, and the LLM’s attempt to utilize the extracted information.

This approach creates a self-correcting feedback loop. The pipeline begins with a cheap parser, evaluates its output at multiple checkpoints, and escalates to a deeper parser only if a check flags the initial parse as inadequate for the specific question. This evaluation is not a single point of failure but a methodical cascade of checks, each progressively more reliable and, crucially, more expensive than the last. Each check addresses a fundamental question: "Did the parser produce enough information to answer the query?"

The initial part of this series focused on catching failures cheaply, before the generation phase. This article explores scenarios where these early checks are insufficient, and the LLM’s own performance during generation—specifically, its self-evaluation (check 7) and groundedness assessment (check 8)—flags a parse that deterministic checks missed. When this occurs, the pipeline escalates the problematic page to a more advanced parser. Two illustrative examples, frequently encountered in enterprise corpora, are examined: Table 3 from the seminal Attention paper (Vaswani et al., 2017), which typically presents as a flat-parsed grid requiring escalation to Azure Layout, and Figure 1 from the same paper, an opaque diagram necessitating a vision LLM for interpretation.

The Generation-Side Checks: LLM Self-Evaluation and Groundedness

Generation marks the final stage in the RAG cascade. It incorporates two critical checks: Check 7, the LLM’s inherent self-evaluation flag embedded within its typed answer, and Check 8, a post-generation groundedness verification performed by a separate LLM or a Natural Language Inference (NLI) model. Check 7 is both the most expensive and least stable of the checks, demanding careful consideration. This article details two successful instances of Check 7 in action (Case A and Case B) and then presents a stress test demonstrating the limitations of relying solely on the LLM signal. Check 8 is discussed as a vital complementary mechanism.

It is important to note that most questions do not require any escalation. For instance, querying the Attention paper for "What are the options for positional encoding?" or the NIST document for "What are the six CSF Functions?" often yields a clean answer from the first generation pass, with context_structured=True, using only PyMuPDF because the answers reside in plain prose. The walkthroughs below focus on the more challenging cases where PyMuPDF’s capabilities are insufficient.

Case A: Flat-Table Escalation, End to End

Consider the question: "What is the value of h for the base model in Table of Variations on the Transformer architecture?" The correct answer, ‘8’, is clearly visible in the ‘h’ column of the ‘base’ row in Table 3 on page 9 of the Attention paper. While the cell is unambiguous, the challenge lies in the table’s flat-parsed structure, which a cheap parser might misinterpret.

Initial Question Processing: The question is direct, bypassing the elaborate parse_question brick (detailed in Article 6) used for typos, multi-part queries, or language detection. For retrieval, keywords like "Table 3: Variations on the Transformer architecture" are sufficient to target page 9.

Parsing with PyMuPDF: The standard pipeline begins with PyMuPDF. This yields a line_df (dataframe of lines) where each line includes its text and bounding box coordinates. For page 9, PyMuPDF captures the table’s cells as distinct lines, losing the tabular context.

Retrieval: Keyword retrieval successfully identifies page 9, as the table caption is unique to that page. The filtered_line_df then contains all lines from page 9, ready for generation.

First Generation Pass (GPT-4.1 on PyMuPDF): The llm_answer_with_evidence brick (Article 8) feeds the PyMuPDF-parsed lines and the question to GPT-4.1, expecting an AnswerWithEvidence schema. This schema includes context_structured, a binary flag crucial for the cascade.

In a real run, the LLM correctly identified the answer as "8" with high confidence (0.99), citing line 25 of page 9. Line 25 indeed corresponds to the ‘h’ cell of the base row. However, the crucial detail was context_structured=False. The LLM, despite providing the correct answer, flagged a structural risk. PyMuPDF had returned the 13 cells of the base row as 13 separate lines, and column headers were similarly fragmented. To connect line 25 to ‘h’, the LLM had to infer column positions from a flattened header, a process it recognized as fragile. This False flag immediately triggers an escalation.

Pipeline Escalation: Re-parsing with Azure Document Intelligence: Upon detecting context_structured=False, the orchestrator activates a structure-aware parser. Azure Document Intelligence (DI) is used in this project, though alternatives like Camelot or Docling would integrate similarly. Azure DI processes only page 9, yielding a new line_df where each table row is represented as a single markdown line, pipe-anchored, accurately preserving the table structure.

Second Generation Pass (Same Model, Trusted Structure): The same LLM call is made, but this time with the Azure-parsed lines. The response is again "8," but crucially, context_structured=True. The LLM now trusts the structural integrity of the input. The citation points to line 7 (the entire table row), and the cell-to-column mapping is unambiguous due to the markdown structure. This feedback loop successfully fired and resolved the parsing ambiguity. The cost incurred was minimal (around $0.003 and four seconds for the Azure DI call), but the gain was significant: a structurally trusted answer and a permanently structured record for page 9 for future queries.

Auditability: Annotation and Cross-Model Comparison: The system allows for auditability by annotating the cited cell on the original PDF. Using the PyMuPDF citation (L25) provides a precise bounding box for the ‘h’ cell, visually confirming the LLM’s claim.

A sidebar analysis comparing three models (GPT-4o-mini, GPT-4o, GPT-4.1) with both parsers (PyMuPDF, Azure DI) on this question revealed that while all models consistently returned the correct answer ‘8’, only GPT-4.1 on PyMuPDF conservatively flagged context_structured=False. The key takeaway is that upgrading the parser offers a durable fix by eliminating structural risk entirely, whereas upgrading the model merely enhances the detection of parsing risk. Production pipelines often leverage both to hedge against weaknesses.

Case B: Figure Escalation, End to End

The question: "What is the architecture of the Transformer?" The answer is primarily conveyed by Figure 1 on page 3 of the Attention paper, depicting the encoder-decoder architecture. The surrounding prose offers some context but cannot replace the diagram’s detail.

First Pass (PyMuPDF Only): Retrieval identifies page 3. PyMuPDF extracts the prose but represents Figure 1 as a type='image' row with a placeholder text, lacking actual content. The LLM, reading this, produces a partial answer with a critical caveat: "Figure 1 is referenced in the prose but its content is absent from the context." It also sets complete_answer_found=false and context_structured=true (as the prose itself is well-structured).

Pipeline Escalation: Vision LLM: The pipeline interprets the caveat "figure referenced but absent" and identifies image_df.image_id=1 as parsing_method='not_parsed'. It then routes the image to a vision-language model (e.g., GPT-4o with vision capabilities). This model analyzes the image and returns a structured description: detailing the encoder’s input embedding and positional encoding, the six identical layers with multi-head self-attention, residual connections, layer normalization, and position-wise feed-forward networks; a similar decoder structure with masked self-attention and cross-attention to the encoder output; and a final linear and softmax layer. This rich description is appended to line_df as new text rows, marked with parsing_method='vision_gpt4o'.

Second Generation Pass: With both the original prose and the structured visual description, the LLM now generates a complete answer, with high confidence (0.92), detailing aspects like residual connections, layer norm placement, and the cross-attention bridge—details the prose alone could not provide. The total cost for this comprehensive understanding is a few cents and 10-30 seconds for the vision call on one image, with other pages remaining on the cheaper PyMuPDF parse.

Stress Tests on the LLM Signal: Unveiling Instability

While the case studies highlight the utility of the LLM signal, a stress test reveals its limitations. Eighteen runs were conducted, involving three questions of varying structural difficulty on Table 3, three models (GPT-4o-mini, GPT-4o, GPT-4.1), and two parsers (PyMuPDF, Azure DI).

The initial hypothesis was that a weaker model, even if unable to answer correctly, could at least flag a broken parse as a "structural canary." The findings contradicted this: out of four wrong answers across the matrix, all incorrectly asserted context_structured=True. For instance, on a question requiring inference about inherited values (Q_inheritance), models fabricated answers from adjacent columns and confidently declared the structure clean. The single instance of cs=False occurred with GPT-4.1 on PyMuPDF for Q_inheritance, where the answer was actually correct. This suggests that when a model misaligns columns or fabricates, it often fails to detect its own error in structural interpretation.

Further investigation into a continuous context_structured_score (0.0-1.0) yielded similar results. Scores consistently clustered between 0.85 and 1.00, regardless of correctness, rendering them ineffective for distinguishing valid from erroneous parses. This indicates that the LLM’s structural verdict is largely bimodal, not truly continuous.

However, asking GPT-4.1 for a one-sentence rationale alongside the flag proved insightful. The rationale consistently and accurately described parsing issues ("headers split across multiple lines," "each cell on its own line," "blank cells indicate inherited values"). The description of the parse was stable, but the verdict (the cs flag or score) built upon it was unstable. This suggests that a deterministic rule applied to the LLM’s rationale ("if rationale contains ‘split across multiple lines’ or ‘blank cells,’ then escalate") would create a more reliable feedback loop than relying directly on the LLM’s binary judgment.

Check 8: Post-Generation Groundedness

Beyond the LLM’s self-evaluation, Check 8 involves a separate LLM or NLI model verifying that every claim in the generated answer is directly supported by the cited chunks. This acts as a crucial safety net, catching fabrications that the generating model might have confidently missed in Check 7. This check is more reliable than the generating model’s self-evaluation because the independent judge has no incentive to defend the answer. It is also a more cost-effective and reliable method for assessing parsing quality than relying on the LLM-as-judge approach of Check 7.

The overall cascade is structured for efficiency: Checks 1 and 2 (document parsing) fire in milliseconds on every page; Check 4 (question parsing) processes the query; Checks 5 and 6 (retrieval) operate after page selection; Check 7 (LLM self-eval) only fires if upstream checks have cleared the page; and Check 8 (groundedness) acts as a final validation post-generation. This hierarchical design ensures that the LLM’s verdict is not the sole, nor the first, line of defense.

Operational Notes: Efficiency and Caching

When Eager Parsing Wins: While lazy parsing (processing only when needed) is generally the default, eager parsing (processing all documents upfront) can be beneficial for specific scenarios:

• Documents with extremely complex, non-standard layouts that are known to break cheap parsers.
• Documents with high query frequency, where pre-parsing saves cumulative processing time.
• Documents that are static and unlikely to change, making the initial investment a one-time cost.
• Documents that are critical for legal or compliance reasons, where absolute parsing accuracy is non-negotiable from the outset.
  The decision hinges on the average query frequency of a document. For typical enterprise corpora, where many documents are queried infrequently or not at all, lazy parsing offers significant cost savings.

The Data Model as the Cache: A key architectural advantage is that the data model itself functions as the cache. When a page is re-parsed with a deeper method (e.g., Azure DI), the results are stored in line_df and page_df with the parsing_method column updated. Subsequent queries touching that page directly access these enhanced records, eliminating redundant parsing. This allows the system to intelligently invest in deep parsing for frequently queried and structurally complex document regions based on actual usage, rather than speculative ingestion-time decisions.

Conclusion and Implications for Enterprise RAG

Parsing quality is a distributed decision across all four RAG bricks, prioritized by cost-efficiency: starting with deterministic signals during document parsing (e.g., character density, flat-table fingerprint, chunk integrity), followed by intent-aware routing during question parsing, then score-gap and context-gap checks during retrieval, and finally, LLM self-evaluation as the last line of defense. The stress test conclusively demonstrates that the LLM’s binary verdict on its own context is insufficient to reliably lead the cascade; a separate, unbiased judge is essential for catching fabrications.

The robust data model, featuring parsing_method as a column across all parsing tables, simplifies mixed parses, escalation, auditability, and caching into straightforward data filtering. This flexible framework can accommodate new failure shapes and checks, enhancing the system’s adaptability. The insights gleaned from this adaptive parsing strategy are crucial for building enterprise RAG systems that are not only accurate but also cost-effective and scalable, capable of navigating the vast and varied landscape of corporate documentation. This continuous self-correction frame will be further explored in Article 11, focusing on cross-references.

Sources and Further Reading

The capabilities and cost implications of advanced parsing are detailed in Auer et al.’s Docling Technical Report (2024). Table extraction as a distinct problem from text extraction is thoroughly grounded by Smock et al.’s PubTables-1M / Table Transformer (CVPR 2022). The cost tier for vision-LLM-per-page (approximately 5-30 seconds on a GPU) is derived from Blecher et al.’s Nougat (Meta 2023). The pattern of using an LLM as a feedback signal to drive escalation aligns with research like Asai et al.’s Self-RAG (ICLR 2024). This article specifically contributes a concrete negative result from an 18-run stress test, highlighting that LLM self-evaluation alone is an unreliable binary indicator of parser quality, and that a separate-LLM groundedness check on the generated answer is necessary to catch what the in-loop signal misses.

Earlier in the series: "Enterprise Document Intelligence"

• What works, what breaks
  • RAG needs a doc-intel architecture, not a single LLM call
  • RAG needs a data model
  • Why cheap parsers are not enough (and expensive parsers are too much)
  • Enterprise RAG needs a feedback loop
• Document parsing
  • The three families of parsing technologies
  • Loop engineering with adaptive PDF parsing: start cheap, pay for a heavier parser only when the page needs it
• Question parsing
  • The question parser: intent, keywords, entities, and language
  • Question parsing: a grammar for the enterprise
• Retrieval
  • Retrieval: the four families of methods
  • Retrieval with a semantic index
• Generation
  • Generation: grounding, hallucination, and the data model
  • Generation: a Pydantic schema for enterprise answers
• One-document pipelines
  • One-document pipelines: from scratch
  • One-document pipelines: the four bricks