RAG Accuracy for Technical Documents: Chunking, Search, and Evaluation

Why do generic AI search tools struggle with technical documents?

Generic indexers split documents in ways that do not follow their structure, so a search returns fragments rather than complete answers. We often see teams start with Microsoft Copilot over SharePoint, find the answers vague or wrong, and assume the model is at fault. In our experience the cause is usually the chunking.

Technical documents are dense and carefully structured. A finding in an inspection report depends on the equipment, the conditions, and the method described in the sections around it. When a chunk boundary falls in the middle of that reasoning, neither half carries the full meaning. The model receives part of the evidence and produces a confident answer from it.

General-purpose tools also give you little control over the process. You cannot change how a document is parsed, which metadata is kept with each chunk, or how results are ranked. For everyday search that is acceptable. When engineers need precise answers, those controls are where accuracy comes from.

How does a RAG pipeline for technical documents work?

A RAG pipeline has two halves. Ingestion turns documents into a searchable index, and query time retrieves the most relevant pieces and passes them to a model. We find most of the accuracy is decided during ingestion.

The stages are:

1. Parse each document so its layout, headings, and tables are recognised rather than flattened into plain text.
2. Chunk it into coherent sections.
3. Enrich each chunk with its source document, section heading, and business metadata.
4. Embed each chunk by converting it into a vector that represents its meaning.
5. Index the chunks in Azure AI Search, keeping both the text and the vectors.
6. Evaluate retrieval against a set of questions agreed with your experts, and repeat whenever the pipeline changes.

What is structure-aware chunking?

Structure-aware chunking splits a document along its own sections, so each chunk is a coherent piece of text that could answer a question on its own. The aim is for every chunk to carry a complete thought, with enough context to be understood without the text around it.

In practice a long technical document might become 30 or 40 chunks, each matching a section or subsection. Each chunk still has to fit the embedding model’s input limit, so long sections are split again within their own boundaries. Each chunk keeps a reference to its document and heading, so answers can cite exactly where they came from.

How do you implement it on Azure?

Azure AI Search supports structure-aware chunking through its Document Layout skill. The skill calls the layout model in Azure Document Intelligence, which returns the document’s headings and content as Markdown sections. A Text Split skill then constrains chunk size within each section. For new pipelines, Microsoft now recommends its Content Understanding skill, which adds semantic chunking and improved table extraction in a single skill.

We have used Azure Document Intelligence in this way, and we have also written our own parsing and chunking code where a document set needed it. Custom code is worth the effort when your documents follow a house template, because the template tells you exactly where each section starts and what it contains.

When is fixed-size chunking good enough?

Fixed-size chunking, which splits text every few thousand characters with an overlap, works well for mixed content with little consistent structure. Our AI Search Accelerator uses it to index file shares and databases where document formats vary widely.

Technical documents are different because their structure carries meaning. When the same section headings appear across hundreds of reports, chunking by section makes it possible to retrieve every ‘root cause’ section or every ‘recommendations’ section across the whole library. Fixed-size chunks cannot offer that.

Why does hybrid search matter for technical content?

Technical documents are full of exact terms that vector search alone can miss, such as part numbers, equipment tags, standards references, and error codes. Vector search finds passages with similar meaning, while keyword search finds exact matches. Azure AI Search can run both in a single hybrid query and merge the results.

We find this combination suits engineering content well. A question about a particular component should return passages that name it exactly, as well as passages that describe the same failure in different words. Azure AI Search’s optional semantic ranker can then reorder the merged results by relevance.

Metadata filters add a further layer of precision. Filtering by document type, site, customer, or date narrows the search before ranking begins. This is how an assistant can check a draft only against documents previously sent to the same customer.

How do you measure RAG accuracy?

Measure it with an evaluation set agreed with your subject matter experts, and run it throughout the project rather than once at the end. Without a shared measure, accuracy becomes a matter of opinion, and changes to chunking or ranking cannot be judged.

What goes into the evaluation set

The set is a list of realistic questions, each paired with the sections a correct answer should draw on and, where useful, the expected answer. Your experts supply the questions and the right sources, because they know the documents. The delivery team turns them into repeatable tests.

Measure retrieval and answers separately

We test retrieval separately from the generated answer. If the right sections are not coming back from the search, no prompt or model change will fix the answer. Retrieval tests show whether chunking and ranking are working. Answer tests show whether the model uses the evidence well.

Running the set on every change to chunking, embedding, or ranking shows whether accuracy improved or regressed. Evaluation is one of the techniques covered in our guide to AI test quality.

Which model do you need for accurate answers?

Usually a mid-tier model. When retrieval returns the right sections, the model’s job is to compare, summarise, and explain evidence it has been given, which does not need the most capable model available. Investment in data preparation pays back more than a more expensive model.

We typically run one capable model for harder reasoning and a cheaper one for simpler steps, with fallback between deployments. Our guide to AI usage governance on Azure explains how we route between models and track usage per user and action.

What can you build on a well-structured index?

Once the index holds well-formed sections with good metadata, the assistants built on top of it are small pieces of software. Most of their value comes from querying the right documents and returning the results with sources.

Examples of assistants this kind of index supports include:

• A document review assistant that compares each section of a draft with how the same section is written in approved documents, and flags content that belongs elsewhere
• A precedent finder that retrieves past cases similar to the one being written up, so the author can check they have covered the same investigations
• A consistency checker that compares a draft with information previously supplied to the same customer and highlights differences
• An analysis assistant that retrieves historical data and past findings to support engineers working through a fault investigation

Why assistants rather than agents?

We recommend assistants that support engineers rather than agents that act on their behalf. The assistant retrieves evidence and makes suggestions, while the engineer remains responsible for the investigation and the document. That keeps accountability clear, and engineers adopt a tool more readily when it supports their judgement than when it replaces it.

Get the index and ingestion right first, then add assistants one at a time. Each new assistant reuses the same index, which is why the foundation deserves the most care.

How does Talk Think Do approach RAG for technical documents?

We start with the documents rather than the model. Early in a project we sample the document set, agree the evaluation questions with your experts, and design the chunking around how your documents are structured.

Our approach typically includes:

• Structure-aware chunking using Azure Document Intelligence, custom parsing, or both
• Azure AI Search with hybrid keyword and vector search, metadata filters, and source citations
• An evaluation set agreed with your experts and run throughout delivery
• Azure OpenAI models chosen for the task, usually mid-tier, with usage tracked per user and action

Azure and .NET have been our core stack since 2014, and we are a Microsoft Solutions Partner. We develop software under an ISO 27001:2022-certified information security management system and hold Cyber Essentials Plus.

For the wider choice between retrieval, fine-tuning, and prompt engineering, read RAG vs fine-tuning vs prompt engineering. If an existing Copilot deployment is returning poor answers, our Copilot Studio rescue service replaces shallow retrieval with an engineered pipeline. Our custom generative AI service covers the full build. To discuss your own document set, book a free consultation.