Why your RAG chatbot answers confidently and wrongly

Almost every broken RAG system I have been asked to look at had the same diagnosis. The model was fine. The retriever was handing it the wrong context, and the model did what models do: it answered anyway.

This matters because the failure is invisible from the outside. A hallucination from a model with no context looks like nonsense. A hallucination from a model given three irrelevant paragraphs looks like a confident, well-structured, completely wrong answer — and it will cite the irrelevant paragraphs as support.

Retrieval is the bottleneck, not generation

When I build these now, the first thing I instrument is retrieval quality on its own, before the model is in the loop at all. Take fifty real questions, write down which documents should come back for each, and measure how often they do. That number is your ceiling. No amount of prompt engineering gets you above it.

On the proctor office assistant I built for a university, that first measurement was humbling: the right document appeared in the top five results for about 60% of questions. The chatbot was being blamed for a search problem.

flowchart LR
    Q[User question] --> E[Embed query]
    E --> V[(Vector store)]
    V --> R[Top-k chunks]
    R --> RR[Rerank]
    RR --> P[Prompt with context]
    P --> L[LLM]
    L --> A[Answer + citations]
    R -.->|measure here| M{{Retrieval eval}}
    A -.->|measure here| M2{{Answer eval}}

Four things that actually helped

1. Stop chunking by character count

Fixed 500-character chunks cut sentences in half and separate a heading from the rule it introduces. A chunk that begins mid-sentence embeds to a vector that means very little, so it is never retrieved, and the information in it is effectively deleted from your system.

Split on document structure first — headings, sections, list items — and only fall back to length limits inside an oversized section. Then prepend the heading trail to each chunk before embedding, so a chunk reading "Submit within 7 working days" carries the context that it belongs under "Complaint procedure".

def chunk_with_context(sections, max_chars=1200):
    """Split on structure, then carry the heading trail into each chunk.

    The trail is what lets a chunk like "Submit within 7 working days"
    be retrieved for "how long do I have to file a complaint".
    """
    for section in sections:
        trail = " > ".join(section.headings)
        body = section.text

        if len(body) <= max_chars:
            yield f"{trail}\n\n{body}"
            continue

        # Oversized section: split on paragraphs, never mid-sentence.
        buffer = []
        size = 0
        for para in body.split("\n\n"):
            if size + len(para) > max_chars and buffer:
                yield f"{trail}\n\n" + "\n\n".join(buffer)
                buffer, size = [], 0
            buffer.append(para)
            size += len(para)

        if buffer:
            yield f"{trail}\n\n" + "\n\n".join(buffer)

2. Hybrid search, because embeddings are bad at names

Dense vectors are excellent at meaning and poor at exact tokens. Ask for form SR-114 and a pure vector search returns things that are semantically about forms. Keyword search returns SR-114. You need both, and combining their rankings is a handful of lines.

3. Rerank the shortlist

Retrieve 30 candidates cheaply, then have a cross-encoder score each one against the question properly and keep the best 5. It is slower per query than similarity alone because the model sees the question and the chunk together rather than comparing pre-computed vectors — and that is exactly why it is more accurate.

4. Let the model decline

A model told to "answer using the context" will always answer. Told that saying "I could not find this in the documents" is a correct and expected response, it starts declining — and a system that declines 8% of the time is far more useful than one that invents an answer 8% of the time.

Build the evaluation set first

Fifty questions with known-correct sources, written before you tune anything, is worth more than any framework choice. Without it, every change is a guess and you will confidently ship regressions. With it, the work becomes ordinary engineering: measure, change one thing, measure again.

1. Collect real questions. Invented ones are too clean and too polite.
2. Record the document each one should be answered from.
3. Measure retrieval alone: is the right document in the top k?
4. Only then measure answer quality, and only against retrievals you know were correct.
5. Re-run the whole set on every change to chunking, embeddings or the prompt.

None of this is exotic. It is retrieval hygiene, an honest evaluation set, and permission for the model to say it does not know. That combination is the difference between a demo and something people trust enough to keep using.