The RAG Stack Behind the Community Chatbot

I shared the backstory of building this chatbot in Part 1. This is the engineering companion, a closer look at how I built it: the pipeline end to end, and the decisions that shaped it, chunking, embeddings, the vector index, reranking, and the confidence thresholds. The code is in the repo if you want to follow along or fork it.

The stack

The whole system runs on five services, and at the volume this community generates, four of them sit inside free tiers.

A few of those choices are decisions anyone rebuilding this would face. The vector store retrieves the top five chunks by cosine similarity through a Postgres RPC (rag/retriever.py), which keeps the database doing the vector math instead of pulling everything into the app. The embedder (rag/embedder.py) tags text with input_type="document" at ingest and input_type="query" at retrieval, which lets Voyage embed the two asymmetrically; a stored message and a user's question are different kinds of text, and telling the model so improves the match. And generation runs Claude Haiku 4.5 at temperature=0, which makes answers deterministic: the same question gets the same answer every time, which matters more than you would think once you start evaluating.

The pipeline, from export to grounded answer

There are two paths through the system, and they share an embedding model but almost nothing else. The ingest path runs once every few months, offline, and turns raw chat history into a searchable index. The query path runs in real time, every time someone messages the bot. Drawing them separately is the clearest way to see where the work lives.

For scale: the corpus is 115,014 WhatsApp messages across seven community groups spanning 2020 to 2026, which the ingest path turns into 7,669 embedded chunks. That came to roughly 5.8 million tokens embedded one time, comfortably inside Voyage's free tier. The rest of this piece is about the decisions inside those boxes.

Chunking text well takes skill

Before anything gets embedded, the raw chat history has to be cut into pieces. This sounds like plumbing, and it is the single decision that most quietly determines whether retrieval works, because you can only ever retrieve a whole chunk. If the right answer is split across a chunk boundary, no embedding model in the world will save you.

The repo cuts WhatsApp messages with a sliding window of 20 messages and 5 messages of overlap (ingestion/chunker.py). The framework behind that choice is a straight tradeoff: smaller chunks raise precision because each one is about one thing, while larger chunks preserve context so an answer is not severed from the question that prompted it. Overlap is the hedge against boundary-splitting, at the cost of storing some text twice. Most production systems land around 200 to 500 tokens per chunk with 10 to 20 percent overlap (Stackviv, Weaviate).

The strategies fall into three families, each with different costs:

1. Fixed-size chunking cuts every N tokens regardless of meaning. It is the fallback: cheap, simple, and blind to where ideas begin and end.
2. Recursive chunking splits on structure first (paragraphs, then sentences) and is the sensible default for most documents (Redis).
3. Semantic chunking groups text by meaning using embeddings, buying maybe two to three points of recall over recursive, but the cost scales with the corpus, so reserve it for a measured retrieval problem rather than a default (F22 Labs).

Chat logs are a hostile chunking target. A clean document has paragraphs and headings that tell you where to cut. A group chat has topic drift inside a single window, names dropped mid-conversation, and a recommendation that arrives three messages after the question that prompted it. A fixed 20-message window will sometimes wrap a restaurant recommendation together with an unrelated argument about parking, and the embedding for that chunk is now a blurry average of both. The repo's own notes flag exactly this: service-provider recommendations embed poorly, and a wider 30-to-40-message window might capture them better (TIERED_CONFIDENCE_SUMMARY.md).

The embedding model is a quality lever you can pull cheaply

An embedding model turns text into a vector, and retrieval is just finding the stored vectors closest to the query's vector. The quality of that model sets a ceiling on everything downstream: if two genuinely related messages land far apart in vector space, no amount of clever prompting recovers the match.

I used voyage-3-lite, the low-cost, low-latency tier at 512 dimensions. The reason to know the tiers is that moving up one is one of the cheapest quality wins available. Voyage reports voyage-3-lite beating OpenAI's text-embedding-3-large by about 3.82 percent on average, while the larger voyage-3 at 1024 dimensions beats it by about 7.55 percent (Voyage). Embedding quality bounds retrieval relevance, and retrieval relevance bounds the hallucination rate, so a better embedder is a lever on the whole system, not just the search step (arXiv 2512.15068).

One practical note if you are comparing models yourself: compare on the MTEB Retrieval sub-leaderboard using nDCG@10, not the headline average. The overall MTEB score blends in classification and clustering tasks that have nothing to do with how well a model retrieves, so a model can look great on the average and middling at the only job you care about here (Ailog).

The index type you never picked still has a tradeoff

Once vectors are stored, something has to find the nearest ones fast. pgvector gives you three ways to do that, and most people accept the default without realizing they made a choice. The three options are flat (an exact, brute-force scan of every vector), IVFFlat (which partitions vectors into lists and only searches the nearest few), and HNSW (a navigable graph that hops toward the closest neighbors).

HNSW is the safer default under roughly 10 million vectors with active writes: it delivers large gains in throughput and tail latency over IVFFlat at equal recall, at the cost of two to five times the memory. Both let you trade recall for speed, but through different knobs: IVFFlat via nprobe (how many lists to scan, roughly linear), HNSW via ef_search (how wide to search the graph). On one benchmark, pushing HNSW recall from about 0.8 to 0.95 cost roughly 31 percent more latency (BigData Boutique, pgvector study).

Reranking, and why I skipped it

Sometimes top-5 retrieval returns chunks that are roughly right but not the best available, the relevant one sitting at rank 7 instead of rank 3. The standard fix is two-stage retrieval, and it is the single highest-leverage upgrade I left on the table.

The framework: the first stage is a fast bi-encoder (the embedding search I already have) that casts a wide net over 50 to 100 candidates, optimizing for recall. The second stage is a cross-encoder reranker that reads the query and each candidate together and scores their relevance directly, optimizing for precision. Bi-encoders are fast because they embed query and document separately; cross-encoders are slower but sharper because they look at both at once, which is why you only run one over a shortlist. Reported gains are large: Cohere Rerank 3.5 lifts BEIR results about 23.4 percent over hybrid search, and one study cites roughly a 40 percent accuracy improvement from adding a reranker (Pinecone, Ailog).

It is on the repo's future-work list alongside hybrid BM25 search, and in hindsight it is what I would add first. A reranker would have rescued some of the answerable-but-rejected questions the confidence system was throwing away, which is the next section.

Confidence thresholding: the hallucination-versus-refusal dial

This is the centerpiece, because it is where the project stopped being a tutorial and started being a judgment call. Once you retrieve chunks, you have a similarity score for each. The question is what to do with that score, and the naive answer (always feed the top chunks to the model and let it talk) is how you get a confident, fluent, wrong answer.

The build uses a three-tier system keyed on the maximum cosine similarity across the retrieved chunks (rag/pipeline.py, TIERED_CONFIDENCE_SUMMARY.md). The thresholds are not magic numbers from a paper; they were tuned against this specific corpus:

# config: confidence thresholds (cosine similarity)
MIN_SIMILARITY        = 0.50   # below this, drop the chunk entirely
LOW_CONFIDENCE_THRESHOLD  = 0.55   # 0.55-0.64 -> answer, but hedge
HIGH_QUALITY_THRESHOLD    = 0.65   # >= 0.65   -> answer with attribution

Those three constants produce three behaviors:

1. 0.65 and above, HIGH. The bot answers normally, with attribution back to the community messages it drew from.
2. 0.55 to 0.64, LOW. The bot answers, but with a "limited match" caveat, shows its provenance, and suggests asking the group to confirm. It speaks, but in a quieter voice.
3. Below 0.55, FALLBACK. The bot declines: "I don't have a recommendation for that yet, try asking the group directly." No improvising from a weak match.

You can watch the dial work across real queries. Asking for Singaporean food scores 0.796 and asking about pineapple tarts scores 0.737, both clean HIGH answers. Indian restaurants comes in at 0.581, a LOW answer delivered with a hedge. Ski resorts (0.272) and yoga studios (0.439) fall through to the fallback, correctly, because the corpus genuinely does not cover them.

I tuned these thresholds to cut hallucinations, and they worked, but they made the bot too conservative. The clearest case: a member had genuinely recommended a good mechanic, the data was in the corpus, and the bot still refused, because the conversational phrasing ("does anyone know a guy for...") scored only 0.30 to 0.54 against a direct query. The information existed. The bot would not surface it. That is a false refusal, and a fixed threshold produces them by design.

It gets worse when you try to generalize. A fixed threshold like 0.5 that delivers around 90 percent recall on one dataset can collapse to roughly 60 percent on another, because what counts as a "confident" similarity score depends entirely on the corpus and the embedding model (arXiv 2512.15068). Thresholds do not transfer. You cannot copy mine and expect them to work on your data; they have to be tuned per corpus.

The tiered design is my considered answer to that problem. Instead of one silent cutoff that either answers or refuses, the middle LOW band lets the bot answer with caveats and provenance, which turns a binary into a graceful degradation. It is a reusable operator pattern: when you are unsure whether to trust a retrieval, do not choose between confident and silent. Add a third state that speaks honestly about its own uncertainty.

What the evaluation showed

I ran a golden set of known question-answer pairs to put numbers on the behavior, and the scores tell a consistent story. Factual accuracy came in at 8.2 out of 10, but completeness was 5.0, relevance 5.6, and overall 4.8. No-answer detection was a perfect 100 percent, catching all six questions the corpus could not answer.

Read those numbers together and the design philosophy is right there: high accuracy, perfect refusal of unanswerable questions, mediocre completeness. The bot is rarely wrong and never bluffs, but it often answers less fully than it could, which is exactly the conservative tuning showing up in the metrics. I chose accuracy over coverage on purpose, because in a community giving advice on immigration and housing, a confident wrong answer is far more damaging than a cautious incomplete one. The evaluation confirms I got the tradeoff I aimed for, including its downside.

Three design choices fall out of that stance, each deliberate rather than accidental:

• Attribution is always hedged. The bot says "A member mentioned..." rather than stating a recommendation as settled fact, because the source is one person's message, not a verified truth.
• Generation is deterministic. At temperature=0, the same question yields the same answer, which makes the system testable. You cannot evaluate a bot whose answers drift run to run.
• It is stateless by design. The system is RAG-only with no conversational memory, so a follow-up like "what about their hours?" fails: the bot has no idea what "their" refers to. Each query is answered fresh, in isolation. That is a real limitation, and a deliberate scope cut for a first version.

Security and PII are now your problem

This is the part a managed tool quietly handles and a custom build does not. The moment you own the stack, you own its safety defaults, and the corpus here made that concrete: it holds real people's names and messages. Asked the right way, the bot could assemble a small biography of a real community member from scattered mentions. That is not a hypothetical; it is what a retrieval system over personal chat history does unless you stop it.

The surface is broader than just names. Sensitive domains like immigration and tax advice need disclaimers, because a hedged bot answer is not legal counsel. Prompt injection and abuse need guardrails. And queries get logged by the messaging provider for around 30 days, so even the questions people ask are data you are now responsible for.

The mitigations I reached for, as a pattern rather than a finished solution: system-prompt rules that forbid profiling or assembling biographies of individuals; PII handling at the data layer, weighed carefully against the fact that stripping names also strips the attribution that makes answers trustworthy; and output filtering as a last line. None of this is solved. It is the kind of work that does not appear in a RAG tutorial and consumes real hours once the system touches real people.

What I would build next

The repo carries a future-work list. Having lived with the system, I would order it like this, with the first three the ones I would do:

1. Hybrid search and a reranker. Add BM25 keyword matching alongside the semantic search so exact terms ("kopitiam", a specific stall name) are not lost to embedding fuzziness, then put a cross-encoder reranker on top. This is the most direct fix for the false refusals.
2. Query expansion. Rewrite a conversational question into a cleaner retrieval query before embedding it, which is what the mechanic example needed: the phrasing, not the data, was the problem.
3. Per-category thresholds. One global cutoff cannot be right for both restaurant recommendations and immigration advice. Different topics deserve different confidence bars.
4. A stronger embedder. Move to voyage-3 at 1024 dimensions for the cheap relevance bump, or add the reranker first and measure whether the embedder is still the bottleneck.
5. Temporal decay. Down-weight stale recommendations, because a hawker stall that closed in 2023 should not outrank one that opened last month.
6. A Singlish glossary. Help the embedder understand that "shiok" and "siok" mean the same thing, and that "rojak" is food, not chaos.
7. Optional conversational memory. Reframe RAG as a tool the model can call rather than a fixed prefix, which would let follow-up questions work, at roughly double the cost per query.

Where this leaves the project

Building this was a great way to learn RAG, and almost none of the learning was about the model. The retrieve-then-generate loop is a weekend's work. The hours went into calibration, deciding when a retrieval is trustworthy enough to speak, and into safety, deciding what a bot is allowed to say about real people. Those are the parts that separate a demo from something you would put in front of a community, and they are exactly the parts no tutorial covers.

If you want to fork it, change the thresholds, or just read how the pieces fit, it is all in the repo. And if you came here from Part 1, the loop closes the way it opened: the engineering was never the hard part. The judgment was.