#004 Top 10 LLM+RAG Anti-Patterns
I find Anti-patterns as a best way to QUICKLY understand a NEW topic
đ LLM + RAG Anti-Patterns with Business Problems & Solutions
1. Stuffing the LLM with Too Many Documents
Problem: Retrieval pulls 50+ long passages â context window bloats â model hallucinates or truncates.
Business Impact: Customer support chatbot gives irrelevant or cut-off answers â high support costs.
Solution:
Measure: Track token usage vs. retrieval relevance score.
Fix: Use Max Marginal Relevance (MMR) or reranking to keep top 3â5 most relevant docs.
2. Embedding Everything Without Normalization
Problem: Raw text with boilerplate, disclaimers, stopwords gets embedded.
Business Impact: Search recalls irrelevant âlegalâ or âfooterâ text instead of meaningful business content.
Solution:
Measure: Audit recall quality with top-k evaluation.
Fix: Clean & normalize text (remove boilerplates, dedupe, split semantically).
3. Over-Reliance on Default Vector Similarity
Problem: Using only cosine similarity, ignoring domain semantics (e.g., âpremiumâ â âsubscriptionâ).
Business Impact: Insurance RAG system retrieves wrong policy clauses â regulatory compliance risk.
Solution:
Measure: Evaluate retrieval F1 with domain-specific benchmarks.
Fix: Fine-tune embeddings or hybrid retrieval (BM25 + dense).
4. Ignoring Recency in Data
Problem: Index not refreshed frequently â outdated retrieval.
Business Impact: Financial chatbot uses last yearâs rates â customers misled, legal exposure.
Solution:
Measure: Track % of queries answered with stale docs.
Fix: Incremental indexing, metadata filters (date-aware retrieval).
5. Chunking Without Overlap or Semantics
Problem: Arbitrary splitting (e.g., every 512 tokens) â broken meaning across chunks.
Business Impact: Medical assistant misses critical context â wrong treatment recommendations.
Solution:
Measure: Evaluate recall on multi-chunk queries.
Fix: Semantic chunking with overlaps & document structure awareness.
6. No Grounding in Retrieved Sources
Problem: LLM answers but doesnât cite sources.
Business Impact: Legal research tool delivers hallucinated case law â damages trust & liability.
Solution:
Measure: Track % answers with cited references.
Fix: Chain-of-thought prompting with âinclude citation spansâ or structured output with metadata.
7. One-Size-Fits-All Prompting
Problem: Same prompt template for FAQs, financial reports, and contracts.
Business Impact: Poor precision in specialized queries (e.g., compliance rules).
Solution:
Measure: Measure answer accuracy across task types.
Fix: Context-specific prompt templates (FAQ mode vs. compliance mode).
8. Ignoring Query Understanding
Problem: Treat user query as raw text â retrieval ignores intent (e.g., âcheapest planâ vs. âmost affordable long-termâ).
Business Impact: Sales chatbot suggests wrong product bundle â revenue loss.
Solution:
Measure: Compare retrieval precision with/without query rewriting.
Fix: Add query rephrasing step (LLM reformulates for retrieval).
9. Lack of Evaluation Pipeline
Problem: No systematic way to measure hallucinations, grounding, latency.
Business Impact: System deployed â business learns problems only via angry customers.
Solution:
Measure: Build RAG eval harness (precision@k, factual consistency).
Fix: Automated regression testing + business KPI dashboards.
10. Overlooking Latency & Cost
Problem: Every query hits embedding store + long LLM call.
Business Impact: High infra bills + slow user experience â customer churn.
Solution:
Measure: Track cost per query & response latency.
Fix: Cache frequent queries, use lightweight rerankers before LLM, and tiered infra (fast embeddings + deep retrieval only when needed).