RAG Implementer

Build production-ready retrieval-augmented generation systems.

Core Principle

RAG = Retrieval + Context Assembly + Generation

Use RAG when you need LLMs to access fresh, domain-specific, or proprietary knowledge that wasn't in their training data.

⚠️ Prerequisites & Cost Reality Check

STOP: Have You Validated the Need for RAG?

Before implementing RAG, confirm:

• Problem validated - Completed product-strategist Phase 1 (problem discovery)
• Users need AI search - Tested with simpler alternatives (see below)
• ROI justified - Calculated cost vs benefit of RAG vs alternatives

Try These FIRST (Before RAG)

RAG is powerful but expensive. Try cheaper alternatives first:

1. FAQ Page / Documentation (1 day, $0)

• Create well-organized FAQ or docs
• Add search with Cmd+F
• Works for: <50 common questions, static content
• Test: Do users find answers? If yes, stop here.

2. Simple Keyword Search (2-3 days, $0-20/month)

• Use Algolia, Typesense, or PostgreSQL full-text search
• Good enough for 80% of use cases
• Works for: <100k documents, keyword matching sufficient
• Test: Do users get relevant results? If yes, stop here.

3. Manual Curation (Concierge MVP) (1 week, $0)

• Manually answer user questions
• Build FAQ from common questions
• Works for: <100 users, validating if users want AI
• Test: Do users value your answers enough to pay? If yes, consider RAG.

4. Simple Semantic Search (1 week, $30-50/month)

• Use OpenAI embeddings + Postgres pgvector
• Skip complex retrieval, re-ranking, etc.
• Works for: <50k documents, basic semantic search
• Test: Are embeddings better than keyword search? If no, stop here.

Cost Reality Check

Naive RAG (Prototype):

• Time: 1-2 weeks
• Cost: $50-150/month (vector DB + embeddings + API calls)
• When: Prototype, <10k documents, proof of concept

Advanced RAG (Production):

• Time: 3-4 weeks
• Cost: $200-500/month (hybrid search, re-ranking, monitoring)
• When: Production, 10k-1M documents, validated demand

Modular RAG (Enterprise):

• Time: 6-8 weeks
• Cost: $500-2000+/month (multiple KBs, specialized modules)
• When: Enterprise, 1M+ documents, mission-critical

Decision Tree: Do You Really Need RAG?

Do users need to search your content?
│
├─ No → Don't build RAG ❌
│
└─ Yes
   ├─ <50 items? → FAQ page ✅ ($0)
   │
   └─ >50 items?
      ├─ Keyword search enough? → Use Algolia ✅ ($0-20/mo)
      │
      └─ Need semantic understanding?
         ├─ <50k docs? → Simple semantic (pgvector) ✅ ($30/mo)
         │
         └─ >50k docs?
            ├─ Validated with users? → Build RAG ✅
            └─ Not validated? → Test with Concierge MVP first ⚠️

Validation Checklist

Only proceed with RAG implementation if:

• Tested simpler alternatives (FAQ, keyword search, manual curation)
• Users confirmed they need AI-powered search (not just you think they do)
• Calculated ROI: cost of RAG < value users get
• Have >50k documents OR complex semantic search requirements
• Budget: $200-500/month for infrastructure
• Time: 3-4 weeks for production implementation

If any checkbox is unchecked: Go back to product-strategist or mvp-builder skills to validate first.

See also: PLAYBOOKS/validation-first-development.md for step-by-step validation process.

8-Phase RAG Implementation

Phase 1: Knowledge Base Design

Goal: Create well-structured knowledge foundation

Actions:

• Map data sources (internal: docs, databases, APIs / external: web, feeds)
• Filter noise, select authoritative content (prevent "data dump fallacy")
• Define chunking strategy: semantic chunking based on structure
• Add metadata: tags, timestamps, source identifiers, categories

Validation:

• All data sources catalogued and prioritized
• Data quality assessed (accuracy, completeness, freshness)
• Chunking strategy tested with sample documents
• Metadata schema validated for search effectiveness

Common Chunking Strategies:

• Fixed-size: 500-1000 tokens, 50-100 token overlap
• Semantic: By paragraph, section headers, or topic boundaries
• Recursive: Split by structure (markdown headers, code blocks)

Phase 2: Embedding Strategy

Goal: Choose optimal embedding approach for semantic understanding

Actions:

• Select embedding model: text-embedding-3-large (1536 dim) for general, domain-specific for specialized
• Plan multi-modal needs (text, code, images, tables)
• Decide on fine-tuning: use domain data if general embeddings underperform
• Establish similarity benchmarks

Validation:

• Embedding model benchmarked on domain data
• Retrieval accuracy tested with known query-document pairs
• Storage and compute costs validated

Model Selection:

• General: OpenAI text-embedding-3-large, text-embedding-3-small
• Code: code-search-babbage-code-001 or StarEncoder
• Multilingual: multilingual-e5-large

Phase 3: Vector Store Architecture

Goal: Implement scalable vector database

Actions:

• Choose vector DB (Pinecone, Weaviate, Qdrant, Chroma, pgvector)
• Configure index: HNSW for speed, IVF for scale
• Plan scalability: data growth and query volume
• Implement backup, recovery, security

Validation:

• Vector store benchmarked under expected load
• Index optimized for retrieval speed and accuracy
• Backup and recovery tested
• Security controls implemented

Vector DB Decision:

• Managed cloud → Pinecone
• Self-hosted, feature-rich → Weaviate
• Lightweight, local → Chroma
• Cost-conscious → pgvector (Postgres extension)
• High-performance → Qdrant

Phase 4: Retrieval Pipeline

Goal: Build sophisticated retrieval beyond simple similarity search

Actions:

• Implement hybrid retrieval: semantic search + keyword (BM25)
• Add query enhancement: expansion, reformulation, multi-query
• Apply contextual filtering: metadata, temporal constraints, relevance ranking
• Design for query types: factual (precision), analytical (breadth), creative (diversity)
• Handle edge cases: no relevant results found

Advanced Techniques:

• Re-ranking: Use cross-encoder after initial retrieval (e.g., cross-encoder/ms-marco-MiniLM-L-12-v2)
• Query routing: Route different query types to specialized strategies
• Ensemble methods: Combine multiple retrieval approaches
• Adaptive retrieval: Adjust top-k based on query complexity

Validation:

• Retrieval accuracy tested across diverse query types
• Hybrid retrieval outperforms single-method baselines
• Query latency meets requirements (<500ms ideal)
• Edge cases and fallbacks tested

Phase 5: Context Assembly

Goal: Transform retrieved chunks into optimal LLM context

Actions:

• Rank and select: prioritize by relevance score, recency, source authority
• Synthesize: merge related chunks, avoid redundancy
• Compress: use LLMLingua or similar for token optimization
• Mitigate "lost in the middle": place critical info at start/end
• Adapt dynamically: adjust context based on conversation history

Context Engineering Integration:

• Blend RAG results with system instructions and user prompts
• Maintain conversation coherence across multi-turn interactions
• Implement context persistence for follow-up queries
• Balance context size vs. information density

Validation:

• Context relevance validated against human judgments
• Token optimization maintains accuracy
• Multi-turn conversations maintain coherence
• Assembly latency <200ms

Phase 6: Evaluation & Metrics

Goal: Measure RAG system performance comprehensively

Retrieval Quality:

• Precision@K: Fraction of top-K results that are relevant
• Recall@K: Fraction of relevant docs in top-K
• MRR (Mean Reciprocal Rank): Average rank of first relevant result
• NDCG: Ranking quality with graded relevance

Generation Quality:

• Faithfulness: Generated content accuracy vs. sources
• Answer Relevance: Response relevance to query
• Context Utilization: How effectively LLM uses retrieved info
• Hallucination Rate: Frequency of unsupported claims

System Performance:

• End-to-End Latency: Query to answer (<3 seconds target)
• Retrieval Latency: Time to retrieve and rank (<500ms)
• Token Efficiency: Information density per token
• Cost Per Query: Combined retrieval + generation costs

Validation:

• Baseline metrics established
• A/B testing framework for config comparisons
• Automated evaluation pipeline deployed
• Human evaluation protocols for ground truth

Phase 7: Production Deployment

Goal: Deploy with enterprise-grade reliability and security

Deployment:

• Containerize with Docker/Kubernetes
• Implement load balancing across RAG instances
• Add caching for frequent queries
• Graceful degradation: fallback to base model on component failure

Security:

• Role-based access controls for knowledge base
• Data masking and PII protection
• Audit logging for compliance
• Prompt injection defense

Monitoring:

• Real-time metrics dashboard (latency, cost, accuracy)
• Query analysis for patterns and failure modes
• Cost tracking and optimization alerts
• Performance profiling for bottlenecks

Validation:

• Production handles expected traffic
• Security prevents unauthorized access
• Monitoring provides actionable insights
• Incident response procedures tested

Phase 8: Continuous Improvement

Goal: Establish processes for ongoing enhancement

Data Pipeline:

• Automated knowledge base updates (real-time or scheduled)
• Quality monitoring: detect data drift and degradation
• Source diversification: add new data sources
• Feedback integration: user corrections and preferences

Model Evolution:

• Evaluate and migrate to improved embeddings
• Fine-tune on domain data regularly
• Upgrade architecture: Naive → Advanced → Modular RAG
• Expand multi-modal support (images, audio, video)

Optimization:

• Analyze query patterns, optimize for common needs
• Improve cache hit rates
• Tune vector indices regularly
• Balance performance vs. costs

Validation:

• Automated improvement pipelines functioning
• Performance trends show improvement
• User satisfaction increasing
• System adapts to changing needs

Key RAG Principles

1. Relevance Over Volume

• Quality curation > massive datasets
• Remove outdated/low-quality content continuously
• Prioritize most relevant info to prevent "lost in the middle"

2. Semantic Understanding

• Use embeddings for true semantic matching, not just keywords
• Recognize query intent (factual, analytical, creative)
• Adapt retrieval strategy based on context

3. Multi-Modal Intelligence

• Handle text, images, code, tables, structured data
• Enable cross-modal retrieval (text query → image results)
• Preserve document structure and formatting

4. Temporal Awareness

• Prioritize recent info for time-sensitive topics
• Maintain historical access when relevant
• Integrate real-time data feeds for dynamic domains

5. Transparency & Trust

• Always provide source citations
• Indicate confidence levels
• Explain why specific information was selected

Standard RAG Response Format

{
  "answer": "Generated response incorporating retrieved information",
  "sources": [
    {
      "content": "Retrieved text chunk",
      "source": "Document/URL identifier",
      "relevance_score": 0.95,
      "chunk_id": "unique_identifier"
    }
  ],
  "confidence": 0.87,
  "retrieval_metadata": {
    "chunks_retrieved": 5,
    "retrieval_time_ms": 150,
    "generation_time_ms": 800
  }
}

Critical Success Rules

Non-Negotiable:

1. ✅ Source attribution for every response
2. ✅ Validate generated content against sources (prevent hallucination)
3. ✅ Filter sensitive data before retrieval
4. ✅ Respond within latency thresholds (<3 seconds)
5. ✅ Monitor and optimize costs continuously
6. ✅ Comply with security policies
7. ✅ Graceful degradation on failures
8. ✅ Comprehensive testing before production

Quality Gates:

• Before Production: >85% accuracy on evaluation dataset
• Ongoing: User satisfaction >4.0/5.0
• Performance: 95th percentile <5 seconds
• Reliability: 99.5% uptime
• Cost: Within 10% of budget

Advanced Patterns

Modular RAG Architecture

• Search Module: Query understanding and reformulation
• Memory Module: Long-term conversation persistence
• Routing Module: Query routing to specialized knowledge bases
• Predict Module: Anticipatory pre-loading based on context

Hybrid RAG + Fine-tuning

• RAG for dynamic, frequently changing knowledge
• Fine-tuning for domain-specific reasoning patterns
• Combine strengths for maximum effectiveness

Related Resources

Related Skills:

• multi-agent-architect - For complex RAG orchestration
• knowledge-graph-builder - For structured knowledge integration
• performance-optimizer - For RAG system optimization

Related Patterns:

• META/DECISION-FRAMEWORK.md - Vector DB and embedding selection
• STANDARDS/architecture-patterns/rag-pattern.md - RAG architecture details (when created)

Related Playbooks:

• PLAYBOOKS/deploy-rag-system.md - RAG deployment procedure (when created)