RAG Copilot
An AI copilot for on-call engineers. Ask "Why is the payments service showing a latency spike?" and it pulls runbooks and live telemetry, diagnoses the cause, proposes a fix, and executes it after human approval.
Designed and built the agent workflow, RAG pipeline, approval flow, Kubernetes integration, evaluation harness, and React UI.
Problem: On-call engineers piece together runbooks and telemetry by hand during incidents, and letting an AI act on production is unsafe without strict limits.
Architecture: A LangGraph workflow retrieves runbooks (RAG over pgvector) and evidence from Prometheus, Loki and Tempo in parallel, then the LLM gives a structured diagnosis and picks an action from a typed list. Approved fixes run on Kubernetes and are verified against metrics.
Technical Decisions: Built the RAG pipeline on PostgreSQL and pgvector: Markdown split by headings, re-ingestion that skips unchanged documents, a vector index with filtered search, and a startup check that refuses to mix vectors from different embedding models. Defended against prompt and query injection: untrusted text is clearly marked in prompts, all model output is validated, queries come from fixed templates, and a person must approve every action. High-risk actions need a second superuser, and each action executes exactly once. Instrumented the AI itself with OpenTelemetry: one trace per run with a span for every step, tool and LLM call, including token usage, plus a saved record of the exact prompts behind every answer.
Outcomes: Shipped a production-style stack with 103 backend tests, 6 browser tests, Docker images and Kubernetes manifests verified on a local kind cluster. An LLM evaluation harness replays evidence recorded from the live system so only the model varies, applies rule-based checks, and reports pass rates per model, including a scenario that plants a malicious instruction in the logs.
>Technologies Used

// System Architecture Diagram
graph TD
%% RAG Copilot agent workflow (LangGraph)
Q["Question from on-call engineer"] --> Classify["classify"]
Classify -- knowledge --> Docs["retrieve_documents"] --> Answer["answer_knowledge"]
Classify -- investigate --> Runbooks
Classify -- investigate --> Evidence
subgraph Gather [Runs in parallel]
Runbooks["retrieve_runbooks<br/>RAG over pgvector"]
Evidence["gather_evidence<br/>Prometheus · Loki · Tempo · deployments"]
end
Runbooks --> Diagnose["diagnose<br/>structured LLM output"]
Evidence --> Diagnose
Diagnose --> Propose["propose_remediation<br/>typed action list"]
Propose --> Approval{"await_approval<br/>human decision"}
Approval -- rejected --> Stop["End"]
Approval -- approved --> Execute["execute_action<br/>least-privilege Kubernetes"]
Execute --> Verify["verify_outcome<br/>metrics back within targets"]
subgraph Platform [Platform]
PG[("PostgreSQL + pgvector<br/>runs, checkpoints, audit trail")]
OTel["OpenTelemetry<br/>trace per run"]
end
Approval -. paused run saved .-> PG
Verify -. spans and token usage .-> OTel
classDef step fill:#121212,stroke:#deec56,stroke-width:1px,color:#fff;
classDef gate fill:#1a1a1a,stroke:#deec56,stroke-width:2px,color:#deec56;
classDef infra fill:#121212,stroke:#38bdf8,stroke-width:1px,color:#fff;
class Classify,Docs,Answer,Runbooks,Evidence,Diagnose,Propose,Execute,Verify step;
class Approval gate;
class PG,OTel infra;





