EvalRAG — an inspectable product experimentation analyst
Turns an experiment question or CSV into a source-grounded launch memo, then checks the recommendation against deterministic policy guardrails.
The problem
A launch recommendation can sound convincing even when retrieval is weak, a sample-ratio check fails, or a key product guardrail regresses.
Interactive trace
Policy checkedRevenue improves, but 7-day retention declines.
Final decision
Investigate further
A product guardrail regressed, so the policy validator blocks a simple launch recommendation.
01
Retrieve guardrail guidance
02
Read experiment evidence
03
Apply launch policy
04
Write traceable memo
Synthetic examples from the project’s starter evaluation set.
What I built
A bounded LangGraph workflow with hybrid playbook retrieval, deterministic SRM, lift, and segment diagnostics, structured memo generation, policy validation, and trace logging.
What I tested
A starter synthetic evaluation suite measures source retrieval, concept coverage, decision accuracy, policy corrections, latency, faithfulness, and context quality.
What changed
The useful product is not a generic chatbot. It is a constrained analyst whose evidence, tool calls, policy overrides, and failure modes can all be inspected.