LangGraph · Agentic RAG · Evaluation

Agents that cite their sources, or say they can't.

I'm Ingyu Koh. I build LangGraph agents with explicit state, bounded tools, verified citations, and a principled abstention path. Then I measure them against a real baseline, so the extra complexity has to show up in the numbers.

View the SEC Filing RAG Auditor → Read the case study

Featured project

SEC Filing RAG Auditor

A reproducible LangGraph agent that answers numeric questions over SEC 10-K facts. It plans, retrieves, grades metadata, verifies provenance, cites an SEC filing, and refuses to guess. It is benchmarked head-to-head against a single-pass baseline that uses the same corpus and retriever.

github.com/ingyukoh/agentic-rag-eval

LangGraph StateGraphFastAPIBM25 SEC EDGAR XBRLpytestDockerGitHub Actions

Python 3.11–3.12 · MIT · CI passing

65.6% → 100%task success, +34.4 pts vs baseline

64frozen questions, including unanswerable cases

60normalized SEC facts: AAPL, MSFT, GOOGL

0guesses on unsupported questions (baseline: 4)

Task success

65.6%

100%

Answerability

93.8%

100%

Value accuracy

65.6%

100%

Citation accuracy

65.6%

100%

Single-pass BM25 baselineAgentic metadata-verified path

Numbers come from python -m agentic_rag_eval.bench. They are committed output, not hand-typed.

The graph

Four LangGraph nodes over a typed AgentState. Every node appends a timed event to the trace, so each answer comes with a record of how it was reached.

Real traces from the committed code

These are output from AgenticSystem.answer() at main, with the retrieval hit list shortened. The second question asks about a company that isn't in the corpus. BM25 still returns plausible revenue facts, but the grader rejects all of them, so the agent abstains.

"What was Apple revenue in fiscal year 2024?"answered

"value": 391035000000, "unit": "USD",
"citation_id": "aapl-2024-revenue",
"source_url": "https://www.sec.gov/Archives/
  edgar/data/320193/000032019324000123/",
"trace": [
  {"node": "plan", "ticker": "AAPL",
   "metric": "revenue", "fiscal_year": 2024},
  {"node": "retrieve", "hit_ids": ["aapl-2024-revenue",
   "aapl-2022-revenue", … 10 more]},
  {"node": "grade",
   "exact_candidates": ["aapl-2024-revenue"]},
  {"node": "verify", "answerable": true}
]

"What was Tesla revenue in fiscal year 2024?"abstained

"value": null, "answerable": false,
"reason": "missing or ambiguous company, metric,
  fiscal year, or SEC fact",
"trace": [
  {"node": "plan", "ticker": null,
   "metric": "revenue", "fiscal_year": 2024},
  {"node": "retrieve", "hit_ids": ["aapl-2024-revenue",
   "googl-2024-revenue", … 10 more]},
  {"node": "grade", "exact_candidates": []},
  {"node": "verify", "answerable": false}
]

Where the baseline fails, and why

Every failure is committed to results/failures.md. The baseline misses 22 of 64 cases, and they fall into two patterns that the graph was designed to fix:

Failure pattern	Cases	Example	Graph node that fixes it
Right company and metric, wrong fiscal year	18	aapl-2024-net-income → returned another year's value	`plan` extracts the year, then `grade` requires an exact match
Answered a question with no supporting fact	4	unanswerable-unknown-company → 391035000000	`verify` abstains unless exactly one fact survives grading

Anti-tuning controls

Data comes from the official SEC EDGAR Company Facts API. Each fact stores CIK, XBRL concept, period, accession number, and archive URL.
data/manifest.json content-hashes the facts, questions, and provenance. If any of them change, tests fail before scoring.
The baseline and the agent share the same corpus and the same BM25 implementation. The only difference is orchestration.
CI runs ruff, pytest, and the full benchmark on Python 3.11 and 3.12.

Known limits

The corpus covers 3 issuers, 5 metrics, and fiscal years 2022–2025.
Planning is deterministic, which keeps the focus on orchestration and verification rather than on hosted-model quality.
The agent doesn't yet handle comparisons, ratios, or free-text accounting questions.
100% applies only to this narrow benchmark. It is not a claim of production reliability.

Reproduce it in about a minute, offline

git clone https://github.com/ingyukoh/agentic-rag-eval && cd agentic-rag-eval
python -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'
ruff check . && pytest -q
python -m agentic_rag_eval.bench        # regenerates results/benchmark.md
docker compose up --build bench         # or run it without a local Python

Experience

Agent engineering work

Independent, public projects. Each one can be cloned, tested, and benchmarked.

Sep 2026 — present

Agentic RAG & Evaluation Engineer

Independent · agentic-rag-eval

Designed and built a LangGraph StateGraph (plan → retrieve → grade → verify) for numeric questions over SEC 10-K facts, with an explicit abstention path and SEC-linked citations.
Raised exact task success from 65.6% to 100% over a single-pass BM25 baseline with the same corpus and retriever, on a content-hashed 64-case benchmark that includes unanswerable questions.
Built the eval harness, which reports task success, answerability, value and citation accuracy, and p50/p95 latency, and writes failure reports that cover every failed case.
Shipped it as a FastAPI service with a node-level execution trace in every response, packaged with Docker and CI (ruff, pytest, and the benchmark on Python 3.11 and 3.12).

Sep 2026

Agent Safety & Tool Routing

Independent · evidence-first-agentic-rag

Built a LangGraph agent with an input guard that blocks prompt-injection patterns before retrieval or any tool call runs.
Added allowlisted tool routing (skill-gap analysis, evidence-quality inspection) and grounded synthesis with source citations.
Designed a pluggable generation layer with a deterministic offline provider and an optional OpenAI Responses provider, so evals are reproducible at zero cost.
Wrote an eval suite covering grounded retrieval, injection attempts, and insufficient-evidence refusals, and compared a top-1 baseline with an agentic top-3 path.

Scope: These are portfolio systems that show current engineering ability. They have not been deployed for a client. I wrote the orchestration, evaluation, and serving code. SEC data, LangGraph, and FastAPI are third-party.

Evidence map