Repo: github.com/midimurphdesigns/issuegraph
Live demo: issuegraph.kevinmurphywebdev.com
Read the full story: Building issuegraph
issuegraph is a GitHub issue triage agent built as a LangGraph state machine. It classifies an issue, routes it to a specialist drafter, loops the reply through a quality guard, and pauses for human approval when its own confidence is low. The live demo runs the real graph server-side and streams every node to the page as it executes. 7 graph nodes, 2 evaluators over a labeled golden set, 1 calibration report checking whether the model's confidence is honest.
How it's built
Next.js 16 App Router, TypeScript strict, zero any. The graph is @langchain/langgraph, the model calls are @langchain/anthropic, evals and tracing are langsmith. Zod validates every LLM boundary and both API request bodies. Upstash provides the rate limiter and, through a derived TCP connection, the Redis checkpointer that lets paused graphs resume across serverless instances.
The classifier is a LangChain chain: a prompt template piped into a model with structured output. The schema forces a category and a confidence score out of every call, so the graph downstream can route on typed data instead of parsing prose.
const classifierChain = prompt.pipe(
model.withStructuredOutput(ClassificationSchema, { name: "classify_issue" }),
);
// returns { category: "bug" | "feature" | "docs" | "question",
// confidence: number, reasoning: string }
The graph wires seven nodes with conditional edges. One router picks the specialist drafter by category. A second router loops rejected drafts back for another pass, bounded so the guard cannot spin forever.
const builder = new StateGraph(TriageState)
.addNode("classify", classifyNode)
.addNode("draftBug", draftBugNode)
.addNode("draftFeature", draftFeatureNode)
.addNode("draftDocs", draftDocsNode)
.addNode("draftQuestion", draftQuestionNode)
.addNode("guard", guardNode)
.addNode("gate", gateNode)
.addEdge(START, "classify")
.addConditionalEdges("classify", routeByCategory, { /* category -> drafter */ })
.addConditionalEdges("guard", afterGuard, { /* approved -> gate, rejected -> redraft */ })
.addEdge("gate", END);
The confidence gate is where the human enters. Above the threshold the run finalizes on its own. Below it, the node calls interrupt(), which checkpoints the whole graph state and stops. The run resumes later, possibly on a different serverless instance, when a human sends back a decision.
function gateNode(state: TriageStateType) {
const confidence = state.classification?.confidence ?? 0;
if (confidence >= CONFIDENCE_GATE) return { status: "auto-finalized" };
const decision = interrupt({
reason: "low confidence, human approval required",
confidence,
draft: state.draft,
}) as { approved: boolean };
return { status: decision.approved ? "human-approved" : "human-rejected" };
}
Resuming is one call: graph.invoke(new Command({ resume: { approved } }), config). The resume value is an object on purpose. LangGraph treats a bare falsy resume as missing input and throws, which I learned by hitting it.
The LangSmith leg
LangSmith does two jobs here: it records what ran, and it grades how well.
Recording is free. Set two environment variables and every chain and graph invocation ships a trace: the full tree of model calls with inputs, outputs, token counts, and latency per step. When the guard rejected a draft and the graph looped back, the trace shows the second draft call nested exactly where it happened. The live demo page mirrors this same shape in the browser, streaming one event per executed node, so what you watch on screen is the trace structure LangSmith records server-side.
Grading is the eval suite. A labeled golden set uploads as a LangSmith dataset, evaluate() runs the full graph over every example, and two evaluators score each result. Category accuracy is a deterministic exact match. Draft quality is an LLM judge, because a free-form reply has no single correct answer to compare against.
const experiment = await evaluate(target, {
data: "issuegraph-golden",
evaluators: [categoryAccuracy, draftQuality],
experimentPrefix: "issuegraph",
});
The calibration report is the part I care most about. The classifier states a confidence with every call, and that number is a claim, not a fact. The report checks the claim against reality: a Brier score across all predictions plus a reliability table that buckets stated confidence and compares it to actual accuracy. This is the same discipline forge applies to its debugging lanes, rebuilt on LangSmith eval data.
── CALIBRATION REPORT ──────────────────────────────────
samples: 8
Brier score: 0.003 (0 = perfect, 0.25 = coin flip)
reliability by confidence bucket:
bucket n claimed actual verdict
0.9-1.0 8 0.95 1.00 calibrated
A clean report on eight easy examples flatters the model, and the honest read is that the golden set is the weakness. The interesting work is growing it with production failures until the reliability table starts telling the truth about where the classifier overclaims.
The demo hardening
The public page triages curated preset issues only. Arbitrary input on a public LLM endpoint invites prompt injection and unbounded spend, so no visitor text ever reaches a prompt. Per-IP rate limits and a global daily budget cap the spend, both failing closed if the limiter is unconfigured in production. Resume requests run no model call, so they stay rate-limited but never burn the daily budget. Thread ids are UUIDs, checkpoints expire after an hour, and production error events are generic while the real errors stay in server logs.
Artifacts worth reading
src/graph.ts. The state machine: routing, the guard loop, the confidence gate withinterrupt().src/run-evals.ts. Dataset upload,evaluate(), and the calibration snapshot the demo page renders.src/calibration.ts. Brier score and reliability buckets in plain TypeScript, unit tested.src/demo/checkpointer.ts. Why serverless interrupts need Redis, and the Upstash env-var derivation.
The trade-offs
The golden set is eight clean examples, which proves the eval plumbing rather than the classifier. The guard and the judge share a model family with the drafters, so the grading has a family bias a production system would break by judging with a different provider. Tracing stays off in production to hold the free LangSmith tier; runs trace locally and in evals. And the CLI accepts any public GitHub issue URL while the hosted demo deliberately does not.