What issuegraph demonstrates
- LangGraph state machines. Seven nodes, shared typed state, conditional edges for routing and a bounded retry loop, a checkpointer for pause and resume.
- Human-in-the-loop interrupts. A confidence gate calls
interrupt()mid-run, the graph checkpoints to Redis, and a human approves or rejects the draft before the run finishes, possibly on a different serverless instance. - LangChain chains with structured output. Prompt templates piped into Zod-schema-forced tool calls, so every LLM boundary returns typed, validated data.
- LangSmith evals. A labeled golden set uploaded as a dataset,
evaluate()running the full graph over every example, a deterministic evaluator paired with an LLM judge. - Confidence calibration. Brier score plus a reliability table that checks the classifier's stated confidence against actual accuracy, the same discipline forge applies to its debugging lanes.
- Public-endpoint hardening. Preset-only input, per-IP rate limits, a global daily budget that fails closed, sanitized production errors.
Why I built it
Most of my agent work runs on the Vercel AI SDK and the raw Anthropic SDK. forge orchestrates four parallel subagents with hand-rolled coordination. Kev-O does retrieval with hand-rolled BM25 and reranking. I like building from primitives because you learn what the abstractions are hiding. LangChain and LangGraph are the other major dialect of agent engineering, and I wanted to put them through the same bar I hold my own orchestration to.
So I built the kind of system I would normally hand-roll, in the framework instead. A GitHub issue triage agent: read an issue, decide what it is, draft a maintainer reply, check the draft, and know when to ask a human instead of trusting itself. Small enough to be honest about, real enough to exercise the parts of LangGraph I actually wanted to test: conditional routing, a bounded loop, and human-in-the-loop interrupts.
The three pieces, in one line each: LangChain is the glue for calling a model, LangGraph turns a set of steps into a runnable graph, and LangSmith records and grades what ran. The one I most wanted to learn was LangGraph, because it is built for exactly the thing this project needed: an agent that stops partway through and waits for a person.
Which is why I do not have a single default. If I had to pick one for a team that ships a lot of different AI features, LangChain plus LangSmith is the safe standardization: the ecosystem is broadest, the tracing and eval story is the most complete, and most agents eventually grow a branch or a loop. But for a single-provider service where I want every token in view, or a durable step pipeline, the lighter tools are not a compromise, they are the better fit. The skill is not loyalty to one. It is knowing which shape the problem is before you reach.
The graph
A pipe is a straight line. Real triage branches, loops, and pauses, and that is exactly what a pipe cannot express. LangGraph models it as a state machine: nodes are functions that read shared state and return partial updates, edges decide what runs next.
const builder = new StateGraph(TriageState)
.addNode("classify", classifyNode)
.addNode("draftBug", draftBugNode)
.addNode("draftFeature", draftFeatureNode)
.addNode("draftDocs", draftDocsNode)
.addNode("draftQuestion", draftQuestionNode)
.addNode("guard", guardNode)
.addNode("gate", gateNode)
.addEdge(START, "classify")
.addConditionalEdges("classify", routeByCategory, {
draftBug: "draftBug",
draftFeature: "draftFeature",
draftDocs: "draftDocs",
draftQuestion: "draftQuestion",
})
.addConditionalEdges("guard", afterGuard, {
gate: "gate",
draftBug: "draftBug",
draftFeature: "draftFeature",
draftDocs: "draftDocs",
draftQuestion: "draftQuestion",
})
.addEdge("gate", END);
A conditional edge is just a function that returns the name of the next node. routeByCategory reads the classification and picks a specialist drafter. afterGuard either advances to the gate or points back at a drafter, and pointing backward is all it takes to make a retry loop. The loop is bounded by a redraft counter in state, because an LLM judge that keeps rejecting drafts would otherwise spin forever.
The classifier that feeds the router is a plain LangChain chain. The part worth knowing is withStructuredOutput: it converts a Zod schema into a forced tool call, so the model cannot reply with prose. It has to fill in the schema, and the chain returns a typed object.
const ClassificationSchema = z.object({
category: z.enum(["bug", "feature", "docs", "question"]),
confidence: z.number().min(0).max(1),
reasoning: z.string(),
});
const classifierChain = prompt.pipe(
model.withStructuredOutput(ClassificationSchema, { name: "classify_issue" }),
);
That confidence number matters later. It is a claim the model makes about itself, and the whole back half of this project is about checking that claim.
The interrupt, and the bug that taught me the most
The gate node is my favorite part of the system. If the classifier is confident, the run finalizes on its own. If not, the graph stops and asks a person.
function gateNode(state: TriageStateType) {
const confidence = state.classification?.confidence ?? 0;
if (confidence >= CONFIDENCE_GATE) return { status: "auto-finalized" };
const decision = interrupt({
reason: "low confidence, human approval required",
confidence,
draft: state.draft,
}) as { approved: boolean };
return { status: decision.approved ? "human-approved" : "human-rejected" };
}
interrupt() pauses the graph and saves a checkpoint: which node was running, what the state held, everything needed to come back later. Resuming is graph.invoke(new Command({ resume: value }), config) with the same thread id. The checkpointer is what makes this work, and it is required, because you cannot resume a run that nothing remembered.
The bug: my first version resumed with a bare boolean, new Command({ resume: false }) for a rejection. LangGraph threw EmptyInputError. A falsy resume value looks like no input at all, so approval worked and rejection crashed. The fix is resuming with an object, { approved: false }, which is never falsy. It is a small bug, but it taught me how the resume plumbing actually works underneath: the resume value is real graph input, not a signal, and it flows through the same input validation everything else does.
The serverless wrinkle is that the pause and the resume are different requests, which on Vercel can mean different machines with nothing shared in memory. So the checkpointer writes to Redis. The demo derives its Redis connection from the same Upstash environment variables the rate limiter uses, and paused runs survive across instances with an hour of TTL.
LangSmith, in practice
Two environment variables turn on tracing, and every chain and graph call ships a full trace tree: each model call with its inputs, outputs, token counts, and latency, nested inside the node that made it. When the guard rejects a draft and the graph loops back for a redraft, the trace shows the second call exactly where it happened. Debugging a non-deterministic system without this is guesswork. With it, you read what actually ran.
The demo page streams the same structure to the browser as the graph executes, one event per node, so watching the pipeline light up on screen is watching the trace shape form in real time.
Evals are where LangSmith earns its keep. The golden set is a list of issues with known-correct categories. It uploads once as a dataset, and evaluate() runs the whole graph against every example.
const experiment = await evaluate(target, {
data: "issuegraph-golden",
evaluators: [categoryAccuracy, draftQuality],
experimentPrefix: "issuegraph",
});
Two evaluators, deliberately different kinds. Category accuracy is an exact match, because there is one right answer and comparing strings is free and objective. Draft quality is an LLM judge, because a good maintainer reply has a hundred valid forms and no string comparison can grade it. Knowing which kind fits which output is most of eval design.
Calibration: checking the model's honesty
The classifier says 0.95 confident. Should anyone believe it?
Calibration is the discipline of checking. Take every prediction, pair the stated confidence with whether the answer was actually right, and compute two things. The Brier score averages the squared gap between confidence and outcome, so confident-and-wrong costs the most. The reliability table buckets predictions by stated confidence and compares each bucket's claim to its actual accuracy.
── CALIBRATION REPORT ──────────────────────────────────
samples: 8
Brier score: 0.003 (0 = perfect, 0.25 = coin flip)
reliability by confidence bucket:
bucket n claimed actual verdict
0.9-1.0 8 0.95 1.00 calibrated
That report is real output from the committed golden-set run, and the honest read is that it flatters the model. Eight clean examples prove the plumbing, not the classifier. The value shows up as the golden set grows with production failures: the buckets where claimed confidence outruns actual accuracy are exactly where the confidence gate should distrust the model and route to a human. forge does this same loop with a learned weight per debugging lane. issuegraph rebuilds the pattern on LangSmith eval data, and the confidence gate is the consumer.
What the framework bought, and what it cost
My other agent projects hand-roll their coordination. forge runs four parallel subagents with my own orchestration. Kev-O does retrieval with my own BM25 and reranking. Building this one in a framework instead was the point, and it made the trade concrete.
What it bought me was real. The checkpointer and interrupt() gave me pause-and-resume across serverless instances for almost no code, which is the kind of thing that is fiddly to get right by hand. Tracing turned on with two environment variables. evaluate() plus datasets replaced a pile of test-harness code I have written from scratch elsewhere.
What it cost me showed up at debug time. When I coordinate everything myself, a failure lands in code I wrote and can read straight through. The false-resume bug earlier in this post is the small example: the misbehavior was inside the framework's input handling, so understanding it meant reading LangGraph's source rather than my own. In my own code it would have been obvious at a glance. That is the honest tradeoff, and it is not a knock on the framework. It is just where the time moves when you let someone else own the control flow.
The demo hardening
The hosted page triages curated presets only. No visitor text ever reaches a prompt, which closes prompt injection and keeps spend bounded. Per-IP rate limits ride a sliding window, a global daily budget caps the worst case at a few dollars, and both fail closed if the limiter is unconfigured in production. Resume requests make no model call, so they stay rate-limited without burning budget. Production error events are generic; the real errors stay in server logs.
The CLI version accepts any public GitHub issue URL, which is the honest split: capability in the repo, restraint on the public endpoint.