Building a Production-Grade Agentic System End-to-End: Orchestration, Memory, Observability, and Deployment

Building a Production-Grade Agentic System End-to-End: Orchestration, Memory, Observability, and Deployment

Picture two versions of the same support agent. One loops a model over a couple of tools and prints an answer, happy to die with the process and invisible to anyone when it misbehaves. The other is a durable, observable, safe, deployable machine that does that work continuously for many users, surviving failures it cannot predict. The second version is not the first version with extra polish; it is built differently from the start, and this article walks that build.

This is a build-through, not a survey. We take one working agent, a support agent that triages tickets and drafts responses, and carry it across the four concerns that turn a demo into something you can run for real users: orchestration, memory, observability, and deployment. Each one shows up here as concrete code and concrete decisions. By the end you should be able to look at any agent architecture and name which of the four pillars it is missing, because the missing pillar is usually the one that fails first in production.

The shape of the whole system#

Let me lay out the architecture before diving into pieces, so you can see how the pillars relate. The agent receives a ticket, runs a classify step to decide the path, routes to tools for account and knowledge lookups, drafts a response grounded in retrieval, and stops at approval gates when the action is consequential. Everything flows through state and a checkpointer so a failed or interrupted run can resume. Every step emits telemetry to a trace, and the output runs through guardrails before delivery.

There is a deep relationship among the pillars, and it is worth stating because it explains why they must be designed together rather than bolted on. Orchestration is what makes the agent's behavior a reproducible machine, and checkpoints make that machine resumable. Memory makes it continue across sessions. Observability makes the failures of the other three seeable. Deployment makes changes safe. None of these works in isolation, and a system built without one of them tends to fail in a way that looks like a bug in the model but is actually a hole in the architecture.

Orchestration: the state machine under the agent#

When I say production-grade orchestration, I mean two specific things beyond a plain loop: explicit graph structure and durable state. The graph structure is what lets you see and control every path through the system, and LangGraph is the right tool because it makes both the graph and the state explicit.

The contrast with a naive loop is sharp. A loop "just works" until you need to pause and resume, interrupt for human approval, run a validation that routes back, or coordinate a branch. In a graph, all of those are first-class edges and nodes. The support agent is a graph in which classifying produces evidence, and a conditional edge routes to the right retrieval and drafting nodes.

from typing import Annotated, TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langgraph.checkpoint.memory import MemorySaver
from langchain_openai import ChatOpenAI

class SupportState(TypedDict):
    messages: Annotated[list, add_messages]
    ticket: dict
    retrieved: list[str]
    draft: str
    needs_approval: bool

def classify(state: SupportState) -> dict:
    model = ChatOpenAI(model="gpt-4o")
    cat = model.invoke(f"Classify this ticket into billing|technical|account:\n{state['ticket']}")
    return {"messages": [cat]}

def retrieve(state: SupportState) -> dict:
    # RAG over the knowledge base
    chunks = vectorstore.similarity_search(state["ticket"]["body"], k=3)
    return {"retrieved": [c.page_content for c in chunks]}

builder = StateGraph(SupportState)
builder.add_node("classify", classify)
builder.add_node("retrieve", retrieve)
builder.add_edge(START, "classify")
builder.add_conditional_edges("classify", route_to_retrieve, {...})
builder.add_edge("retrieve", "draft")
app = builder.compile(checkpointer=MemorySaver())

The checkpointer is what makes orchestration production-grade rather than toy-grade. It serializes the state at every step, so if the process crashes mid-run, or a human pauses to review for an hour, the run resumes from its checkpoint rather than starting over. In production this checkpointer lives in Postgres or Redis, not memory, so it survives restarts. This durability is the difference between an agent that can be part of a real workflow and one that can only answer a chat in the moment.

Memory: persistence is the boring foundation#

Memory in this system is not a feature you add; it is the thing that makes the agent's history real. Every ticket maps to a thread, and a returning ticket must see prior context, the account's history, past resolutions, the customer's mood. Real memory requires a persistence layer that survives the individual request, and it must be chosen for concurrency and durability, not for demo convenience.

The distinction that matters in production is between the record, everything that happened, and the context, what the model actually sees on this turn. Holding everything in context is both expensive and dilutive, so production memory compresses: the model sees a window of recent turns plus distilled summaries of the older ones, and the full record lives in storage for audit.

import psycopg
from psycopg.rows import dict_row

class ThreadStore:
    def __init__(self, dsn: str):
        self.conn = psycopg.connect(dsn, row_factory=dict_row)

    def save_event(self, thread_id: str, role: str, content: str):
        with self.conn.cursor() as cur:
            cur.execute(
                "INSERT INTO thread_events(thread_id, role, content) VALUES (%s,%s,%s)",
                (thread_id, role, content),
            )
        self.conn.commit()

    def load_summary(self, thread_id: str) -> str:
        with self.conn.cursor() as cur:
            cur.execute(
                "SELECT summary FROM thread_summary WHERE thread_id = %s", (thread_id,)
            )
            row = cur.fetchone()
        return row["summary"] if row else ""

The discipline here is to decide consciously what the model sees and what it does not. A production system that dumps the entire history into every request is paying for tokens and burning context for no benefit. The compression strategy, which turns the last N turns into the working context and older turns into a summary, is a design decision you should test, because it directly changes answer quality and cost. Memory is the layer where a production system quietly outspends a demo a hundredfold if it is not engineered, because every user with a long history costs more per request.

Observability: you cannot fix what you cannot see#

I have never met a production agent where "just look at the answer" was enough, because by the time a user complains, the request is gone and all you have is a vague report. Observability is what turns that vague report into a specific, diagnosable failure, and for an agent it means tracing the full path: which classifier output, which chunks were retrieved, which model call, which tools ran, which guardrail gate fired, how long each step took, how many tokens it cost.

The trace is the causal chain, and it is the tool you use to answer the only question that ever matters in production: why. Why did this ticket get a wrong answer? The trace shows you the retrieved chunks were the wrong ones. Why was it slow? The trace shows where the seconds went. Why did the guardrail block it? The trace shows the gate decision. Without the trace, every one of these is a black box and every fix is a guess.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("support-agent")

def handle_ticket(ticket: dict):
    with tracer.start_as_current_span("support.handle") as span:
        span.set_attribute("ticket_id", ticket["id"])
        span.set_attribute("ticket_category", classify(ticket))
        with tracer.start_as_current_span("support.retrieve") as rspan:
            chunks = retrieve(ticket)
            rspan.set_attribute("chunks", len(chunks))
        with tracer.start_as_current_span("support.guardrail") as gspan:
            allowed = run_guardrails(chunks, ticket)
            gspan.set_attribute("allowed", allowed)
        span.set_status(Status(StatusCode.OK))

Beyond the raw trace, observability includes two quiet but important companions: metrics and evaluation in the loop. Metrics give you the aggregates, request rate, p95 latency, error rate, token cost per request, and they are how you notice a slow drift in cost or a spike in failures before users do. Evaluation in the loop means sampling a fraction of live responses, scoring them with the same LLM judges you used offline, and feeding the results back into your eval set. This closes the loop: your offline harness was built on assumptions about real traffic, and live eval continuously corrects those assumptions. This is the pillar that most distinguishes a maintained production system from a demo that "seems fine."

Deployment: changing a live system without breaking it#

Deployment is the pillar where production-grade bites hardest, because the moment the agent is live you cannot edit the prompt and rerun. Real users are on the current version, and a regression is not a notebook annoyance, it is a support ticket of its own. So deployment in this build means controlling change and scale deliberately.

The mechanism we use to change the support agent without breaking the people already on it is a feature flag wired to the eval harness from the observability section. A new prompt or chunking strategy runs in shadow on copied traffic, its outputs compared against the current version by the same LLM judges we use offline, and the validation decides whether it ships. Only a version that measurably wins, or at least does not regress, gets flipped live; in this codebase the rollout flag and the eval score are tied together, so promotion is a decision the eval makes explicit rather than a toggle flipped on a hunch.

import json, random
from redis import asyncio as aioredis

async def should_use_new_version(redis, flag_key: str, pct: float) -> bool:
    # Read the rollout percentage for this flag
    cfg = json.loads(await redis.get(flag_key) or "{}")
    rollout = cfg.get("rollout_pct", 0.0)
    return random.random() * 100 < rollout

The other deployment requirements are the familiar infrastructure classics, applied to an agent: caching repeated requests to cut cost, batching where requests are fungible, and provider fallback so the agent keeps working when a single model vendor is degraded. And unlike most systems, agent deployment has an operational dimension: because agent behavior is not fully predictable, you need the evaluations running continuously in production, you need alerts tied to the real quality signals, not just latency, and you need the ability to roll back a prompt change the moment a quality metric dips. A production agent is deployed software with an eval harness attached, and it is deployed and operated that way.

The cost of skipping a pillar#

Let me close the technical tour by being concrete about what each skipped pillar costs, because that is the argument that actually convinces. Skip orchestration and checkpoints and you have an agent that cannot survive a crash, cannot pause for human approval, and cannot be reasoned about as a machine; it is a loop that dies with the process. Skip memory and every returning user is a stranger, your agent repeats itself, and your token bill grows without bound as histories pile into every request. Skip observability and you are flying blind, every "it gave a weird answer" is a week of guesswork, and you will make changes that silently regress quality because you have no way to see it happen. Skip deployment discipline and your first prompt tweak regresses one slice of traffic, and you cannot even tell which, because there is no shadow comparison and no quality alert.

Each pillar, skipped, produces a failure that looks unrelated to the architecture and gets blamed on the model. But the model is rarely the villain in these failures. The architecture let it down. And because the four pillars are intertwined, the failure of one tends to cascade, an unobservable system cannot diagnose its own checkpoint losses, and an uncheckpointed system cannot use the durable state that memory expects.

The whole being more than the sum#

The discipline of building a production agent end-to-end is that all four pillars have to be present for any of them to be worth having, because they form one coherent machine and not a checklist. Orchestration drives the behavior and makes it durable. Memory gives that durability meaning across sessions. Observability makes the machine's behavior visible so you can maintain it. Deployment makes it possible to change and scale the maintained machine without breaking it. Remove any one and the machine develops a hole that no amount of careful attention to the other three can patch.

When I evaluate whether a team has really built a production-grade agent, I do not look at the model or the number of tools. I look for the four pillars, and specifically for whether they are integrated: checkpoints in a real store, memory compressing into context, traces covering every node, and evals running on live traffic with flags controlling rollout. That combination, boring and unglamorous as it sounds, is what separates the agents that teams trust with real work from the demos that convinced them to try. The model gets the credit, but the four pillars are the system that actually earns it.