Lessons learned from a year of running Pydantic AI in production

Building and running a modular agentic system that serves more than 9k active users taught us a few things about LLM orchestration and the ecosystem that grew around the different provider APIs. The current framework landscape feels similar to the rise of frontend frameworks in the mid-2010s: lots of trending libraries, lots of opinions, and a new recommendation every other week. Yet swapping the underlying framework rarely translates to value for end users.

For us, the framework decision was not about which library could produce the demo in the least time. Most frameworks can call a model, register a tool, and stream a response. The harder question was whether the framework would still be pleasant to operate after thousands of real conversations, multiple model migrations, custom agents, per-user permissions, and the usual long tail of provider edge cases.

To avoid optimizing for the wrong thing, we built several MVPs with commonly suggested frameworks and evaluated them against a few criteria:

• Developer experience and familiarity: what is the learning curve for a team of Python developers with a data science and data engineering background?
• Modern standards: how well does the framework integrate with emerging protocols like Model Context Protocol?
• Observability: can we easily distinguish model behavior from engineering faults?
• Models and providers: how easy is it to switch between model families and provider APIs?
• Out-of-the-box capabilities: what “batteries” are included in the framework for advanced use-cases?

The key lesson from these MVPs was that the framework should not become the product. The product is the user experience, the quality of the final answer, latency, reliability, and the trust users have in the system. The framework is only useful as it makes those things easier to build, maintain and debug.

Why Pydantic AI stood out

Pydantic AI stood out mostly because it felt like Python. Not Python wrapped around a graph DSL, not Python as a configuration language, but regular typed Python with familiar development loops.

That mattered more than we expected. Our team already used Pydantic heavily for data validation and backend development, so Pydantic AI’s mental model mapped well to the rest of our stack: agents, dependencies, tools, structured outputs, model settings, and reusable capabilities. Pydantic AI’s dependency system is type-safe, tools are registered as regular Python functions, and tool arguments are validated with Pydantic before being passed into application logic.

The type system was not just a developer-experience detail. It changed how we designed the boundaries of the system. Tool inputs became typed contracts. Agent dependencies became explicit. Structured outputs became application objects instead of “JSON-ish strings we hope parse later”. Many bugs that would otherwise have appeared as strange model behavior became ordinary validation errors.

That distinction matters in production. When an LLM application fails, the default assumption is often “the model did something weird”. Sometimes that is true. But often the problem is mundane: malformed tool output, stale context, a missing auth scope, a provider timeout, or a schema mismatch introduced during a refactor. Strong typing does not make the model deterministic, but it reduces the number of places where ordinary engineering bugs can hide behind stochastic behavior.

One year in production

After a year, the main thing we appreciate is not any single feature. It is that the framework stayed out of the way while the model ecosystem changed quickly.

The agent layer is not where we want to spend most of our engineering energy. We want to spend that energy on product behavior, tool quality, evaluations, permissions, and user experience. Pydantic AI gave us a stable place to put those concerns without forcing us to rewrite the application every time a provider changed a request parameter.

Model support without owning every abstraction ourselves

Provider abstraction sounds straightforward until you operate it. Models differ in tool-calling behavior, streaming semantics, context windows, native features, retry behavior, and reasoning controls.

Thinking mode is a good example. Different providers expose reasoning capabilities through different settings and defaults. Before using Pydantic AI’s unified thinking support, this was exactly the kind of abstraction we would have had to build and maintain ourselves. Instead, the common interface gave us a portable way to configure reasoning behavior while still leaving room for provider-specific settings when needed.

Observability became part of the product

No agentic system should go to production without traces. Logs are not enough. A single user message may trigger multiple model calls, tool calls, retries, validations, fallbacks, and summarization steps before a final answer is produced.

Pydantic AI’s OpenTelemetry-based instrumentation was one of the strongest production features for us. With Logfire instrumentation enabled, a trace is generated for each agent run, with spans emitted for model calls and tool execution. And due to it being an open protocol, we could use the existing monitoring and tracing infrastructure without introducing a new tool.

We needed to answer questions like:

Question	What it points to
Did the model call the wrong tool?	Prompt, tool description, or model issue
Did the tool return bad data?	Backend or integration issue
Did structured output validation fail?	Schema or model-output issue
Did latency come from the model or tools?	Routing and performance issue
Did the user lack permission?	Auth and product behavior issue

This failure taxonomy made incidents much easier to discuss. “The agent failed” is not actionable. “The model selected a valid tool, but the tool returned a permission-filtered empty result and we did not explain that to the user” is actionable.

Context management became its own subsystem

The context window is not just a model limit. It is a product constraint.

In early prototypes, it is tempting to append every message and tool result to the conversation history. That works until it does not. Long-running users eventually accumulate large tool outputs, files and historical details that may or may not still matter. Sending everything is expensive and can exhaust the model’s context window.

Our approach evolved into a tiered context strategy: keep recent turns, summarize older conversation blocks after a threshold, and compress huge message payloads such as pdfs with hundreds of pages immediately.

Pydantic AI made this quite easy with their latest harness update and allowed us easily keep up with provider api requirements. For example when slicing or summarizing history, tool calls and tool returns must remain paired, otherwise provider errors can occur.

CodeMode helped with orchestration-heavy tasks

Most tool-calling loops are simple: the model requests a tool, the application executes it, the result goes back to the model, and the model decides the next step. This is easy to reason about, but inefficient when the task is naturally procedural.

For example, if an agent needs to fetch ten records, compare them, and transform the result, a naive tool loop may require many model round trips. CodeMode, part of Pydantic AI Harness, wraps tools into a single run_code tool powered by the Monty sandbox. The model can write Python that calls multiple tools with loops, conditionals, variables, and asyncio.gather inside one tool call.

Agent Specs made custom agents manageable

One of our requirements was that users should be able to create custom agents without us generating bespoke Python classes for every configuration.

Agent Specs helped here. Pydantic AI supports declarative agent definitions in YAML or JSON, including model, instructions, capabilities, and related configuration. The docs frame this as a way to separate agent configuration from application code and allow non-developers or domain experts to configure agents.

That gave us a clean separation: product code defines which capabilities exist, specs define how those capabilities are assembled, user configuration chooses the allowed behavior, and the runtime validates and loads the result.

Where we would not use Pydantic AI

We would choose Pydantic AI again for agentic chat applications and we even migrated many existing services written in other frameworks to Pydantic AI.

For pure API-style services with little or no agentic logic, using the provider SDK directly is often faster and gives more control. Image generation endpoints for example do not need an agent framework.

The distinction we use is simple: if the application needs to manage an agent loop, tools, structured outputs, retries, history, model switching, and observability, Pydantic AI is a strong default. If the application is just a thin wrapper around one provider endpoint, the provider SDK may be the better abstraction.

Conclusion

The biggest lesson from running Pydantic AI in production is that agent frameworks should be boring in the right ways.

They should make common production concerns explicit: types, tools, dependencies, model settings, retries, traces, evals, context, and configuration. They should make provider changes survivable without forcing every product team to maintain its own abstraction layer. They should help developers debug real failures instead of hiding the agent loop behind magic.

Pydantic AI worked well for us because it matched the way we already build Python services. It gave us enough structure to move quickly, enough type safety to keep the codebase maintainable, and enough escape hatches to handle the parts of LLM engineering that are still changing.