What OpenAI and Google engineers learned deploying 50+ AI products in production

We worked on a guest post together and the core idea stuck with me: building AI products is not the same as building traditional software. Today I’m joined by Aishwarya and Kiriti to share what they’ve learned shipping dozens of AI deployments, with one goal for this episode—save you time, pain, and false starts; so, on the ground, what’s going well and what’s not?

This year feels different—skepticism is way down and teams are rethinking flows instead of just slapping chat on their data. Execution is still messy though, because the AI lifecycle breaks old handoffs and forces tighter, shared feedback loops across PMs, engineers, and data folks.

Let’s hit the two big differences you’ve been emphasizing that change how we build.

First, non‑determinism on both sides: users express intent in countless ways, and models respond probabilistically, so neither inputs nor outputs are predictable. Second, autonomy versus control: the more decision‑making you hand to an agent, the more governance you give up, so the system must earn trust before you expand its freedom.

Start small and train up, like preparing for a big hike by doing shorter trails first. Begin with low‑risk tasks under strong human control, then increase autonomy only as you gain confidence.

Can you walk through a concrete example of that progression?

Take support: begin with AI suggestions for human agents to accept or edit, learn where it fails, then let it reply directly for common cases, and only later add actions like refunds or creating tickets. Stacking complexity slowly keeps it controllable.

Think of it as behavior calibration. Constrain autonomy where risk is high, like pre‑authorizing simple tests while keeping invasive procedures human‑reviewed, and log human actions to power a steady improvement loop without breaking trust.

You’ve shown similar step‑ups for coding assistants and marketing tools, and the throughline is that user inputs and model outputs both vary, so you build confidence in stages.

The wild part is also the good part—natural language lowers friction and feels human. The hard part is mapping that fluidity back to reliable outcomes.

And when teams jump straight to full autonomy, they often stall out or label the whole approach a failure.

Problem‑first thinking keeps you grounded. Tight scope with limited autonomy clarifies what you’re actually solving and reduces the blast radius when things go wrong.

Reliability is the top blocker—roughly seventy five percent of enterprises cite it—and that’s why most wins so far live in productivity helpers, not end‑to‑end automation.

We also touched on adversarial risks like prompt injection and jailbreaks, which only get scarier as agents act on the world.

Security will loom large as these systems go mainstream, since non‑deterministic behavior widens the attack surface for injected instructions.

I’m still optimistic—adoption is early, and with sensible human checkpoints you can focus on process gains while containing risk.

What working patterns do you see in teams that consistently ship successful AI products?

It’s a triangle of leadership, culture, and technical craft. Leaders must rebuild instincts by getting hands‑on—one CEO I worked with blocked early mornings for AI deep dives—while culture should empower subject matter experts rather than scare them, and teams need a surgical grasp of workflows, mixing deterministic code with ML, building data flywheels, and ignoring one‑click agent hype because enterprise reality takes months.

On that note, leaders who live in the tools seem to drive the biggest gains; let’s talk eVals, since that’s been a hot debate—do they solve it or are they overrated?

It’s a false choice. Offline evals encode your product judgment, while production monitoring surfaces real failures via explicit and implicit signals like regenerations or feature toggles; you need both because each catches different classes of issues.

Also, the word “evals” now covers everything from expert error notes to PM checklists to model benchmarks, which confuses teams; focus on building actionable feedback loops, and remember LLM judges are brittle in complex domains, so sometimes user signals plus quick fixes beat elaborate judges.

How does the Codex team handle this balance in practice?

We use guardrail evals for core behaviors and lean hard on customer feedback, A/B tests, and hands‑on trials, because coding agents are highly customizable and no static set covers all integrations. We ship, watch real usage, and iterate quickly.

Let’s unpack your continuous calibration, continuous development framework that ties all this together.

We created CCCD after painful end‑to‑end agent projects, including one we had to shut down, and seeing incidents like a refund policy hallucination. Continuous development means scoping capability, curating examples, aligning on desired behavior, setting metrics, and deploying, while continuous calibration means inspecting traces, spotting new error patterns, adding or refining metrics when needed, and fixing issues; start with low autonomy and human‑in‑the‑loop, then progress from routing to copilot drafts to autonomous resolution, logging edits to fuel the flywheel, and constrain autonomy by action count or by topic risk.

How do you know when it’s time to step up autonomy?

Advance when surprises taper off. Expect to recalibrate after model changes or shifts in user behavior—for example, underwriters moved from simple lookups to deep historical questions, which required a very different system.

What feels misunderstood or underappreciated right now?

Multi‑agent setups are often misapplied; supervised orchestrators with sub‑agents can work, but peer‑to‑peer chatter is hard to control in production. Coding agents are still underused outside tech hubs and will unlock big gains.

Chasing tools is overrated; what’s underrated is deep problem understanding, strong product taste, and thoughtful design.

Paint the next year or two—what’s coming by the end of twenty twenty six?

Proactive background agents will plug into the real places work happens and nudge you with completed steps or timely prompts, like waking up to reviewed PRs or triaged tickets.

Richer multimodal experiences will get us closer to human conversational richness and unlock value in messy inputs like handwritten forms and ugly PDFs.

For builders who want to level up, what single skill should they invest in?

Cultivate taste, judgment, and ownership—shipping is cheap, insight is scarce—and act with agency, like a teammate who replaced a pricey tool with a simple internal app to rethink our process.

Add persistence to that; your willingness to endure the messy iterations becomes a moat, because the hard‑won knowledge compounds.

Final reminder: obsess over customers and data flows. Most of the real work is mapping workflows and edge cases, not chasing model novelties.

Lightning round time—quick picks and favorites.

Book: When Breath Becomes Air for its lived wisdom; show: rewatch Silicon Valley, it’s timeless; product: WhisperFlow’s concept‑aware dictation is surprisingly useful; motto: be bold enough to try even when the odds look bad.

Book: the Three‑Body trilogy for its sweeping perspective; game: Expedition 33; tools: Recast for speed and caffeinate to keep long runs alive; motto: you connect the dots only in hindsight.

What I admire about Kiriti is his calm, grounded judgment, the way he anticipates pitfalls, and his kindness; he lets the work speak.

What I admire about Ash is her gift for teaching—she makes complex ideas feel simple and empowering, and she pairs that with relentless follow‑through.

Find my writing on LinkedIn, grab our free GitHub resource list, and check out our Maven course on building enterprise AI products; we also share lots of free sessions.

I’m also on LinkedIn, and my DMs are open if you want to talk about coding agents or tricky product use cases.

Thanks for listening; subscribe on your favorite app and consider leaving a rating so others can discover the show.