MCP (Model Context Protocol) gets a lot of attention right now for the right reasons: it gives you a standardized contract for connecting LLMs to external tools. Instead of every team building their own bespoke function-calling layer, you have a protocol — defined inputs, defined outputs, defined error shapes. That's genuinely useful.
But the protocol isn't what breaks production agents. What breaks them is everything the protocol doesn't define: what happens when the agent fires the same tool twice, what happens when step 4 of 6 fails, and who's accountable when the agent does something unexpected.
We built a production MCP-based workflow engine that runs internal operations autonomously — DB queries, Slack notifications, report generation, record updates. It's been running for about 18 months. Here's what I've learned.
Idempotency is not optional
Agents retry. Networks drop. Users sometimes click twice. If your MCP tools aren't idempotent, you will send duplicate Slack messages, create duplicate records, or charge a card twice. The LLM doesn't know it already ran that step — it might re-invoke a tool if the previous call's result was ambiguous or the context window got truncated.
Every tool in our MCP server accepts an optional idempotencyKey parameter. The agent generates a key at the start of each workflow run (timestamp + intent hash). Tool implementations check this key against a Redis set before executing — if it's already there, they return the cached result without re-running. This eliminated an entire category of production incidents.
Every tool needs a dry-run mode
Before we deploy any new agent workflow, we run it in dry-run mode against production data. Dry-run tools log what they would do without actually doing it — which lets us verify the agent's reasoning chain is correct before it has real-world effects.
This sounds obvious. It wasn't obvious to us until an agent correctly classified a report generation request and then tried to send that report to every customer on the account list instead of just the requester. The tool would have caught this in dry-run. We hadn't built dry-run yet. We built it the next day.
The audit trail is the product
The hardest cultural challenge with autonomous agents isn't getting them to work — it's getting people to trust them. The way you earn that trust is not by making the agent more capable; it's by making everything the agent does completely transparent and inspectable.
Every tool invocation in our system emits a structured event: timestamp, workflow ID, tool name, input parameters (sanitized), result, and the LLM's reasoning for why it called that tool. These events are stored in PostgreSQL and surfaced in an admin dashboard. When something goes wrong — and it will — any team member can pull up the exact sequence of decisions the agent made and understand precisely what happened.
The audit trail also provides a natural rollback surface. For any workflow that modifies state, we log the before-state alongside the after-state. Rolling back a failed workflow is a query, not a support ticket.
The thing MCP doesn't solve
MCP gives you a clean interface for tool invocation. It doesn't give you workflow orchestration — how you chain multiple tool calls, how you handle partial failures mid-workflow, how you maintain state across a multi-step operation that might take 30 seconds.
We built a lightweight workflow engine on top of MCP that handles this. Each workflow is defined as a sequence of steps with explicit success/failure transitions. The engine executes steps sequentially, persists state to the database between steps, and can resume from any checkpoint if the process is interrupted. The LLM drives the tool selection within each step; the workflow engine ensures the overall sequence is correct and recoverable.
That's the part nobody puts in the MCP tutorial. The protocol is the easy part. The workflow layer is where the engineering actually lives.