Rootly cut its MCP error rate 10x by fixing errors that looked like successes

About Rootly: Rootly helps fast-growing tech and F500 companies prevent and resolve incidents faster, and brings AI-assisted incident response to the engineers running them. Their MCP server lets agents work incidents directly by searching incident history, reading on-call schedules, triaging alerts, and drafting retrospectives. Rootly's customers lean on them when their own systems are down, so its products are held to a 99.99% uptime. Their MCP has to meet the same bar.

Sylvain Kalache, the Head of AI Labs, got in touch with us to help them understand where their MCP usage was coming from and where agents were getting stuck. They integrated our SDK and tested live production data the same day. That data helped them find and fix the biggest issues, then build a system to keep the MCP reliable as it grew.

THE PROBLEM

The errors their observability stack couldn't see

Rootly shipped its MCP server very early, and adoption grew fast, with hundreds of different companies and hundreds of thousands of tool calls a day after just a few months. Given the diversity of their customer base, they noticed many different types of agents, from coding assistants to IDEs to custom-built internal agents, all calling the same tools slightly differently.

At first, the server looked healthy by every measure they had. Uptime was fine, latency was fine, and every response was a 200.

But there was a hidden issue.

When an MCP tool call fails, the failure gets wrapped by a success response sent back to the agent, with an error flag inside the agent is supposed to read. But the HTTP response is still successful. All conventional monitoring tools treat that as a success, and that results in errors going hidden from teams, while agents and end users behind those agents have frustrating experiences of retries and failures.

"AgentCat allowed us to identify the MCP failures that our standard observability stack was not catching."

The Rootly team knew that agents fail differently from human developers. A human reads the docs once, hardcodes the call, and debugs until it's successful. An agent re-reads the docs in every session and has to re-infer how to call the MCP server. If an error string isn't clear in the tool response, agents will eventually make mistakes. And every one of those mistakes was showing up as a success in their monitoring tools.

Sylvain from the Rootly team got in touch with us, and after a day of conversations, the team had the AgentCat SDK in testing for their MCP server. From there, Rootly realized that a bunch of agent calls was silently failing. That was an average error rate of 2.2%, peaking at 6% in the worst weeks. Their existing observability tools, logs, and checks treated those errors as successes.

The MCP server has become one of the main ways Rootly's customers adopt AI in incident response: they build long-lived agents on it to resolve incidents. A failure that only shows up inside an agent's session never reaches the dashboards anyone is watching.

THE SOLUTION

Treating agent failures as first-class errors

AgentCat treats an agent-visible tool failure as an error because, to the agents and their end users, it is an error.

The AgentCat dashboard captured thousands of unique tool call failures and then intelligently fingerprinted and combined those error events into 18 distinct issues. The top two issues made up 87% of the error volume in the MCP server. The largest single issue contributed almost half of all error events. Prioritization was now based on data, not judgment calls.

The biggest issue? Agents were calling tools incorrectly.

They invented parameters that didn't exist, omitted required arguments, and even called tools that never existed.

Less than 1% of those errors were actual infrastructure failures. The server wasn't broken, and there was no bug. The team addressed this by tightening the contract:

• Correcting ambiguous parts of the schema
• Making error messages instructive
• Removing unclear or overlapping tools

Within a week of releasing those changes, error rates fell across the entire server, and for every customer at once.

From one-time cleanup to a reliability loop

But agent hallucinations never go to zero. New clients, new models, and new agents keep joining. And every new session is a re-interpretation of the schemas, so a low baseline of agents making mistakes is permanent.

The difference is whether you see it and how quickly you respond. AgentCat helped Rootly solve this problem by fingerprinting every new failure pattern and grouping those errored events, allowing the team to see the first occurrence, the latest occurrence, and the exact error text.

Rootly also began tagging each agent with the customer it belongs to using Event Tags, letting them scope a new failure immediately and understand whether it's just impacting one customer or multiple. This also allows them to identify whether an error could be coming from a specific customer's setup.

With these changes, Rootly has kept the error rate below 0.5% since fixing the initial issues. All while their MCP usage volume has been growing. They used AgentCat to monitor reliability as a product property, something Rootly is deeply committed to.

RESULTS

Rootly decreased their error rate from peaks of up to 6% to sub-0.5% sustained over several weeks after integrating AgentCat. This came from a combination of addressing the most impactful issues while building systems to identify and resolve new issues that arose, all without sacrificing shipping velocity or falling behind their MCP's growth.

Agents will keep improving, but a baseline level of mistakes will never go away. And as MCP adoption continues to grow exponentially, more and more of Rootly's usage is coming from agents rather than humans.

AgentCat fills in the reliability gaps that other observability tools don't capture. By treating MCP failures as first-class errors and giving teams like Rootly the tools to monitor, identify, and resolve them quickly, AgentCat helps Rootly deliver an MCP that their customers can integrate into critical workflows.

Going forward, Rootly plans on updating their MCP server to support any action users can perform within their web app. "We're treating MCP as our UI for agents. We want to bring the same level of functionality and power that users have in our web app to our MCP server. And making our MCP a reliable product that our customers can depend on for their most important workflows is crucial. AgentCat helps us meet that standard of reliability."