What One Production Incident Really Costs You (With a Calculator)

Ask an engineering leader what incidents cost and you’ll usually get one of two answers: “Not much, we fix them quickly,” or a huge downtime-per-minute figure from an analyst report that doesn’t feel real for their company.

Neither one helps you decide anything.

This post gives you a way to calculate the cost for your own team, from numbers you already know. Most leaders who run it are surprised, and not because the fix is expensive. It’s everything around the fix.

Two questions to start

Before the breakdown, answer these two:

1. How many production incidents did you have last month? Count anything that pulled an engineer off planned work: customer-reported bugs, alerts that needed investigation, failed deploys, slow endpoints.
2. How long did a typical one take to resolve, from first alert or report to confirmed fix?

Keep those two numbers handy. We’ll use them in the calculator below.

Where incident time actually goes

Every incident goes through the same phases, whether or not your team names them:

Phase	What’s happening	Typical share of time
Detect	An alert fires or a customer complains	Small
Triage	Is it real? How bad? Who owns it?	Small–medium
Investigate	Find the error, then the trace, then the logs, then what the user did, then what changed	Largest by far
Fix	Write and ship the change	Medium
Verify	Confirm it’s actually resolved	Small
Follow-up	Postmortem, ticket, customer reply	Small–medium

In most teams we talk to, investigation takes about half the incident, often more. And investigation is mostly not thinking. It’s gathering context:

• Open the error tracker. Find the stack trace.
• Copy the timestamp. Open the logging tool. Search around that time.
• Find a request ID. Open the tracing tool. Find the slow span.
• Ask in Slack: “Did anyone deploy something around 2:40?”
• Open the session replay tool to see what the user actually clicked.
• Open Git to find what changed in that service.

That’s five or six tools and a lot of copying timestamps between tabs, before anyone has written a line of the fix.

The formula

Here’s the cost model. Each part uses a number you can find or estimate in a few minutes.

1. Direct engineering time

Incidents per month × Hours to resolve × Engineers involved × Loaded hourly cost

“Loaded hourly cost” means salary plus benefits, taxes, and overhead, divided by working hours. For many SaaS teams it’s somewhere between $60 and $150 per hour depending on location and seniority.

2. The interruption tax

Every incident pulls people out of focused work, and getting back in isn’t instant. Research by Gloria Mark at UC Irvine found it takes about 23 minutes to get back to an interrupted task. In practice, on-call engineers lose more than that, because the rest of the day is fragmented.

Incidents per month × Engineers involved × ~0.5 hours recovery × Loaded hourly cost

3. Customer impact

This one is harder to pin down, so keep it simple and conservative:

(Annual revenue ÷ 8,760 hours) × Hours of customer-facing impact × % of customers affected

This doesn’t capture churn, which is often the real cost: a customer who hits the same bug twice in a month starts evaluating alternatives. If you have a sense of incident-driven churn, add it. If not, leave it out and remember the number is a floor.

4. Opportunity cost

Every engineer-hour spent on incidents is an hour not spent on the roadmap. You don’t need to price this separately. It’s the direct time from step 1, seen from the product side. But it’s the line that matters most to your board.

Calculator

A worked example

A team with 8 incidents a month, 3 hours each, 2 engineers involved, at $90 per hour, with $5M annual revenue and about 10% of customers affected:

Component	Monthly
Direct engineering time (8 × 3 × 2 × $90)	$4,320
Interruption tax (8 × 2 × 0.5 × $90)	$720
Customer impact ($5M ÷ 8,760 × 24 hrs × 10%)	$1,370
Total	≈ $6,410 / month (~$77K / year)

That’s 48 engineer-hours a month, more than a full working week of one engineer, spent on incidents. About half of it went to gathering context.

And this team is doing fine. Three hours per incident is respectable. The point isn’t that they’re failing; it’s that the cost is invisible because it’s spread across many small interruptions instead of one big outage.

How to bring the number down

There are three levers, and they’re not equally easy to pull.

Fewer incidents. Better testing, safer deploys, feature flags. Worth doing, but it’s slow, and incidents never go to zero.

Fewer people per incident. Clear ownership and good runbooks help. But in hard incidents you often need the second engineer, because no one person has all the context.

Less time investigating. This is the biggest lever and the most overlooked. If the error, the trace, the logs around it, the user’s session, and the deploy that preceded it were already connected on one screen, half the incident goes away. That’s not a faster engineer; it’s removing the tab-switching.

The last lever is where tooling matters. If your errors live in one product, logs in another, traces in a third, and session replays in a fourth, investigation will always be slow. The tools don’t share context, so your engineers have to carry it between them.

Where Modulis fits

Modulis was built around the investigate phase. It deploys inside your own cloud and correlates logs, traces, metrics, errors, session replay, and Git metadata automatically. When something breaks, you start from a single view: here’s the error, the trace it came from, the logs around it, what the user did, and the commit that changed that code path.

AI-assisted root cause analysis then runs on your own LLM key, so your code and telemetry never leave your environment.

For the example team above, cutting investigation from 1.5 hours to a few minutes per incident gives back around 20 engineer-hours a month. That’s before counting faster customer recovery.

FAQ

What counts as a production incident? Anything that pulls an engineer off planned work to investigate production behavior: an alert, a customer report, a failed deploy, a performance regression. Using a wider definition gives you a more honest cost.

What is a good MTTR? It depends on severity and system complexity, but the trend matters more than a benchmark. Track median time to resolve by severity level month over month. If it isn’t going down as you add tools, the tools aren’t helping.

Why not just use an industry cost-of-downtime figure? Those numbers are averaged across companies of very different sizes and usually only count full outages. Most of your incident cost comes from frequent, smaller issues, which only your own numbers capture.

Should I include churn in the calculation? If you can estimate it, yes. It’s often the largest cost. If you can’t, leave it out and treat your result as a minimum.

How do I reduce investigation time without changing tools? Standardize on trace IDs in every log line, make sure every deploy is tagged with a version you can search for, and keep a single “where to look first” runbook per service. These help a lot. Removing the need to switch tools at all helps more.

---

Run the calculator, then start free on one application and compare your next incident. Or book a 20-minute walkthrough and bring your numbers.