Database Index Mishaps Amplify Outages, Impact API Availability

15 October 2026

A missing database index can force a full table scan, sparking cascading delays and exhaustion of API connection pools, a post-incident review has found. After a long-running query took seconds per request, the Dispatch API's 60-connection pool reached its limit, causing 40% of requests to fail during a 41-minute period.

The root cause was a missing index that forced the affected query into a linear scan, locking up each connection until resolved. A developer attempted to handle the pressure spike with an aggressive horizontal pod autoscaler (HPA) setting, but that only amplified the issue by spinning up more pods before the index was fixed.

Postmortem diagrams show how an unindexed query demand can drive requests from milliseconds to seconds, as sequential scans exhaust CPU and I/O resources.

Azure SQL Database documentation warns that missing indexes are a top source of performance bottlenecks. A SELECT operation depending on a scan "will execute more slowly and use more CPU, Memory, and Disk resources, than an equivalent query that uses an index," according to Microsoft's troubleshooting guide.

The Dispatch incident confirms those concerns, though awareness and responses to the missing index took time. Figuring out the problem required debugging the work queues and clearing duplicates. "It took a lot of effort to see what queries were running in `pg_stat_activity` and what errors were happening in the logs," according to a post-incident report.

The Dispatch team eventually identified and indexed the suspect table, freeing up the 60-connection pool in 7 minutes. Developer logs show the fix almost immediately restored response times, though reports of impact continued to arrive for hours.

The Trade-Off of Indexing

SQL Server documentation cautions that indexes themselves are not a free performance win. The same pattern that speeds up SELECTs by replacing linear scans with seeks must be reviewed against the write impact. "An increase in the number of indexes will improve the query retrieval time. However, it slows down the speed for the Data Manipulation Language (DML) operations such as INSERT and UPDATE," according to Microsoft's index design guide.

This trade-off should be part of performance tuning and incident reviews. "Adding too many indexes will cause excessive time to perform DML queries," the guide says. "The needed indexes should be determined according to the actual requirements of the business."

The risk of unindexed queries driving availability problems suggests that teams may want to monitor for missing indexes in addition to operational metrics. Microsoft suggests checking for index plan warnings if the optimizer believes a missing index could improve query performance.

Indexes should be validated before addition, as missing index warnings are only rough estimates based on single query analysis. "The SQL Server Query Optimizer analyzes a query and determines if a missing index could improve performance. But a missing index recommendation is only an estimate," Microsoft documentation warns.

Key Lessons

The key lesson is to combine feedback paths on resource exhaustion with detailed operational telemetry on both availability issues and query plan performance. Dispatch described resolving the availability problems in minutes, thanks to a timely fix. However, while that prevented more severe issues, no process improvements were implemented.

Developers and site reliability engineers should collaborate more closely in previewing change plans. CPU, I/O, Memory, and Disk metrics should be included in any alerting or performance monitoring. An indexing warning policy could create a validated feedback loop between immediate operator concerns and background indexing risks.

The ultimate risk could be more important to share with technical leadership. In 2015, Stripe experienced 90 minutes of degraded performance and customer complaints when a poorly managed index drop, intended to improve write times, had unforeseen latency effects.

Another takeaway should be preparing for worst-case scenarios. Dispatch restored availability and resolved the pending batch, but post-incident timelines showed lingering issues even after identifying the root cause. The lesson is that addressing one problem can reveal others, requiring workflows to handle both shifting resource demand and system vulnerabilities.

The root cause here applies to how teams evaluate real-world performance, not just immediate concerns during any given quarter.