Your warehouse does not break at peak. It slows down, and payroll absorbs the difference.

The failure I am describing has no alert attached to it, and that is not an oversight in anyone’s monitoring. It is a category error. Monitoring is built to answer a binary question — is the thing up. Degradation is continuous. A query that used to return in two seconds and now takes seven minutes was never down on any single day, and there is no day on which it should have paged anyone.

At the fulfillment operator in C08 — millions of order-related records a day — that is exactly what happened. Analytical queries had drifted from two seconds to seven minutes, and were now colliding with operational writes on the same database. Nobody had made a mistake. The orders table had reached 3.2 billion rows with no partitioning, VACUUM was running more than six hours, competing with production writes, and the system was doing precisely what a system in that state does (C08).

At peak, that arithmetic stops being a performance metric. Queue buildup becomes a missed cutoff, and a missed cutoff is not a chart — it is a shipment that does not go out.

Three mechanics, and they compound in this order.

The table outgrows the maintenance window. Autovacuum on a table of that size cannot finish inside the quiet hours it is given, so dead tuples accumulate, the visibility map goes stale, index-only scans stop being index-only, and every subsequent pass has more work than the last. The planner’s estimates drift with the statistics. Nothing in that sequence throws an error.

Reads and writes fight over the same instance. Someone in operations opens the volume report during the outbound push. The report is not wrong to exist. It is wrong to be served from the instance that is trying to commit scan events.

Manual workarounds harden into the process. This is the one that hides the other two. When the pick-list screen crawls, a supervisor keeps a spreadsheet. When the report times out, someone runs an export before the shift. Within a quarter the workaround is the process, and nobody remembers that it started as a symptom.

The fix in C08 was structural rather than clever: partition the orders table by month with pg_partman and migrate all 3.2 billion rows incrementally, in parallel, with no downtime; split into an operational tier for writes and current reads and an analytical tier for reporting and history, with a read-replica pool and PgBouncer in front; move changes into ClickHouse by CDC for the analytical side. VACUUM now runs per partition in 15–20 minutes with far less impact on production, and analytical queries went from 4–7 minutes → 8 seconds (C08).

The position: at logistics volumes, the operational tier and the analytical tier are two different systems. Running both on one instance is a decision, not a default, and peak is when you find out you made it.

A forecast can improve while stockouts worsen, and the two refuse to square until you look under the model. The ML team does its job — better features, cleaner training set, new architecture, beautiful dashboards. What nobody touches is the master data underneath: new SKUs entering with the wrong category code, new lanes inheriting the previous lane’s transit time, a distribution center that quietly stopped reporting receipts on time.

The model forecasts reality faithfully. The data has stopped being reality.

This is the same failure mode as the slow query, expressed in a different currency. Nothing errored. No record was rejected. A transit time of four days on a lane that now takes six is a perfectly valid number. It is simply false, and every downstream decision inherits it — safety stock, appointment scheduling, the promise your customer service team makes on the phone.

Contrast C12, the B2B distributor running 82,000 SKUs, where the inputs were fixed as part of the work rather than assumed. Forty percent of that catalog had intermittent demand, which is exactly where standard models fail; lead times across twelve suppliers ranged from 3 to 45 days with no per-supplier reliability model behind purchasing. Building the supplier lead-time distribution into safety stock, rather than only improving the forecast, is why the numbers moved in the same direction for once: stockouts 12% → about 4%, and $2.3M in working capital released in the first year (C12).

Smarter models on rotting data is a worse business than a plain model on clean data. The next hire after your ML lead is a data steward — someone whose only job is keeping the inputs honest.

Operations absorbs degradation before finance sees it, because absorbing things is what a good operations team does. The absorption has a price and it is booked somewhere else: overtime, a fifth shift lead at peak, a temp line that grows a little every season, dock doors standing idle while the system thinks and the trailer behind them accruing detention on someone’s clock.

Here is the diagnostic I would run before any of the technical ones. Plot overtime hours against order volume, month over month, for two years. If labor is growing faster than volume and nobody changed the process, the software is the variable. That is a chart your controller can produce this week and it costs nothing.

There is a second consequence that is easier to ignore and harder to defend. US warehousing employs roughly 1.85 million workers across 23,757 establishments, and the BLS injury rate is 4.8 per 100 full-time employees (2024). A system that forces manual workarounds at peak is not only a cost problem — people move differently around a process that is fighting them.

The 3PL hub in C09 was moving 35,000+ shipments a day with monitoring that watched servers. CPU was fine. Memory was fine. A dropped WMS integration or an interrupted EDI feed was discovered two to four hours later, when orders stopped moving. Reliability was felt rather than measured: no SLOs on order ingest, carrier API or EDI flows, no runbooks, no incident process, and night incidents closed personally by the CTO after a phone call — 45 to 90 minutes to respond. A four-hour failure in the peak window stops outbound entirely and costs $80,000–$120,000 in SLA penalties (C09).

What we put in its place was OpenTelemetry across the services with business signals on one dashboard — order throughput, carrier API latency, EDI 945 success rate, scan event frequency, shipment creation rate — and an SLO per critical integration — warehouse order sync, carrier rate APIs and EDI 945 — with error budgets reviewed monthly. Then tiered on-call across three time zones, forty-plus runbooks, and monthly chaos exercises.

P1 MTTR went 45–90 minutes → 11 minutes. Availability of critical integrations reached 99.6% against 94.1% the year before. 100% of night and weekend incidents were closed with no client involvement, and one night carrier-connectivity failure was caught in time to avoid $90,000 in SLA penalties (C09).

The position: an SLO on the warehouse order sync is worth more than an uptime number on a server, because latency is the thing that turns into a missed cutoff and uptime is not. Instrument the promise you sold, not the machine you rented.

Four measurements, all of which you can take without hiring anyone.

The p95 of the query behind your busiest operational screen, at peak, against the same query at 3am. Not the average — the average is dominated by the fast case and will tell you everything is fine. If the ratio is worse than a few times, that is your answer.

The age of your oldest table statistics and the last time autovacuum completed a full pass on your largest table. If nobody knows, that is also an answer.

Dock appointments that slipped past their window with no incident logged. This is the number that converts a software problem into detention and into your customer’s OTIF scorecard, and it is usually sitting in the yard or dock scheduling system already, unqueried.

Overtime hours divided by orders shipped, by month, for twenty-four months. The cheapest of the four and the most persuasive to the person who approves the budget.

One caveat, stated honestly. If your WMS is a vendor product and the slow path is inside it, some of this is their problem and not yours to fix. Knowing precisely where that line falls is worth the measurement on its own — it changes what you are entitled to ask for on your next renewal.