Engineering RFC · Worked example
A cache that can't fix the tail
A six-node cache cluster proposed to bring catalog p99 under 150 ms. At the expected 90% hit rate, every miss still costs at least 384 ms.
- Objective
- p99 catalog read latency under 150 ms at 18,000 req/s.
- Hidden assumption
- That a 90% hit rate moves the p99. The slowest 1% of reads are all misses.
- Flip threshold
- Hit rate ≥ 99.0%. The RFC expects 90%.
The full document, both reviews
Source draft
591 charsRFC: Redis cluster in front of the catalog database
Platform team · status: proposed
Adversarial review
pinned · quoted · computedPinned objective
p99 catalog read latency under 150 ms at 18,000 req/s.
Underneath: The success metric is a tail percentile; the proposal reasons about the average request.
Hidden assumption
A 90% hit rate moves the p99. It does not: p99 is set by the slowest 1% of reads, and with 10% of reads missing, that slowest 1% is all misses.
“We expect a 90% hit rate based on the 80/20 access pattern we see in logs.”
If it’s wrong: The cluster ships, average latency falls, and the SLO still fails at the same 380–450 ms tail.
Flip threshold
Cache hit rate at which p99 can fall under 150 ms
| Miss path: Postgres + cache lookup | 384–454 ms |
| Every miss vs the 150 ms SLO | over, always |
| p99 ≤ 150 ms needs misses ≤ | 1.0% of reads |
| p99 at the expected 90% hit rate | ≥ 384 ms |
At 90% the SLO fails. The decision turns on hit rate above 99%, not on the cluster.
Cheapest test: Replay one peak hour of read logs against an LRU of the planned size and measure the miss rate before provisioning.
Is your next proposal built on a number like this?
Audit it before it ships. Free, two to three minutes, the draft is never stored.