How browserfoundry Serves 1M+ Web Searches a Week at a 600 ms p99

Short version: browserfoundry's search tier (/search/lite and /search/fast) served 1,084,000 requests last week with a server-side p99 of 587 ms. Volume isn't the hard part. The hard part is the tail: every search depends on upstream engines we don't own, and their slowest 1% is several times slower than their median. We get from a 1.48 s p99 to under 600 ms with five things, in order of how much they bought us: hedged requests with a delay pinned to the primary engine's rolling p90, latency-aware routing across engines and egress, a freshness-classed cache with request coalescing, deadline propagation so no stage spends time the request no longer has, and admission control that sheds load at the gateway rather than letting queues build. This post walks the request path stage by stage, with the numbers.

What's in scope: the two search endpoints that agents call inside a loop, /search/lite and /search/fast. Deep search (15 to 50 seconds by design), search + content, answer and the agentic endpoint run on the same front half of the path but have their own latency targets, and they aren't in these numbers. Latency is measured server-side, from the moment the gateway accepts the request to the moment the last byte of the response is written. Network time between you and our edge isn't in it; that's yours and the internet's.

Latency percentiles, before and after

• v3 · August
• v4 · September

Data

Percentile	v3 · August	v4 · September	Change
p50	248 ms	148 ms	−40%
p90	612 ms	341 ms	−44%
p99	1,480 ms	587 ms	−60%
p99.9	2,950 ms	1,020 ms	−65%

Throughput is not the hard part

Start with the arithmetic, because it sets expectations. A week has 604,800 seconds. A million requests a week averages 1.65 requests a second. Even the busiest hour, 20,412 requests during a customer's overnight batch job, is under six a second. Daytime peaks are around 12,400 an hour. The gateway is sized for 50 times the average, and none of this is a throughput problem. A single modest machine can terminate TLS, authenticate and serialize JSON at many times this rate.

One week of search traffic

Data

Day	Requests	Busiest hour (UTC)	Requests that hour
Mon	159,892	15:00	12,342
Tue	196,152	04:00	20,412
Wed	159,432	16:00	12,435
Thu	193,781	02:00	19,937
Fri	158,299	16:00	12,400
Sat	107,588	15:00	8,398
Sun	108,857	15:00	8,308

So why is a 600 ms p99 hard? Because a search request is not served from our own data. It's served by fanning out to upstream search engines and indexes over the public internet, through egress we manage, and those upstreams have a latency distribution we don't control. Measured on its own, a single engine looks like this:

• p50: 248 ms
• p90: 612 ms
• p99: 1,480 ms
• p99.9: 2,950 ms

The median is fine. The tail is six times the median, and it isn't a fixed set of slow queries you can find and fix: the same query to the same engine is fast one minute and slow the next. Variance comes from things like their cache state, their garbage collector, a cold TLS session on our side, a congested path between two networks, or an egress IP that a target has started to rate-limit.

Agents make the tail matter more than it does for a human searcher. An agent searches, reads, critiques and searches again, often 10 to 50 times per task. If 1% of calls take 1.5 seconds, a 30-call task has a 26% chance of hitting at least one of them (1 − 0.99³⁰). The p99 of one call becomes the p75 of the task. That's the number agent builders feel, and it's why we set the target at p99 and not at the median.

The design rule that follows: assume every upstream call can be slow, and make slowness cheap to route around.

The request path

Every search request takes the same path. Each stage has a slice of the 600 ms budget, and everything to the right of the router runs under a deadline that only shrinks.

The request path

Data

Stage	What it does	Budget
Edge	TLS termination, region pin from X-Region	8 ms
Gateway	API key auth, per-key token bucket, adaptive admission	4 ms
Compiler	Normalize the query, assign a freshness class	3 ms
Cache	In-process L1, regional L2, request coalescing	2 ms
Router	Pick the primary engine by latency score and breaker state	1 ms
Engines	Primary call, hedge after the rolling p90, first valid answer wins	≤ 520 ms
Merge	Validate, dedupe, rank, serialize	12 ms

The budget is deliberately lopsided. Our own code, from edge to router and from merge to the last byte, gets about 30 ms in total. The remaining ~570 ms belongs to the upstream call and its hedge. Every millisecond our stages waste comes straight out of the time we can give a slow engine before we have to give up on it, so the fixed stages are held to their budgets in CI with a latency regression test on a replayed sample of production queries.

The rest of this post goes through the stages in order.

Edge and gateway: admission before work

The edge terminates TLS, reads the X-Region header if you've pinned one (EU, US or APAC), and forwards to a gateway in that region. Requests without a pin go to the nearest region by latency.

The gateway does three things before any real work happens.

1. Authenticate. API keys are looked up in an in-process cache backed by the regional store, so auth costs a hash and a map read on the hot path. Revoking a key is propagated with a pub/sub message, not by waiting for a TTL.

2. Rate-limit per key, per endpoint. Every key has a token bucket per endpoint, sized by plan. Buckets let you burst (an agent that fires 20 searches at once is normal) and refill at a steady rate. When a bucket is empty you get a 429 with a Retry-After header that tells you exactly when the next token lands, plus X-RateLimit-Remaining and X-RateLimit-Reset on every response so well-behaved clients never hit the wall.

3. Admit or shed. This is the one that protects the p99. Rate limits protect us from one customer; admission control protects every customer from the system's own load. The gateway keeps an adaptive concurrency limit per region: the number of requests it will allow in flight at once. The limit moves with measured latency, using a gradient in the style of Netflix's concurrency-limits library:

// Runs every 250 ms per region. minRtt is the lowest latency seen recently
// (the no-queueing baseline); sampleRtt is this window's p90.
function nextLimit(limit: number, minRtt: number, sampleRtt: number): number {
  const gradient = Math.max(0.5, Math.min(1, minRtt / sampleRtt)); // < 1 when queueing
  const headroom = Math.sqrt(limit);                               // room to probe upward
  const next = limit * gradient + headroom;
  return Math.max(MIN_LIMIT, Math.min(MAX_LIMIT, Math.round(next)));
}

When latency rises because work is queueing, the gradient drops below 1 and the limit shrinks. When latency is at its baseline, the limit grows by about √limit each window and probes for more capacity. A request that arrives when the region is at its limit isn't queued. It's either spilled to another region that has room (if you haven't pinned one) or rejected straight away with a 503 and a short Retry-After.

Rejecting fast is the point. A queue converts overload into latency for everyone; a fast rejection converts it into a retry for a few. We learned this the hard way, which is the last section of this post.

Batch jobs and the slow endpoint never compete with interactive traffic: they enter through a separate lane with its own concurrency limit, and that lane yields whenever the interactive lane is above 80% of its limit. The two overnight spikes in the traffic chart are batch jobs; you won't find them in the latency chart.

The query compiler

Before we can cache or route a query, we need to know what it is. The compiler does two jobs in about 3 ms.

Normalization makes equivalent requests produce the same cache key. It applies Unicode NFKC, lowercases, collapses whitespace, strips tracking parameters from any URLs in the query, and sorts operators like site:. Then the request parameters that change the result (result count, mode, region, language, includeMedia) are folded into the key. Parameters that don't change the result, like a client's request ID, are left out:

const key = sha256([
  "v4", endpoint, region, lang,
  normalize(query),            // NFKC, lowercase, whitespace, operator order
  bucket(maxResults),          // 1-10, 11-20, 21-50: fetch the bucket, slice per request
  mode,
].join("\u0000"));

Bucketing maxResults is a small trick with a big payoff. Clients ask for 5, 8, 10 or 20 results more or less at random. Fetching the top of a bucket and slicing per request lets all of those share one cache entry.

Classification assigns each query a freshness class, which decides how long a cached answer is allowed to live. A small classifier handles it, using features like time-sensitive terms ("today", "latest", a year), named-entity types, the presence of a price or a version number, and how often the same normalized query is seen. It's tuned to fail toward fresh: a query that might be news is treated as news.

The cache: two tiers and one flight per key

The cache has two tiers. L1 is an in-process LRU on each gateway instance: small (tens of thousands of entries), sub-millisecond, and lost on deploy. L2 is a regional key-value store shared by every instance in the region: larger, and around 1 to 2 ms away. A lookup checks L1, then L2, then goes upstream. Hits are written back to L1 so the next request on that instance doesn't leave the process.

Entries are stored with two timestamps, fresh until and usable until. Between the two, we serve the cached answer immediately and refresh it in the background (stale-while-revalidate). The caller gets a cache-hit latency, and the next caller gets a fresh result. Classes with no revalidate window (news, long-tail) are never served stale.

Cache hit rate by query class

Data

Query class	Hit rate	TTL
Navigational	71%	6 h
Evergreen reference	52%	24 h
Product and pricing	34%	2 h
Long-tail agent questions	12%	1 h
News and live events	9%	5 min

Weighted by traffic, 31% of search requests are cache hits, and a hit is answered in about 20 ms end to end. That's the lower step in the latency distribution further down. The hit rate is lower than you'd see for a consumer search engine because agent queries are long and specific: half of all queries we see are seen exactly once a week.

Request coalescing matters more than the hit rate suggests. When an agent framework retries, or many copies of the same agent run in parallel, identical queries arrive within milliseconds of each other, before the first one has filled the cache. Without coalescing, each copy goes upstream. With it, the first request for a key becomes the leader and later ones wait on its result:

const inflight = new Map<string, Promise<Result>>();

async function coalesced(key: string, fetch: () => Promise<Result>): Promise<Result> {
  const running = inflight.get(key);
  if (running) return running;                 // join the flight already in the air
  const p = fetch().finally(() => inflight.delete(key));
  inflight.set(key, p);
  return p;
}

This is per instance. Across instances, an L2 lock with a 50 ms lease does the same job, so a burst of 40 identical queries spread across a region's instances turns into one upstream call and 39 short waits.

Coalescing has one sharp edge: if the leader is slow, everyone waiting on it is slow too. So waiters don't wait on the leader forever. They inherit their own deadline (next section), and when it's close they detach and make their own call.

Deadlines that only shrink

Every request carries an absolute deadline, set once at the gateway: accept time plus 600 ms, minus a safety margin for serialization and the write. Every stage reads the time remaining and does only what fits:

type Ctx = { deadline: number; signal: AbortSignal };

const remaining = (ctx: Ctx) => ctx.deadline - performance.now();

async function callEngine(ctx: Ctx, engine: Engine, q: Query): Promise<Result> {
  const budget = remaining(ctx) - MERGE_RESERVE_MS;   // keep time to merge and write
  if (budget < engine.minUsefulMs) throw new DeadlineExceeded();
  return engine.search(q, {
    timeoutMs: budget,
    signal: AbortSignal.any([ctx.signal, AbortSignal.timeout(budget)]),
  });
}

Three rules follow from this.

Timeouts are derived, never configured. No stage has a fixed "5 second timeout". A stage that starts late gets less time, so a slow early stage can't push the whole request past its deadline.

Work that can't finish isn't started. Each engine has a minUsefulMs, the shortest time in which it has historically been able to answer. If less than that remains, calling it only adds load, so we don't.

Cancellation propagates. When the deadline passes, or when another engine has already answered, the abort signal fires and the outstanding upstream connections are torn down. Without that, a hedged system quietly does all the work twice.

When the deadline does pass without a valid answer, the response is a 503 with a Retry-After. It is never a 200 with an empty result list. That rule has a story attached, at the end.

Routing: picking the engine most likely to be fast

A cache miss goes to the router, which picks the primary engine for this query. We run several upstream engines and indexes for the search tier, and for any query class some are better than others. "Better" here means likely to return a valid, complete result soon.

Each (region, engine, query class) triple keeps two numbers:

• an exponentially weighted moving average of latency, with a short half-life (about 30 seconds) so it reacts quickly to a degrading upstream,
• a success rate over the same window, where "success" means a valid result (more on validity below), not just an HTTP 200.

The score is the expected latency, penalized by the failure rate, since a failure costs a retry:

const score = (s: Stats) => s.ewmaLatencyMs / Math.max(0.05, s.successRate) + s.inflight * s.ewmaLatencyMs * 0.1;

The last term adds a load penalty for requests already in flight to that engine from this instance. Without it, every router would pile onto whichever engine looks fastest and immediately make it slow.

Picking the single best score has a well-known failure mode: herding. Every instance sees the same "best" engine and sends everything there at once. We use the power of two choices instead: sample two eligible engines at random (weighted by their quality for this query class) and send the request to the one with the better score. It's nearly as good as always picking the best, and it spreads load enough to stop herding entirely.

An engine is only eligible if its circuit breaker is closed. Each (region, engine) pair has a breaker with the usual three states. Closed means normal operation. Open means the engine is skipped entirely, which happens when its valid-result rate over the last 10 seconds falls below 80% with at least 50 samples. Half-open comes 5 seconds after opening: 5% of eligible traffic is let through as probes, and the breaker closes once the probes succeed. Breakers trip on valid-result rate rather than error rate, which catches an upstream that is "up" but returning empty pages.

Hedging: the biggest single win

Routing makes the median better. Hedging is what fixes the tail.

The idea comes from Dean and Barroso's The Tail at Scale. Send the request to the primary engine. If it hasn't answered after a delay d, send a second copy (the hedge) to the next-best engine. Take the first valid answer and cancel the other. If d is set at the primary's p90, only 10% of requests ever send a hedge, and those 10% are exactly the ones that were heading for the tail.

The latency of a hedged request is roughly min(T₁, d + T₂), where T₁ and T₂ are the two engines' latencies. Because the engines' slow moments are close to independent (they're different systems on different networks), the chance that both are slow is the product of two small probabilities. That's what collapses the tail:

Where the tail went

• One upstream engine
• Full path (cache + routing + hedging)

Data

Percentile	One upstream engine	Full path
p50	248 ms	148 ms
p90	612 ms	341 ms
p99	1,480 ms	587 ms

Choosing d is a trade between tail latency and upstream load. Smaller delays cut the p99 further but send more duplicate requests, and duplicates cost money and count against upstream rate limits. We swept d in a shadow test, replaying a day of mirrored traffic against both engines without returning the shadow results to anyone:

Choosing the hedge delay

Data

Hedge delay	p99	Extra upstream load
150 ms	431 ms	+58%
250 ms	482 ms	+26%
350 ms	546 ms	+14%
420 ms	587 ms	+10%
500 ms	655 ms	+7%
600 ms	746 ms	+5%

The curve flattens below about 400 ms. Going from 420 ms to 150 ms would buy another 156 ms of p99 at almost six times the extra load. We run at 420 ms, which on a typical day is the primary engine's rolling p90.

In production d isn't a constant. It's recomputed every 10 seconds per (region, engine) from a rolling HDR histogram, so when an engine slows down its hedge fires sooner in absolute terms while still firing on about 10% of its requests:

// per (region, engine), refreshed every 10 s from a 60 s rolling histogram
const hedgeDelayMs = (h: Histogram) => clamp(h.valueAtPercentile(90), 120, 450);

async function hedged(ctx: Ctx, primary: Engine, backup: Engine, q: Query) {
  const race = new AbortController();
  const c = { ...ctx, signal: AbortSignal.any([ctx.signal, race.signal]) };
  const first = callEngine(c, primary, q).then(validate);
  const second = sleep(hedgeDelayMs(primary.hist), c.signal)
    .then(() => hedgeBudget.take() ? callEngine(c, backup, q).then(validate) : never());
  try {
    return await Promise.any([first, second]);   // first *valid* result wins
  } finally {
    race.abort();                                 // cancel the loser's connection
  }
}

Two safeguards keep hedging from making a bad day worse.

A hedge budget. Hedges draw from a token bucket that refills at 15% of the primary request rate. In normal conditions about 10% of requests hedge and the bucket stays full. When an upstream degrades and every request starts looking slow, the bucket empties and hedging stops amplifying load at exactly the moment the upstreams can least afford it. Without the budget, hedging turns a partial outage into a full one, because it doubles traffic to systems that are already struggling.

Promise.any, not Promise.race. The first settled result isn't good enough; we want the first valid one. If the primary returns a fast empty or malformed page, validate throws, and we keep waiting for the hedge. A fast wrong answer is worse than a slightly slower right one.

"Valid" is not "200"

validate is small and boring, and it's the most important function in the path. An upstream answer counts as valid only if all of these hold:

• the payload parses against the schema for that engine,
• the result list is non-empty, unless the query was a site-restricted or exact-match query where zero results is a legitimate answer,
• the results aren't a known block page, CAPTCHA interstitial or consent wall (fingerprinted by structure, not by text, so it survives localization),
• at least half the results have a title, a URL and a snippet,
• the result set isn't byte-identical to the previous answer from that egress IP for a different query, which is the signature of a cached block page.

An invalid answer counts as a failure for the router's success rate and the circuit breaker, and the request carries on to the hedge or a retry within its deadline. Only results that pass go to the merge stage, which dedupes by canonical URL, ranks, slices to the requested maxResults and serializes. Merge has a 12 ms budget, and it's the one stage that has never been near it.

Egress: the milliseconds before the request

Before any upstream can answer, we have to reach it. For an upstream call, the cold path pays for DNS, a TCP handshake, a TLS handshake and, through a proxy, a CONNECT round trip, all before the first byte of the query is sent. Across continents that can cost more than the upstream's own median.

So the egress layer works hard to never be cold:

• Connection pools are pre-warmed. Each instance holds open HTTP/2 connections to every upstream it routes to, kept alive with periodic pings, and requests are multiplexed over them. A new instance warms its pools before it's put in rotation, so a deploy or a scale-up doesn't create a cold-start tail.
• TLS sessions are resumed when a connection does have to be rebuilt, and DNS answers are cached in-process for their TTL, with the next lookup done in the background before they expire.
• Egress IPs are scored, much like engines are. Each IP in the pool tracks its valid-result rate and latency per upstream. An IP that starts drawing block pages from one upstream is rested for that upstream (not for all of them) and put back once probes succeed. Sessions that need a stable IP keep one; everything else rotates.

None of this is novel, but it's worth around 100 ms at the median of cache misses, and it's the reason the egress line in the latency budget stays close to zero.

Three providers, none above 45%

The compute under all of this runs on three independent providers, and no provider carries more than 45% of traffic. That cap is the lesson from July: when a single provider is underneath everything, its bad day is your bad day, whatever the architecture above it.

• Active health probes every 5 seconds, per region and per provider. These are synthetic searches through the full path, not TCP pings, so they catch a gateway that is up but can't reach anything.
• Per-region circuit breakers at the routing layer between regions. A region that's failing probes stops receiving spilled traffic from other regions, so trouble in one region can't spread to the others by overflow.
• Pre-warmed standby capacity. Spare instances on a second provider run with warm pools and filled L1 caches, and take traffic within 20 seconds of a breaker opening. Cold standby would take minutes and would start with empty caches, a cold-start tail on top of an incident.
• Queue backpressure everywhere. Every queue has a bound. When it's full, the producer gets an error immediately rather than an unbounded wait.

Region pinning with X-Region is honored through all of it. A request pinned to the EU fails over between providers within the EU and never spills to another region. If you pin, you trade some resilience for data residency, and the API makes that trade explicit rather than quietly breaking it.

Measuring p99 honestly

A p99 is easy to get wrong in a way that flatters you. Four rules we follow:

Never average percentiles. The p99 of a week is not the mean of 10,080 per-minute p99s. Averaging hides exactly the bad minutes you care about. Each instance records latency into an HDR histogram (3 significant digits, 1 µs to 60 s); histograms are shipped every minute and merged, and percentiles are only computed from the merged histogram. Every number in this post comes from merged histograms.

Measure what the client waits for. The clock starts when the gateway accepts the request, not when a worker picks it up. Starting late leaves out time spent waiting in a queue, which is exactly the latency that grows under load. It's the gateway's view of coordinated omission.

Count failures in the latency. A request that times out at 600 ms and returns a 503 is recorded at 600 ms, not dropped from the histogram. Dropping failures makes the p99 improve during an outage.

Look at the tail per customer. A global p99 can be fine while one customer's workload, with long queries or an unusual region, sits at 2 seconds. We track the p99 per API key, and the on-call dashboard sorts customers by how far their p99 is above the global one.

Alongside real traffic, synthetic probes run fixed query sets from each region every 15 seconds. Real traffic tells us what customers saw. Probes tell us about the quiet hours, when there isn't enough traffic for a percentile to mean much.

What broke, and what we changed

Between July 28 and 30, an upstream compute provider lost an availability zone to a fiber cut, and our EU-West region went from healthy to returning errors on 34% of requests. We published the full report; here is the shape of it:

July 28–30: the incident that shaped v4

Data

Time (UTC)	What happened
Jul 28 14:22	latency anomaly, no drops
Jul 28 16:05	retries mask TCP resets, <1%
Jul 29 06:55	SEV-1 declared, 34%
Jul 29 08:22	Provider B online, 6%
Jul 29 12:30	US-East restored, <1%
Jul 30 09:00	fully stable

Three failures lined up, and each one maps to a part of the design above.

One provider underneath everything. All three regions ran on a single upstream provider. When it lost capacity in US-East, there was nowhere independent to go. → Three providers, none above 45%, with warm standby.

Overflow without backpressure. Traffic from the degraded region was rerouted to EU-West, which had no limit on how much it would accept. Its queues grew to 12 times their normal depth, and its p99 went past 8 seconds. → Adaptive admission at the gateway, bounded queues, and per-region breakers that stop spill into a struggling region.

Timeouts dressed as success. When EU-West timed out, a bug in the error path returned HTTP 200 with an empty result list. Clients that check status codes, which is most of them, saw success and moved on with no data. This was the worst part of the incident, because the failure was invisible to the people it was hurting. → A deadline that ends in a 503 with Retry-After, never an empty 200, and a validator that treats an unexpected empty page as a failure all the way up.

The 34% peak lasted about 75 minutes, until standby capacity on a second provider came online and brought errors down to 6% within 12 minutes. That standby was cold in July. It's warm now.

What's next

The path above gets the search tier to a 587 ms p99. Three projects aim to move it further:

• Speculative prefetch for agent loops. Agents are predictable: after searching for X, they often search for a narrower version of X. We're testing whether prefetching the likely next query into L1, within the same customer's rate limit, turns a share of the next misses into hits.
• Per-class hedge delays. One hedge delay per engine leaves value on the table. News queries are more variable than navigational ones. Splitting the rolling histogram by query class should let us hedge earlier where it pays and later where it doesn't.
• Pushing the p99.9 under a second. p99.9 is at 1,020 ms today. Most of what's left is requests where both engines were slow, which points at a third, cheaper engine for a last-chance hedge under its own small budget.

If you're building an agent and the tail of your search calls is hurting your task times, the free tier has 1,250 credits a month, and every response includes latency_ms so you can check these numbers against your own traffic.