Text engine
tantivy-based autocomplete: text-analysis parity, then what full production scale actually costs and how that cost was cut.
What's working (crates/farnsworth-text-spike)
Tested against a real 4,315-doc POI sample (FR/GB/US):
- NFKC normalization + lowercase + accent folding (OpenSearch's
icu_normalizer/icu_foldingequivalent) - Native prefix matching (
PhrasePrefixQuery) and typo tolerance (FuzzyTermQuery) - "cheeky caf" correctly ranks "Cheeky Cafe" top; "shcool" (typo) still finds schools - Index-time-only synonym expansion via a custom
TokenFilter(position-preserving, not a string hack) - "st joseph" matches "Saint-Joseph" et al., mirroring OpenSearch'ssynonym_graphbeing onindex_analyzeronly - A merged multi-language field (one field, every language variant as a separate occurrence) instead of OpenSearch's N-separate-fields-per-language approach - "united states" and "états-unis" return identical results through a single query. This directly targets the root cause found in the baseline spike: one query clause per token instead of N per-language branches, so the fan-out cost disappears structurally.
Full-scale result (13.9M real POIs, all countries). Build throughput is great: 102,168 docs/sec (136.2s for the full corpus). Peak memory: 38.4GB. Query latency did not hold up at first glance - the same queries that ran in single-digit-to-low-hundreds of microseconds at 4,315 docs take 30-200 milliseconds at 13.9M docs.
Isolated with a dedicated benchmark (src/bin/bench.rs, 15 iterations per variant, median timing) rather than left as a guess: FuzzyTermQuery (typo tolerance) is ~97% of that cost.
| Variant | Median |
|---|---|
Prefix-only (PhrasePrefixQuery) |
3.3ms |
Fuzzy-only (FuzzyTermQuery) |
135.4ms |
| Combined | 143.3ms |
Prefix matching - the actual autocomplete/type-ahead mechanism, the thing that matters most for the product - scales fine at full scale (comparable to or better than OpenSearch's own current p50). Typo tolerance is the entire bottleneck.
Tested two ways to tone the fuzzy cost down, one confirmed, one disproven - not guesses:
| Variant | Median | Max |
|---|---|---|
| Fuzzy, all tokens (baseline) | 287.6ms | 328.3ms |
| Fuzzy, skip tokens <=3 chars | 408.9ms - worse, and far more variable | 1.73s |
| Fuzzy, last token only | 88.5ms - ~3.25x faster | 477.3ms |
"Skip short tokens" was a wrong guess (dropping a short-token clause made the BooleanQuery slower and much more variable, not faster - not retrying that, reconfirmed unstable across a third run too). "Fuzzy-match only the last token" (the word most likely still being typed) is a real, measured win, and matches the UX case that actually matters.
Wired in and confirmed end-to-end. build_autocomplete_query now uses last-token-only fuzzy (the old behavior kept as build_autocomplete_query_naive for comparison); all 8 existing correctness tests still pass unchanged. Full-scale, production query builder, not isolated pieces:
| Variant | Median |
|---|---|
| Combined, naive | 147.6ms |
| Combined, tuned | 56.8ms - ~2.6x faster |
Autocomplete's core mechanism (prefix matching + typo tolerance on the word being typed) now runs in tens of milliseconds at full 13.9M-doc scale, not hundreds. Note: absolute numbers carry real run-to-run system noise (the same baseline measured 135-288ms across different runs) - the qualitative findings and win direction are stable across every run; exact multipliers are approximate.
Also confirmed, closing out the last text-analysis gap: word_delimiter_graph parity. Checked tantivy's SimpleTokenizer source directly - it splits on any non-alphanumeric character, keeps alphanumeric runs together, which is exactly what OpenSearch's real config does with every token-multiplying flag disabled. Proved with 6 tests against the actual edge cases (Saint-Germain -> [saint, germain], 3M -> [3m] not split, McDonald's -> [mcdonald, s], 24/7 -> [24, 7], case changes don't split, and a real search for "saint germain" correctly finding "Saint-Germain-des-Prés"). No custom filter needed - tantivy's default already matches.
Memory, decomposed (build_persistent.rs / serve_only.rs)
The 38.4GB build-time peak RSS from the full-scale run was one opaque number. Dug into what it's actually made of.
First hypothesis (wrong, corrected honestly): suspected a double-storage bug like the one fixed in the geo spike (records held alive via borrow instead of being consumed). Fixed it (into_iter/into_values) - no measurable improvement (38.4GB -> 40.98GB, within noise). Unlike the geo case, this loop never actually cloned the data twice; the fix only changed when memory was freed, not whether it existed twice, and tantivy's own writer buffers dominate peak memory regardless of that timing.
Real cause, found by isolating a minimal build (name + public_id only, no multi-language field) against the same full corpus:
| Measurement | Value |
|---|---|
| Build-time peak RSS, name-only | 2.69GB |
| Build-time peak RSS, name + multi-language field | ~38-41GB |
| On-disk index size (LZ4) | 1.45GB (111.6 bytes/doc) |
| On-disk index size (Zstd) | 1.31GB (101.2 bytes/doc, ~9.4% smaller, free) |
| Steady-state serving RSS (fresh process, mmap-open, real queries) | 114MB |
The ~15x gap between name-only and name+multi-language builds is the real cost of indexing ~20 language variants per document across 13.9M docs - genuine extra work, not a bug. But the number that actually matters for production is the last row: opening the persisted index in a completely fresh process (no JSON loading, no indexing) and running real queries costs 114MB RSS, not gigabytes - mmap only pages in what queries actually touch. Build cost and serving cost are entirely different budgets; every earlier "how much memory does this need" number in this investigation was a build-time number, not a serving one.
Zstd docstore compression (vs tantivy's default LZ4) is a free ~9.4% size win - it only touches stored-field retrieval, not the hot query-matching path. No further free compaction is available: shrinking further would mean dropping position indexing (breaks PhrasePrefixQuery, autocomplete's core mechanism) or dropping stored text (breaks returning result names) - real feature trade-offs, not free lunches.
Concurrent throughput (concurrent_bench.rs)
Tested whether the 114MB steady-state number and single-query latency hold under real concurrent load, on an 18-core machine:
| Threads | Throughput | p50 | p99 |
|---|---|---|---|
| 1 | 27 q/s | 35ms | 70ms |
| 2 | 47 q/s | 40ms | 101ms |
| 4 | 90 q/s | 43ms | 61ms |
| 8 | 150 q/s | 51ms | 78ms |
Throughput scales sub-linearly (5.56-6x at 8 threads on 18 cores, not 8x). Verified WHY rather than guessing: a first check (system-wide top sample) looked like it contradicted CPU-boundedness (~85% idle) - turned out to be a timing bug in the check itself, sampling after the run had already finished. The reliable check: wrapping the whole run in /usr/bin/time -l gives precise total CPU-seconds, not a timing-sensitive snapshot - 143.74 user-CPU-seconds measured against a 150-CPU-second theoretical ideal (1+2+4+8 threads x 10s, if every thread did real work the whole time) = 95.8% of ideal. That rules out lock contention or blocked-waiting (which would show well below ideal) and confirms this is genuinely CPU-bound. The sub-linear throughput instead comes from each individual query getting modestly slower under 8-way parallelism (p50 34ms -> 46ms, +35%) - consistent with memory/cache-bandwidth pressure from multiple cores sharing a large (~1.3GB) mmap'd structure, a normal effect at higher concurrency, not a bug.
Memory stays essentially flat: ~141MB peak across the whole concurrent run, barely above the 114MB single-threaded number - confirms empirically (not just assumed from tantivy's Send+Sync design) that mmap pages are genuinely shared across threads by the OS, not duplicated per-thread. Serving concurrent load doesn't multiply memory cost; it does hit a real, expected CPU/cache-bandwidth ceiling worth knowing for capacity planning.
Exact matches were tying with same-prefix substrings of longer words
Found against the real 8.8M-record localities dataset (see localities.md): search("paris", 5) returned "Hamilton Parish" ahead of the actual city "Paris". Root cause, verified by printing raw scores, not guessed: PhrasePrefixQuery/FuzzyTermQuery/a basic TermQuery all score a match as a flat constant - no BM25 term-frequency/length differentiation between an exact match and a same-prefix substring of a longer word. Every real hit for a plain query came back with the identical text_score (2.0 for a two-clause match) regardless of match quality - "parish" legitimately starts with "paris", so it matched the same two clauses (prefix + fuzzy) as "Paris" matching itself, and text score alone couldn't tell them apart. Whatever tie-break axis happened to be active (type_boost, popularity) was doing ALL the differentiation, silently, the entire session - this was never actually BM25-flavored relevance ranking.
Fixed in build_autocomplete_query (farnsworth-text-spike): an extra Occur::Should clause with a BoostQuery-wrapped exact TermQuery on the last token, EXACT_MATCH_BOOST = 5.0. An exact whole-token match now outscores a prefix/fuzzy-only match outright, before any tie-break axis even runs - "paris" now correctly returns "Paris"/"París" first (text_score jumped from the flat 2.0 to 70.5 on the real dataset), with type_boost only still breaking ties among the genuinely-exact matches (admin-level "Paris" over city "Paris" - a real, correct outcome of production's own sort chain, not a bug).
Query-time fix, no reindex needed - EXACT_MATCH_BOOST lives in the query construction, not the persisted index, so the existing ~/.cache/farnsworth-data/world-index benefited immediately (poi buckingham now correctly ranks the exact match "Buckingham" first, was previously tied with every "Buckingham "-prefixed POI).
Country and type filtering
Real production filters pois on exactly two recall axes (localities/elasticsearch/builders/poi_query_builder.py): Terms("country_codes", values=...) (skipped when the parsed query is worldwide) and build_filters_for_parsed_types (include via a Terms clause, exclude via must_not on sub_types). Farnsworth had zero filtering before this - country_code was a schema field that existed but was never populated, sub_types didn't exist on the schema at all.
PoiSearchIndex::search() takes a SearchFilters<'a> { country_codes, include_sub_types, exclude_sub_types } (all &[String], empty = no-op, matching every other skew/filter's "default changes nothing" convention). Both country_code and the new sub_types field are written once per real value on a document - the same multi-occurrence-per-field pattern the merged multi-language field mechanism already used - so a POI genuinely spanning two countries (a real border-region case) or carrying multiple real sub-types matches a filter on ANY of its values, via TermSetQuery.
Filters are AND-ed directly into the query itself (Occur::Must/Occur::MustNot clauses alongside the text query), not applied after TopDocs::with_limit - the same over-fetch-pool lesson popularity's fix already taught: a post-hoc filter can exclude a valid match the collector never even considered (the Buckingham Palace bug, on a filtering axis this time instead of a ranking one).
farnsworth-cli's REPL exposes it: poi <query> in <cc,cc> types <t,t> not <t,t>.
Matching on more than just the name
Direct feedback: LocalitiesIndex/GeocodeIndex only ever matched a query against a record's name (or street_name) - a query like "nantes france" or a postal code couldn't find a record whose own name doesn't contain that text, even when it's genuinely the right place. Checked what real production actually matches against (textual_localities_query_builder.py's build_full_context) rather than guessing: production corroborates the primary name match against full_indexed_string, a real per-language field already in the export data - "name § locality § region § country" for a locality (e.g. "Paris § Brant § Ontario § Canada"), "street, postal code, region" for an address (e.g. "Route de Dévé, 97133, Saint-Barthélemy"). It was already in the parquet schema, just never imported.
Wired up by repurposing a field that turned out to have been sitting dormant: country_multilingual existed in the shared schema since the very first multi-language POI work, documented as "unused by the current importer" - checked and confirmed genuinely never queried anywhere either, not just unpopulated. Renamed to context and given real data: LocalityRecord/StreetRecord gained a context: Vec<String> (every full_indexed_string language variant), written once per value - the same multi-occurrence-per-field pattern as country_code/sub_types.
build_autocomplete_query gained an context_field: Option<Field> parameter. When given, it adds prefix + fuzzy-last-token clauses against the context field too - deliberately without the EXACT_MATCH_BOOST clause that applies to the primary name field. That boost is for an exact match on the thing itself; giving a context match the same weight would let a query that only touches a country/region name outrank a genuine exact name match, which production's own tiered design (name gating is the primary tier, full_indexed_string context only corroborates) avoids. PoiSearchIndex doesn't pass a context field yet - deferred, not yet wired, country_multilingual's original intended use case.
2 tests per index (locality.rs/geocode.rs) prove both halves of the real behavior: a query matching only the context field (a country/postal-code term absent from the name) still finds the record; a context-only match never outranks a genuine exact name match on a different record.
Wire protocol: a negative result, worth having
Compared serde_json vs bincode for a realistic AutocompleteResult-shaped payload (5 hits, crates/farnsworth-wire-spike), 100,000 iterations:
| JSON | bincode | |
|---|---|---|
| size | 1265 bytes | 780 bytes (62%) |
| round-trip | 2.9us/op | 1.0us/op |
bincode wins on paper (2.9x faster, 38% smaller). It doesn't matter: both are microsecond-scale, dwarfed by query execution (3.3-56.8ms) and the ~6ms transport overhead the baseline spike measured at p50. JSON serialization is roughly 0.1% of that 6ms - not where the overhead comes from. Building a custom wire protocol would solve a problem that doesn't exist at this scale; the baseline spike's transport overhead is more likely HTTP/connection-level (headers, handshake), not payload encoding.