Architecture

Design sketch, being validated incrementally against real production data. This is a learning project (see index), not a committed prod-replacement initiative - read-only, no write/patch-path port. Tracked as a shaped roadmap epic in the localities repo (roadmap/farnsworth/).

Status

• Baseline latency spike: done. The original premise (OpenSearch's p99 tail is a JVM/GC problem) was disproven by direct CloudWatch/Datadog evidence - zero old-gen GC collections, comfortable memory/CPU headroom. Root cause instead traced to query-plan cost scaling with token count in the per-language dis_max fan-out (confirmed via N_TOKENS-correlated latency: p99 climbs ~13x from 1 to 10+ tokens).
• milli spike: done, not viable. Compiles as a git dependency with two non-obvious workarounds (pinned rustc 1.98.1, forced smartstring/serde feature), but its indexing API is Meilisearch's raw server internals (LMDB txns, bump allocators, thread pools) - not a stable, documented embedding surface.
• tantivy text-analysis parity: done. Full-scale (13.9M real POIs) surfaced a real query-latency issue, which was isolated, fixed, and confirmed end-to-end - see below.
• Geo index: done, clean win at full production scale. See below.
• Major finding: ranking is not Farnsworth's job at all. See below - this significantly shrinks the project's real scope.
• Wire protocol spike: done. bincode wins on paper (2.9x faster, 38% smaller) but doesn't matter - serialization is microsecond-scale, dwarfed by query execution and whatever the real transport overhead actually is. See below.
• Integration capstone: done. Text search and geo filtering compose correctly into one search path - proven with tests, not just asserted. See below.

What's working on tantivy (crates/farnsworth-text-spike)

Tested against a real 4,315-doc POI sample (FR/GB/US):

• NFKC normalization + lowercase + accent folding (OpenSearch's icu_normalizer/icu_folding equivalent)
• Native prefix matching (PhrasePrefixQuery) and typo tolerance (FuzzyTermQuery) - "cheeky caf" correctly ranks "Cheeky Cafe" top; "shcool" (typo) still finds schools
• Index-time-only synonym expansion via a custom TokenFilter (position-preserving, not a string hack) - "st joseph" matches "Saint-Joseph" et al., mirroring OpenSearch's synonym_graph being on index_analyzer only
• A merged multi-language field (one field, every language variant as a separate occurrence) instead of OpenSearch's N-separate-fields-per-language approach - "united states" and "états-unis" return identical results through a single query. This directly targets the root cause found in the baseline spike: one query clause per token instead of N per-language branches, so the fan-out cost disappears structurally.

Full-scale result (13.9M real POIs, all countries). Build throughput is great: 102,168 docs/sec (136.2s for the full corpus). Peak memory: 38.4GB. Query latency did not hold up at first glance - the same queries that ran in single-digit-to-low-hundreds of microseconds at 4,315 docs take 30-200 milliseconds at 13.9M docs.

Isolated with a dedicated benchmark (src/bin/bench.rs, 15 iterations per variant, median timing) rather than left as a guess: FuzzyTermQuery (typo tolerance) is ~97% of that cost.

Prefix matching - the actual autocomplete/type-ahead mechanism, the thing that matters most for the product - scales fine at full scale (comparable to or better than OpenSearch's own current p50). Typo tolerance is the entire bottleneck.

Tested two ways to tone the fuzzy cost down, one confirmed, one disproven - not guesses:

"Skip short tokens" was a wrong guess (dropping a short-token clause made the BooleanQuery slower and much more variable, not faster - not retrying that, reconfirmed unstable across a third run too). "Fuzzy-match only the last token" (the word most likely still being typed) is a real, measured win, and matches the UX case that actually matters.

Wired in and confirmed end-to-end. build_autocomplete_query now uses last-token-only fuzzy (the old behavior kept as build_autocomplete_query_naive for comparison); all 8 existing correctness tests still pass unchanged. Full-scale, production query builder, not isolated pieces:

Autocomplete's core mechanism (prefix matching + typo tolerance on the word being typed) now runs in tens of milliseconds at full 13.9M-doc scale, not hundreds. Note: absolute numbers carry real run-to-run system noise (the same baseline measured 135-288ms across different runs) - the qualitative findings and win direction are stable across every run; exact multipliers are approximate.

Memory, decomposed (build_persistent.rs / serve_only.rs)

The 38.4GB build-time peak RSS from the full-scale run was one opaque number. Dug into what it's actually made of.

First hypothesis (wrong, corrected honestly): suspected a double-storage bug like the one fixed in the geo spike (records held alive via borrow instead of being consumed). Fixed it (into_iter/into_values) - no measurable improvement (38.4GB -> 40.98GB, within noise). Unlike the geo case, this loop never actually cloned the data twice; the fix only changed when memory was freed, not whether it existed twice, and tantivy's own writer buffers dominate peak memory regardless of that timing.

Real cause, found by isolating a minimal build (name + public_id only, no multi-language field) against the same full corpus:

The ~15x gap between name-only and name+multi-language builds is the real cost of indexing ~20 language variants per document across 13.9M docs - genuine extra work, not a bug. But the number that actually matters for production is the last row: opening the persisted index in a completely fresh process (no JSON loading, no indexing) and running real queries costs 114MB RSS, not gigabytes - mmap only pages in what queries actually touch. Build cost and serving cost are entirely different budgets; every earlier "how much memory does this need" number in this investigation was a build-time number, not a serving one.

Zstd docstore compression (vs tantivy's default LZ4) is a free ~9.4% size win - it only touches stored-field retrieval, not the hot query-matching path. No further free compaction is available: shrinking further would mean dropping position indexing (breaks PhrasePrefixQuery, autocomplete's core mechanism) or dropping stored text (breaks returning result names) - real feature trade-offs, not free lunches.

Concurrent throughput (concurrent_bench.rs)

Tested whether the 114MB steady-state number and single-query latency hold under real concurrent load, on an 18-core machine:

Throughput scales sub-linearly (5.56-6x at 8 threads on 18 cores, not 8x). Verified WHY rather than guessing: a first check (system-wide top sample) looked like it contradicted CPU-boundedness (~85% idle) - turned out to be a timing bug in the check itself, sampling after the run had already finished. The reliable check: wrapping the whole run in /usr/bin/time -l gives precise total CPU-seconds, not a timing-sensitive snapshot - 143.74 user-CPU-seconds measured against a 150-CPU-second theoretical ideal (1+2+4+8 threads x 10s, if every thread did real work the whole time) = 95.8% of ideal. That rules out lock contention or blocked-waiting (which would show well below ideal) and confirms this is genuinely CPU-bound. The sub-linear throughput instead comes from each individual query getting modestly slower under 8-way parallelism (p50 34ms -> 46ms, +35%) - consistent with memory/cache-bandwidth pressure from multiple cores sharing a large (~1.3GB) mmap'd structure, a normal effect at higher concurrency, not a bug.

Memory stays essentially flat: ~141MB peak across the whole concurrent run, barely above the 114MB single-threaded number - confirms empirically (not just assumed from tantivy's Send+Sync design) that mmap pages are genuinely shared across threads by the OS, not duplicated per-thread. Serving concurrent load doesn't multiply memory cost; it does hit a real, expected CPU/cache-bandwidth ceiling worth knowing for capacity planning.

What's working on geo (crates/farnsworth-geo-spike)

Tested at full production scale: 13,914,016 real POIs, all countries, no sampling shortcuts.

• rstar (R-tree), two-step query design matching how real geo search engines (including OpenSearch's geo_distance internally) do it: a fast, coarse degree-based bounding-radius prune via locate_within_distance, then an exact haversine distance filter/sort on the smaller candidate set.
• Build: the full 13.9M-point corpus indexed in 8-9s.
• Memory: peak RSS 13.76GB for the full corpus (corrected down from an initial 15.5GB after fixing a real double-storage bug in the spike - see the roadmap for detail).
• Query: NYC-area radius search (1/10/50km) returns correctly distance-sorted real POIs in 3-29ms. Sparse rural Montana queries in sub-100 microseconds, correctly scaling hit count with radius (0 -> 2 -> 5 hits).
• Point-in-polygon (geo crate's Contains, real GeoJSON admin boundaries): 3,000/3,000 shapes parsed with zero failures, 99.8% self-containment sanity check, correct reverse point-in-polygon lookups.

This is the piece an independent fresh-eyes review flagged as standing on its own merit regardless of the text-engine outcome - unlike the JVM/GC-removal argument, which the baseline spike disproved. The full-scale result confirms that read cleanly: unlike text search, geo held up completely at real production scale with no surprises.

Major finding: ranking is not Farnsworth's job

Read localities' actual ranking implementation directly (localities/suggestions/processor/, not just docs). OpenSearch's native _score (BM25/dis_max/rank_feature) is discarded entirely for autocomplete/geocode ranking - it only affects recall (which candidates come back at all). Actual relevance ordering is a fully separate, engine-independent step: every candidate runs through DEFAULT_PIPELINE, a chain of Python scorers that re-tokenize the returned field values and fuzzy-compare them token-by-token against the input, producing a composite 0-1 coverage score - computed identically regardless of whether the candidate came from OpenSearch or HERE.

This reframes Farnsworth's real scope, for the better:

• rank_feature/distance_feature-equivalent scoring is low priority - it only needs to be good enough for recall, not final relevance ordering
• Porting anything to Farnsworth does not require replicating OpenSearch's scoring formula or matching BM25 numerically. It requires good recall (the right candidates in the pool) and returning correct field values in the expected shape - the existing Python ranking pipeline can run unchanged on top
• Geo ranking is similarly unaffected: LocationScorer's geo_bias_score is also computed downstream, independent of the engine - consistent with the geo spike already being evaluated purely on recall/distance-filtering correctness

Also confirmed, closing out the last text-analysis gap: word_delimiter_graph parity. Checked tantivy's SimpleTokenizer source directly - it splits on any non-alphanumeric character, keeps alphanumeric runs together, which is exactly what OpenSearch's real config does with every token-multiplying flag disabled. Proved with 6 tests against the actual edge cases (Saint-Germain -> [saint, germain], 3M -> [3m] not split, McDonald's -> [mcdonald, s], 24/7 -> [24, 7], case changes don't split, and a real search for "saint germain" correctly finding "Saint-Germain-des-Prés"). No custom filter needed - tantivy's default already matches.

Wire protocol: a negative result, worth having

Compared serde_json vs bincode for a realistic AutocompleteResult-shaped payload (5 hits, crates/farnsworth-wire-spike), 100,000 iterations:

bincode wins on paper (2.9x faster, 38% smaller). It doesn't matter: both are microsecond-scale, dwarfed by query execution (3.3-56.8ms) and the ~6ms transport overhead the baseline spike measured at p50. JSON serialization is roughly 0.1% of that 6ms - not where the overhead comes from. Building a custom wire protocol would solve a problem that doesn't exist at this scale; the baseline spike's transport overhead is more likely HTTP/connection-level (headers, handshake), not payload encoding.

Integration capstone (crates/farnsworth-core)

Combines the text index and geo filtering into one query path (PoiSearchIndex) - a single in-process Rust type, not a service. Text-match candidates first (over-fetched), then geo-filter to within a radius of an optional bias point. Matches the ranking finding above: geo bias affects recall (exclusion), not final ordering.

4 tests prove the composition works, not just each piece alone:

• Two identically-named POIs (Paris vs NYC) - the geo filter correctly excludes the far one
• A geographically adjacent but textually unrelated POI is correctly excluded - proximity alone isn't enough
• A real match gets included with the correct distance
• No bias point cleanly falls back to pure text search

Real-data demo (src/bin/demo.rs) against the 12,724-POI sample: "cafe" with no bias returns matches spread across the whole sample; biased near NYC at 10km correctly narrows to the one actual nearby cafe; tightening to 1km correctly returns 0 (too tight for this sparse sample, not a bug).

Full workspace: 32/32 tests passing (4 geo + 4 polygon + 6 interpolation + 4 core + 14 text-spike). This is as far as the learning project's read-only scope goes - write-path, shadow deployment, and prod cutover remain explicitly out of scope.

Address interpolation (crates/farnsworth-core/src/bin/interpolation.rs)

The full 11-country address migration was too large for this learning project's remaining scope - bounded instead to validating the one genuinely new mechanic: house-number interpolation, against real French address data.

Read localities' actual algorithm directly (localities/suggestions/suggestion.py), not guessed: filter known addresses to the target number's parity (falling back to the other parity if empty), sort, binary search, and linearly interpolate lat/lon between the two bracketing known addresses by numeric ratio - a straight line between two known points, not geometric interpolation along street geometry. Out-of-range numbers snap to the nearest known endpoint.

Reproduced exactly in Rust and validated with holdout testing against real data: for every street with 3+ known addresses of the same parity, hold one out, interpolate it back from its remaining neighbors, measure the real error against its actual location. 48,312 holdout predictions across 3,836 real French streets:

Not a Farnsworth-specific finding - this is the accuracy profile of the algorithm itself, wherever it runs, measured with real ground truth for the first time. Typical accuracy is good (building-scale). The tail reveals a real, general limitation: some streets have numbering that doesn't correlate smoothly with geographic position, where linear number-based interpolation can be meaningfully wrong regardless of engine.

Details and geocode (crates/farnsworth-core)

Asked directly: how likely are autocomplete/details/geocode to actually work on Farnsworth? Honest answer at the time - autocomplete strong (this session's main focus), details untested but low-risk, geocode a real gap (text matching and interpolation math both proven separately, never connected). Built both missing pieces.

Details (PoiSearchIndex::details): exact lookup by public_id via a TermQuery on the already-indexed field - the same mechanism a real point lookup would use, not a side shortcut. 2 tests.

Geocode (src/geocode.rs) - the actual missing glue: parses a street number out of free text (leading or trailing), text-matches the street name (reusing the proven autocomplete query), then interpolates the number's position with the exact algorithm above. 12 tests (parsing, exact match, interpolation, street-only fallback, disambiguating streets, unknown streets).

Real-data demo against the FR sample:

"Route de Menerbes"        -> street-only, general location
"210 Route de Menerbes"    -> exact (210 is a known address)
"500 route de menerbes"    -> interpolated (not a known number)
"route de nantes"          -> correctly resolves to the DIFFERENT street

Reverse geocode (GeocodeIndex::reverse_geocode): given a point, find the nearest known address - pure spatial nearest-neighbor (rstar, the same crate and mechanism proven at full 13.9M-point scale in the geo spike), not text search. 3 tests. Real-data round-trip check: forward-geocoded "210 Route de Menerbes" to a coordinate, then reverse-geocoded that exact coordinate back - correctly returns street="Route de Menerbes" number=210 distance=0.000km.

Full workspace: 45/45 tests passing. Caveat: still scoped to one country's schema (FR), and the number-splitting is a simple leading/trailing heuristic, not a full address parser across all 11 countries' formats.

Full world import (crates/farnsworth-importer)

Native Rust Parquet reader (the parquet crate directly) - replaces the duckdb-to-JSON export step used throughout this session's earlier spikes. No external duckdb dependency, reads real files directly.

Run against the REAL world dataset, not samples: all of pois.parquet plus all 24 address parquet files across all 11 countries.

Total run: 506s (~8.4 min), peak build-time RSS 52.7GB (build cost, not steady-state - see the memory decomposition above; build and serving are different budgets).

World-coverage queries against the combined index - not a single-country sample:

• Cross-country geocode works automatically, no country parameter: "Route de Nantes" (FR), "Oxford Street" (UK), "Via Roma" (IT), "Calle Mayor" (ES) all correctly resolved from the SAME 9.3M-street index, purely via text matching.
• Reverse geocode gives genuinely correct real answers: near Paris -> "Place de l'Hôtel de Ville-esplanade de la Libération" 0.02km away (Paris City Hall Square). Near London -> "Charing Cross" 0.04km away.
• Plain POI text queries with no bias (cafe/school/hotel) returned Portuguese results - expected, not a bug: consistent with the ranking finding above, raw text score has no geographic intelligence by itself.

This directly answers whether autocomplete/details/geocode/reverse-geocode work at real world scale: they do, against the real full dataset, not samples or assumptions.

Still open

Nothing from the original roadmap - all 7 items are done, plus details/geocode/reverse-geocode built on direct follow-up. Full 11-country schema migration and a real address parser remain out of scope for this learning project.

Verified, not a risk after all: cross-language over-counting in the merged text field - checked tantivy's actual indexing source, it inserts an automatic position gap between values of the same field, so phrase queries can't spuriously span two language variants. Dedup of identical text across language codes at index time already covers the main over-counting case.

Read path (target shape, unchanged)

• localities - admin areas, geo_point + geo_shape, per-language autocomplete
• pois - geo_point, rank_feature/distance_feature scoring, search-as-you-type
• addresses-* - 11 country-specific indices, interpolation-range geometry

Write path

Explicitly out of scope (see index). OpenSearch keeps the live document patch/audit-trail job.

Deployment shape

Single-node, in-memory. The full dataset (~26.4M docs) fits in RAM on one box - no distributed-cluster machinery planned. tantivy's default MmapDirectory already keeps segments memory-mapped rather than heap-resident.