Architecture
Design sketch, being validated incrementally against real production data. This is a learning project (see index), not a committed prod-replacement initiative - read-only, no write/patch-path port. Tracked as a shaped roadmap epic in the localities repo (roadmap/farnsworth/).
Detail lives on dedicated pages: text-engine.md (tantivy parity, memory, wire protocol), geo.md (R-tree, interpolation, geocode), world-scale.md (full import, popularity skew), localities.md (towns/villages/capitals, type_boost), ranking.md (the ranking math), persistence.md (on-disk file formats). This page is the overview and the pieces that don't belong on any of those.
Status
- Baseline latency spike: done. The original premise (OpenSearch's p99 tail is a JVM/GC problem) was disproven by direct CloudWatch/Datadog evidence - zero old-gen GC collections, comfortable memory/CPU headroom. Root cause instead traced to query-plan cost scaling with token count in the per-language
dis_maxfan-out (confirmed viaN_TOKENS-correlated latency: p99 climbs ~13x from 1 to 10+ tokens). - milli spike: done, not viable. Compiles as a git dependency with two non-obvious workarounds (pinned rustc 1.98.1, forced
smartstring/serdefeature), but its indexing API is Meilisearch's raw server internals (LMDB txns, bump allocators, thread pools) - not a stable, documented embedding surface. - tantivy text-analysis parity: done. Full-scale (13.9M real POIs) surfaced a real query-latency issue, which was isolated, fixed, and confirmed end-to-end - see text-engine.md.
- Geo index: done, clean win at full production scale. See geo.md.
- Major finding: ranking is not Farnsworth's job at all - later superseded, see below. Correct for the shape of the original spike; turned out not to hold once Farnsworth had no downstream Python pipeline of its own to lean on.
- Wire protocol spike: done. bincode wins on paper (2.9x faster, 38% smaller) but doesn't matter - serialization is microsecond-scale, dwarfed by query execution and whatever the real transport overhead actually is. See text-engine.md.
- Integration capstone: done. Text search and geo filtering compose correctly into one search path - proven with tests, not just asserted. See below.
- World-scale import and popularity skew: done. Full 11-country real dataset imported and served; popularity now ranks during collection, not as a post-hoc resort. See world-scale.md.
- Localities index: done. The last of the three real indices (after POIs and addresses) - 8.8M real countries/admin-levels/localities/postal-codes.
type_boostturned out to be a real sort tie-break, not a saturation term like popularity - a genuine finding, not the pattern that was assumed going in. See localities.md. - Exact-match scoring fix: done. Found via the localities work - autocomplete query clauses all scored matches as flat constants, so exact matches tied with same-prefix substrings of longer words. See text-engine.md.
- Country/type filtering: done. Real recall filters (
country_codes,sub_typesinclude/exclude) onPoiSearchIndex, mirroring production's two real filter clauses. See text-engine.md. - Merged POI+locality search: done. Resolved a real open question (tantivy has no cross-
Indexquery API - matches production's own_msearchfan-out architecture) and found a real, documented limitation (cross-index score comparability). See world-scale.md. - REST API: done. Real HTTP server (
crates/farnsworth-server, axum) over the persisted world index - queryable from curl, not just the CLI REPL. See rest-api.md.
Major finding: ranking is not Farnsworth's job - correct at the time, later superseded
Read localities' actual ranking implementation directly (localities/suggestions/processor/, not just docs). OpenSearch's native _score (BM25/dis_max/rank_feature) is discarded entirely for autocomplete/geocode ranking in production - it only affects recall (which candidates come back at all). Actual relevance ordering is a fully separate, engine-independent step: every candidate runs through DEFAULT_PIPELINE, a chain of Python scorers that re-tokenize the returned field values and fuzzy-compare them token-by-token against the input, producing a composite 0-1 coverage score - computed identically regardless of whether the candidate came from OpenSearch or HERE.
This reframed Farnsworth's scope at the time:
- rank_feature/distance_feature-equivalent scoring looked low priority - good enough for recall, final relevance ordering being someone else's job
- Porting anything to Farnsworth didn't look like it required replicating OpenSearch's scoring formula or matching BM25 numerically
- Geo ranking looked similarly unaffected:
LocationScorer'sgeo_bias_scoreis also computed downstream in production, independent of the engine
What actually happened once real multi-candidate queries were tested live: that framing assumed the existing Python DEFAULT_PIPELINE would keep running on top of Farnsworth's recall, the way it does on top of OpenSearch's today. Farnsworth has no such downstream step of its own - it is the whole stack in this spike, recall and ranking both. Real bugs surfaced immediately once that gap mattered: decoy candidates (e.g. a short "Paris" outranking the real "Rue de la Paix Paris" street) that production's own DEFAULT_PIPELINE would have reordered correctly, with nothing downstream of tantivy's own BM25/type_boost order here to catch it. So DEFAULT_PIPELINE's real algorithm - fuzzy_score/fuzzy_score_cascade, name_coverage, geo_bias_score - ended up ported into Farnsworth itself after all (crates/farnsworth-core/src/ranking.rs), not skipped. The original finding about production's own architecture (OpenSearch's _score genuinely is discarded there) stays true and is still the reason Farnsworth never tried to numerically match BM25/rank_feature - it's the "nothing runs downstream of Farnsworth" assumption that didn't hold.
See ranking.md for the real ported algorithm and the precise formula Farnsworth implements (popularity skew, the fuzzy-score gap-closing fix) and where it still deliberately diverges from production's own math (the distance term).
Integration capstone (crates/farnsworth-core)
Combines the text index and geo filtering into one query path (PoiSearchIndex) - a single in-process Rust type, not a service. Text-match candidates first (over-fetched), then geo-filter to within a radius of an optional bias point. Matches the ranking finding above: geo bias affects recall (exclusion), not final ordering.
4 tests prove the composition works, not just each piece alone:
- Two identically-named POIs (Paris vs NYC) - the geo filter correctly excludes the far one
- A geographically adjacent but textually unrelated POI is correctly excluded - proximity alone isn't enough
- A real match gets included with the correct distance
- No bias point cleanly falls back to pure text search
Real-data demo (src/bin/demo.rs) against the 12,724-POI sample: "cafe" with no bias returns matches spread across the whole sample; biased near NYC at 10km correctly narrows to the one actual nearby cafe; tightening to 1km correctly returns 0 (too tight for this sparse sample, not a bug).
This is as far as the learning project's read-only scope goes - write-path, shadow deployment, and prod cutover remain explicitly out of scope. See geo.md for details/geocode built on top of this, and world-scale.md for the same path proven at full world scale.
Still open
Nothing from the original roadmap - all items are done, plus details/geocode/reverse-geocode, the full world import, popularity skew, the localities index, and merging all three into one ranked search, all built on direct follow-up. Full 11-country address schema migration and a real address parser remain out of scope for now.
Verified, not a risk after all: cross-language over-counting in the merged text field - checked tantivy's actual indexing source, it inserts an automatic position gap between values of the same field, so phrase queries can't spuriously span two language variants. Dedup of identical text across language codes at index time already covers the main over-counting case.
Read path
All three real indices exist, and are now merged into one ranked search (see world-scale.md and localities.md):
localities- countries, admin levels, towns/villages/cities, postal codes. Done, see localities.md. Geo_shape polygon containment (admin boundaries) proven separately in the geo spike, not yet re-integrated here.pois- geo_point, popularity ranked in-collector, search-as-you-type. Done, see world-scale.md.addresses- one merged index across all 11 countries (production keeps 11 separate ones - a deliberate, documented divergence, see world-scale.md), interpolation-range geometry. Done, see geo.md.
Write path
Explicitly out of scope (see index). OpenSearch keeps the live document patch/audit-trail job.
Deployment shape
Single-node, in-memory. The full dataset (~26.4M docs) fits in RAM on one box - no distributed-cluster machinery planned. tantivy's default MmapDirectory already keeps segments memory-mapped rather than heap-resident.