Localities: towns, villages, capitals
The third and last of the three real indices (crates/farnsworth-core/src/locality.rs), after PoiSearchIndex and GeocodeIndex. Closes the last gap in architecture.md's "Read path" - localities was always one of the three real indices, but only pois and addresses got ported earlier in this session.
What it covers
Real production splits this into four physical files sharing one shape (names, public_id, type_boost - confirmed against all four, not assumed from one), imported via a single import_localities():
| File | Real rows | Kind |
|---|---|---|
countries.parquet |
247 | countries |
admin_levels.parquet |
33,053 | admin regions/states/provinces |
localities.parquet |
2,217,978 | towns/villages/cities/hamlets etc |
postal_codes.parquet |
6,571,553 | postal codes |
All four imported and indexed together: 8,822,831 real records, built in 17.8s.
type_boost is a sort tie-break, not a rank_feature - a real finding, not an assumption
The original shape of this roadmap item assumed type_boost would behave like popularity - a static per-record signal blended into the score via a saturation curve. Reading the real code first (per this project's own rule) found something different: localities/patch/derivation/type_boost.py computes the value (country=100, admin_level=90, postal_code=80, locality 5-70 by sub-type - city=70 down to hamlet=10), but localities/services/autocomplete.py and geocode.py use it as a plain field sort, after _score:
sort=[
DslFieldSort("_score", "desc"),
DslFieldSort("capital_boost", "desc"),
DslFieldSort("type_boost", "desc"),
DslFieldSort("population", "desc"),
]
(capital_boost/population are the same two fields found dead/absent earlier - see world-scale.md - so only _score then type_boost are real, live tie-break keys here.)
LocalitiesIndex::search() reproduces this exactly, not with an invented formula: TopDocs::tweak_score returns a tuple (text_score, type_boost) as the collector's score type (tuples implement PartialOrd lexicographically already, no custom comparator needed) - text relevance decides first, type_boost only breaks an exact tie, same as the real compound sort.
Superseded by a later finding, kept above for the real history it is: that tuple collector is no longer the final word on ranking order. Once the real production fuzzy-score algorithm was ported (fuzzy_score_cascade in ranking.rs - see ranking.md), search() gained the same "rue de la paix paris"-class bug every other index had: a short, single-token exact match ("Paris" alone) could outrank a longer, more relevant candidate if it wasn't even in the BM25-ordered pool the collector kept. So search() now does overfetch after all - crate::ranking::overfetch_window(limit, tokens.len()), same real, measured, token-count-scaled window as GeocodeIndex/PoiSearchIndex - then re-ranks that pool by fuzzy_score_cascade instead of trusting raw (text_score, type_boost) order for display. The tuple collector still decides which candidates make the overfetched pool, it just no longer decides their final order.
3 tests prove the tie-break behavior specifically: a real tie (two "Springfield"-prefixed names) resolves by type_boost; a country (type_boost=100, the real max) does NOT beat an exact different-name match just because its boost is higher (text relevance always wins first); missing type_boost defaults to 0 and never wins a tie.
Real-data finding that turned out to be a real bug - fixed
LocalitiesIndex::search("paris", 5) against the real 8.8M-record index initially returned "Hamilton Parish" (an admin-level record) ahead of the actual city "Paris". First read as "not a bug, just how fuzzy/tie-break interacts" - that assessment was wrong. Digging further (raw score printout, not just top-5 names) found the real cause: text scoring here was flat constant for every match, exact or not - "parish" legitimately starts with "paris", so it tied the actual city on text score, leaving type_boost (which only breaks ties, correctly) to decide between two candidates that should never have been tied in the first place. Root cause and fix documented in text-engine.md - an exact-match boost clause now makes "Paris" outscore "Hamilton Parish" outright. type_boost still correctly breaks the remaining tie among real "Paris" matches (an admin-level "Paris" over the smaller city, per production's own real sort chain).
No longer standalone - now merged
LocalitiesIndex is one of the three real indices search_merged() draws from (see world-scale.md) - POI, locality, and address candidates do now compete for the same result slots, with a reserved-slot allocation keeping one kind from crowding the other out. type_boost still only differentiates within the locality family (country vs. city vs. hamlet); cross-kind ordering is kind_priority's job, not type_boost's.
Same real exact-match/radius-bypass pattern GeocodeIndex needed also landed here: a population-based bypass alone (LOCALITY_POPULATION_BYPASS_FLOOR) excluded genuinely small villages - "Boujan-sur-Libron" (~3,000 population) wasn't reachable once geo-biased elsewhere, even on an exact name match. Fixed with the same name_coverage >= 0.9 exemption GeocodeIndex uses (via fuzzy_score_components, see ranking.md) - an exact or near-exact name match bypasses the radius regardless of population now, not just regardless of popularity.