On-disk format
build_persistent writes two kinds of file into an index directory: tantivy's own segment files, and postcard sidecar files for the structures tantivy doesn't know how to persist (join tables, R-trees). open reads both back. Layout below is for ~/.cache/farnsworth-data/world-index-v3/{poi,localities,geocode} produced by farnsworth-cli build.
tantivy segment files (poi/, geocode/, and LocalitiesIndex)
One schema (build_schema() in farnsworth-text-spike), shared by all three indices: public_id (STRING|STORED), name (custom analyzer, indexed+stored), country_code (STRING|STORED, real country_codes: varchar[] - written once per value, matching a POI's real multi-country membership on border regions), context (custom analyzer, indexed - real full_indexed_string: map(varchar,varchar) source data, every language variant written once per value; used by LocalitiesIndex/GeocodeIndex for admin-hierarchy corroboration, see text-engine.md, not yet wired for PoiSearchIndex), popularity (f64, FAST only - not stored, not indexed for text search), type_boost (i64, FAST only), sub_types (STRING, indexed only, real sub_types: varchar[] - same multi-occurrence-per-field pattern as country_code), lat/lon (f64, FAST - real POI location, written at index time so it survives without a sidecar round-trip; not used for geo filtering itself any more, see h3_cell below), and h3_cell (u64, INDEXED only - the real geo pre-filter, see world-scale.md). Each index only ever writes the fields it needs; an unset field reads back as tantivy's default (0.0/0/absent) for every other index - e.g. GeocodeIndex never sets popularity/type_boost/sub_types/h3_cell, PoiSearchIndex never sets type_boost/context, LocalitiesIndex never sets popularity/country_code/sub_types/h3_cell.
(This field was named country_multilingual before this doc's own note said "unused by the current importer" - true until this point, so it was renamed to context and repurposed for full_indexed_string rather than adding yet another dormant field. Same mechanism as always intended (a merged multi-occurrence-per-language text field), just finally given real data.)
Per-segment files, named by a random segment UUID:
| Extension | Contents |
|---|---|
.term |
term dictionary |
.idx |
postings (inverted index) |
.pos |
term positions, for phrase/prefix queries |
.fieldnorm |
per-doc field length, for scoring |
.fast |
fast fields - popularity per doc, read directly inside search()'s TopDocs::tweak_score collector (see world-scale.md), not via a side lookup |
.store |
doc store - the actual stored field values (public_id, name) |
Plus directory-level files: meta.json (segment list, schema, opstamp), .managed.json (tantivy's file-tracking manifest), .tantivy-meta.lock / .tantivy-writer.lock (empty lock files, held during writes only).
Multiple segments exist because writer.commit() was called once per import batch at world scale - tantivy doesn't merge automatically unless a merge policy runs.
poi/ sidecars
locations.postcard-HashMap<String, LatLon>,public_id -> {lat, lon}. The geo join tablesearch()uses to compute exact haversine distance/exclusion after theh3_cellpre-filter narrows the candidate pool; tantivy has no native geo type here so this lives outside the index. ~1GB at 13.9M POIs.popularity.postcard- a singlef32: the precomputed corpus-mean saturation pivot (mirrors OpenSearch's own default when no explicit pivot is configured - see ranking.md). Per-record popularity itself is NOT here - it's thepopularityFAST field inside the tantivy segments above, read directly insidesearch()'s collector.
geocode/ sidecars
known_numbers.postcard-HashMap<String, Vec<KnownNumber>>,public_id (street) -> known address numbers on that street.KnownNumber { number: i64, location: LatLon }. Used for forward-geocode interpolation (bracket the target number, interpolate between the two nearest known ones of matching parity).street_location.postcard-HashMap<String, LatLon>,public_id -> street's own location. Fallback when no number is given or resolvable ("street-only" accuracy).reverse_index.postcard-RTree<GeomWithData<[f64; 2], ReverseGeoPayload>>(viarstar's ownSerialize/Deserialize), every known address point across every street flattened into one spatial index, for reverse geocoding (point -> nearest known address).ReverseGeoPayload { street_name: String, street_number: i64 }. Largest file (~5.9GB at 9.3M streets) since it duplicates street_name per point rather than referencing the street join table.
localities/ sidecars
locations.postcard-HashMap<String, LatLon>,public_id -> {lat, lon}, same shape and role aspoi/locations.postcard. Added after real integration feedback reported Farnsworth's API returned no coordinates at all -LocalitiesIndexoriginally had no location data anywhere, real or persisted (see localities.md's history and rest-api.md).
type_boost itself still needs no sidecar - unlike popularity (an additive score component, needs a corpus-mean pivot computed and persisted alongside it) or the geo join tables (tantivy has no native geo type), type_boost is read directly from its own FAST field inside search()'s collector with no side state needed - see localities.md for why it's a plain sort tie-break, not a scoring formula. Location, by contrast, is only ever read per-hit after collection (never inside the collector itself), so a FAST field would be paying for a capability it doesn't need - the sidecar HashMap PoiSearchIndex already used was the right fit, not a new pattern.
All sidecar files use postcard::to_allocvec/from_bytes - no custom framing, whole file is one encoded value. Migrated off bincode (previously bincode::config::standard() + bincode::serde::encode_to_vec/decode_from_slice): bincode is now unmaintained (its 3.0.0 release ships only a compiler error - the maintainer stopped after a harassment incident), so persistence moved to postcard instead, a drop-in swap via the same serde::Serialize/Deserialize derives every sidecar type already had. farnsworth-wire-spike still depends on bincode deliberately - it's a frozen historical benchmark comparing bincode against JSON as a wire format (see text-engine.md), not live persistence code, so migrating it would corrupt what the benchmark is actually measuring.
Profiling query-time cost against a real persisted index - a real, reusable workflow
Not persistence itself, but learned while profiling farnsworth-server against a real persisted world-index-v3 this session, and not written down anywhere else: samply (a sampling profiler, macOS via dtrace) needs more setup than samply record alone to get real symbol names instead of raw hex addresses.
samply record --save-onlydoes NOT symbolicate - resolution is normally lazy, done later bysamply load's local symbol server. Use--unstable-presymbolicateinstead to get a.syms.jsonsidecar with real resolved symbols baked in ({string_table, data: [{debug_name, symbol_table: [{rva, size, symbol}]}]}), parseable directly (binary-searchframeTable.address, an RVA, against each library's sortedsymbol_tableranges).- Presymbolication still needs real debug info to resolve against. A release build has none by default - add a dedicated profile to the workspace
Cargo.toml:[profile.profiling] inherits = "release" debug = true - Even with
debug = true, macOS still won't resolve anything: unlike Linux ELF, DWARF isn't embedded in the Mach-O binary - a separate.dSYMbundle is required (dsymutil target/profiling/farnsworth-server), and it goes stale (silently) if the binary is rebuilt after it - re-rundsymutilafter every rebuild, don't trust an old one. samply record --duration Ndoes not stop the recording if the server process never exits on its own -axum::servehas no shutdown handler. Kill the real child process directly (kill -TERM <child_pid>, found viaps/lsof, not samply's own wrapper PID - killing the wrapper orphans the child and the recording never finalizes).- A profile taken right after server startup is dominated by startup cost, not query cost - this project's own startup does real multi-GB postcard sidecar deserialization (see above) that can dwarf actual query-time sampling if not excluded. Cut samples by cumulative
timeDeltasagainst the server's own logged"Opened in Xs"line before drawing any conclusion from the profile. - Two real port collisions hit this workflow before it was fixed properly: an unrelated process already squatting the hardcoded port (caught via "Address already in use" in the samply log, confirmed via
lsof -i), and the server's own previous invocation still holding it. The real fix was a CLI[port]argument onfarnsworth-server(see rest-api.md), not sed-hacking the hardcoded default for one profiling run.
Real result from this workflow: FAST-field reader reconstruction-per-candidate (segment_reader.fast_fields().f64("popularity").first_or_default_col(...), rebuilt on every hit instead of once per segment) measured at ~11.7% of real query time before the per-segment caching fix (HashMap<u32, Arc<dyn ColumnValues<T>>> keyed by segment ordinal, in both PoiSearchIndex::search and LocalitiesIndex::search), confirmed gone (~0.7%, unrelated leftover) in the re-profiled result.
Compatibility
open() unconditionally reads every sidecar file by its constant filename (LOCATIONS_FILE, POPULARITY_FILE, KNOWN_NUMBERS_FILE, STREET_LOCATION_FILE, REVERSE_INDEX_FILE). Adding or renaming a sidecar, or changing the tantivy schema (both have happened - the bincode->postcard rename, and adding the h3_cell/lat/lon fields), makes previously-built indices unopenable: a schema field-count mismatch panics (index out of bounds, confirmed directly against a real stale index after the h3_cell schema change - Fields field ordinals are built fresh from the code's current schema, not validated against what's actually on disk), and a missing/renamed sidecar file errors on open() rather than defaulting. There's no schema-version marker; a stale index directory must be deleted and rebuilt via farnsworth-cli build (~12.5min at full 32M-record world scale, after parallelizing the parquet import across files - see world-scale.md) after a sidecar- or schema-changing code change.