Search your app's data

Task: find records by the words in them — notes that mention “invoice”, titles that start with “roa” — without reading every record in the app.

Each node that holds a replica of the app keeps a full-text index (tantivy) of the tables that declare search fields, in memory, built from the state it holds. The index is derived data: it is never replicated, never written to disk, and never part of the app’s state. A query reads the page of hits, never the table.

Declare and query

Declare the fields to index on the table itself, and the host keeps the index from the app’s state:

#[derive(Record)]
#[table(key = "id", search = ["title", "body"])]
pub struct Note {
    pub id: String,
    pub title: String,
    pub body: String,
}

#[app::view]
pub fn find_notes(&self, text: String) -> app::Result<Vec<Note>> {
    let found = self.notes.search(&text, SearchOpts::default().limit(20))?;
    Ok(found.items.into_iter().map(|f| f.record).collect())
}

• The state ABI lists the fields (state.search: table to fields, camelCase). Text fields are indexed; a list of strings is joined; anything else is skipped.
• Every record of the table is one document. Each replica derives its index from the record keys the entries it applied touched, else from a scan of the table, before a view reads; nothing enters the log.
• Table::search(text, SearchOpts) queries through the WIT search interface (SearchMode::{Words, Prefix, Substring, Fuzzy}, a cursor and a limit of 1 to 100) and reads each hit’s record back; a hit whose record is gone is skipped. Only views search; any other run traps HostDenied. A call of a declared view runs as a strong view, so it may search too.
• The test harness (TestApp) answers searches with a plain scan: records whose declared fields contain every word of the query, all scored the same. Assert which records match, not their order.

Limits

• One index per table, over the fields its #[table(search = [..])] names.
• A query is at most 256 bytes and 4 KiB encoded; a page at most 100 hits; a cursor at most 10,000 deep.
• Fuzzy matching is edit distance 1 on words: no stemming, no synonyms, no phonetics. Substring search needs 3 characters.
• Scores are per app (each index has its own term statistics).
• A stale view on a replica that is behind searches the state that replica has applied.
• A new release rebuilds the indexes from state, on the first view after it, about 0.1 ms a document. So does the first view after a restart or after the app moves to a node: indexes that survive a restart are not done yet.

Operating it

There is nothing to configure: search is on for every app that declares an index, and its indexes live in memory on the nodes that serve its views. A node keeps the touched entities of the last 4096 entries per app (TOUCHED_RING); a view after a longer gap rebuilds from a scan.