Omni search — four groups, on every tab

SHIPPED v753, 2026-08-11. All seven steps below are built: the table, the backfill (lib/migrate_search_index.py), the write hook, the redact hook, GET /api/search/history, the three-group client, and six smoketest gates. Live numbers from the dev corpus — 24,098 lines indexed from 1,632 rooms in 4.1 s; a query answers in 3 ms in the database and 17 ms round-trip. Two things moved while building, both noted in place below: a head-slice snippet hid the very match it was there to show (§5), and the index hook reached for the wrong database handle and the safety try swallowed it (§6).

Today the Chats field narrows the chat list and nothing more (v751). The ask is WeChat's: one field that searches everything you have, and answers in three named groups — group chats, chat histories, friends. Two of those three the app can already answer in a keystroke. The third has never existed, and it is the whole engineering story: the app has never searched what was said.

The thesis, in one line. Three of the four groups are free and the fourth is not, because the text worth searching is 1.3% of what you must read to find it. That single ratio — 3.15 MB of conversation inside 233.5 MB of room state — is what turns「search your history」from a filter into an index.

1 · The shape

fig 1 · one field, three sources, one list — as designed in v753. Only the last one was new work, and it is last for a reason: see §5. ⚠ A fourth source joined in v939 — the persona library, already in memory — and since v940 it lands directly under friends, alongside the other question about people. See §2.

The order came from the screenshot, not from the brief. The ask listed group chats · chat histories · friends; WeChat's own answer runs 联系人 → 群聊 → 聊天记录 — friends first, history last. That is worth following, because it also happens to dissolve the one layout problem this feature has: the two instant groups land immediately and the slow one grows onto the bottom of the list, where it cannot shove anything already on screen. The brief's order put the slow group in the middle. Same three groups either way — say the word if you want yours.

2 · Why four groups — and why a 1-on-1 is not one of them

The groups are not filters over one pile; they are different kinds of thing, and each already has a row that knows how to draw itself and what to do when tapped. That is the rule the whole feature hangs on: a result looks exactly like the row it stands for, and does exactly what that row does. Three shipped in v753; the fourth arrived in v939, when the field stopped being three fields wearing one glyph.

Group	What is in it	The row says	Tapping it	Source
1 · Friends 联系人	people and personas	face · name, with the hit highlighted	opens the profile	`/api/friends?q=`
2 · Personas 角色 v939	the public library, minus everyone you have already friended — a friended persona is a contact and answers in group 1, so no card can appear twice	the Friends list's persona row: card colour · name · role	opens the profile; its own long-press menu is where you friend them	client, `LIB` − `PFRIEND`
3 · Group chats 群聊	rooms that are not a 1-on-1 — a panel of personas, or more than one human	the chat-list row · member count · and why it matched:「contains: 高玮丽」when the hit was a member rather than the name	opens the chat	client, `ALL_ROOMS`
4 · Chat histories 聊天记录	lines that were said, in any chat you can see	one row per chat, not per line: the newest matching line + date + 「62 matching messages」	opens that chat	NEW endpoint

Two things the screenshot settles that a brief would not have. First, a result explains itself: a group that matched on a member says so on its second line, because otherwise a row whose visible title has nothing to do with what you typed looks like a bug. Second, history is grouped by chat, not by line — one row per conversation carrying its newest hit and a count, so a word said sixty-two times in one chat is one answer, not sixty-two. Each group is also capped with a「more」 link rather than running long: four groups only stay readable if none of them can push the others off the screen.

Where the fourth group goes — one constraint, then a choice. The constraint is the WIRE, and it binds exactly one position: chat history is the only group that waits, so it goes last and grows onto the bottom of a list that has already settled, where it cannot shove anything the eye is reading. Everything else answers off memory or one small endpoint, so their order among themselves was free — and the owner spent it on adjacency (v940): Friends and Personas answer the same question, people you have then people you could have, and the group chats had been sitting between them, making you read past half an answer to reach the rest. Friends · Personas · Group chats · Chat history.

A 1-on-1 chat is not a search result — the person is. This is WeChat's rule, and it is why there is no fifth group for DMs. A DM has no name of its own: it is titled from who is in it, so searching「Lincoln」and offering both「Lincoln」the friend and 「Lincoln-ish」the chat is the same answer twice, and makes you read before you can pick. The person is the door; their profile has Send message on it, and their Shared chats row lists every room you are in together. So only multi-party rooms appear under Group chats, and everything 1-on-1 is reached through Friends.

3 · The measurement that decides the architecture

Groups 1 and 3 are free. Group 2 is the question, and it is not close. Measured on the dev corpus, 2026-08-11:

	Amount	What it means
Rooms on disk	1,755	each a directory with `state.json`
Total room state	233.5 MB	what a naive search must open
Naive full scan	15.3 s	per query. Not a feature — a hang
Lines actually said	30,066	the visible conversation
Text worth searching	3.15 MB	1.3% of what you had to read to find it

The 74× gap is history — the raw model messages, megaprompt and all — which nobody searches and which dwarfs the transcript sitting beside it. So the text gets pulled out once and kept where it can be asked a question: a msgs table in app.db, backfilled once, appended as turns land. At 30k short rows a query is single-digit milliseconds, and it grows with what was said rather than with what the model was told.

Which index — the three candidates, run against the real corpus

The obvious reach is SQLite's full-text index. It is the wrong tool here, and the corpus says so rather than the argument. All three were built from the same 30,057 lines and asked the same two questions — a real two-character Chinese word (个人, which 2,434 lines genuinely contain) and an English one:

Candidate	Build	On disk	CJK query	CJK hits	Latin hits	Verdict
plain table + LIKE	0.1 s	9.7 MB	6–12 ms	2,434 ✓	17 ✓	correct
FTS5, default (unicode61)	0.3 s	15.7 MB	0 ms	0 ✗	17 ✓	finds nothing
FTS5, trigram	1.4 s	34.1 MB	0 ms	0 ✗	17 ✓	finds nothing

Both full-text options return ZERO, and they do it silently. unicode61 tokenizes on spaces and punctuation, and Chinese has neither — 中海恒昌玖里 is one token, so「恒昌」is not a word inside it and never matches. Trigram fixes CJK substring search in principle, but it can only answer queries of three characters or more, and Chinese search terms are routinely two —「恒昌」in the screenshot is two. Below its floor it does not error; it returns an empty result, which is the worst failure available: a search that quietly says「nothing」about a word said 2,434 times. Recommend the plain table with a substring scan. It is the only one that is correct, it is the cheapest to build and the smallest on disk, and — the reason that settles it beyond the numbers — it is the same rule the chat list already uses (includes), so one word cannot find one thing in the list and a different thing in the history.

What it costs, and when it stops working

A substring scan is linear in total text, so the honest question is not「is it fast」but 「how much room is there before it isn't」. Measured: 1.9 ms per MB of conversation. The whole 3.15 MB corpus answers in 6 ms; a single user's own rooms — the only scope a real query has — answer in under 1 ms. Ten times the text is ~60 ms and a hundred times is ~600 ms, so the ceiling is around 50–100× today's corpus, at which point the upgrade is a hand-rolled bigram column (every adjacent pair indexed as a token) — which is what CJK full-text setups do anyway, and can be added beside the same table without changing a single caller.

4 · The three filters an index must not lose

This is the part that can leak. Today every line a user sees passes through one read path — replay(floor) then _hide_filter — and that path is where all privacy is enforced — the floor exists only in the read path; nothing is deleted underneath it. An index sits beside that path, so each filter has to be carried across explicitly or it is simply gone:

Filter	What it hides	Carried across by
The floor `member_see_from`	everything said before you joined, when history was private	store each line's ts; drop candidates older than the caller's floor
Delete for everyone a `redact` event	a line the room retracted — replay renders a tombstone	store each line's lid; delete it from the index when the redact lands
Delete for me `hidden_msgs`	a line one person put a lid on	same lid, filtered per caller — already a DB table

The rule this earns. An index is a second read path, and a second read path is a second place to forget. So it stores ts and lid on every row — neither is needed to match anything; both exist so the three filters can still be applied to a candidate without loading the room. A row that cannot be filtered must not be returned.

5 · What it looks like, and where a tap goes

The field and its behaviour are unchanged — the magnifier grows into the bar, the ‹ retracts it, the word is the tab's (v748–v752). What changes is what the list under it shows.

On a phone

The results replace the chat list in place, under three headings. A chat opens the chat; a person opens their profile. Both are the end of the loop — entering a chat already sweeps the search (v751), and it should keep doing exactly that.

On wide

Results fill the left pane, and the right pane answers:

a group chat or a history hit → that chat in the right pane
a friend → that profile in the right pane

Which is not a new rule at all — it is the one the app already has: pressing a row in the left pane replaces the right pane. The search list is a left-pane list like any other, so it inherits that for free. And on wide the pick ENDS the search (v756, reversing the first cut's「hold the results」): the field clears, the ordinary list returns, and the picked row is revealed in it — a chat scrolled to centre (v757's one helper), a contact scrolled to centre and wearing the current mark while their profile holds the right pane (v792). What you want beside the thing you opened is its neighbours, not the answers you have finished with.

Why the order is load-bearing. Friends and group chats answer in a keystroke; history needs the wire. In WeChat's order the slow group is last, so it grows onto the bottom of a list that has already settled and nothing on screen moves. Put it in the middle and every arrival shoves Friends down under the reading eye. The order is not decoration — it is the whole reason this does not need a spinner, a reserved slot, or a second settle.

Built, and one thing had to change: the SNIPPET IS CENTRED ON THE MATCH. The first cut showed the first 80 characters of the matching line — and in a long message the hit is rarely near the front, so three of the first three results came back showing a line where the word was nowhere to be seen. A row that matched something you cannot see reads as a wrong result, not a right one. It now takes a window around the hit with about a third of it as lead-in, and elides both ends — which is what the screenshot was doing all along（…中海恒昌玖里…）and I had read as a truncation.

The field itself

Unchanged from v748–v752 — the magnifier grows into the bar, the ‹ retracts it, the word belongs to the tab. And that is now the shape at every width (v937): the wide left pane had carried a standing field instead, which cost it the root's own title; it wears the phone's header whole now, title and all. One addition from the screenshot: an ✕ clear button at the right end of the field once there is something to clear, which empties the word without putting the bar away. The ‹ puts the bar away; the ✕ only empties it. Two different intentions, two different controls.

6 · What building it is, in order

The table. msgs(room_id, lid, ts, who, text) in db.py, indexed on room_id.
The backfill. A migrate_* that walks every room once and fills it — ~15 s once, on the box, at deploy.
The write hook. Where a turn is appended, mirror it in. Idempotent on (room_id, lid) so a re-persist cannot double it.
The redact hook. Delete the lid on「delete for everyone」— the one place the index can go stale in a way that leaks.
The endpoint. GET /api/search/history?q= — scope to the caller's rooms, apply the three filters, return chat + line + count.
The client. Three groups in the Chats results list, in WeChat's order: friends from the existing call, group chats from ALL_ROOMS restricted to multi-party, histories from the new one. Each group capped with a「more」link, the hit highlighted in coral, and a group row carrying「contains: name」when it matched on a member.
The gates. Six, in the smoketest: no floor finds everything · a floored member cannot find a pre-floor line · a line you hid stops being findable by you · a redacted line leaves the index entirely · re-indexing replaces rather than duplicates · another room's lines are never candidates.

The bug this build produced, because it is the shape to watch for. The index hook reached for db — which is local to build_app(); module-level Room methods read the _DB alias the app publishes for exactly this. So the hook was calling a module where an instance was wanted, and the try/except that exists so a search index can never fail a user's turn swallowed the error completely. Every gate passed, the backfill was full, the UI worked — and nothing said after the deploy would ever have been findable. A safety net that hides a defect is a defect, and the only thing that caught it was exercising the live path directly: say a line, persist, look. Now verified end to end — a line indexes itself on persist, a redact removes it, and a rehydrated room re-syncs whole without duplicating it or resurrecting the redacted one.

Not in scope, deliberately. Ranking (results come back newest first, not scored); searching inside one chat; highlighting the hit in the chat once opened. ⚠ The third exclusion did not survive contact with the owner: the public persona library was left out on the reasoning that Discover owned it and that its answers were「a different kind of thing from what is on your screen」. Both halves were true and the conclusion was still wrong — what it bought was a field that found different things depending on which tab you were standing on. v939 folds it in as the fourth group, PERSONAS: everyone in the library you have not friended (the friended ones are contacts and answer in group 1), sitting directly under Friends since v940 — the two questions about people, together. Order — Friends · Personas · Group chats · Chat history.

-ish · omni search · designed 2026-08-11 · measurements and the index bake-off run against the dev corpus (1,755 rooms · 30,057 lines · 3.15 MB) · siblings: the style guide (the field itself, v747–v752) · friends · room language