Making My Note Search Actually Good

In my last post about this project, I built a semantic search pipeline over my Obsidian vault — pgvector, Ollama embeddings, an MCP server that lets Claude Code rummage through my notes. It worked. 141 lines of Python, zero API costs, and it found things that keyword search never could.

I also wrote this, in the "What I'd Do Differently" section:

I haven't built any kind of hybrid search that combines vector similarity with keyword matching. For queries where you do remember an exact term, BM25 + vector fusion would be strictly better.

Well. Here we are.

The thing that pushed me over the edge was noticing how often vector search almost got it right. Ask for "snapraid parity disk failure" and it'd find the SnapRAID config doc, sure. But ask for "docker compose services" and it'd happily return my server migration plan from 2023 alongside the actual Docker notes — because conceptually, they're in the same neighborhood. Meanwhile, keyword search would have nailed "docker compose" instantly. Each approach has blind spots that the other fills perfectly.

So I did the thing. Hybrid search with Reciprocal Rank Fusion. Full-text search via PostgreSQL's built-in tsvector. Neighbor chunk expansion. And while I was in there, wikilink graph traversal — because my notes link to each other and those relationships are signal, not noise.

The Problem With Pure Vector Search

Vector search is great at understanding what you mean. It maps your query into a 768-dimensional space and finds the notes that live nearby. "How do I back up my server" correctly surfaces my Server Migration Plan (similarity: 0.631), even though the note never uses the word "backup" — it talks about tar archives, home folder restores, and FSTAB configurations. Conceptually close. Vector search gets that.

But it also has a specific failure mode: when you know exactly what you're looking for, it can be too clever. You search for "zigbee mqtt" and it returns home automation concept notes ranked above the services catalog that literally lists Zigbee2MQTT, Mosquitto, and ESPHome with their port numbers. The services catalog is what you wanted. Vector search thought you wanted vibes.

Keyword search has the opposite problem. It's great when terms appear literally in the text. But ask it "how do I back up my server" and it matches on common words like "server" and "back" — one of my test runs returned a lecture transcript about single-page applications as the top result. Keyword search found the words. It did not find the meaning.

Neither approach is wrong. They're just incomplete.

The Fix: Reciprocal Rank Fusion

The idea behind hybrid search is simple: run both searches, then combine the results. The trick is how you combine them. You can't just average the scores — vector similarity ranges from 0 to 1, while PostgreSQL's ts_rank_cd produces scores on a completely different scale. The numbers aren't comparable.

Enter Reciprocal Rank Fusion (RRF). Instead of caring about scores, it cares about ranks. Each result gets a score based on its position in each list:

rrf_score = 1/(k + rank_in_list_1) + 1/(k + rank_in_list_2)

The constant k (I use 60, which is standard) prevents top-ranked results from dominating too aggressively. A result that appears in both lists gets both contributions summed together, which naturally surfaces it to the top. A result that only appears in one list still gets a score, but it has to compete against the doubly-boosted ones.

It's elegant because it's entirely rank-based — no score normalization, no hand-tuned weights, no "well the vector score is cosine similarity but the keyword score is tf-idf so we need to..." None of that. Just positions in sorted lists.

def _rrf_fuse(vector_results, keyword_results, limit):
    scores = {}
    data = {}

    for rank, row in enumerate(vector_results):
        doc_id = row[0]
        scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (RRF_K + rank + 1)
        data[doc_id] = row

    for rank, row in enumerate(keyword_results):
        doc_id = row[0]
        scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (RRF_K + rank + 1)
        data[doc_id] = row

    ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)[:limit]
    return [(data[doc_id], rrf_score) for doc_id, rrf_score in ranked]

That's the whole fusion algorithm. 15 lines.

PostgreSQL Native Full-Text Search (No New Dependencies)

For keyword search I had a choice: bring in a Python BM25 library, or use PostgreSQL's built-in full-text search. I went with Postgres.

Here's why: I already have pgvector running on PostgreSQL for the vector side. Adding a tsvector column and a GIN index means both search modalities live in the same database, queried by the same connection, in the same transaction. No new services, no new dependencies, no synchronization headaches.

The schema change is three lines:

ALTER TABLE documents ADD COLUMN content_tsv tsvector;
UPDATE documents SET content_tsv = to_tsvector('english', coalesce(heading, '') || ' ' || content);
CREATE INDEX documents_content_tsv_idx ON documents USING gin(content_tsv);

PostgreSQL's to_tsvector handles stemming, stop word removal, and tokenization. The GIN index makes lookups fast. And websearch_to_tsquery on the query side understands quoted phrases, OR operators, and negation — for free. With a plainto_tsquery fallback for queries that trip up the parser.

The query:

SELECT id, file_path, heading, content, chunk_index,
       ts_rank_cd(content_tsv, q) AS score
FROM documents, websearch_to_tsquery('english', %s) q
WHERE content_tsv @@ q
ORDER BY score DESC
LIMIT %s

No Elasticsearch. No Typesense. No new container. Just Postgres doing what Postgres does.

Neighbor Chunk Expansion

Here's a subtle problem with chunk-based RAG: your search finds the right chunk, but the answer spans two chunks. Maybe the heading is in chunk 3 and the code example you need is in chunk 4. Vector search finds chunk 3 but you're left without the context that makes it useful.

Neighbor expansion fixes this. After RRF fusion selects the top results, I look up the adjacent chunks (chunk_index ± 1) for each top hit and append them to the results:

def _expand_neighbors(conn, results, max_expand=5):
    seen_ids = {r[0] for r in results}
    expanded = []

    for i, row in enumerate(results):
        expanded.append(row)
        if i >= max_expand:
            continue

        file_path, chunk_index = row[1], row[4]
        # Fetch chunk_index - 1 and chunk_index + 1
        for neighbor in fetch_adjacent_chunks(conn, file_path, chunk_index):
            if neighbor[0] not in seen_ids:
                seen_ids.add(neighbor[0])
                expanded.append(neighbor)

    return expanded

Neighbors are marked with a score of 0.0 and tagged *(neighbor)* in the output, so you know they weren't independently matched — they're along for the ride because something near them was relevant.

In the worst case (every top-5 result has two neighbors), this adds 10 extra chunks to the response. At ~6000 characters per chunk, that's about 60KB of additional context. For an MCP tool feeding into Claude's context window, that's fine.

Wikilink Graph Traversal

This was the bonus feature I didn't plan to build until I was already elbow-deep in the indexer code.

Obsidian notes link to each other with [[wikilinks]]. These links encode relationships — "this note is related to that note" — and my original pipeline completely ignored them. Out of 742 indexed chunks, 64 contained wikilinks. That's structured knowledge sitting right there, unused.

Extracting Links

The indexer now extracts wikilinks from each file using a regex (after stripping code blocks to avoid false positives):

WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")

It handles display text aliases ([[Target|display text]]), heading links ([[Note#Section]] → Note), and filters out attachments (images, PDFs) and code artifacts (targets containing ${}=*()). The extracted links go into a dedicated note_links table:

CREATE TABLE note_links (
    id SERIAL PRIMARY KEY,
    source_path TEXT NOT NULL,
    target_name TEXT NOT NULL,
    target_path TEXT,      -- resolved after all files are indexed
    updated_at TIMESTAMP DEFAULT now()
);

After indexing all files, a resolution pass matches target_name against indexed file stems to populate target_path. In my vault: 136 links extracted, 62 resolved to actual indexed files. The unresolved ones are mostly references to notes I haven't written yet (Obsidian lets you link to things that don't exist — it's a feature, not a bug).

The Graph Search Tool

The new graph_search() MCP tool does BFS traversal of the link graph — both outgoing and incoming links, up to 3 hops deep:

> graph_search("Digital_Garden/Obsidian-RAG/index.md", depth=2)

# Wikilink Graph: Digital_Garden/Obsidian-RAG/index.md

## Hop 1 (3 links)
- → Database Schema.md (via [[Database Schema]])
- → Indexer.md (via [[Indexer]])
- → MCP Server.md (via [[MCP Server]])

## Hop 2 (1 links)
- → Infra_RELOADED/Node-RED.md (via [[Node-RED]])

Total: 4 links across 2 hops

This is useful for a completely different reason than search. When I'm working on something and want to know "what else is connected to this topic," graph traversal shows me the structure of my own thinking. It's not finding similar content — it's following the relationships I explicitly created.

The Comparison: Does Hybrid Actually Help?

I ran a set of test queries across all three modes. Here's what I found:

"how do I back up my server" — Vector Wins

Pure vector crushes this. Keyword catastrophically misunderstands the query. Hybrid inherits the right answer from vector, but the keyword noise doesn't help.

"home automation zigbee mqtt" — Hybrid Wins

This is where hybrid shines. Vector finds the conceptual HA docs. Keyword finds the services catalog that literally lists Zigbee2MQTT and Mosquitto with port numbers. Neither alone gives you the full picture. Hybrid gives you both.

"docker compose services overview" — Hybrid Wins

Vector finds the main Docker doc. Keyword finds the monitoring stack with its docker-compose service tables. Hybrid merges both into the most useful result set.

"snapraid parity disk failure" — Vector Wins (Keyword Fails Completely)

This one surprised me. I expected keyword to shine on technical terms like "snapraid" and "parity." Instead, it returned nothing — these terms either don't appear frequently enough in the corpus or don't co-occur in a way that scores with ts_rank_cd. Vector carried the entire result set.

The Takeaway

Hybrid search isn't always better than vector-only. For my vault (236 files, 742 chunks), pure vector search is already quite good. But when keyword search does contribute — like finding the exact services catalog entry for "zigbee mqtt" — the RRF fusion correctly surfaces it alongside the conceptual matches. You get the best of both worlds at the cost of one additional SQL query per search.

The real win is having the option. The mode parameter defaults to "hybrid" but I can drop to "vector" or "keyword" when I know which one will work better.

What Changed (The Boring Part)

For anyone following along technically:

Schema additions: - content_tsv tsvector column + GIN index on documents - New note_links table with indexes on source_path and target_path

Indexer changes: - Computes content_tsv server-side via to_tsvector('english', ...) on INSERT - Extracts [[wikilinks]] from each file (code block filtering, attachment filtering) - Stores links in note_links, resolves targets to file paths post-indexing - Cleans up stale links when files are removed from the vault

MCP server changes: - search_notes() gains a mode parameter: "hybrid" (default), "vector", "keyword" - New graph_search() tool for BFS wikilink traversal - Hybrid mode: 3x candidate fetch from both engines → RRF fusion → neighbor expansion

Dependencies added: Zero. PostgreSQL's built-in FTS and the existing psycopg driver handle everything. No new Python packages.

The indexer went from 191 to 276 lines. The MCP server went from 141 to 292 lines. Still no Kubernetes.

What's Next (For Real This Time)

The "What I'd Do Differently" section from the last post is getting shorter. Hybrid search: done. But there are still things that would make this better:

• Cross-encoder reranking — Use a small cross-encoder model to rescore the top-N results. RRF is good at combining two ranked lists, but a cross-encoder actually looks at the query-document pair together and can catch subtle relevance signals that both vector and keyword search miss.
• Query-dependent mode selection — Automatically detect whether a query is better suited for keyword or vector search based on its structure. Quoted terms and specific identifiers → keyword-heavy. Natural language questions → vector-heavy. Right now the user picks, but the system could be smarter.
• Chunk overlap — The current splitter creates non-overlapping chunks at heading boundaries. A sliding window with ~20% overlap would reduce the chance of splitting relevant context across chunk boundaries (and reduce the need for neighbor expansion).

But again — for a personal vault of 236 files, this is more than good enough. The hybrid search handles the cases where pure vector stumbled. The graph traversal reveals structure I forgot I built. And it still costs nothing to run.

The Stack (Updated)

Everything from the original post, plus:

• Full-text search: PostgreSQL tsvector + ts_rank_cd (built-in, no extensions needed beyond pgvector)
• Fusion algorithm: Reciprocal Rank Fusion (k=60)
• Graph storage: note_links table in the same PostgreSQL database
• New MCP tool: graph_search() for wikilink traversal

Total infrastructure added: one SQL column, one table, and zero new services.

Sometimes the boring solution is still the right one. You just make it slightly less boring.