Sourcefinder

Status: paused. Dogfooding this against my own company’s actual sources (four of them, two with their own agentic search already) made it clear the design below is a bit over-engineered for that real use case — most of the machinery here (clusters, the foundation tree, night-shift probes) earns its keep at dozens-to-hundreds of sources, not four. Leaving the writeup as-is as a design record; may pick it back up if a use case at the right scale shows up.

Every information source in a modern organization is, individually, searchable. Confluence has a search API; wikis, ticket trackers, and code hosts expose MCP servers; databases are reachable with a line of code. What nothing provides is the layer above: something that tells an agent — or a new teammate — which of the twenty searchable places exist, what each one actually contains, and which to try first. That meta-index lives in people’s heads, and it walks out the door when they do.

Sourcefinder is my attempt to build that layer: an MCP server that behaves like the organization’s reference librarian. It has a day job and a night shift: it looks for new datasources, and it updates the description of any existing one when something fundamental changes.

The role: a librarian at the desk, and a librarian after hours

Sourcefinder tells you where to look, how to look, what to read first, and when there’s nowhere left to look. It remembers what actually worked, keeps its own map honest — and it never does the looking for you.

That last clause is the load-bearing decision, and the whole job splits cleanly along it into a daytime role and a night-time one.

Online — at the desk. Sourcefinder holds no credentials, runs no searches, ingests no content. A session has a lifecycle:

1. where_can_i_find(question) → a structured answer, not a bare list. It opens with “read if you’re new”: the prerequisite material for the subject — tutorials, READMEs, glossaries, the docs without which nothing else will make sense — emitted as a resolved reading path (each pointer expanded root-first through the foundation tree, below), included always and first, because Sourcefinder never knows how much the client already knows and must assume they are brand new. Then the candidate list: as many relevant sources as are plausibly worth trying, ordered by confidence rather than capped — and when Sourcefinder knows where the answer lives, that source is pinned at the top and flagged as such. Each entry carries per-source affordances (“semantic search via MCP,” “keyword-only JQL,” “ask Dana”), example queries with expected result shapes, gotchas, and combination playbooks (“for outage questions, search the runbooks and the postmortems, join on service name”).
2. The agent does the actual work — searches, queries, reads.
3. report_outcome(session, found?, sources_tried, what_worked, what_failed) → the truth about whether the mentoring helped.
4. Failed sessions return to follow_up for new ideas: untried sources, reformulated queries, different modalities (text → database → ask a human → git history).

Mid-session drift. The loop above assumes every turn arrives at a natural checkpoint to ask “do I need Sourcefinder again?” — but the client only ever sees the reading list and candidates it was handed at query time, never the raw index, so it has no way to check a newly-surfaced term against anything but memory. And the trigger rarely lands at a clean boundary: a debugging agent three tool calls into an unrelated investigation hits a service name it’s never seen, with no natural moment to stop and ask.

Two obvious fixes don’t hold up. Forcing a follow_up call every turn buries the episodic log in null events exactly where the night shift needs signal, and serializes latency across the whole session. Handing the client the raw index so it can self-check fails for a different reason: even generous context windows show real “lost in the middle” degradation on a list injected once and left to compete with dozens of intervening tool results — a model noticing forty turns later that a term is uncovered is a bet, not a guarantee.

The fix that holds treats tool schemas differently from transcript content. A tool’s description is resent on every call and typically prompt-cached, so — unlike a message buried in history — it never recedes. Sourcefinder ships a condensed index of the index as part of its own tool documentation: one line per source, discriminator only, no card body — a table of contents the client always has, refreshed for free every turn. Noticing still isn’t acting, so the second half is structural, not advisory: every follow_up call carries a required field, unrecognized_terms, populated or explicitly left empty. That trades “remember to check” — the kind of standing vigilance models quietly drop over a long trajectory — for “answer one narrow question,” which they do reliably.

Every advice payload ends with a targeted ask-back — “when you’re done, one line back with session X; even a failure is useful” — aimed at the specific claim Sourcefinder is least sure about. Every client becomes a sensor on the card’s weakest assertion, and the ask costs the client nothing: free-form text, one call, never required structure.

Offline — the night shift. Between sessions, Sourcefinder works only from its own logs, in a loop:

• Collect. Every request and every piece of feedback lands, raw and append-only, in an episodic log that no one ever sees.
• Verify. Feedback is a claim about a card (“I couldn’t find what you said was there”) — and claims get checked before anything changes: re-run probes, deep-check the source, cross-examine other episodes. Feedback never edits a card directly. That keeps mistaken feedback from corrupting the map, dedups three clients hitting the same stale promise into one confident fix, and makes the feedback channel impossible to steer Sourcefinder with. The payoff for this discipline is proven: synthetic LLM-generated labels recently matched human-supervised routing quality (SLAT) — verified task outcomes are better labels still.
• Correct & restructure. Verified corrections become incremental card edits. The request corpus is then treated the way a DBA treats a query workload: a card absorbing most sessions gets split and deepened, a topic with no card of its own gets one, routing lines clients keep ignoring get rewritten. The foundation tree is maintained the same way: proposed prerequisite edges verified before they land, over-prescribed ones pruned (below). “Optimal” isn’t a vibe — it’s first-suggestion hit rate, computed from the log.
• Extrapolate & report. Projected future question classes — from episodes, from watcher signals, from the calendar of things the org is starting — are diffed against a coverage matrix (question classes × best internal source × best external source × gap). Gaps that survive become Sourcefinder’s one outbound artifact: a ranked report of unmet information needs with evidence attached, filed for humans to handle. Recurring questions with no primer are themselves a gap — a domain everyone asks about and no document teaches is onboarding debt. The symmetry is the point: clients ask Sourcefinder for help finding things; Sourcefinder asks the org for help creating them.

Two memories sit underneath: the raw episodic log, and the curated index — updated only by this loop, only in delta edits. Never monolithic rewrites; recent self-improving-context research (ACE) shows that path collapses accumulated knowledge.

The index: one markdown file, in context before you ask

What Sourcefinder knows is a single markdown file with a strict convention — and the whole thing is loaded into the LLM’s context before the client’s question arrives, prompt-cached, so a question always meets a fully-loaded map. That choice is deliberate. Routing is a judgment task — matching an intent against messy prose descriptions — which is what LLMs are good at and embedding similarity is bad at: ReSLLM showed an LLM can rank ~150 candidate sources from descriptions alone — though it scored each resource in turn; Sourcefinder makes the stronger bet that the same judgment works list-wise, with the whole map in view at once. Fidelity is low-risk for the same reason: recommendations are transcribed from text sitting in context, not recalled from model weights.

The convention — one ## section per source, fixed fields:

1
## historical-performance        ← stable id; future file boundary
2
- what/what-not: metric trends over time — NOT raw query data
3
- kind: confluence-space · scope: internal · trust: primary
4
- contains: weekly dashboards, trend writeups (~200 docs, active)
5
- read-first: metrics-glossary
6
- how: /search API, semantic works well; browse → space tree for reports
7
- example: "Q2 churn by segment" → trend report T-88   [probe ✓ 9/1]
8
- combines: analytics-warehouse (raw queries) · supersedes: perf-old
9
- health: helpful 12 · harmful 1 · last_verified 9/1

The first line is the discriminator — what it’s for and what it’s not for — because at hundreds of cards, discrimination beats description. Example queries double as the canary probes (below). And overlap between sources is declared, not guessed. The read-first field doesn’t name documents; it points at a node in the file’s small foundation block and is resolved into a root-first reading path at answer time — the next section is about that block.

On overlap. Realistically 10–20% of an org’s sources cover similar ground — old spaces vs. new, wiki vs. dashboard. Undeclared twins force the router to guess freshness every session; declared ones make it a lookup: supersedes, preferred, see-also, not-for turn ambiguity into an answer. Twins that haven’t declared themselves are the night shift’s problem — clients bouncing between similar sources are the detector.

Honest scale. 200–500 cards at ~300 tokens each is a 100–150k-token file: comfortably inside a 1M-token context, near the edge where routing quality needs discipline (class-grouped sections, discriminators first, ordered-not-capped answers). The gauge is in the log — hit-rate@1 and hit-rate@k from a standing eval suite of 50–100 labeled questions, replayed on every index edit. When those numbers sag, each ## becomes a card file and the routing table becomes a generated index; if the sheet ever outgrows attention (~1–2k sources), embed the routing lines, rerank to a top-20, and let the LLM still make the call. Same mental model throughout — the file format is chosen so every step of that escalation is mechanical.

Clusters. Some sources are really source groups — a Confluence instance with fifty spaces, a GitHub org with two hundred repos — where the parent’s own kind/scope/trust/how is real and useful, but a specific child is sometimes what the client actually needs. Giving every child a full card defeats the point of a summarized top level; giving none means the access details a client needs never surface. So a cluster card carries the edge, not the content:

1
## confluence                     ← cluster card: too many spaces for one card
2
- what/what-not: company Confluence instance — general kind/scope/trust only, not a
3
  specific space
4
- kind: wiki · scope: internal · trust: primary
5
- children: 47 spaces · see child-index: confluence-spaces   [derived, not authored]
6
- how: browse via space directory; each space gets its own how-to once expanded
7
- health: helpful 8 · harmful 0 · last_verified 9/5

The children themselves live in a separate fragment, loaded only when a client actually asks for it — the same escalation named above, applied locally: a ## splitting into its own file the moment one cluster, not the whole index, outgrows a single card. Each child declares the edge from its own side, extends: confluence, and states only what differs:

1
### historical-performance-space         ← child of confluence, in confluence-spaces
2
- extends: confluence
3
- what/what-not: metric trends over time — NOT raw query data
4
- contains: weekly dashboards, trend writeups (~200 docs, active)
5
- example: "Q2 churn by segment" → trend report T-88   [probe ✓ 9/1]

kind/scope/trust, how, and read-first are inherited from confluence unless a child overrides them; what/what-not, contains, and example are always per-child, because those are exactly the fields a routing decision hinges on and the ones no parent summary can honestly state on a child’s behalf. Resolution is the same walk-to-root logic the foundation tree already uses for parent: — one primitive, reused, not two. And the parent’s children count is never hand-maintained: it’s counted at index-build time from however many entries declare extends: confluence, so it can’t drift the way a manually edited number would.

Drilling into a known cluster is a different act from finding one in the first place, though, and the difference matters for where each mechanism lives. Drill-down — “this candidate is a cluster, expand it” — belongs in the answer for a card the client already has in hand; it’s local, and the payload says so directly. Finding a cluster the client didn’t know to ask about is the mid-session-drift problem above, solved the same way as any other new topic: the condensed index-of-index has to carry an entry for every cluster’s top card, same as any flat source, so a client noticing something unrelated can still detect that a source area exists — via the usual follow_up check — before drilling into it. Clusters are visible at the routing layer; only their children stay hidden until asked for.

Read-first is a tree, not a list

The “read if you’re new” section of every answer deserves its own structure, because background isn’t a list — it’s ordered. A glossary is no use before the deck that explains why its distinctions matter, and that deck presupposes the org’s vocabulary. So Sourcefinder models read-first material as a small prerequisite tree — the foundation — whose roots are the few documents everyone should read and whose leaves sit just above the sources. Cards point at foundation nodes; the reading order is resolved at answer time by walking each pointer up to its root and emitting the chain root-first.

Store edges, not chains. Foundation nodes live in their own block in the same file, above the ## source sections:

1
### metrics-onboarding            ← foundation node (the tree's interior)
2
- what: how we define and instrument metrics — prerequisite for analytics sources
3
- parent: (root)
4
- pointer: Confluence "Metrics Onboarding" deck
5

6
### metrics-glossary
7
- what: cohort vs. segment, MRR vs. bookings
8
- parent: metrics-onboarding
9
- pointer: glossary page, same space

…and the card above says just read-first: metrics-glossary. Replace the glossary and one node is edited; every downstream card inherits the fix — delta edits all the way down. A card may carry several pointers with independent roots (a small forest, merged and de-duplicated into one ordered reading list). And roles attach to edges, not documents: the same Confluence space can be a searchable source and someone’s prerequisite, so a source that turns out to be foundational is promoted into the interior without losing its card.

Depth is a metric. Two to four levels is the healthy range; anything deeper is really a missing primer — onboarding debt made visible as a chain-length outlier. The fix for a deep chain is not a longer chain; it’s a gap report asking for the document that should sit at its root.

Where the edges come from, in descending order of trust:

• Harvested structure. Most real sources publish their own hierarchy — Confluence page trees, doc-site breadcrumbs, “prerequisites” sections, README → docs/ chains. The watcher already visits these hosts for change signals; the same pass collects candidate parent edges for free.
• LLM-proposed edges. Where structure is implicit, a model reading the source’s own pages proposes them. The output shape matters: strata, not a chain — must-read-first / context / target — judged pairwise (“would a reader who skipped this be misled, or just fine?”) and topologically sorted. Asking for a total order invites over-ordering; relevance graphs are dense, prerequisite graphs are sparse. Two caveats keep this honest. First, the model is not a novice: a frontier LLM reads the dashboard docs fine without the glossary, so its own reading experience systematically under-detects exactly the primers this tree exists to point at. Second, every proposed edge is a claim, verified like any other before it touches a card — and one probe is even experimental: a fresh model instance answers a comprehension question about the child with and without the claimed parent in context, and an edge that doesn’t move comprehension is decoration. The proxy reader shares the model’s surplus knowledge, so that probe is a floor, not ground truth.
• The log. The only signal that knows what a real newcomer needed. Sessions that failed on vocabulary the chain should have covered mark a missing edge; clients who skipped the chain and succeeded anyway mark an over-prescribed one. The eval suite reads it directly: found-rate, conditional on whether the path was actually read.

The roots converge — two hundred sources share a handful of foundations — so the “read if you’re new” prefix is nearly identical across answers: the most prompt-cache-friendly content the server emits, and org-wide onboarding consistency for free.

And before it looks otherwise: this is not the up-front taxonomy that No taxonomy up front, below, argues against. No foundation node exists up front. Each was promoted on demand, because a real datasource admission needed it — the tree grows from use, not from a committee.

How the index stays current

Two channels feed it, continuously:

Clients. Feedback is the high-signal channel. “I looked at the reference you suggested and couldn’t find what you said was there” is overpromise detection — the failure mode probes are worst at catching. “I found a better source — there’s now a Confluence space for historical performance” is discovery: it arrives attached to proven demand, becomes a draft card, gets probe-verified and background-checked before promotion (the full admission flow below), and may supersede the source it beat.

Sourcefinder itself. A quiet crawler watches company datasources for change, not content: new repositories in the org’s Git host, recently-updated doc spaces, new channels, new projects — cheap signals that touch cards (change rate, last verified) and trigger verification. It never re-ingests what it watches. A low-friction human tip line covers what no crawler can see, because new projects start as conversations, not as API events.

Probes. Underneath both, the verified example queries in every card run on a schedule against the real sources — catching schema drift, permission changes, and quietly-dead sources. Descriptions are claims; the probes exist to test them.

How a new source gets admitted

A suggested datasource passes two gates before it becomes a card: claim verification (the promotion flow above) and a background coverage check — can a newcomer, armed with the current foundation tree, actually understand what this source holds? The check is grounded, not vibes. Sample the candidate’s representative pages; extract every assumed-knowledge item — the jargon, acronyms, service names, “see the T-88 report” — each with the page it appears on; then check each item against the foundation tree in one in-context pass. The interesting items are org-specific, which is exactly what the LLM doesn’t already know. Three outcomes, three moves:

• Covered → the card’s read-first points at the deepest foundation node it needs. Done.
• External gap (“what is a cohort?”) → add a foundation pointer to public material — no authoring, no org burden, and link-checkable by probes like any other claim.
• Internal gap → if a document somewhere covers it, promote it to a foundation node; the cheapest discovery channel is asking the suggester “where did you learn this?”, which rides the existing feedback loop. If nothing does, that’s the strongest gap report Sourcefinder can file — “unusable by newcomers until someone writes the primer,” evidence attached — and the source is admitted anyway, flagged “background: missing — senior-friendly only”, so routing answers say so honestly. Admission is not binary.

One honest wrinkle: reading sampled content presses on the thin principle. V1 runs on what the suggester supplies with their feedback (two or three page titles or excerpts is one more ask-back line) plus the structure the watcher can already see — page trees, breadcrumbs, and section names are often enough for assumption extraction. A scoped read-only night-shift credential, used only for admission checks, is the deliberate amendment if that proves too thin — and it would be the first credential Sourcefinder holds.

What “getting smarter” means

In this design the intelligence lives in the index, not the model — the router runs with reasoning off because the hard thinking was already done, at night. So getting smarter never means a smarter model; it means the map improving along five levers:

1. Breadth. More sources only in the sense that matters: question classes with a good answer. The unit of growth is the class, not the card; a source’s value is its exclusive coverage — the classes only it can answer — and duplicates are negative value, not zero, because they spend attention budget on ambiguity. Sometimes getting smarter means removing: demoting dead sources, merging declared twins.
2. Better descriptions — more discriminating, not more informative. At hundreds of cards, a description’s first job is to exclude, and honesty beats richness: an overpromising description routes worse than a plain one.
3. A truer picture of what’s searchable — kind, scope, trust, size, freshness, and the negative space: what the source doesn’t have. Expectations set before the query runs.
4. Clearer how-to instructions — the testable kind of clear, since every example query doubles as a probe and a stale instruction announces itself.
5. Better background coverage — findable isn’t understandable; the foundation tree is the fifth lever, and it’s why breadth alone was never the goal.

Every failed session decomposes into exactly one of them:

“Optimal isn’t a vibe — it’s first-suggestion hit rate” now has a steering wheel: rank the levers by failure counts in the log, pull the heaviest first. The night shift’s work queue is that ranking.

Why build this (and why nothing like it exists — as far as I can find)

The problem is measurable: in a recent benchmark of multi-source information seeking (InfoMosaic-Bench), 22% of frontier-model failures came from selecting the wrong source or tool — not from reasoning. Routing is the bottleneck.

The ingredients are all published, separately. Federated-search resource selection is classic IR (ReSLLM again). Data catalogs like OpenMetadata and DataHub are pivoting agent-facing — by ingesting and owning the metadata, exactly the heavy, platform-shaped direction this project avoids. MCP registries route agents to public tools. A 2026 wave of “librarian” agents (EMBL’s) knows where knowledge lives — but does the searching for you. Proactive-agent research (ProAct) anticipates needs — for one dialogue, not an org’s catalog. Experiential learners like ExpeL distill what worked — inside one agent’s memory.

What I can’t find, in products or papers, is the synthesis: thin, tool-poor routing advice over sources it doesn’t control, spanning the company boundary, kept honest by probes and verified feedback, and grown toward the questions the org will ask next. The pieces are all claimed; the shape isn’t. (Survey current as of September 2026 — I’d love to be pointed at prior art.)

No taxonomy up front

Microsoft’s Viva Topics — auto-curated topic pages across an org — retired in 2025, never having escaped its “bootstrap conundrum”: it needed organizations to define taxonomies up front, and organizations wouldn’t. Taxonomies are a human problem. So Sourcefinder asks for nothing up front: question classes come from what the org is, coverage grows from what people failed to find, and cards earn their keep by being probed and used — or they get demoted.

Status

Design is done; the build is starting. V1 is deliberately small:

• A FastMCP server with the three tools (where_can_i_find, report_outcome, follow_up).
• The router runs on GPT-5.6 Luna with reasoning off — routing is a judgment task the index has already done the hard thinking for, so per-query latency is decode-bound (~1–3s at 500 sources) and cost is under a cent. Terra at low effort is the escalation tier when hit-rate sags; the model is a config option, and the eval suite decides when that changes.
• One markdown index (the format above — source cards plus the foundation block) loaded wholesale into context, plus a SQLite episodic log.
• A handful of seeded cards, a hand-bootstrapped foundation tree, one watcher, the probe loop — and the routing eval suite (labeled questions → hit-rate@1/@k, each failure decomposed into the lever it indicts) from day one.
• No ingestion. No connectors. No credentials to anything but its own notes — admission checks run on what the suggester supplies and the structure the watcher can already see.

If it works, the payoff isn’t a better search engine. It’s an org whose knowledge of its own knowledge stops living in the heads of the people who are about to leave.