Connecting a local RAG store to your tools with MCP

The pattern: wrap your vector store in an MCP server so any MCP-aware client — an IDE, an agent, a chat app — can retrieve from it without a custom integration. Your documents never leave your machine; the client just calls a search tool.

The shape

Client (IDE / agent)
   │  MCP  (stdio or HTTP)
   ▼
MCP server  ──►  embedding model (local or API)
                     │
                     ▼
                 vector DB (pgvector / Qdrant / Chroma)

Minimal server

An MCP server for RAG exposes one tool and, optionally, resources:

• search_documents(query, k) → embeds the query, runs nearest-neighbour, returns the top-k chunks with source metadata.
• Optionally expose each document as an MCP resource so the client can pull the full text on demand.

Keep chunking and embedding inside the server. The client should not know or care which embedding model or DB you use — that is the whole point of the protocol.

Decisions

Choice	Local-first pick	When to deviate
Embedding model	A small local model on the NPU or GPU	Use an API model if recall matters more than privacy
Vector DB	pgvector if you already run Postgres; Qdrant otherwise	A managed DB only if the corpus is huge
Index	HNSW	Flat/brute-force under ~50k vectors — simpler, exact
Transport	stdio for a local IDE; HTTP for a shared server	—

Failure modes

• Stale index — re-embed on document change, or the client retrieves deleted content.
• Chunk too large — the client's context window fills with three chunks. Aim for 200–500 token chunks with overlap.
• No source metadata — the model can't cite, and you can't debug a bad answer.

The rule

One MCP server, one search tool, all retrieval logic behind it. If you find yourself exposing "embed this" or "which DB" to the client, you have leaked an implementation detail the protocol exists to hide.