How to Build a Local MCP Memory Server with sqlite-vec for AI Agents

Key Takeaway: Zero-Cloud Persistent Recall for Coding Agents

Autonomous agents lose context the moment a terminal session terminates or when conversation transcripts exceed context window limits. Sending proprietary source code snippets to cloud vector databases like Pinecone introduces latency, network egress bills, and data compliance risks.

Building a local Model Context Protocol (MCP) memory server using sqlite-vec and local CPU embeddings via FastEmbed gives Claude Code, Cursor, and custom agent loops persistent semantic recall. Search queries resolve in milliseconds on local Linux NVMe storage with zero external API calls.

+-----------------------------------------------------------------------+
|  AI CODING CLIENT (Cursor / Claude Code / CLI ReAct Loop)             |
+-----------------------------------+-----------------------------------+
                                    |
            JSON-RPC over stdio     | tools/call: search_memory, store_memory
                                    v
+-----------------------------------------------------------------------+
|  PYTHON FASTMCP MEMORY SERVER                                         |
|  - FastEmbed ONNX Runtime (BAAI/bge-small-en-v1.5, 384 dims)          |
|  - In-process vector serialization (struct.pack float32)             |
+-----------------------------------+-----------------------------------+
                                    |
            C Extension Direct Call | SELECT rowid, distance FROM vec_items
                                    v
+-----------------------------------------------------------------------+
|  SQLITE-VEC STORAGE ENGINE (~/.local/share/agent-memory.db)           |
|  - Pure C vector search extension loaded via sqlite3.enable_load_ext  |
|  - Fast k-NN distance ranking on local NVMe disk                      |
+-----------------------------------------------------------------------+

Why sqlite-vec Beats External Vector Databases for Local Agents

External vector databases like Qdrant or Milvus require dedicated Docker containers, 500MB+ memory overhead, and open TCP ports. For local developer tooling, single-file SQLite databases deliver superior operational simplicity.

Because sqlite-vec runs in-process as a pure C SQLite extension without HTTP or gRPC serialization overhead, SIMD-accelerated k-NN scans over tens of thousands of 384-dimensional float32 vectors complete in a few milliseconds on local NVMe storage.

Step 1: Install Dependencies and sqlite-vec on Linux

Install fastmcp, sqlite-vec, and fastembed. Using fastembed runs quantized ONNX models on CPU without requiring PyTorch or CUDA runtimes:

pip install fastmcp sqlite-vec fastembed

Verify that Python loads the sqlite-vec shared library without symbol lookup errors:

python3 -c "import sqlite3, sqlite_vec; conn = sqlite3.connect(':memory:'); conn.enable_load_extension(True); sqlite_vec.load(conn); print('sqlite-vec version:', conn.execute('SELECT vec_version()').fetchone()[0])"

Expected output:

sqlite-vec version: v0.1.6

Step 2: Implement the MCP Memory Server (server.py)

Create memory_server.py. This script registers two MCP tools: store_memory and search_memory:

import os
import sqlite3
import struct
from pathlib import Path
from typing import List
import sqlite_vec
from fastembed import TextEmbedding
from mcp.server.fastmcp import FastMCP

DB_PATH = Path.home() / ".local" / "share" / "agent_memory.db"
DB_PATH.parent.mkdir(parents=True, exist_ok=True)

mcp = FastMCP("sqlite-vec-memory")
embed_model = TextEmbedding("BAAI/bge-small-en-v1.5")

def get_connection():
    conn = sqlite3.connect(str(DB_PATH))
    conn.enable_load_extension(True)
    sqlite_vec.load(conn)
    return conn

def init_db():
    conn = get_connection()
    with conn:
        conn.execute("""
            CREATE TABLE IF NOT EXISTS memory_docs (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                category TEXT NOT NULL,
                content TEXT NOT NULL,
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            )
        """)
        conn.execute("""
            CREATE VIRTUAL TABLE IF NOT EXISTS vec_memory USING vec0(
                doc_id INTEGER PRIMARY KEY,
                embedding float[384]
            )
        """)
    conn.close()

init_db()

def serialize_f32(vector: List[float]) -> bytes:
    return struct.pack(f"{len(vector)}f", *vector)

@mcp.tool()
def store_memory(category: str, content: str) -> str:
    """Store architectural decisions, error workarounds, or user preferences into persistent memory."""
    embedding = list(embed_model.embed([content]))[0].tolist()
    conn = get_connection()
    with conn:
        cursor = conn.execute(
            "INSERT INTO memory_docs (category, content) VALUES (?, ?)",
            (category, content)
        )
        doc_id = cursor.lastrowid
        conn.execute(
            "INSERT INTO vec_memory (doc_id, embedding) VALUES (?, ?)",
            (doc_id, serialize_f32(embedding))
        )
    conn.close()
    return f"Stored memory record id={doc_id} under category='{category}'."

@mcp.tool()
def search_memory(query: str, limit: int = 4) -> str:
    """Perform semantic search over stored memories to recall past context and decisions."""
    query_vector = list(embed_model.embed([query]))[0].tolist()
    conn = get_connection()
    cursor = conn.execute("""
        SELECT d.id, d.category, d.content, v.distance
        FROM vec_memory v
        JOIN memory_docs d ON v.doc_id = d.id
        WHERE v.embedding MATCH ? AND k = ?
        ORDER BY v.distance ASC
    """, (serialize_f32(query_vector), limit))
    rows = cursor.fetchall()
    conn.close()

    if not rows:
        return "No relevant memories found."

    results = []
    for row_id, cat, content, dist in rows:
        results.append(f"[{cat}] (score: {dist:.3f})\n{content}")
    return "\n\n---\n\n".join(results)

if __name__ == "__main__":
    mcp.run()

Step 3: Configure Claude Code and Cursor Clients

Register the server in Claude Code by editing ~/.claude/claude_desktop_config.json or ~/.cursor/mcp.json:

{
  "mcpServers": {
    "local-memory": {
      "command": "python3",
      "args": ["/home/user/.local/bin/memory_server.py"]
    }
  }
}

Verify that the client detects both tools via claude mcp list:

claude mcp list

Expected output:

local-memory:
  - store_memory: Store architectural decisions, error workarounds, or user preferences.
  - search_memory: Perform semantic search over stored memories.

Verification Workflow: End-to-End Agent Recall

You can verify persistent vector recall across separate Claude Code or Cursor sessions using this three-step workflow:

1. Store Turn: Instruct the agent to store a project-specific constraint:

   > store_memory(category="database", content="Postgres migration v14 requires running ALTER TABLE with CONCURRENTLY disabled on replicas.")

2. Session Restart: Close the IDE or clear the active conversation context so no in-memory history remains. Context memory is now empty.
3. Recall Turn: In a fresh session, prompt the agent to query its vector store:

   > search_memory(query="postgres migration replicas rule")

   Confirm that sqlite-vec returns the matching record with the lowest cosine distance score and that the agent applies the constraint to its generated SQL migration script.