with-context-mcp: Persistent Memory for AI Agents, With Supervision

One of the first things you notice when building with AI agents is that they're stateless by default. Every session starts from zero. The agent you were working with yesterday, the one that understood your project structure, your conventions, and the decision you made about the database schema, is gone. You explain it again. And again.

The obvious fix is to give agents persistent memory. The less obvious problem is that naive persistence creates its own issues: notes that accumulate without structure, context that grows until it's noise, no way to review what the agent has actually recorded. You trade amnesia for something closer to hoarding.

with-context-mcp is built around a different approach: structured, supervised note-taking for AI agents, backed by Obsidian.

How it works

with-context-mcp is an MCP server that exposes over 33 tools for managing notes in an Obsidian vault. Agents can create, update, search, and organise notes across sessions, but the notes live in a vault you own and can inspect. The supervision layer isn't an abstraction; it's your actual Obsidian instance, with all the search and organisation tools that come with it.

The server connects via the Obsidian Local REST API plugin. Once running, any MCP-compatible client (Claude Desktop, OpenCode, Cursor, Windsurf, Cline, Continue.dev, GitHub Copilot) can use it.

Setup in Claude Desktop or OpenCode looks like this:

{
  "mcp": {
    "with-context": {
      "type": "local",
      "command": "npx",
      "args": ["-y", "with-context-mcp"],
      "environment": {
        "OBSIDIAN_API_KEY": "{env:OBSIDIAN_API_KEY}",
        "OBSIDIAN_API_URL": "{env:OBSIDIAN_API_URL}",
        "OBSIDIAN_VAULT": "{env:OBSIDIAN_VAULT}",
        "PROJECT_BASE_PATH": "{env:PROJECT_BASE_PATH}"
      },
      "enabled": true
    }
  }
}

The agent then has access to the full note management toolset under your specified vault and project path. You don't need to configure individual tools or tell the agent when to use them. It picks them up automatically when it needs to document something, track a decision, or organise context.

The supervision angle

The design choice to use Obsidian as the backing store is deliberate. An agent that writes notes to a proprietary black-box format gives you persistence without visibility. Notes in Obsidian are plain markdown files in a folder on your machine: readable, searchable, editable, and deletable by you at any time.

You can see exactly what the agent is recording. You can correct it. You can organise it. When something goes wrong, and with agentic workflows something always eventually goes wrong, you have a readable audit trail rather than an opaque state blob.

Installation

Prerequisites: Obsidian running with the Local REST API plugin enabled.

npx -y with-context-mcp

Set the environment variables, point it at your vault, and the agent has persistent, structured memory across sessions.

The broader point

Persistent memory for agents isn't a feature. It's infrastructure. The question isn't whether your agents should remember things across sessions, but how that memory is structured, where it lives, and whether you can inspect and correct it. with-context-mcp is a pragmatic answer to that question: standard tools, a format you already know, and full visibility into what the agent is recording.

Source and full documentation on GitHub.