"Every Claude Code Memory System Compared"

AI and Coding
1

“Every Claude Code Memory System Compared”

I pasted a Gemini-generated summary below the video. I haven’t tried these techniques yet.

Let me know if you’ve tried anything like this.

Claude Memory Systems: Techniques & Levels

This document summarizes the memory management strategies for Claude (specifically Claude Code) as discussed in the video * “Every Claude Code Memory System Compared”*. The systems range from simple native files to complex, multi-tool databases.

Core Memory Principles

Regardless of the level, every memory system answers one question: How does Claude pull the right context at the right time?

  • Storage: Where the memory lives (Local vs. Cloud, Markdown vs. Vector).
  • Retrieval: How Claude gets the data (Automatic injection vs. Manual tool calls).

The 6 Levels of Claude Memory

Level 1: Native Implementation

The simplest method using Claude’s built-in features.

  • claude.md: A system prompt in file form. Best for brand rules, coding styles, and project constraints.
    • Technique: Keep it under 200 lines to avoid “Context Rot.” Reference external files for larger docs.
  • memory.mmd: Claude’s auto-memory system that indexes feedback and project history.

Level 2: Structured Injection (The “Hook” Method)

Improving reliability using session hooks.

  • Technique: Use a session_start hook to inject a memory.mmd index into the context automatically.
  • Structure: Organize memory into folders: /general, /domains, and /tools.
  • Maintenance: Use a command like “reorganize memory” to get Claude to deduplicate and clean up its own notes.

Level 3: Semantic Search (Vector Memory)

Moving beyond keyword search to “meaning” search.

  • Mem Search (by Zilliz): Extracts the “Open Claude” architecture for use in Claude Code.
  • Technique: Uses a user_prompt_submit hook to search a local vector database and inject the top 3 most relevant memory chunks into every prompt before you even hit enter.

Level 4: Verbatim Recall (Mem Palace)

For when you need word-for-word accuracy of past decisions.

Level 5: The “Second Brain” (Knowledge Base)

Best for deep research rather than operational tasks.

  • LLM Wiki (Andrej Karpathy’s method):
    • Raw Folder: You drop source docs (PDFs, transcripts).
    • Wiki Folder: Claude owns this; it writes and cross-links markdown files to build a private Wikipedia.
  • Recall: A hosted version that uses browser extensions to auto-build knowledge graphs.

Level 6: The Universal Brain (Multi-Tool)

For users who switch between Claude, ChatGPT, and Cursor.

  • Open Brain (by Nate Jones): A central Postgres/Supabase database.
  • Technique: Connect all AI tools to one “Thoughts” table via MCP (Model Context Protocol).
  • Benefit: Your memory is portable; if a new AI tool comes out in 6 months, you just plug it into your existing “brain.”

Recommendations for Getting Started

  1. Just starting? Use Level 1. Master claude.md and keep it lean.
  2. Using it daily? Move to Level 2. Set up a session hook to inject your project index.
  3. Losing track of old notes? Implement Level 3 (Mem Search) for semantic retrieval.
  4. Need cross-platform memory? Explore Level 6 (Open Brain) to keep your data in a private database you own.
2 Likes
2

I was reading about this one and saw a link to this database. I haven’t used it.

Edit: I also stumbled on this:

1 Like
3

I have tried many, many approaches to getting Claude in particular and Codex, Gemini, GLM-5 via opencode, to stop making the same mistakes. When I call them out on a mistake that was clearly preventable by the CLAUDE.md, AGENTS.md, or other docs/agent-instructions.md that I’ve asked them to read, they respond “You are right, I should have done that” – no good explanation for why they chose to ignore instructions.

There is a lot of research in this area, and I am trying to implement some of it. Agents do better with examples (In-context learning and In-context reinforcement learning). A less-expensive LLM can get a better result in multiple attempts than a more-expensive LLM in one attempt.

One problem I observe is that the more info you give an agent, the less it retains/uses (context rot). I’ve used MCP, Rag, direct agent CLI tools in the path, and even Recursive Language Model (RLM) to try to direct agents.

My current approach is to limit the info provided per agent invocation. To avoid long context-compaction delays, I kill the agent session after a step or two, and my tool tracks what is the current goal and what is the next goal. This has worked pretty well for me (I’v generated a ton of tools and demos recently).

My new approach is agent coordination. Say I want an agent per repo for a system of interrelated tools, I have a (smart) coordinator agent orchestrate, but delegate to each (less-smart?) worker agent to focus on updating one repo, for one task at a time. I’ve built a lot of tooling around this, including a shared agent Wiki and events/mailboxes, and a web ui (not published yet).

Concrete example: I have an emulator for an embedded CPU, written in Rust, that runs a p-code VM written in C, that loads an interpretted language, BASIC, written in Pascal, that runs a Smalltalk interpreter, written in BASIC (like the original). That is half-a-dozen repos. When the Smalltalk repo agent needs a BASIC feature like DIM, the basic repo agent implements that in Pascal, but runs out of symbol table, so the p-code repo agent changes the size of the symbol table, etc.

Another use-case, say I want to generate a youtube video from some inputs (an idea, an image, a repo). The multi-agent set up can divide and conquer. One takes the idea, reads the repo, and writes a script. Another agent creates SVG slides from the script. Another agent genrates narration per slide. Another agent reviews all of this. Another agent converts the narration text to audio via a TTS model. Another agent checks the generated audio using whisper to turn it back to text that can be compared against the narration text (and make corrections). Another agent turns SVG slides into short video segements with audio narration added. Then another agent updates a preview site to show me each segment: its thumbnail, its narration text, the whisper text, an audio player, and a video player. I can accept or provide feedback. When all segments are approved, another agent concats everything into one video that can be previewed or uploaded to YouTube (where I can edit the subtitles).

I’ve done most of the above manually (acting as the coordinator) and I want to delegate coordination to one agent.

If you implement the techniques you/your-link describes and still get less-than-great results, let me know and I might be able to provide tooling that focuses an agent on a task with examples of what and what not to do.

1 Like
4

Learning how to work with multiple agents like that is on my to-do list. I’m manually using multiple agents, but I’m not sure how to automate it.

1 Like
5

I did a blog/vlog on the history of Wikis, and took that repo (6 wiki implementations) and added a couple of agent-friendly REST APIs to the git-backed wiki, so that agents could edit the same page and communicate a feature request or announce a new feature.

This worked okay, but each agent would stop, waiting for its request to be handled or stopping after a feature was implemented. I had to manually “poke” each agent to: (1) describe your blocker on a wiki page; (2) check the wiki page for blockers you can fix; (3) update the wiki page when a feature is impllemented (to unblock); (4) check the wiki page for a feature you were waiting for; (5) give feedback on the wiki if needed.

So I was still manually tabbing from agent to agent, and looking at the wiki (I hate manual repetitive work).

So I looked into what it would take to implement a PTY and control an agent through its TUI but collect all the UIs on one web page and add code that checks each TUI to detect when an agent is idle (usually stopped to ask me a question or to announce that it is done). In addition to the Wiki (broadcast) I added event mailboxes (single recipient) to make it easier to direct a request to an agent, or send a “done, try it” message back to the requester. Now the code can “poke” agents for me while I watch. I only have to intervene to answer questions or provide direction. This is almost working–still fixing UI issues and trying different demos. I hope to produce a video about this soon (using this to create the video using different agents for slides, narration, compositing, splicing,…)

One downside (not directly related to this because I’m not using it heavily yet) is that I’ve run out of Claude Max plan daily/weekly quotas repeatedly (I’m working on too many projects at once). So I am also planning ways to divide the coordinated, multi-repo/agent, work across different agents and plans, including local GPU LLMs (which have so far not been good enough for non-trivial coding tasks–but I’m slowly working on that, too, using fine tuning and the aforementioned inference time “training”.

At the end of the day I am trying to build a team of power efficient agents that collaborate on a non-trivial project, like producing a video, implementing a new programming language, developing an RTS or TD game, generating music, sound effects, etc. Ideally this would run entirely locally using solar power. Sunlight + Idea + AI-cluster = profit?

1 Like
6

Sounds interesting. What kinds of AI tools do you use for video creation?

7

I used mainly Claude Code to call various services, like VoxCPM for TTS, whisper for STT verification, MuseTalk for lipsync, Nightcafe.studio for text-to-image, text-to-video, image-to-video, but mainly Claude generates SVG and html to produce slides. I use my own tools to generate intro/outro music. I use “vhs tape” to record command line demos and OBS to record browser demos.

I use Claude with ffmpeg to sample frames from vhs or OBS videos to summarize what was recorded, and generate narration for eventual TTS using a clone of my voice.

Claude calls image magick and ffmpeg to manipulate, to resize, and for compositing, concatenating, etc.

I also have a couple of text-to-video models from huggingface running locally, but for 5 second intro/outro videos I usually just use one of the cheaper models on NightCafe’s web site, like Seedance 1.5 Pro to create this RTS cut-scene

1 Like
8

Thanks, I’ll check some of those out. There are so many AI tools that it’s hard to keep track of everything.

9

It’s almost like you need an AI agent to keep track of all the AI stuff :slight_smile:

1 Like
10

It’s almost like you need an AI agent to keep track of all the AI stuff :slight_smile:

Pretty much the conclusion I’ve come to. I’ve had some success with very long running sessions with Claude, mirroring what one of the Codex developers is doing. I try to proactively manage context with compaction, but a recent major fail has me restarting sessions at the moment.

1 Like