Stop Taking Notes. Start Building a Brain That Thinks Back.
Here’s a number that stopped me cold when I first encountered it in the cognitive-load literature: the working memory capacity of a healthy adult brain is roughly four chunks of information at once. Not forty. Not four hundred. Four. Everything else you think you’re holding in your head while you’re in a meeting, reading a paper, or sketching an architecture diagram — most of it is already gone by the time you reach for it.
I built a second brain because I believed that number. This is what I learned building it.
The Problem Isn’t That You’re Disorganized
The productivity-tool industry has spent twenty years selling the premise that your problem is a filing system. Get the right tags, the right folders, the right capture ritual, and the information you need will surface when you need it. Tiago Forte codified this into a methodology — Building a Second Brain, organized around the PARA framework (Projects, Areas, Resources, Archives) — and it genuinely helps. Obsidian has a devoted following of people who have built intricate, cross-linked knowledge graphs. Notion can hold almost anything. Mem tried to apply AI-flavored search on top of notes. Rewind tried to capture literally everything that crossed your screen.
I’ve used most of these. The honest evaluation: they’re all sophisticated inboxes. PARA gives you a principled way to decide where a note goes. Obsidian’s graph view is genuinely satisfying to look at. But satisfaction and utility aren’t the same thing. The core experience of all these tools, when you’re not in the mood to maintain them, is a gradually accumulating archive that you periodically feel guilty about. The information went in. It didn’t come back out in a form that was useful at the moment you needed it.
The ceiling these tools hit isn’t a UX problem. It’s an architectural one. They are retrieval systems that require you to know what you’re looking for. That’s not how memory works. Real memory is associative, context-sensitive, and often surfaces things you didn’t know you needed. Human memory also, of course, loses things constantly and at random — which is exactly where this story starts.
The Extended Mind and Why Offloading Is Legitimate
Philosophers Andy Clark and David Chalmers published a paper in 1998 that still generates arguments: “The Extended Mind.” The core claim is that cognition isn’t bounded by the skull. When you use a notebook to remember a phone number, the notebook isn’t a crutch — it’s part of the cognitive system. The thinking is happening across the person and the tool together. Otto, their thought-experiment character with memory difficulties, isn’t diminished by his notebook; his notebook is his memory.
Cognitive-offloading research in the decades since has largely supported this view empirically. Offloading information to external systems frees up working memory for the tasks working memory is actually good at: reasoning, synthesis, generating new ideas. The guilt that productivity culture attaches to “needing to write things down” is backwards. Writing things down and then trusting what you wrote is a legitimate cognitive strategy, not evidence of inadequate mental discipline.
What the extended mind thesis was gesturing at in 1998 becomes a practical engineering problem in 2025: if cognition can extend into the tool, the tool had better be capable of doing cognitive work — not just storing symbols.
Attention Fragmentation Is the Felt Problem
Before getting to the engineering, it’s worth being specific about what information overload actually feels like from the inside, because “information overload” has been a cliché long enough that it’s easy to hand-wave past it.
The real experience isn’t being overwhelmed by too much to read. It’s the tab that’s been open for three weeks because it contains something you know you’ll need but haven’t processed yet. It’s the meeting where someone references a decision made six months ago, and you remember it was discussed but can’t reconstruct what was decided or why. It’s the research thread you were pulling on last month that you can’t find the thread end for. It’s the low-grade, persistent sense that your knowledge is scattered across a dozen tools, several email accounts, some Slack workspaces, a notes app you migrated away from two years ago, and a folder on a laptop that might be in storage.
This fragmentation has a measurable cost. Studies on context-switching have shown that recovering focus after an interruption can take more than twenty minutes. The fragmentation isn’t just storage inefficiency — it’s actively consuming the attention you need for the work itself.
The specific thing I wanted a second brain to solve was this: I should be able to say, out loud, something like “what did I conclude about vector database options last time I looked at this?” and get a real answer based on my actual work, not a generic web search result. That’s a very different requirement than “store my notes and let me search them.”
Why Now: RAG and Agentic Memory
The reason this wasn’t buildable until recently is that making a system that “thinks back” at you requires two things that only recently became accessible to someone building outside a large research lab.
The first is retrieval-augmented generation. RAG is the architecture that lets you take a language model — which knows a lot about the world in general but nothing about your specific context — and ground its responses in a dynamically retrieved set of your actual documents and notes. Instead of querying a static keyword index, you embed your content and query into the same vector space, retrieve what’s semantically relevant to your question, and feed that into the model’s context window. The result is a system that can answer “what did I think about X” by actually reading what you wrote about X, not by predicting what someone-like-you might have thought.
The second is agentic memory architecture: the idea that the system doesn’t just answer queries but actively maintains a structured representation of what it knows about you, your projects, your decisions, your current context. Rather than treating your vault as a passive document store that gets queried on demand, an agentic system can update its understanding of your state as things change, flag when new information is relevant to something you’re working on, and synthesize across sources you wouldn’t have thought to cross-reference yourself.
Together, these two architectural patterns are what make the difference between “smart notes app” and something that actually extends your cognition rather than just your storage. The technology is mature enough now that both patterns can be implemented on hardware you own, running locally, without sending your personal knowledge to a third-party server.
What I Built
LUMIS — the system I’ve been building — is my working implementation of these ideas. It runs locally. It ingests from my actual notes vault. When I ask it something about my own work, it reads my actual work to answer. It maintains a structured memory file that tracks context about current projects and recent decisions, so that queries can be resolved against both long-term stored knowledge and short-term working state. There’s a production pipeline — a thing I call Prism — that handles research, drafting, and review as coordinated agent tasks rather than requiring me to manually assemble everything.
I’m not describing this to pitch it. I’m describing it because the gap between “I have notes” and “I have a system that thinks with me” is the gap I’m trying to characterize in concrete terms. Building LUMIS is how I learned where that gap actually is, and what it takes to close it.
Two things were harder than I expected. The first was getting the retrieval quality high enough that the system surfaces genuinely relevant context rather than just semantically adjacent noise. Cosine similarity over embeddings will find documents that are “about the same topic” — that’s not the same as finding the document that’s actually relevant to your specific question. The second was that agentic memory requires active maintenance decisions: what to summarize, what to keep verbatim, what to discard. The system doesn’t automatically know what’s important. Neither do I, always.
What This Arc Is About
This post is the first in a series. Subsequent posts will go deeper into architecture, specifically the RAG implementation, the memory update pipeline, and — in the third post — the ownership question: why running this locally rather than in a cloud service isn’t just a privacy preference but a different relationship with your own knowledge.
The short version of the argument: your cognitive extensions should extend you, not whoever owns the server. That’s not a cliché about privacy. It’s a claim about what “second brain” actually means. A brain that belongs to someone else isn’t yours.
The case for building this starts with the number I opened with: four chunks. Working memory is a scarce resource. What you offload it to matters enormously. That’s why this is worth building carefully, and why I’m writing about what the building actually looks like.