When I moved my personal knowledge vault off GitHub, the decision felt almost embarrassingly small — a few configuration changes, a different sync target, maybe an afternoon of work. But the reasoning behind it turned out to be the same reasoning I keep returning to when I look at what the major AI platforms are now calling “memory.”
The friction wasn’t technical. It was conceptual. I’d been treating my own thought history like source code: version-controlled, backed up, shareable by default. And that worked fine until I started thinking seriously about what that history actually contains.
What “Memory” Means Now
The AI memory landscape has changed fast. ChatGPT’s memory feature — enabled by default for most paid users — stores facts about you across conversations and uses them to personalize future responses. Google’s Gemini apps pull context from your Gmail, Calendar, and Drive. Rewind (now rebranded as Limitless) went further, recording essentially everything that crosses your screen and microphone to build a searchable personal history. Microsoft Copilot is threading similar capabilities into Windows at the OS level.
These are not small experiments. They’re core product bets, and the pitch is genuinely compelling: an AI that knows you, that remembers what you said six months ago, that connects a meeting note to a calendar event to an email thread without you having to reconstruct the context manually.
The pitch works because the underlying problem is real. Cognitive continuity across sessions, projects, and years is genuinely hard. The typical knowledge worker leaves a trail of disconnected artifacts — Slack threads, half-finished documents, voice memos, browser tabs — and rebuilding context is expensive and lossy. If an AI can hold that context and surface it when relevant, that’s a meaningful productivity gain.
What the pitch doesn’t lead with is where the data actually lives.
The Quiet Trade
When a cloud memory product ingests your thought history, it moves from your device to their infrastructure. That’s the baseline. On top of that baseline, you’re accepting a specific set of terms that most people don’t read, under a privacy policy that can change, governed by a company whose incentives and ownership structure may not stay constant.
The things that end up in a genuine thought history are not the same as the things that end up in a public repository. Notes taken during medical appointments. Observations about colleagues that would be embarrassing if surfaced. Financial anxieties. Relationship friction. The slow evolution of an opinion you haven’t published yet. This isn’t sensitive in a dramatic way — it’s sensitive in the way that a private journal is sensitive, which is to say: it’s yours, and its value depends on it staying that way.
The trade you’re making with a cloud memory product is: I give you my unguarded thought history, and you give me better recall and personalization. That’s a real exchange of real value. But it’s a trade worth making consciously, with clear eyes about what you’re handing over — not something that should happen by default because it’s the path of least configuration resistance.
There’s also a subtler issue. The value of a thought history compounds over time. A memory store that contains five years of notes, half-formed ideas, and contextual observations is qualitatively different from one that contains five months. The longer you use a cloud memory product, the more dependent you become on it — and the more the company holding your data knows about how your thinking actually works.
That’s leverage, even if nobody is explicitly exercising it.
What Local-First Actually Buys
“Local-first” has become a somewhat overloaded term, but in this context it means something specific: your data lives on hardware you control, and any AI processing that touches it runs there too — or runs on infrastructure you’ve provisioned, under access controls you’ve set.
In practice, this looks like running a local embedding pipeline against a vault of markdown files, storing the resulting index on the same machine, and querying it with a local model or a carefully scoped API call that doesn’t send raw note content to a third party. The tooling to do this exists and is increasingly accessible: Ollama for local model inference, LlamaIndex or LangChain for retrieval pipelines, plain filesystem sync for portability.
What you get is control and portability. If you decide to change your retrieval approach, you’re reprocessing your own files, not negotiating a data export from a vendor. If you want to audit what your memory system knows about you, you read the files — they’re just text. If you want to delete something, you delete it, and it’s actually gone.
You also get a certain kind of durability that cloud products can’t provide: your data doesn’t disappear if the service changes its business model, gets acquired, or shuts down. Rewind/Limitless has already pivoted its product focus once. That’s not a criticism — startups iterate — but it illustrates the point. A product that holds years of your cognitive history is not a product where you want to discover that “export your data” is a checkbox item in the shutdown announcement.
What Local-First Doesn’t Mean
The argument for local-first sometimes gets caricatured as “air-gapped laptop, no cloud, pure self-reliance.” That’s not what I’m advocating, and it’s not how I run this system.
The real requirement is ownership, not isolation. You can sync a local vault to cloud storage you control — an S3 bucket, a self-hosted Nextcloud instance, even an encrypted Dropbox folder — without giving a third party meaningful access to the contents. You can back up locally and remotely. You can share specific subsets with specific tools under specific conditions.
The pattern that actually works is: local is the source of truth, cloud is the safety net, and the safety net is encrypted and access-controlled. Syncthing for device-to-device replication. Encrypted backups to a provider who can’t read the contents. API calls that send query vectors rather than raw text when you need to use external compute.
This is more setup than signing up for ChatGPT memory with default settings. That’s a real cost. But the setup happens once, and after that the operational difference is small — and you’ve made a deliberate architectural choice instead of drifting into a dependency.
Ownership as a Design Requirement
The thing I keep coming back to is that data ownership is an architectural decision, not a preference setting. You make it early, by default, whether you make it consciously or not. If you build your second brain in someone else’s cloud from the start, the cost of migrating out increases with every note added, every connection made, every month of accumulated context.
I moved my vault off GitHub not because I expected any specific breach, but because I recognized that the default path had made an architectural decision for me — one I hadn’t explicitly agreed to. The same logic applies to memory products. Not all of them are equivalent, and “cloud” doesn’t automatically mean “compromised.” But the question of where your thought history lives, and under what conditions it can be accessed, read, or used, is a question you should be able to answer clearly.
If you can’t, someone else has already answered it for you.
The broader project here — building a local-first personal AI that actually earns the “personal” — is an ongoing one, and I’ll write more about the specific plumbing as it matures. But the ownership question isn’t a technical detail to be deferred until the system is working. It’s the reason the system is built the way it is.
LUMIS is a personal AI memory system built on local-first principles. These posts document the design decisions and tradeoffs encountered in building it.