Build the second brain before you build the agent

Everyone wants the agent. The ones that actually earn their keep are standing on a pile of your decisions, meetings, and notes. Build the pile first, and the agent has something to stand on.

Andrej Karpathy posted something on X that named a shift I've been feeling. A growing share of his time with AI is no longer spent writing code. It is spent building personal knowledge bases, "manipulating knowledge" instead. The model is the easy part now. What you feed it is the hard part, and the valuable part.

Most CEOs I talk to want to skip straight to the agent. They have read about the AI agent that runs the Monday brief or chases the pipeline, and they want that. So they wire one up, point it at an empty folder, and wonder why the output feels generic.

The honest answer

An agent is only as good as what it can recall. Build the recall layer first.

An agent with nothing to remember is a clever stranger. It is smart, it is fast, and it knows nothing about your business, your customers, your last board meeting, or the decision you made three weeks ago and already half-forgot. So it guesses. The output comes back fluent and generic, which is the worst combination, because it looks right and isn't.

The CEOs getting real leverage out of agents did the unglamorous thing first. They built a second brain: a place where the company's decisions, meetings, and context live in a form an AI can read. Then the agent had something to stand on, and the same model that felt generic started sounding like it had been in the room.

This is the part everyone skips, because the agent is the exciting part and the capture layer is plumbing. But the plumbing is where the leverage hides. Two CEOs can run the identical agent on the identical model; the one whose agent has read a year of decisions and meetings gets answers the other one never will. The moat was never the agent. It is what you let it remember.

What a second brain actually is

It is not a new app you have to feed by hand. It is the stuff your company already generates, captured somewhere an AI can read it.

Start with the richest source you already have: your conversations. Every 1:1, board call, and customer call is full of decisions, commitments, and context that evaporates the moment the call ends. Get those meeting transcripts into a folder an agent can read, and months later you can ask your meeting history anything. Add your own notes, your strategy docs, the running list of what you decided and why. That is the brain, and it grows on its own once capture is automatic.

When I sit down before a customer call now, I ask my own meeting history what we last promised them and where things stood. It answers in seconds. That only works because the conversations were captured first; the agent is reading the pile, not inventing one.

Once the pile exists, the questions you can ask it are the leverage. What did I commit to in last week's leadership sync. What has this customer complained about twice. What did we actually decide about pricing in March, before everyone's memory of it drifted. None of that lives in your head reliably, and none of it lived anywhere an agent could reach until you captured it. The format barely matters at the start. A folder of plain text files an agent can search beats a perfect system you never fill.

Why agents fall flat without it

Two failures, same root cause.

The first is the empty-folder problem above: the agent never knew your business, so it guesses. The second is subtler. Even a well-briefed agent works inside a context window, the slice of text it can hold at once. On a long task, what it needs scrolls out of view. (This is why most agents fall apart on real work.) A second brain fixes both: the agent pulls what it needs on demand instead of holding everything in its head, the same way retrieval lets it look things up rather than memorize them. Give it persistent memory across sessions and it stops re-learning your business every morning.

The order is the whole point. Build the brain, then the agent reads from it. Do it the other way around and you are asking a stranger to run your company from memory.

Want to maximize your AI leverage? Upgrade to Pro.

What to do next

Pick the one source that holds the most of your thinking. For most CEOs that is your meetings. Turn on capture this week so every call lands in a folder as readable text, and do nothing else with it yet. In a month you will have a brain worth pointing an agent at. Tell me which source you reach for first. I love seeing where people start.

Related

• Why most AI agents fall apart in real work
• Ask your meeting history anything
• Get your meetings into markdown
• Persistent memory across Claude Code sessions
• What is a context window?
• What is RAG?