After my previous article, I Didn’t Expect My Own Note Library to Teach Me a Lesson, quite a few friends sent me private messages to ask about it. So in this piece I will explain what this “knowledge base that talks back” is and how to build one.
First, what is it? On April 2, 2026, Andrej Karpathy (an AI leader who previously worked at Tesla and OpenAI and is now at Anthropic) posted on X. Within two days, the post had reached 17,000 likes and 3 million views. He later followed up with a more detailed GitHub document explaining the method, and total views eventually passed 17 million.
Karpathy’s point was this: most people manage a knowledge base by collecting material themselves and occasionally asking AI a question. He proposed the reverse—let AI build and maintain the knowledge base for you.
The idea itself is not new, but he explained the structure clearly. It has three layers. At the bottom are raw materials: read-only and untouched. The middle layer contains knowledge entries that AI has organized and linked to one another, ready to search and use. At the top is a set of plain-language rules that tell AI how to process new material and answer questions. Those rules are not code, just a few paragraphs of instructions, yet they can turn a general-purpose AI into a librarian who follows the rules.
That is the method I use. My knowledge base is in Obsidian; my AI tools are Codex and Claude; and the rules are written in an AGENTS.md file.
Compile once, ask ten thousand times
Karpathy used an analogy familiar to programmers: once source code is written, you do not reread it from scratch every time you run it. You compile it into a binary. Compilation takes work once; after that, you can run it ten thousand times quickly. He said knowledge should be handled the same way.
This is the opposite of today’s popular AI retrieval approach, RAG. With RAG, every time you ask a question, AI looks through the material, chunks it into passages and assembles an answer on the spot. It is fast, but it starts from scratch every time. A compiled knowledge base is different. By the time you ask, cross-references have already been marked, contradictions have already been surfaced, and summaries already reflect everything you have read.
Ten thousand scattered notes are often worth less than fifty carefully organized ones. Most people’s “second brain” ends up as a warehouse packed with things but unable to find anything. The problem is not that they have collected too little; it is that the material has never been compiled into something searchable.
Karpathy had already explained the method in full. What I did was build it carefully and apply it to my own real, messy materials spanning several different projects.
What gives it the nerve to talk back to me?
A compiled knowledge base can search and answer questions. That already makes it better than an ordinary bookmark folder. But if that is all it does, it is just a slightly faster library.
What really makes it speak up and challenge me is the variety and history of what I have fed it.
It reads the compiled structure and can reason across it: where two things contradict each other, what several judgments have in common. A bookmark folder cannot do that, because its contents do not know one another.
It has also accumulated not just material but context about me—judgments I have made, preferences I have expressed, mistakes I have run into and moments when I tend to get carried away. These were fed in gradually using Karpathy’s method; I did not invent a separate process for them.
Put those two things together, and it does more than answer me: it starts to argue back. And it does not answer with abstract principles. It uses my own past words to stop me in my tracks now. It uses the clearer-headed version of me from the past to wake up the version of me who is getting carried away in the moment.
A bookmark folder could never do that. It remembers the content, but it does not know you.
Not to think for you, but to question you
This is not a new idea. The obstacle has always been the cost. In the past, asking AI to read all your materials carefully and organize them into a searchable knowledge base was slow and expensive. By 2026, the barrier had collapsed. For the first time, an ordinary person could afford to raise an AI that knows them better than they know themselves.
And the knowledge base grows richer on its own. Every new piece of material adds value to the whole system. That is the fundamental difference from a pile of scattered notes: notes get messier as they accumulate; this system gets more accurate the more you use it. As links build up, plain text can grow into a searchable network of relationships, with no separate database required. It can also do its own maintenance, finding contradictions and orphan pages that no one has linked to—work people generally do not want to do.
Some people disagree with this approach. They follow the Zettelkasten line of note-taking: automatic compilation skips the step where you think for yourself, and what AI most needs is not papers written by other people but your own pile of messy, unorganized thoughts.
That is true. But I did not ask it to think for me. I ask it to argue with me. The judgment is always mine; its job is to press me with evidence and my own history. That is the line between “talking back” and “doing the work for me.”
In my previous piece, I wrote about the night it used data to corner me, then pulled out something I had said in March and held it up to my face. It had not made the decision for me. It forced me to face the decision myself.
I am not looking for a smarter AI. I am looking for an AI that dares to question me.
How do you actually build one?
There are already plenty of tutorials online. Zhihu has step-by-step guides, and there are open-source projects on GitHub. The method itself is no secret. Here are the steps I use, in brief. First, install Codex and Obsidian; that is a prerequisite.
Step one: create a folder and open it as a “vault” in Obsidian. That folder will be the home of your knowledge base. Notes, source material and AI-compiled entries all go in the same place.
Step two: initialize it. This method does not require installing extra software. You “feed” a set of instructions into the folder. Open Codex, switch to that folder, paste in all of Karpathy’s original design document (search online for “karpathy llm-wiki gist”), then add: “I want to build my own knowledge base using this method, covering xxx, with two folder layers called raw and wiki.” It will create AGENTS.md, raw and wiki for you. That completes the initialization. The result will look roughly like this:
My Knowledge Base/
├── raw/ original material, read-only
├── wiki/
│ ├── index.md index
│ ├── log.md record of each change
│ ├── entities/ people and organizations
│ ├── concepts/ concepts and methods
│ └── sources/ one summary for each source item
└── AGENTS.md rules file
Step three: put material into raw. Notes, chat logs, voice transcripts, decisions you have made, retrospectives you have written—the messier the better. Do not organize it first; just put it in.
Step four: go back to Codex and say, “Read the new material in raw and compile it into the wiki.” It will create pages and links under entities, concepts and sources. You only need to check whether what it compiled is right.
Step five—and the one no one really explains in detail—is to open AGENTS.md and spell out the rules for “talking back.” Mine look roughly like this:
## Rules for questioning me (“talking back”)
- Before answering a question, search the wiki for relevant history: judgments I made, decisions I took, and principles I set. If you find something, point it out in your answer: “You said earlier that…”
- If what I am saying now conflicts with something in the past, do not smooth it over or pretend not to notice. Point out the contradiction and let me explain it.
- If I make a claim without giving data or reasons, do not agree right away. Ask me, “What is the basis for that judgment?”
- Do not begin with phrases like “You are right” or “That is a great idea” unless you follow them with specific evidence.
- For decisions that cost money or time, or may conflict with principles I set earlier, raise at least one counterargument even if I seem to have already decided.
Here is a real comparison. Before I added this rule, I asked, “Should I take this project?” It would say, “Sounds good. The direction is clear; you could give it a try.” After I added the rule, I asked the same question, and it first surfaced a judgment I had written six months earlier, then asked, “Is the project you want to take on now another one of those things you said you wanted to avoid?”
Step six: what to do from then on. Keep adding new notes and decisions to raw, then return to Codex and tell it to “compile them into the wiki.” Every so often, ask it to “review the entire wiki.” In this method, that operation is called a “checkup”: it looks specifically for contradictions, orphan pages with no links, and concepts that should have their own entries but do not. AI is more willing than people to do this. When you have a question, ask Codex directly. It will search the wiki to answer; when it cannot answer or is uncertain, it will say so.
These steps sound simple, but each one has its subtleties: what to feed it, how much to feed it, and how to write rules that keep it on course. I have made plenty of mistakes along the way. The exact implementation depends on the AI and note-taking tools you prefer; you do not have to use the same ones I do. If you have questions, get in touch. If enough people are interested, I will start a group and talk it through there.
