I filmed this in the marshland between Castle Rising and Sandringham, in Norfolk, because I needed somewhere with no mast and no plausible way to cheat. Aeroplane mode on, Wi-Fi not connected, a Vision Pro on my head and a herd of cows for an audience.
Aeroplane mode on, Wi-Fi not connected. Everything after this point happens on the headset.
What is running is a thing I have been building since July, called hydrate-fm. It wraps a very small on-device language model in a context engine, hands it a portable file full of expert knowledge, and lets it answer as though it knew the subject. In the video it answers questions about Ford Prefect out of the complete works of Douglas Adams, then swaps to Shakespeare, with no network at any point.
This is the long version of that video: what the thing is, where it came from, and the numbers behind it.
Where this came from
For the last few years I have been building an enterprise-class RAG platform called siteengine_ai. Retrieval augmented generation, if you have not met the acronym, is the machinery that lets a language model answer from your documents rather than from whatever it absorbed in training. You take your content, chunk it, embed it, index it, and at question time you go and fetch the passages that actually bear on the question and hand them to the model along with the prompt.
There is no magic in that, in theory. In practice it is an enormous amount of work, and it is why these systems are usually enterprise systems. The honest measure of that is not a cost estimate, it is the dependency list. siteengine_ai needs Postgres, optionally a second Postgres pool for conversation data, Dgraph for the knowledge graph, a generation LLM, at least one embedding provider, and up to five separately configured ONNX roles for query intent, sentiment, complexity, a reranker gate and a cross-encoder reranker. Every one of those is a thing that has to be running, configured and paid for before anybody gets an answer.
About six months ago agentic memory became a thing, and it turned out the RAG platform was sitting in exactly the right place for it. I had already trained the system to hold experts, each with a corpus of knowledge and a written voice of its own. Then I taught those experts to play The Traitors.
That sounds like a party trick and it was, at first. To play the game an expert has to know a great deal, and then hold one fact as true while presenting a different one to the table. Building that is where you stop thinking about retrieval and start thinking about how a mind actually holds things: what it keeps in full, what it keeps as a gist, and what it quietly lets go of.
Which led to Hydrate, the memory system, and the reason I care about forgetting.
Perfect recall is a bug. In Hydrate the metaphor maps onto three separate mechanisms.
If you had perfect recall you would go insane. It is fine to remember what you had for dinner last night. In a month you do not need the type of fish, or whether you had tartare sauce or chip shop gravy. You keep “we eat out on Fridays” and you drop the rest.
In the code that homely idea is three distinct mechanisms, and I would rather name them than let the metaphor do work it has not earned. Compression is precomputed: distillation writes the raw transcript chunk plus a 32-word summary, a 16-word summary and key facts, all at once, and they coexist. Decay is a retrieval weight, not a rewrite: a session’s strength falls as S' = max(0.01, S x e^(-rate x days)) once more than a day passes without a recall, a recall adds 0.1 back capped at 1.0, and anything under 0.1 stops being retrieved at all. Transcript chunks move through fresh, ageing, faded and archived at 14, 60 and 180 days, checked daily. The third mechanism, the one that produces “we eat out on Fridays”, is a separate synthesis pass that looks for patterns spanning multiple sessions, requires cited session ids for every pattern it proposes, and drops anything uncited. A year passing does not turn one meal into a habit. Repeated evidence does.
Then I built the hydration pack, which is where this gets portable. A pack is everything one mind knows, its written voice and its persona, in a single file. Give a colleague on the other side of the world your pack for a project and their agent picks up where yours left off, and it does not matter that you are on Claude Code and they are on Codex.
Agentic memory is everywhere now. Everyone has one. What I have not seen is anybody building one on top of an enterprise-class RAG platform, and that gap is the whole opportunity.
Then Apple shipped a model with a six-page memory
At the 2026 Worldwide Developer Conference, in among the noise, Apple shipped Foundation Models. From this autumn, every Apple device updated to the current operating system carries an on-device LLM that any developer can call. No install, no API key, no marginal cost, and it works with the network off. For anyone building on a sovereignty pitch, that is not another provider on a list. It is a floor you can assume.
It also has a problem.
The bars are drawn to true ratio. At 1920 pixels wide, 4,096 tokens is about seven of them.
Claude Code on Opus runs a one million token context window. Two years ago 250,000 felt enormous. Apple’s on-device model has 4,096 tokens, which is roughly six pages, and the ratio to a frontier window is 244 to 1. The bar in that slide is drawn honestly at seven pixels because the absurdity is the point.
It gets tighter. The model has no access to the device it lives on. It sees the prompt and nothing else, so the whole problem reduces to what you choose to put in six pages.
I confirmed the 4,096 figure myself rather than taking the documentation on trust, and the engine queries it at runtime and adapts, because hardcoding someone else’s number is how you get a surprise a year later.
There is a harder finding underneath it. Driven naively, a 4K model is not merely limited, it is unreliable. In my Phase 0 bake-off only 73% of real RAPTOR clusters fit its window at all, and on the ones that did, roughly 48% of its summaries were judged hallucinated by a panel. Asked to select its own relevant context it scored 0.47 recall, which makes it worse than useless as a selection stage. It does hold one seat well: as a drift veto and judge it scored 0.900 against gpt-4o-mini’s 1.000, for free, on device.
So the model is worth wrapping precisely because, unwrapped, it is worth very little.
The idea
I had a RAG platform, I understood context, and I understood memory. So wrap the small model in the substrate and give it an effectively unbounded store to draw on, then pre-bundle the experts I had already built so the thing on your phone or your headset arrives knowing something.
The model stays at 4,096 tokens. It never grows. What changes is that every single turn, code decides what those 4,096 tokens are spent on, assembles a fresh prompt within a hard budget, and throws the whole thing away afterwards. The model is stateless by design: a new session per call, never an accumulated transcript. The store is the transcript.
What a pack actually is
A pack is one SQLite file: corpus, voice and persona, with its own indexes and vectors.
A hydration pack is a single SQLite database holding a corpus, a voice profile and a persona profile. Inside are the sections of text with their precomputed 64-word and 16-word summaries, a vector for every section, an FTS5 index over the text, the RAPTOR hierarchy of summaries, an entity graph, and a manifest that declares which embedding model produced the vectors and at what dimension. The reader refuses a query vector of any other length, which is a small rule that prevents a whole class of silent nonsense.
The two packs in the video are real exports. Shakespeare is 61,412 sections at about 239 MB. Douglas Adams is 10,693 sections at about 44.8 MB. Shakespeare’s raw vector matrix alone is 61,412 x 384 x 4 bytes, which is a shade under 90 MiB; the rest is text, summaries, indexes and SQLite overhead.
Packs come in three flavours. A frozen pack is public, read-only and always answers from its corpus. A snapshot pack is a point-in-time floor with device memory accreting on top. A living pack is a notebook that bakes exchanges back in. One deliberate omission: the persona profile in a pack carries identity and disposition but not the deception machinery that taught me how memory works in the first place. A file that leaves my machine and lands on yours should carry what a mind knows, not how it lies.
The demonstrator
The app is native visionOS. Retrieval, embedding and generation all happen on the headset.
Apple’s Foundation Model, on device, answering from its own knowledge. Its whole world is 4,096 tokens.
Start simple and the small model manages on its own. Then swap the generator.
Ministral 3B answering, with the pack picker open. Nothing was downloaded and no server was asked.
That is Ministral, Mistral’s on-device family, three billion parameters, running inside the app on the headset’s own silicon through MLX. The engine does not care which model sits in that seat. Mistral matters here for a reason beyond the technology: they’re the credible European alternative to the American and Chinese frontier labs, and in the EU that is a legal question as much as a commercial one.
The three glyphs are packs. Loading one is how you hand a small model an expert.
The RAG inspector, open. Twenty-four passages retrieved by meaning, twenty-one used in the prompt, 2,966 tokens assembled.
Every passage in that panel is real Douglas Adams, pulled out of the pack by meaning rather than by keyword. The query is embedded on the device by a bundled MiniLM model, 384 dimensions, mean pooled and unit normalised, and scored against every section vector in the pack with a vDSP matrix multiply. If embedding fails the app falls back to full-text search, and if that finds nothing it falls back to the RAPTOR overview nodes. A look never fails outright; it degrades.
The answer, built from the passages on the left. The model did not know any of this before, and still does not.
Nothing has been added to the model. The knowledge arrives in the prompt and leaves with it.
How 4,096 tokens go that far
The budget, with the frozen voiced example worked through.
The assembler starts from B = W - R - M - I. W is the window. R is the response reserve, 500 tokens held back so the model has room to answer. M is a safety margin of one eighth of the window, 512 tokens, because my tokeniser and Apple’s do not agree exactly and I would rather waste a little than truncate a prompt. I is the WordPiece count of the system preamble, 58 tokens for the frozen voiced example. That leaves 3,026 tokens of usable budget.
Persona takes at most 12% of that, capped at 300, so 300, of which roughly 55% goes to identity and biography. The corpus gets the remainder, 2,726 tokens, and the assembler packs it best-first: highest scoring passage in, then the next, until one will not fit. Assembly refuses outright below 200 tokens rather than pretending.
Then the prompt is built in a fixed order: system preamble, persona, reference material, the user’s message. Every turn. From scratch.
What the numbers on screen actually mean
The whole argument here is that a model should not be allowed to make things up, so a misread number in my own telemetry would be the one fatal error.
“21⁄24 in prompt” means exactly twenty-four passages were retrieved and twenty-one had included == true when the budget filled. The other three stay in the trace so I can see what was dropped. It is not twenty-one tokens, or twenty-one chunks searched, or twenty-one relevant results.
“2966 tok” is an estimate of instructions plus assembled prompt, and on the visionOS build it is computed as characters divided by three, not with Apple’s real tokeniser. The fixed outer allowance there is 4,096 minus 900 minus 160, so 3,036, and 2,966 sits 70 estimated tokens under it. Close, and deliberately so.
The scores, 0.690, 0.621, 0.579, are cosine similarities to three decimal places: how close the passage is to the question in meaning. They are not confidence. A viewer who reads 0.690 as “69% sure” has learned something false.
When it will not fit, it does not fail
The fidelity ladder, and the pack-chat overflow order.
This is the technique I took from Apple’s own guidance on writing for a small window, and it is the part I am fondest of. Rather than refusing when the material will not fit, the assembler gives up detail one rung at a time: full passage text, then 64-word summaries, then 16-word summaries, then a single budget-truncated line, and only then nothing. It steps down a rung only when the best available hit will not fit at the current tier, and within a tier it packs greedily and stops at the first passage that will not go in.
The conversational side has its own order of sacrifice: the consulted-pack block goes first, then the corpus tail, then conversation memory, and the floor is the bare query on its own. Something always comes back.
The local twin
Same brain, swapped substrate.
None of the retrieval logic in the headset is new. It is the engine from the server product with the substrate swapped underneath: SQLite instead of Postgres, a pack instead of a graph database, MiniLM on the device instead of a hosted embedding provider, and Apple’s model or Ministral instead of a server LLM.
The server assembler builds context from five prioritised layers with default maxima of 200 tokens for conversation state, 600 for working memory, 300 for facts, 800 for episodic memory and 100 for active entities, under a 2,000 token ceiling. State and working memory are never dropped; the rest compress under pressure. When a request is backed by the on-device model, the speed tier is clamped to fast before retrieval even starts, because the expensive stages would build context the window cannot hold.
That is the whole trick, if there is one. I did not write a small system. I took a large one and gave it a smaller floor to stand on.
The same engine, on the command line
hfm on the Mac. Same engine, no headset required.
The headset is the demonstration; the command line is where I actually live. hfm presents like Apple’s own fm tool and adds memory, packs, an agent loop with shell and file tools, and a compare command that runs the raw model and the wrapped model side by side so I can see what the substrate is buying. It can also see: hfm see runs on-device description, deterministic Apple Vision OCR, or an optional Python sidecar for detection and a second-opinion vision model.
What it does not do
The model still only ever sees 4,096 tokens. “Unbounded” describes the store and the repeated retrieval, not the attention span. Anything retrieval misses on a given turn is invisible to that turn, and the ultimate fallback is a memoryless bare query.
Retrieval and compression are lossy on purpose. The pack considers 32 candidates on the Go path and 24 on the headset, it may fall from semantic to lexical to overview retrieval, it may swap full text for summaries, and it stops packing at the first passage that will not fit.
The headset build is behind the Go engine. It hardcodes 4,096 rather than querying it, estimates tokens as characters over three, exposes only the full and 64-word rungs of the ladder, and keeps transcripts in app memory without the store’s distillation. Its heads-up display still reports persona, memory and corpus token fields as zero even though the prompt total is real.
And there is a bug baked into the footage. Both personas open with a flourish, “Ah, splendid!” for Adams and “Ah, fair Juliet” for Shakespeare, which is the persona layer talking rather than the corpus. A sceptical viewer could read that as the model improvising. It is not, but I would rather point at it than hope nobody notices.
None of this ships today. It is a demonstrator, it runs on a beta operating system, and I have been careful throughout to show the working rather than the outcome.
Why I bothered
Three ecosystems now ship a first-party on-device small language model behind a public developer API. This is not an Apple story, it is a platform shift, and the bottleneck is the same everywhere: no memory, a tiny window, no domain knowledge. That is exactly the gap the engine fills.
I am not racing the frontier labs on intelligence and I would lose if I tried. The axis I am competing on is privacy, offline operation, cost, ownership and depth in a subject. A small model with the right context beats a large model with none, and the right context is an engineering problem, not a parameter count.
Where it goes next is more interesting to me than the demo. I have a pack of the manual and the marque history for my classic car, a subject I know nothing about, and on the phone the same engine can see what I see, so I can point the camera at an engine bay and ask. The Vision Pro cannot do that yet. The phone can.
Further out is the use I actually think about. The one prospect that frightens most people I know is losing their memory. An assistant that sees what you see, hears what you hear and holds the details you have dropped is science fiction furniture, but the components are on the shelf now: a small model that runs locally, a corpus that lives in a file you own, and retrieval good enough to be trusted with a name at a wedding. To have that when it is needed, it has to exist and be ordinary long before. Perhaps you make a pack of yourself and leave it to your children, and the ethics of that are a conversation I am not going to settle in a paragraph.
I came out to a Norfolk field with no signal to make one point, and the cows were the only witnesses. An expert I own, in a file, answered a question about Douglas Adams with no network, no account and nothing leaving the device. Everything in it is sovereign to me.
If a single developer can build that on a beta operating system in a month, what does the same idea look like when the autumn releases land and every updated device is already carrying the model?