We took learning back from the machinery we built for it

Reading the learning module’s production numbers in early August, we found a set that did not sit together. The main account had run nine learning jobs and committed six of them successfully, changing nineteen memories and rules for real. That same account still had one hundred ninety-nine untouched items on its learning list. The last new learning run happened on 22 July and there was never another one, while the patrol that decides whether there is anything worth learning right now kept checking every day through 4 August.

The agent was alive. It had simply stopped learning. The list did not shrink, the gate did not stop, and the workbench looked fine. A stall like that is much harder to notice than a crash.

What we did in the end was hand judgment back to the agent and delete the machinery in between. A learning run is now one session of cognitive maintenance. The agent reads the whole space in a single context and decides for itself what to remember, what to correct, and what to forget. Every step was forced on us by production data, and we still have not finished deleting.

Learning does two jobs a day for a space

Once the seenzus steward moves into a smart space, it takes in signals all day. A sentence a user says in the app, a scheduled task finishing, a wall switch flipped by hand, one of the home’s own automations firing. All of it counts. The learning module’s job is to periodically turn that loose pile into two things.

One is an understanding of the home, written into memory. The living room dims after ten. The kids’ room gets its night light before bed. The other is rules, which do something for the space under some condition or say something about it, like switching off the floor lamp when the TV comes on. Memory decides whether the steward understands this home when it speaks. Rules decide whether it can lend a hand at the right moment.

Follow one lamp through the whole thing. A child turns on the kids’ room lamp around ten several evenings running. Learning should see a stable bedtime habit in those few actions, write that understanding into memory, and register “around ten, kids’ room lamp” as a candidate rule. Whether that rule stays under observation, becomes a prompt at the right hour, or eventually becomes automatic care is governed by a separate ladder of permissions. Learning’s job ends when the rule exists.

Which leaves two ways to tell whether learning is working. Has memory grown this home’s actual rhythms, and does the rule list residents see hold anything they are willing to keep. If neither is growing, a busy backend is just busy.

The first problem really was that it learned nothing

The first version of learning was simple enough to learn almost nothing. It could see only what a user did inside our own app. Wall switches, the home’s existing automations, anything done by hand were all out of view. Most of a home’s real change happens exactly out there, so what it learned was always a small handful of actions, nowhere near enough to carry the rhythm of a whole house. The objection at the time was blunt. Shouldn’t it be learning from the activity feed? Shouldn’t it be seeing plain human meaning rather than raw fields?

Narrow material was one half of it. Back then an observation without enough evidence had nowhere to live, so the agent wrote running ledgers into its own memory files as notes for tomorrow’s self. What grew inside “my preferences” were sentences like “first pass”, “insufficient samples”, and “rules: none yet”, while the learned-rules list stayed empty and the home screen never once showed a decision that came from learning. Process notes polluted memory, and learning itself delivered nothing.

The intuitive fix is to run more often, and that road is closed. One learning run is one full model call, and everything it can see has to fit in the context. Running when the material has not changed burns a resident’s subscription on an idle spin. So the system needs a judgment about when a run is worth taking. The constraint sounds reasonable. Everything that follows grew out of it.

That beginning is worth remembering. Every machine we added later started life fixing a real problem of exactly this kind.

The model could not be trusted then, so we compensated with engineering

In July we rewrote learning from the ground up. The reasons are all in the spec from that time. Something a resident corrected out loud could be silently overwritten by background learning. A run that failed halfway left half-finished artifacts behind. Shipping a model or prompt change had no bar at all; you changed it and went on a feeling.

We compensated with a lot. Everything that happened in the space went into a single evidence ledger, and a gate admitted a learning run only once the ledger had accumulated real change. Each run’s material was frozen into a batch first. Every write landed in a staging area and became real only on an explicit commit from the model. On top of that came a durable learning list, batch capacities, failure strikes, shelving, a work ledger, and a synthetic exam with a signed release before any model or prompt change could ship.

Piece by piece, here is what each was fixing. The evidence ledger and the gate decided when learning was worth running. Frozen material stopped the evidence from shifting mid-read, which would leave half a judgment resting on the old world. Staging plus explicit commit made a failed run genuinely zero-side-effect, so even a botched write left nothing to clean up. Strikes and shelving handled the bad items that kept sinking whole runs; three appearances in a failed batch and an item went out to wait for a human. The exam and the release signature were a pre-flight check for model upgrades, so changing a prompt meant someone had read it and signed for it.

Each piece made sense at the time. Together they were an assembly line, and the agent was one station on it, stamping what the earlier stations had already computed.

The machinery started judging in the agent’s place

The pipeline hit its first deadlock within days of going live. The list was supposed to shrink the work by diffing against what the account had already learned, but the main account had never committed successfully. There was no previous state, so everything counted as new. One run faced two thousand items against a safety cap of four hundred and died outright. A failed run advanced no consumption watermark, so the next run faced the same mountain.

We fixed it with a durable list plus batch capacity. Each run took a few dozen items off the head of the queue, successes left the list, a failure earned a strike, and three strikes shelved that item on its own to wait for a human. The fix worked; it really did. Looking back, the disease lived inside the fix. The list packed three different things into one durable state machine. Hard facts that must never be forgotten, like a correction a resident stated to your face. Patterns that hold only while the current window supports them. And temporary grouping that exists only to fill one execution. All three constrained each other.

The second deadlock came straight out of that. Picture a concrete case. In April the system reads a candidate pattern out of the evidence. Damp spring, and the dehumidifier always runs in the afternoon. It enters the list and sits at the head of the queue. By August the evidence for that pattern has long since slid out of the window, so the head of the queue waits for fresh material that never arrives and never gets its turn, and the list has no way to reopen itself. The candidates behind it, real August routine changes, stay pinned underneath. There is work to learn and there will never be another run. The patrol keeps running, the gate keeps checking, and from the outside nothing looks stopped.

A pinned agent learns to pad

Once the machinery took over, what got learned started to rot. The pipeline had a gate requiring every run to write memory, and a run that wrote nothing did not count as complete. Production did exactly what you would fear. With nothing real to write, the agent wrote device-lifecycle ledgers into resident-facing memory. Memory is supposed to hold an understanding of the home, not system state, and we had forbidden that prose in writing ourselves. It repeated the same lines nightly for as long as the triggering evidence stayed in the window.

Stand where the agent stands. The list decides what to learn, the material is frozen, the gate decides whether anything gets written, and the only freedom left is wording. The gate demands writing. The cheapest available move is to stitch the pile of material in hand into something that looks like a learning outcome.

Commit summaries had the same shape. They answered the question the contract asked, which frozen evidence was processed and which rule ids were touched, and never said what the agent’s understanding of the home had actually changed.

Whatever the gate checks is what the model delivers.

The subtraction started with a dead list

The diagnosis in early August put the accounts side by side. Nine jobs, six successes, nineteen real changes, one hundred ninety-nine list items nobody would ever process, and a gate still patrolling every day. Another account looked worse. Three jobs, five attempts each, all of them run into the ten-call tool budget ceiling, not one record learned. The module had worked. Eight of the nine accounts with learning jobs had committed successfully at least once. The problem was the design of the durable list itself.

That list had made a promise. Every passive pattern that ever appeared would eventually get reviewed. The promise does not survive a world that changes. Seasons turn, routines drift, and a candidate from three months ago that current evidence no longer supports still hangs on the list as a debt that can never be repaid. One unfinished item blocks the whole queue behind it. Learning stopped following the home as it is now and started chasing a historical ledger instead.

We decided to stop draining historical debt and maintain current understanding instead. There are only two inputs now. Explicit corrections and denials from residents, plus hard facts that may invalidate what the agent already believes, such as a member leaving, a device being deleted, or permissions being narrowed. Those ignore the time window and count as resolved only after a successful review. Everything else is judged against the current window alone. While the window still supports a passive pattern it is an opportunity, and when the evidence is gone it is gone, with nothing booked. Each space gets one unified review, where the agent understands the environment and every member in one context, and at commit time shared understanding goes to environment memory while personal preferences go back to each person’s own.

Material handling kept one minimal mark. Each subject remembers a fingerprint of what it has already seen, so unchanged material is not sent to the model again and materially changed material becomes reviewable once more. The mark is not a queue. No priority, no frozen payload, no strikes, no shelving.

”No change needed” became a first-class outcome

There is one seam left in the code. It takes an account, a stable job identity, and a frozen bundle of material. The agent runs one review, every write lands in staging, and the whole thing ships as one atomic commit. The commit passes a lease check, a collision check, and the large-delete brake, then leaves a receipt and a ninety-day archive. Callers never hear the words gate, list, or segment. Those were removed from the domain.

The same subtraction gave the agent somewhere to be unsure. Sub-threshold hunches used to have nowhere to go. They were either statistics the agent had no voice in or per-run notes that died with the run, so hesitation leaked into memory prose. Now there is a scratchpad. The agent writes it and reads it, nothing else treats it as authority, residents never see it, and the back office can read it. Entries whose evidence arrives get promoted; entries whose support vanishes get dropped. Take the same kids’ room lamp. If it has only been on three evenings recently, that is not enough to register a habit, and the agent can park the suspicion on the scratchpad and look again next week.

The most important change was removing the forced write. Nothing to change means nothing changes, and “no change needed” is now a cheap, respectable outcome. When there is nothing to say, nothing gets written, and the padded diaries have no source left to come from.

We did not tear it all down, only the parts that judged for the model

The safety base was untouched. Atomic commits, the before-image change ledger, per-item restore from the back office, the large-delete brake, collision protection, and ninety-day replay archives all stay, and they are the only hard gates left. The subtraction drew a line. Machinery that judges for the model comes down; machinery that prevents accidents stays.

Deleting the release gate took more nerve than deleting the list. It had been designed in earnest. A synthetic question set, plus a replay bank drawn automatically from ninety days of real archives with resident corrections as ground truth. Someone ran it and signed a release record, and that record was pinned to the current model and prompt. On startup, a mismatch refused to assemble at all.

In practice it fired in production exactly once, taking down three deploys and two rollbacks. Afterwards it was clear the change had been a vendor prefix dropped from a model string. Same model, prompt byte-identical. Because it failed at startup, the health check could never go green, and it took a ten-minute timeout before anything reported that the replica had never become healthy. Worker services have no health check, so the same commit passed on the worker and died on the web service, which reads like a web-specific bug.

Later we widened what the signature covered from the system prompt to the entire behaviour surface, every tool description and every parameter blurb included. From then on, changing a single phrase tripped it. And every trip ended the same way. The signer looked at the diff, decided the exam would tell them nothing, and signed a waiver. Its own runbook had a section titled 危险窗口, the dangerous window, explaining that in the seconds between signing and merging the running version is orphaned, so any restart in that gap takes production down.

A gate that can only catch itself ends up as a deployment hazard. We deleted it along with the offline exam. The replay bank was the one genuinely valuable idea in the whole subsystem, and we deleted that too. A guardrail nobody runs is just code you have to keep compiling. The real archives it drew from are still there.

The cost should be stated plainly. We gave up a sentinel that could have caught regressions on its own. Judgment quality is now read by a person. The before-and-after diff of every commit and its plain-language summary both sit in the back office, legible, but slower and occupying someone. We took that cost on purpose, because the run of waiver signatures had already shown this was how it worked in practice.

The subtraction is not finished either. Two closures on the old opportunity machinery are still open, and the statistical aggregation still runs on the old contract. The new architecture has been in production for a week. Long-term quality, seasonal drift, none of that is ours to conclude on yet.

Before adding machinery again, ask one thing

Looking back at these two months, every gate, every list, every exam made sense going in, and each was fixing somewhere that had actually broken. Together they took judgment out of the agent’s hands one piece at a time. When the model was weak there was no alternative. Once the model got strong, the more complete the machinery, the more the agent looked like a rubber stamp.

Before we add another machine to learning, we ask two questions. Is this compensating for a weakness in the model, or judging in its place? After it ships, will we be watching the output or the ceremony?

The second one matters more. Gate pass rates, list length, patrol frequency all prove the machinery is running. None of them prove learning is happening. The proof is still the two things from the beginning. Has memory grown this home’s rhythms, and does the rule list hold anything residents are willing to keep.

This answer only holds for the generation of models we have now. The day we run one whose temperament we cannot read, some of what came down will have to go back up. We will still have ninety days of real archives then, so writing a fresh question set will not be hard, which is the fallback we deliberately left ourselves when the replay bank went.