For a long time, I have been building a complex AI system.
It began as a fairly specific product. Over time, it grew to include execution, tools, memory, permissions, long-running tasks, external capabilities, and multiple product forms. As it became larger, I spent less of my time asking how to add another feature and more of it asking a different class of questions: Where should a capability belong? Who has the final authority to interpret it? Which decisions should be left to the model, and which guarantees must come from software? Will the mechanism that helps us ship today become a boundary we cannot escape tomorrow?
These questions look like architecture questions. In practice, they changed how I think about software itself.
Traditional software is built around instructions. A programmer defines the path in advance, and the machine follows it precisely: if A happens, do B; if the input is C, return D. Much of software’s value has come from turning human judgment into stable, repeatable steps.
AI changes this arrangement. For the first time, machines can decide what to do next across a meaningful range of situations. We no longer need to enumerate every path. We can describe a goal, provide an environment and feedback, and let a model form a plan, choose tools, observe the outcome, and adjust.
That can create the impression that as models become more capable, software becomes less important.
My experience has led me to the opposite conclusion. Software is not disappearing. The object of its responsibility is changing.
In the past, software told the machine what to do next. Now a model can choose the next step, but software still has to answer a different set of questions: What is true? What is allowed? What counts as complete? Who is responsible when something goes wrong? How does an intelligent action become an outcome people can trust?
I have come to believe this:
Models generate possibilities. Software turns possibility into order.
Software is no longer merely the instruction set around AI. It is becoming the institutional layer of the world AI inhabits.
The principles below emerged gradually as I built, discarded, rebuilt, and verified different parts of this system. They are not an architecture guide for a particular product, nor are they a universal methodology. They are simply the ideas I now consider worth carrying into whatever I build next.
1. Define What You Are Building Before Today’s Interface Defines the Destination
The most dangerous drift in a long-running project is not getting one feature wrong. It is gradually forgetting what you set out to build.
Products always begin through a concrete entry point. It might be a chat interface, a desktop utility, an editor, a browser, or a workbench. Because these things are visible and usable, they easily begin to define the entire project in reverse: the project becomes whatever screens exist today, and its future becomes an expansion of whichever feature is currently most mature.
But the current product is often a vehicle for validating the destination, not the destination itself.
I learned to separate three things deliberately: what I ultimately want to build, the common foundation required to support that goal, and what users can genuinely use today.
The three must connect, but they must never impersonate one another.
If you speak only about the destination, the project becomes a vision with no deliverable. If you speak only about the present, the project becomes trapped by its current interface and temporary implementation. If there is no common foundation, every new product form has to reinvent execution, state, permissions, and memory.
This distinction changed many of my decisions. I stopped deciding where a feature belonged solely by asking where it could be implemented fastest. I began asking whether it was an experience unique to the current product or a system capability that future products would also need. A temporary implementation can be imperfect, but it should not be allowed to redefine the long-term goal.
The first thing a long-lived system needs is not a roadmap. It is a sense of layers: what is the direction, what is the foundation, and what is merely the way the system reaches users today.
2. Ask “Who Owns This?” Before “How Do We Build It?”
Much of the complexity in software is not caused by too much code. It is caused by responsibility that has no home.
When a concept has no clear owner, it naturally begins to replicate. The interface keeps one version of the state, the backend keeps another, and a configuration file keeps a third. None is obviously wrong, but none can give the final answer when they disagree.
The system then starts growing synchronization logic. A change in one place has to notify the other two. A notification is occasionally lost, so recovery is added. Recovery order proves unstable, so compensation is added. Compensation creates another state, so reconciliation is added. The code works harder and harder while the system becomes more difficult to explain.
I used to be drawn immediately to the question of how to connect everything. Now I ask different questions first:
- Who does this concept truly belong to?
- Who creates it, interprets it, and ends it?
- If it is disabled, migrated, or deleted, which part of the system understands the full meaning of that change?
- Do other components receive the truth, or merely a view projected from it?
Only after those questions have answers does the connection strategy matter.
“Ownership” sounds like a code-organization term, but I think it expresses a much broader principle: anything that persists needs one party responsible for its entire lifecycle. Otherwise, everyone can change it, but no one truly understands it.
This has also given me a strong preference for a single final interpreter of every fact. Other components may observe, cache, present, and use that fact, but they should not invent equally authoritative versions of it.
Complex systems are not afraid of having a lot of state. They are afraid of having too many authorities.
3. Consolidate Mechanisms, Not User Experiences
As a system grows, creating a new mechanism for every new requirement is both the most natural and the most dangerous choice.
A capability is slightly different, so it gets a new registration path. A use case is somewhat unusual, so it gets a new mode. A piece of state does not yet have an obvious home, so it gets a separate store. A new execution path is inconvenient to integrate, so it bypasses the existing path and creates another.
Each decision looks reasonable in isolation. Together, they produce parallel systems for discovery, permissions, loading, updates, disabling, and recovery. The real cost is not a few extra classes. It is that every future capability has to answer the same questions again across all of those parallel mechanisms.
I gradually learned to distinguish between two kinds of diversity: the diversity users experience and the diversity the system must internally maintain.
Users can and should encounter very different products, entry points, and interactions. Different contexts deserve their own language and mental models. But underneath those experiences, capabilities can enter the system, receive permissions, and acquire lifecycles through a small number of stable mechanisms.
Consolidate the mechanism, not the experience.
The hard part of this principle is resisting the temptation to create something new because it is easier to wire up. Before introducing another mechanism, I now ask whether the current one truly cannot express the need, or whether a new one is simply more convenient in the short term.
“It looks a little different” is usually not enough reason to create a new axis in the system.
Stable mechanisms are not there to make architecture diagrams look cleaner. They reduce future cognitive cost: models, developers, and products no longer have to guess between several almost-equivalent ways to do the same thing.
4. Software Should Constrain Consequences, Not Prescribe the Model’s Thought Process
This is where my thinking changed the most.
When model behavior is inconsistent, the natural impulse is to encode the ideal workflow in software. Require research before planning, planning before execution, and one completed stage before the next can begin. Run verification after every change. Add another runtime gate whenever the model appears likely to take the wrong path.
These systems are usually designed with good intentions. But they often turn into rigid pipelines. A model that could adapt its approach to the size and shape of the task is forced to complete every stage even for a simple request. Multiple guardrails can also interact and create a deadlock the model has no way to escape.
I began using a simpler test to decide what belongs in software and what should remain with the model: does the rule constrain amethod, or aconsequence?
“Read before editing,” “present a proposal before implementation,” and “remember to verify at the end” are methods. They belong in prompts, skills, and context, where the model can understand and apply them flexibly.
“Do not access data without authorization,” “do not cause irreversible loss without confirmation,” and “do not bypass a permission the user explicitly disabled” are boundaries on consequences. They cannot depend on the model remembering them. Software must enforce them.
Mechanical guardrails against repetition, idling, and loops should also be used sparingly. A first attempt should be allowed. When the system blocks an action, it should explain why. And it must provide a clear path to recover rather than trapping the model behind the guardrail.
This changed how I understand the division of labor between models and software:
Give goals and methods to the model. Give non-negotiable boundaries to software.
If software prescribes every step of the reasoning process, we have merely rebuilt a workflow engine with AI inside it. If every boundary lives only in a prompt, we have handed real responsibility over to probability.
A good system avoids both extremes.
5. Keep Reality Unambiguous Instead of Making the Model Sound Certain
A model’s reasoning can remain open. Its language can vary. Its plans can be revised. But once it begins acting on the real world, some facts cannot be ambiguous.
Did a task actually start? Did a change really occur? Is a permission enabled? Which session owns a result? The answer to these questions should not change depending on which screen is looking at them.
I increasingly think of software as the reality layer of an AI system. A model can offer interpretations of the world, but software records what actually happened in it.
That means an interface can express state but cannot become the state itself. Parameters attached to a request can express intent but cannot temporarily overrule durable authority. A cache may improve performance but cannot acquire the same status as its source. When the system cannot establish a fact, it should stop honestly rather than guess the most convenient answer to preserve a feeling of smoothness.
Traditional software also needs consistency, but AI makes it even more important. Models are exceptionally good at continuing from incomplete information and producing explanations that sound plausible. When the underlying facts have already diverged, intelligence at the top can make the contradiction feel more coherent and allow the problem to remain hidden for longer.
My principle is therefore simple: allow the model to be uncertain, but do not allow system facts to become ambiguous.
When the system does not know, it should say that it does not know. When authoritative data is damaged, it should refuse to act. When two sources conflict, it should require the conflict to be resolved rather than silently choosing the easier one.
Intelligence can work with ambiguity. Institutions must preserve reality.
6. Reversibility Is a Prerequisite for Autonomy
We often define AI autonomy as asking the user fewer questions. But what determines whether a user is willing to let go is not the number of confirmation dialogs. It is whether the system lets them see, stop, correct, and recover.
If every step requires confirmation, AI becomes a semi-automatic tool that demands constant clicking. If nothing requires confirmation and there is no record or path back, autonomy becomes a loss of control.
I began paying more attention to a different set of qualities: Is the process visible? Can the action be interrupted? Can changes be reviewed? Is the history traceable? After failure, can the system recover to a state whose meaning is clear?
Reversibility is not an extra layer of insurance for when something goes wrong. It is a precondition for granting authority in the first place.
When an action is easy to undo, the system can give the model more room to try. When an action is irreversible, the system should tighten permissions and return the decision to the user. The degree of autonomy should not be determined by how optimistic we feel about a model. It should be determined by whether the consequences can be recovered from.
This also changed my definition of “done.” Completion is not a green status in the interface. It means the system has made the result, the remaining risk, and the path to recovery clear. When a long-running task is interrupted, the animation cannot simply disappear. When a change is rejected, half of the execution state cannot remain behind. When a capability is disabled, recovery logic must not quietly bring it back the next time the application starts.
The more independently we want AI to act, the more carefully we must design the way it stops and returns.
7. Do Not Just Fix Bugs. Make Each Failure Leave Behind a Principle
Any long-running project will encounter many failures. What separates projects is not whether failures occur, but the form in which they remain afterward.
The weakest response is to make the immediate symptom disappear. A better response is to identify and fix the root cause. The most valuable response goes one step further: What class of failure was this? Through which other paths could the same failure return? Is the system missing a rule, a single owner, or a structure that can prevent the problem from growing back?
I developed a habit of recording more than the symptom and the fix. I also record the root cause, the owner, and the evidence that would prevent a regression. A problem described only by its symptoms, with no responsible boundary and no evidence, has not yet become knowledge.
A patch fixes one failure. A principle reduces an entire class of failures.
If a synchronization bug is repaired by adding one more notification, it can return later under a different event order. If the failure instead establishes the principle that every piece of state has one owner, future design naturally contains fewer synchronization chains. If an old entry point creates permission ambiguity, another special case will only add another branch. Establishing that one intent has one canonical entry point gives the system a durable decision rule.
I think this matters even more now that AI is participating in software development. Code is being produced faster, and local fixes are becoming cheaper. Without principles, a system can accumulate plausible duplicate implementations at unprecedented speed.
AI can help us write patches faster. Humans still have to decide what each failure should teach the system.
The real accumulation of value in a project is not its volume of code or the number of tickets it has closed. It is whether the cost of past failures has been converted into principles that prevent the same cost from being paid again.
8. Do Not Mistake the Appearance of Completion for Evidence
AI is very good at creating a sense of completeness. It produces well-structured explanations, syntactically valid code, and reassuring summaries. But the hardest parts of engineering often live outside that surface.
An interface showing no error does not prove that nothing continues running in the background. A capability working in a development environment does not prove that it exists in the artifact delivered to users. Validation on one platform does not prove identical behavior on another. Code passing a type checker does not prove that an old authoritative path has disappeared from the rest of the system.
Over time, I have asked “Is it done?” less often and “What evidence would be sufficient to prove that it is done?” more often.
Evidence is not one universal green indicator. Different layers answer different questions. Code checks establish local structure. Runtime validation establishes the real execution path. Artifact inspection establishes what users will actually receive. Recovery testing establishes whether the system preserves meaning under non-ideal conditions.
In the AI era, validation is not only about preventing models from writing incorrect code. It also counters a subtler risk: a result can be so logically convincing that everyone becomes willing to believe it already exists in reality.
One of my most important rules is not to infer underlying facts from surface state, not to infer whole-system correctness from a local pass, and not to present an unverified future as a capability delivered today.
Respecting reality sometimes means admitting, “This is not finished yet.” That honesty does not weaken a product. It gives the system a reliable point from which to keep evolving.
Software Is Shifting from Instructions to Institutions
Looking back, what I carried away from this work was not one particular architecture. It was a different view of AI software.
In the past, software tried to enumerate paths. In the future, more of those paths will be generated by models in the moment. Software no longer needs to prescribe every action for intelligence, but it does need to provide a stable world: clear identities and ownership, boundaries that cannot be bypassed, state that preserves reality, ways to exit and recover, and evidence that establishes whether an outcome is real.
This is similar to the relationship between institutions and individuals.
Institutions do not think on behalf of every person, nor should they turn every action into a fixed procedure. They define shared reality, rights, responsibilities, and what happens when something goes wrong. Good institutions leave room for action while preventing freedom from coming at the expense of order.
Software is beginning to play a similar role in the age of AI.
It is not the model’s brain, and it should not compete for the model’s job. It is the law, ledger, boundary, and memory of the world the model inhabits. It turns a clever action improvised in the moment into a stable capability. It turns an accidental success into a repeatable product. It allows users to entrust more important work to AI without trading blind trust for convenience.
I do not believe AI will make software less important. On the contrary, the more capable models become, the more software must answer the questions models should not decide for themselves.
What is true?
What is allowed?
Who owns it?
How do we return after failure?
What evidence proves completion?
Models explore the answers. Software protects the questions.
We are moving from an era in which people wrote every step for machines to one in which people establish principles and machines explore paths. The role of software is shifting with it: from describing actions to establishing order.
Models generate possibilities.
Software makes those possibilities worthy of trust.