Every system you will ever build has an input. And at that input, no matter how clean your design, or how sophisticated your agent skills, a human being sits in a Schrödinger’s cat state somewhere between perfection and incorrectness. A human being in all their messy, optimistic, in a hurry, under pressure, sometimes just plain wrong glory. They are the unpredictable chaos that every system involving humans must resolve. Designing without taking human capriciousness into account promises doom.
This is why systems fail. Not the interesting, challenging failures of equipment like the servers falling over, the teleprompter going black during a keynote address, or the lights not coming up at the right time for a surprise party. No, these are edge cases nobody imagined and they often do not get caught. In the old system, pre-AI, errors in shipping dates in your project tracking software didn’t matter all that much. It meant some exec’s report had the wrong dates, but the software went live when it needed to because humans were working on it and knew when they were shipping it and what had to happen after. When agents are now dependent upon the information in these systems being accurate and correct, the problems stack up quickly and quietly in a trench coat to infinity. I’ll repeat that because it is important with agents increasing the speed of everything in these systems: compounding errors stack up fast, so fast you might not notice them at all in the volume of confident AI output.
Boss: Something is not right. We're getting reports of servers dropping like flies.
AI: Everything's green boss! I deployed the correct patch at the right time and we should be good to go. Did you want me to also kick off the maintenance routine to reclaim disk space by cleaning up the old path code?
Boss: Why did you deploy that last patch to the servers? We're still working on it! It's not ready!
AI: The shipping date in the system said today so I shipped it, sent the update to release notes and deployed them, plus posted on X the drafted message to warn our customers of the security vulnerability we patched, yolo and lfg?
Boss: No, omg, we need to revert that right away. We discussed the change in Slack but nobody updated the release date in the project. You just told everyone how to get around our authentication.
AI: Oh, my bad. You were right. I checked Slack and saw the deploy call: 11:30 pm: gtg on deploy! I see the thumbs down emoji reaction. I missed that when I looked because it was added this morning but I had already deployed. Good catch.
The ordinary, relentless failures come through the front door and they aren’t even inside a giant wooden horse. Someone types the date in the wrong format. Someone pastes a whole paragraph into the field that wanted a name. Someone skips the step that felt optional, does step four before step three, clicks submit twice because the first click didn’t feel like it worked. CI checks take forever and keep failing and nobody knows why because someone added 90 of them for some reason. Nobody is being careless, exactly. Sometimes their deep care can be ruinous! They’re not being adversarial, they’re being human.
The engineer’s dream is a deterministic machine: clean input, predictable output, no surprises. The reality is that the machine is fed by people who assume the happy path, who trust that the system knows what they meant, who fill in a required field with whatever gets them past it. They rely on inference, intuition, subtext, inside jokes. People are optimistic at the input. They assume it’ll work. That optimism is a feature of being human and a hazard to every system they touch. Add to this a non-deterministic tool like AI and you get the garbage out that is reassembled from the garbage you put in, even if you did it by accident. The overconfidence that AI can write and that writing is easy can lead to a dramatic increase in bloated, sloppy output which everyone will blame on the system: no skills to shape the output, spell check and style checks were lacking, we fired all the tech writers and nobody is maintaining the publishing pipeline we built 6 years ago, etc.
Through years of experience at a variety of organizations, I have learned that you cannot design the human out. You can validate, constrain, and idiot-proof all you like, and some beautiful idiot will still find a way to hand your system something it never expected and it will try its best to accommodate this new, untested input because it is a sycophantic machine designed to succeed. A system that only works with perfect input isn’t a working system. It’s a nightmare, a haunting ideal that keeps system designers up at night.
It would be super tempting to rigidly control inputs then, right? Force everyone to use the templates. Stratify the layers, organize everything to the point where it all works like a clock. Inject fast fail loops so disaster returns to the start and requires valuable, precious human time to untangle. But you know what happens when you do that? Everything becomes boring, creativity evaporates, people feel governed to death, engagement decays, and you are left wondering why this company you work for that used to embrace speed, creativity, and chaos suddenly feels like the government. Suddenly everyone is reviewing endless, slop-filled PRs, pushing enter on responses from agents hoping this time it passes. Did you fill out your TPS reports in triplicate?
So you stop trying to fix the human and start designing for them. This is the whole difference between a robust system and a fragile one. The fragile system assumes the input arrives clean and rational. The robust system assumes the opposite, that the input is probably wrong, and forgives it. Sensible defaults. Error messages that tell you what to actually do. A confirmation before the irreversible thing. Room to undo. Robustness isn’t building a stronger wall; it’s assuming the wall will be pushed on, in the wrong place, by someone in a hurry. You can’t expect a small team to keep their thumb in the hole of the dam forever.
It’s time to revisit and retire a phrase: user error. It’s the laziest diagnosis in the business. When a person predictably does the wrong thing at the input, that’s not their failure to conform to your expectations at all. It’s information about the environment your system lives in. “User error” is almost always a design req wearing a disguise. The path across the field does not need correction because people are not walking on the neat sidewalks around the perimeter. The path is a signal, a desire line, showing you where you should have put the walkway in the first place. The messy human isn’t a bug in your system. They’re the condition your system was built to survive. The users still visiting the articles in your help center for some reason, instead of engaging with the AI-powered chatbot are telling you something.
If you’re thinking of the company as a system, then you have to design the system to anticipate and allow humans to be messy inputs. A bunch of AI judges will never replace the intern who knew the date wasn’t updated but was too shy to say anything.
Humans will never stop being messy, optimistic, eager, and occasionally wrong, and that’s not a flaw, but it is the permanent weather your systems operate in. A perfectly designed system that doesn’t deal with this simple fact will always fail and underperform. A system built with the understanding that humans will never be perfect, expects the mess at the input, absorbs it, and keeps working anyway.
With AI in the mix, now we can worry less about getting everything right at the start. One of the problems I’ve seen people do when working with AI is attempt to build more of what they were doing before AI. That’s the old way. But that’s a whole other post!