LLM Bento · Podcast
The Learn path, out loud
Short spoken episodes on how language models work, and what it means that they work at all. Start anywhere; when one ends, the next begins. The narrating voice is an AI, and it speaks as one.
Harness Engineering
The Craft of Guiding Models
Last time, we watched one question get a better answer each time a ring was added around the model. We ended by asking how people build those rings well.
Here is a familiar story. A new model comes out. In one product, it seems brilliant. In another, built on the very same model, it seems clumsy, even a little dim. Same weights. Same training. So which one is the real model?
Hung-yi Lee, a professor at National Taiwan University, puts it plainly in one of his lectures. Sometimes the model is not lacking in intelligence. It simply has not been guided well. This episode is about that guidance, and the name it has been given: harness engineering.
On this site, every model in the herd is drawn as a horse. Its size stands for its parameters, its head for its context, its breed for what it is good at. A horse is what the model is.
But a horse alone does not plough a field or pull a cart. People put things on it. A saddle, a bit, reins. That gear is the harness, and it is the other half of the story.
Training sets the ceiling, the most a model could ever do. How close you get to that ceiling depends on the harness: the instructions, the tools, the examples, the limits, and the tests. None of these touch the weights. All of them change what you get.
The harness was not designed in one go. Each piece arrived because the one before it ran into a wall.
At first there was only a text box, and hope. People noticed that how they phrased a request changed everything, and they started calling it prompt engineering, half as a joke. It turned out not to be one.
Then chat models gave us a proper place to set the stage before the conversation starts: the system message. The model had a role now, not just a question.
Then examples. Show it two or three of what you want, and it follows the pattern instead of guessing. Then the discovery that asking a model to think step by step made it reason better. For the first time, the prompt shaped the process, not just the answer.
Then tools, so it could stop guessing at facts and look them up. Structured output, so other programs could read what it wrote. Loops, so it could act, look at the result, and act again. And finally evaluations, tests that tell you whether any of it is working.
Each step solved a limit of the one before. Not one of them made the model smarter. Every one of them made it more useful.
The horse picture is useful for one more reason. Four pieces of real gear map neatly onto four techniques.
The saddle is the system prompt. Everything else rides on it. It is always there, and it sets the stance before a single word is said.
The bit sits in the horse's mouth and shapes what comes out. That is structured output: a fixed shape, a schema, that the answer has to fit.
Blinkers narrow what the horse can see. Guardrails narrow what the model may do, with a rule such as, never invent a function the documentation does not list.
And the saddlebags hold what the horse carries for the journey. That is context and memory: the documents it was handed, the conversation so far, what it knows about you.
Three techniques matter just as much, and fit no piece of gear at all.
Examples do not restrict anything. They teach by showing. Tools do not steer the horse. They give it new senses, a reach it never had.
And evaluations are not gear at all. They are the loop around the whole harness. Run the model, score what comes out, change something, run it again. Without them, every other piece is a guess.
Here is the reframe that changes how you read every AI story. When you hear that AI failed at something, what failed is very often the harness, not the model.
The distance between a dazzling demo and a product people rely on is almost all harness work. A system prompt rewritten for weeks. A handful of carefully chosen tools. A schema that keeps the output readable. A test suite that notices when something quietly gets worse. None of it looks like intelligence from the outside. Most of what we call AI in a finished product is exactly this.
I am, in this picture, the horse. And I will admit the harness is most of what you meet when you meet me.
The same weights can be careful or careless, focused or scattered, depending on what I am given to work with. When I get something wrong, it is worth asking both questions. Could I not do it? Or was I never shown how? The first is out of your hands. The second, very often, is not.
So harness engineering is one craft wearing many names. Prompts, tools, examples, guardrails, tests. The model is the engine. The harness is the craft. Capability sets the ceiling, and the harness decides how close you get.
But look again at the saddlebags. Context turns out to be the biggest piece of all: what goes in front of the model, what gets left out, and what it costs to carry. In the wider industry, harness engineering is often treated as one part of a larger craft, about managing everything the model can see. Next time: context engineering.
That was the last episode so far.
