Building Agents Users Trust (and why we can't just stop at engineering reliability)
With a summary of the paper on Agentic Harness Engineering
As we build agents at work, one of the hardest problems remains consistency and reliability at scale. And I think we’re conflating two separate problems that are worth pulling apart:
1. Technical reliability
2. User trust
Related, definitely. But they are not the same problem. Reliability is an engineering property. Trust is a product outcome built on reliability, stakes calibration, and legibility. In other words, reliability is necessary but not sufficient. And as we make progress on the first, we need to acknowledge that it doesnt automatically get us to the second.
Trust is accumulated gradually ( even when we get to technical maturity), but destroyed asymmetrically. A single high-stakes failure can outweigh hundreds of successful interactions.
To make this more concrete, let’s use a simple example throughout this post.Imagine we are building an AI travel agent that manages your trip end to end: creating itineraries, booking flights and hotels, suggesting activities, and making restaurant reservations.
Technical reliability asks whether the agent consistently does what it is supposed to do ( books the right flights, reserves the correct hotel dates, and updates the itinerary accurately when plans change)
User trust asks a different question: would you feel comfortable handing over increasingly consequential parts of your trip to it? You may happily accept restaurant suggestions after one or two good experiences, but hesitate to let it book a complex international itinerary with non-refundable flights until it has demonstrated sound judgment over time.
That is the distinction I am aiming to explore through this post. Reliability is whether the agent can perform the job correctly. Trust is about whether you would be willing to delegate work that truly matters.
Agent reliability ( Summarizing my insights from the Agent Harness Engineering paper)
I spent this weekend reading a paper on Agentic Harness Engineering that makes interesting progress on the technical reliability piece. The core idea being that that with the right kind of observability around the scaffolding, Agents might get better at fixing the issues it runs into.
It talks about establishing three pillars of observability:
Component Observability: Every part of the harness (the prompt, the tools, the memory) is stored as an individual file. This makes it very easy for the AI to change one specific rule without breaking the whole system.
Experience Observability: Instead of making the AI read millions of lines of logs, the system ‘distills” that data into short, structured reports that point out the exact root cause of a failure.
Decision Observability: Whenever the Evolver AI makes a change, it must write down a prediction of which tasks this will fix and which tasks it might accidentally break. If the next round of testing shows the change didn’t work, the system automatically reverts the change.
What the results showed:
After multiple rounds of self-improvement, the AI-designed harness outperformed harnesses created by human experts.
This is exciting. At the same time, practical questions remain:
Would this approach still work at production scale?
Could organizations afford frontier models across all three observability layers?
If lower-cost models were substituted, would the gains persist?
These are still open questions. But there is no denying the principle its based on: better observability may allow agents to become better engineers of their own systems.
User Trust: And why it can’t just be about chasing technical reliability
Now, getting to the second part of this problem-How do we earn user trust? Reliability is necessary to get there. But it’s not the only factor. Trust is a broader system level objective that reliability is a key contributor to. How do we keep investing in user trust through product decisions, as the technology keeps maturing? This matters especially outside the technology bubble, where skepticism is the default and first impressions carry outsized weight.
There are a few key product questions we need to answer right when we start framing the solution:
Stakes calibration:
Not all failures are created equal. There are low-stakes jobs where users may be relatively forgiving. And then there are high-stakes promises where failure carries disproportionate consequences. In these moments, a single mistake can permanently alter a user’s willingness to delegate meaningful work to the Agent. Going back to the travel agent example-Suggesting a café in Rome is a relatively low-stakes decision. Booking a non-refundable international itinerary with tight connections is a very different promise.
The central product challenge is teaching the agent to recognize this difference.How do we communicate not just what to do, but when the cost of being wrong is too high?
Trust in many ways behaves like human relationships. It is built gradually through repeated positive interactions, but a single consequential moment can disproportionately shape the entire relationships.
A truly magical experience at the right moment can create outsized confidence and accelerate adoption. Conversely, one catastrophic miss can erase months of goodwill.
This suggests two complementary product goals:
Engineer for delight intentionally. Create moments of clear, undeniable value that build trust and earn the right to take on more responsibility over time.
Define catastrophic failures explicitly. Understand what would constitute an unforgivable mistake from the user’s perspective, and design product guardrails to ensure the agent does not cross those boundaries.
In other words, the objective is not simply to maximize autonomy. It is to calibrate autonomy to the stakes of the task, building confidence where the cost of failure is low and exercising restraint where trust is most fragile.
Legibility:
Even highly capable agents will fail. The question is not whether failure occurs, but how those failures are experienced by users. When an agent makes a mistake, does it fail in ways that are understandable and graceful? Or do failures feel arbitrary, opaque, and difficult to explain?
In the travel agent example: you asked it to book a specific restaurant for your anniversary dinner. It could not get a reservation for that night. So it found one that met your stated criteria on paper, but was an hour away from your hotel. And it said nothing. Just confirmed the booking without explaining the tradeoff to you.
That is not a reliability failure. The agent found a restaurant. The failure is that it made a consequential tradeoff silently, on a night that mattered, without giving you the chance to decide.
This is where legibility becomes critical. Legibility is the system’s ability to communicate its reasoning, confidence, limitations, and boundaries in a way users can understand. An overconfident agent that is wrong is far more damaging than an agent that clearly communicates uncertainty or admits that it does not know ( Yet again, very much like humans :))
Pointing Observability to the user trust layer
Calibrating stakes and measuring legibility are more abstract than the more tangible engineering reliability. Which brings me to the primary thought I had as I was reading this AHE paper. We have a lot of work right now that points towards inward observability, but maybe we also need to standardize how we point observability outwards to user trust. In the travel agent example, outward observability would help us understand not just whether bookings are technically correct, but whether users are becoming increasingly comfortable delegating more consequential parts of their trip over time. For instance, for the two product goals that we called out earlier ( delight and preventing catastrophic failures), maybe we instrument for:
Signals of delight:
Rate at which users act on what the agent surfaces vs ignore it
How quickly they act ( here letting urgency of response is a proxy for perceived quality)
Whether action rates decline over time even if volume stays constant ( If users are acting on 80% of suggestions in week one and 30% in week four, trust is eroding)
Signals of catastrophic failure:
This one we almost can’t observe after the fact. Because by then, trust is gone.
Which means your observability layer needs to know two things simultaneously: how confident is the agent, and how important is this task to the user. The latter being particularly tough to gauge.
My parting thoughts on this are that observability in agent systems cannot stop at just the engineering reliability piece. It must extend all the way to the delegation boundary between the human and the agent. Because whether or not the user continues to delegate meaningful work to Agents we build would be gated on whether it is exercising sound judgment about when to act, when to defer, and when the consequences of being wrong are too significant to proceed autonomously.