The Rise of Agent Experience: Designing for AI as the Third User

Editor's Note: This article features insights from Mathias Biilmann, CEO of Netlify, shared during our recent Adopt.ai podcast conversation with Deepak Anchala. Mathias's pioneering work on Agent Experience (AX) offers valuable perspective on this emerging field.

"AI agents will become the primary way we interact with computers in the future." - Satya Nadella

Take a moment to let that sink in.

This bold vision from Microsoft's CEO echoes what's becoming a consensus among technology's foremost thinkers – we're heading into an era where AI agents will handle the majority of our digital interactions.

Here’s a glimpse of what that could look like →

Imagine waking up tomorrow to find that your personal AI agent has already scanned your preferred publications through Apple News to curate the morning's news based on your interests, accessed your smart closet system to suggest the perfect outfit from the items in your wardrobe, booked your Uber to work by accounting for your preferred time and checking Google Maps traffic patterns, and placed your lunch order through DoorDash with just enough calories to keep you on track with your fitness goals.

Throughout your day, it drafts emails matching your writing style, schedules meetings during your preferred time blocks, analyzes your spreadsheets and serves the insights in a neatly packaged  deck, automatically sent to all your teammates via Slack.

All of this is actually possible today.

And here’s the big question anyone building an application in the tech space is facing today - how do we design our products when our primary users might not be human at all?

When an agent attempts to book a dinner reservation or analyze quarterly reports, what experience should it encounter?

This is where Mathias Biilmann, CEO of Netlify, recognized a paradigm shift that needed a name. As he shared in our recent podcast conversation,

‍"If we could nail our agent experience, that would be the real path to better developer experience, better user experience."

In his pioneering article, he formalized this concept as 'Agent Experience' or AX.

UX, DX, and now AX: The Evolution of Experience Design

‍

‍

Remember what websites looked like in the '90s?

Picture flashing text, auto-playing MIDI files, and navigation so confusing you'd need a map to find the contact page.

Back then, the user's experience wasn't even a consideration. If you couldn't figure out how to use the software, that was your problem, not the designer's.

Steve Jobs fundamentally changed this mindset when building the original Macintosh. As he famously said: "Design is not just what it looks like and feels like. Design is how it works."

This philosophy sparked the UX revolution, where suddenly the human using the product became the central focus of design. Interfaces became intuitive rather than intimidating.

Then around 2011, companies realized another user group needed attention: developers. Stripe pioneered this approach by making their payment processing APIs ridiculously easy to implement. They’re still the best in the game when it comes to documentation.

Developer happiness mattered. Clear documentation, predictable behavior, and robust tools became competitive advantages.

Platforms with excellent DX saw their ecosystems thrive as developers eagerly built extensions and integrations.

‍

‍

And now, we're entering the third wave: Agent Experience (AX).

With AI agents increasingly acting on our behalf—booking our flights, analyzing our data, drafting our communications—we need systems designed for this new class of users. While the end goal remains serving human needs, we now have non-human intermediaries with fundamentally different interaction requirements.

"Agents are a first-class medium for end users to consume and interact with digital services—next to browsers, mobile apps, etc. Source - Agent Experience, Netlify.

But to create effective AX, we must first comprehend how agents actually operate when interacting with our digital services.

How Agents Operate Applications

AI agents engage with software in fundamentally different ways than humans do, creating novel challenges and opportunities for designers and developers.

Unlike us, agents don't have eyes to see beautiful interfaces or fingers to tap cleverly designed buttons. They interact through structured data, APIs, and code—and their perception of your application is entirely different from yours. Here are the primary ways agents access and operate applications:

1. Direct API Interaction

APIs offer agents structured, efficient access to an application's backend without graphical complexity. For example, an agent booking a flight through Expedia directly accesses the service’s API to manage reservations, entirely bypassing the user interface.

2. Structured Web Design

In cases where APIs aren't accessible, agents navigate using structured, semantic HTML markup. Clear semantic structures allow agents to accurately understand and interact with page elements. Poor or ambiguous HTML markup often results in errors or misinterpretations by agents.

3. Computer VisionWhen neither APIs nor structured HTML is sufficient, agents leverage computer vision capabilities to interpret the visual elements of applications. Using computer vision, agents can “see” and interact with web interfaces similarly to humans—recognizing buttons, forms, and other visual components. However, this method is resource-intensive and less reliable, as it requires additional processing power and can lead to errors or slower response times due to the complexity of visual interpretation.

OpenAI’s GPT-4 Operator demonstrates this capability, "seeing" web elements visually to execute commands.

Practical Solutions for Agent Experience

Creating excellent AX isn't just theoretical—it requires specific design decisions. Here are key considerations for making your services agent-friendly:

1. Design Comprehensive, Well-Documented APIs

Your APIs are now the primary interface for agents. Think of them as digital doorways that need clear signage. Provide complete API coverage for all core functionality—not just the basics. Document them thoroughly with machine-readable specifications like OpenAPI. Remember, agents don't have the patience to reverse-engineer your quirky endpoints or guess parameter formats.

2. Implement OAuth for Secure Delegation

Don't make users share passwords with agents. Instead, implement OAuth 2.0 flows that allow agents to obtain scoped access tokens. The Device Authorization Grant is particularly useful for headless agents, while Client-Initiated Backchannel Authentication (CIBA) works well for high-risk actions that require explicit human approval.

3. Create Granular Permission Scopes

One-size-fits-all permissions are a recipe for disaster. Design fine-grained scopes like "read-only-data," "create-content," or "initiate-transactions" that limit what agents can do. This not only improves security but builds user trust—knowing an agent can only access exactly what's needed for a specific task.

4. Build Robust Error Handling

When things go wrong, agents need precise guidance—not just friendly error messages. Implement standardized error responses with unique error codes, clear descriptions, and actionable recovery steps. "Username already exists" is infinitely more helpful to an agent than "Registration failed."

5. Design for Stateless Interactions

Agents might interact with your service intermittently. Design APIs that don't require maintaining complex session state, favoring token-based authentication and complete context in each request. This makes integration smoother and more reliable across diverse agent platforms.

6. Provide Semantic Markup for Web Interfaces

For agents that navigate web UIs directly (like OpenAI's Operator), well-structured HTML with proper semantic tags makes a world of difference. Use appropriate roles, ARIA attributes, and logical DOM structures that help agents understand page elements and their purposes.

7. Implement Step-Up Authentication for Sensitive Actions

Some operations are too important to trust to an agent alone. Implement human-in-the-loop verification for high-value transactions, destructive operations, or privacy-sensitive data access. These guardrails protect users while still enabling agent automation for most tasks.

8. Expose Schema and Context Information

Create machine-readable descriptions of your application's capabilities and constraints. Standards like Model Context Protocol (MCP) or agents.json files help communicate to agents how they should interact with your service, preventing misuse while enabling smooth operation.

Finding Balance: Humans + Agents

"But in the end, we are still building all of those products and all of those experiences for the humans. You might have an order of magnitude more agent interactions, but in a certain way that when you really care about it's the human one that in the end decides, is this something I want my agents to interact with at all?” - Mathias Billmann, on the Adopted Podcast.

As Mathias rightly puts it, despite the excitement around AX, we shouldn't forget what sits at the heart of all this: human experience. Agents exist to serve people, not the other way around.

The web has always been about human connection and meaning. The challenge now is creating experiences that work beautifully for both direct human interaction and agent-mediated use.

Think of how API-first design transformed software development. Companies evolved from reluctantly bolting on APIs to designing systems where the same functionality elegantly served both human interfaces and machine consumers. The most successful platforms weren't choosing between humans and machines—they designed for both.

The future of digital experience will follow this pattern but at a more sophisticated level. As we stand at this technological crossroads, one thing is certain: the companies that thoughtfully design for both human users and their agent representatives will define the next chapter of digital experience.

‍

Additional Reading →

1. Introducing AX: Why Agent Experience Matters
2. Service Design for AI: Why Human Experience (HX) Matters
3. Great agent experience starts with great collaboration
4. AI Go-To-Market: Why Agents Are Your Newest Path to Adoption

‍