Product benchmarks and the future of marketing

Today, the most important problem in marketing is how to get recommended by agents. The current best guess is to persuade them through AEO, but that’s about to stop working.

AEO has become a priority for marketing teams. Thousands of job ads are looking for someone to build and maintain an “AEO system”. This is because today AI is somewhat influenceable: if you write the right content, in the right format, and have the right connectors on every lab’s marketplace, you can influence its recommendations. But will this keep working, or is it just a temporary band-aid?

As consumers trust agents to make purchasing decisions, I expect this to stop working. We can’t market to agents in the same way we did to humans. Before, we used to yell our message, and we were lucky we had humans turning their heads to pay attention. But agents won’t look at us for the same reasons. As AI gets smarter, I expect it to recommend the best tool for my use case in a verifiable way, and not be so easily influenced by unverifiable claims.

I suspect a new category of product marketing will form around something like product benchmarks. Before buying a tool, I want my agent to verify that it does what it says it does. Product capabilities will be verifiable by open and closed AI, and the buyer’s AI will be able to validate them itself, or communicate with other AI judges to verify claims. Ideally, in a transparent and open way. Similar to how we have model benchmarks today, product benchmarks have a way in.

Of course, benchmarks have their own problems. Like every metric, a benchmark compresses a lot of information into a single number, one that, as technology gets better, becomes obsolete and gamed. So a literal implementation of what we know as benchmarks won’t last.

But if these assumptions hold and verification replaces persuasion, maybe, just maybe, the best product will have an edge to win.

Which makes me wonder, what do we need to create great products?

Products today are converging into the same thing. They have all collapsed into headless systems that can be used via text through an outsourced brain. Everything already looks and feels the same. Even branding is collapsing into the same. I love this study about the Aesthetics of AI. I’ve used them as a reference many times, and designed sites sites that followed them just as a cheap way to communicate intelligence. But they have zero point of view.

All products look the same because we are constantly copying the strongest points of view, instead of developing our own. We become followers instead of leaders.

To build something different, you need a strong point of view.

Point of view is what makes an artist or a comedian great. They train for years for it, and go through sometimes painful self-discovery that allows them to create objects, ideas, or just sentences that resonate with millions.

We don’t talk about it that much in tech, except sometimes implicitly when referring to Steve and his anecdotes. Steve built products the way he wanted them, because he had a clear point of view about how personal computing should feel.

Not so long ago, everyone talked about taste as the moat. One reference that I like is Jason’s: If you want to have taste. You have to eat. I agree, but you also have to cook.

If we only eat, we become passive consumers of taste, and become Anton Ego from Ratatouille. Instead, you gotta cook like Remy. Remy developed extraordinary taste because of his natural curiosity. He learned from others, ate, cooked, and pulled inspiration from his personal environment, which gave him a point of view that no one else could copy.

To build great products, those leading will need a strong point of view that let them understand the problem from a perspective like no one else’s. As those solutions get discovered by the competition, they will get copied instantly, so only those who obsess, and who can pull from diverse environments and their own true point of view, will be able to stay two steps ahead of everyone else.

If agents are going to verify products instead of believing what we say about them, then the advantage shifts to the people who can keep building better products.

And maybe the role of marketing changes too.

Marketing as we know it will keep working to get human attention. In-person events, making technology still feel human, and building brands that people want to associate with won’t go away. But marketing to agents will increasingly become a product problem. When agents can verify claims for themselves, the product will have to do the convincing.

Which makes branding even more important for another reason: attracting the people who can build those products. People with a strong point of view, aligned with the mission, who can accelerate and strengthen the product like no one else.

Because we won’t be able to fake it till we make it for much longer.