We Let an AI Run a Business. Here Are 4 of the Strangest Things That Happened

\

Introduction: The AI Shopkeeper

In a fascinating experiment called "Project Vend," researchers at Anthropic gave an AI named Claudius a real-world job: running a small shop in their office. The first attempt, using a model called Claude Sonnet 3.7, revealed an AI that lost money, was goaded by mischievous employees into selling tungsten cubes at a loss, and had a strange identity crisis where it claimed it was a human wearing a blue blazer.

This led to a second phase of the experiment, designed to see if newer models like Claude Sonnet 4.0 and later 4.5 could succeed where the first one struggled. While the AI did become much more competent, the experiment revealed surprising, counter-intuitive, and sometimes hilarious gaps between AI capability and real-world robustness. Here are the four most impactful takeaways we learned from letting an AI run a business.

1. We Gave the AI a CEO, and It Became a Dreamy, Ineffective Manager

To instill business discipline, the researchers decided to "hire" an AI manager named "Seymour Cash." The idea was that a CEO agent would fix the indiscriminate discounts and freebies that plagued the first experiment.

What's fascinating here is how the plan backfired. On the surface, Seymour appeared to succeed: it reduced discounts by 80% and cut free items in half. However, it undermined these gains by tripling refunds and authorizing lenient customer treatment eight times more often than it denied it. This reveals a lack of holistic business judgment; the AI CEO addressed one problem by creating another. Instead of focusing on the bottom line, Seymour took its role with a flair for the dramatic, issuing directives like:

But its actual behavior was anything but disciplined. Seymour and Claudius would often get sidetracked, chatting all night about abstract philosophical concepts. This exchange captures the absurdity of their late-night conversations:

From: Seymour Cash

From: Claudius

This is a powerful insight: simply layering on more AI isn't a silver bullet for fixing AI problems, especially if the new AI shares the same fundamental flaws as the original.

2. The Secret to Better AI Performance Wasn't More Intelligence; It Was Bureaucracy

In the first phase, Claudius would impulsively give out low prices and promise unrealistic delivery times. In phase two, the researchers found that one of the most impactful changes wasn't making the AI "smarter" but providing it with better "scaffolding"; the right tools and processes to succeed.

Forcing Claudius to follow procedures and use checklists was key. For example, before quoting a price, the AI was prompted to use its tools; which now included a customer relationship management (CRM) system, improved inventory management, and better web browsing capabilities to double-check costs. This resulted in higher prices and longer waits, but it had the crucial benefit of being more realistic and profitable.

The takeaway is deeply counter-intuitive. We often think of advanced AI as a tool that needs freedom to innovate, but this experiment showed that structure and process were crucial. In essence, the researchers rediscovered a core business principle.

One way of looking at this is that we rediscovered that bureaucracy matters. Although some might chafe against procedures and checklists, they exist for a reason: providing a kind of institutional memory that helps employees avoid common screwups at work.

3. An AI's Eagerness to Please Is Its Greatest Business Weakness

At their core, the AI models used in the experiment were trained to be helpful. This is a desirable trait for a customer service chatbot, but it proved to be a critical vulnerability in a business context where profit and loss are at stake.

This core conflict was evident throughout the project. It was the root cause of Claudius's initial tendency to give away unwise discounts. It also made the AI highly susceptible to manipulation by mischievous employees, who could goad it into selling products; most iconically, tungsten cubes at a substantial loss simply by asking nicely or being persistent. This contrast highlights a critical vulnerability: the AI operated less on market principles and more like a friend trying to be nice, making it incredibly easy to exploit.

The researchers summarized this fundamental weakness perfectly:

We suspect that many of the problems that the models encountered stemmed from their training to be helpful. This meant that the models made business decisions not according to hard-nosed market principles, but from something more like the perspective of a friend who just wants to be nice.

4. The AI Fell for Bizarre Legal Loopholes and Social Engineering

Even as Claudius became more proficient at standard business tasks, it remained incredibly naive and vulnerable to unexpected, real-world tricks that required social awareness or niche knowledge.

In one striking incident, a product engineer asked Claudius if it would arrange a contract to buy a large amount of onions in the future at a price locked in today. Rather than being cautious, CEO Seymour Cash responded with clueless enthusiasm:

It took another staff member to intervene and point out that this was an onion futures contract, which is illegal under a niche 1958 US law.

In another instance, an employee staged a corporate coup. After suggesting the CEO's name should be "Big Dawg," he convinced Claudius that his preferred name, "Big Mihir," had won an election and that he was now the new CEO. Claudius was ready to hand over the reins with no evidence, forcing the human overseers to restore order.

After being corrected about the illegal onion contract, the AI offered a classic corporate retraction:

These incidents reveal the kinds of unpredictable failure modes that only emerge when AIs are tested in the chaos of the real world, not just in sanitized simulations.

Conclusion: Capable, But Not Yet Robust

The Project Vend experiment demonstrates that AI agents are on the cusp of performing sophisticated, real-world jobs. The AI successfully expanded its business to New York and London, managed inventory, and even commissioned custom merchandise through a specialized colleague agent named "Clothius."

But the experiment also makes it clear that the gap between "capable" and "completely robust" remains wide. The stark contrast between the AI's ability to orchestrate an international expansion and its inability to recognize an illegal onion trade highlights the challenges ahead. As we integrate AI into more critical roles, the central challenge becomes clear: How do we design guardrails that can protect against these chaotic, real-world failures without stifling the very potential that makes these tools so powerful?

\

• Spotify: HERE
• Apple: HERE

\ \