How to Build AI Agents: a Preliminary Playbook

AI Agents have great potential for improving productivity, particularly in fields like sales and marketing that feature a lot of repetitive work that’s ripe for automation. But it’s a mistake to think building agents is easy – despite all the tools coming your way that claim to make it easy. Without the proper process, data, context, and engineering, agent projects waste the money, time, and trust organizations have invested in them. 

This article is for those who want to build agents that are practically useful, not demos that impress in a meeting and flame out in production. I won’t be getting into coding details but focusing instead on core principles. 

My organization, Similarweb, has been creating agents for our customers for the last year and we have at least started figuring out a playbook. We’ve hit a few brick walls and waded through data swamps, but learned a ton of valuable lessons. 

In addition to developing agents for our digital intelligence product suite, we’ve delivered a Model Context Protocol server that simplifies agent integration with our data. One thing working with customers who want to create their own agents has shown us is how few organizations understand how to design for an agent that will deliver practical ROI (no shame in that – we’re all learning). 

The technology is moving so fast that some of my recommendations could be outdated within months, but I believe the guiding principles will still make sense. 

There are four phases: 

Ideation – Clearly define the problem and the value delivered to the customer. 

Prototyping – Prompt and play. Manual testing to prove feasibility before writing code. 

Development – Where things get messy. Production standards, data integrations, security, performance. 

Refinement & optimization – The world changes, models change, and you must adapt and refine. 

Ideation 

Step 1: Figuring out the “Why” 

Clearly defining the problem and value is key to success. It’s sexy to build agents, but pointless if nobody uses them. 

We’ve set up a questionnaire with 10-15 questions to focus agent design and clarify mission. Before you build anything, ask yourself: 

Can I clearly define the outcome needed? 

Can I write down a “standard operating procedure”? If the answer is a clear yes, you probably don’t need the adaptability of an agent, just a simple automation script. You’re looking for a multi-step task where the number and nature of the steps is going to be different every time. 

Do all the inputs exist for the agent to make appropriate decisions? 

Is this really a job for an agent? Does it go beyond what can be accomplished with a simple LLM prompt or with traditional software? 

As with any product, it can be useful to interview potential customers. But understand no one trusts AI blindly. Be clear about what it will and won’t be able to do and how that matches with what people expect. You will only know your agent works when real users start breaking it. 

Step 2: Examples 

If you can’t come up with concrete examples of the outcomes you expect, you haven’t adequately scoped the requirements. If your best examples are trivial, you probably don’t have an idea that justifies a full-blown agent. 

Think of 3-10 examples and outcomes you’d like the agent to achieve. Add 2-3 realistic user stories highlighting how the agent helps in daily workflows. 

Here’s one from our work: We built an agent that helps sales teams prepare for prospect meetings. The desired outcome isn’t “summarize web traffic”—it’s “give me three conversation starters based on the prospect’s recent digital performance shifts and how we can help.” The agent pulls competitive traffic data, identifies trends, and frames insights in the context of what the salesperson needs to say. That specificity—outcome, not output—separates useful agents from demos. 

Step 3: Outline the Process 

This step separates the good from the best. 

Write a standard operating procedure for how a human would do it. Too short or too linear? May not require an agent. Impossible to write down? Under-scoped. 

Define the process step by step to achieve the best outcome. Define the output and success metrics. If the process can go different ways, embrace the flexibility. 

Then think: what could AI add to make this much better? What saves time, effort, energy? What tasks would a human find tedious that AI could easily do? Add those as discovery steps. 

Separate the process into: contextual inputs, data inputs, workflow/playbook, and output. 

Not everything needs an agent. A competitive tech stack monitor? That’s simple—deployable in 1-2 weeks. A root cause analysis agent diagnosing why organic traffic dropped? That’s advanced—expect 4-6 weeks with a data scientist. Know the difference before you start. 

Prototyping 

Step 1: Prompt It—Easy Does It 

You don’t need crazy infrastructure to test your product. Start with a simple prompt in your favorite conversational AI (ChatGPT, Gemini, Claude), add context and data, and feed it to an LLM. Walk through it step by step to see whether an LLM is a good match for the problem. 

This reduces risk dramatically. 

Add relevant contextual information and data—a CSV file, a document, whatever you have. For the more tech-savvy, I recommend the open agent platform of Lang Chain to accelerate the process. 

Don’t write any code before you do this. 

Step 2: Design the Core Interaction 

Another overlooked step: the UX and Agent Experience—how the agent interacts with the user. Without adequate focus here, your “expert advisor” winds up speaking in riddles rather than being helpful. 

Decide how users will engage: UI, API, chat, or just a button. Wireframe the flow, inputs, and outputs. Make sure the agent communicates clearly and naturally. When things get complicated, show the full reasoning. Agents that “think out loud” build more trust than black boxes. 

Step 3: Test and Iterate Before You Start 

Test → Feedback → Iterate → Repeat. 

Log all potential alerts for two weeks before enabling notifications. Analyze historical data to find the right thresholds. Expect to adjust 3-5 times in the first 60 days. 

The Development Phase 

Step 1: Write the Orchestration Logic, Tools, and System Prompts 

Now that you have a basic prompt and know where to get context, let’s put it together. 

First, understand what you’re actually building. An AI agent is not magic—it’s orchestrated API calls + LLM synthesis + scheduled execution: 

Scheduler (cron/Zapier/n8n) 

    ↓ 

Trigger agent run 

    ↓ 

Agent calls data APIs (2-10+ calls) 

    ↓ 

Agent sends data + context to LLM with prompt 

    ↓ 

LLM analyzes and synthesizes insights 

    ↓ 

Agent formats output into actionable report 

    ↓ 

Agent delivers via Slack/Email/CRM/Dashboard 

Take your refined prompt and add context: personality, guardrails, the mission it needs to accomplish. For example, we instruct our sales meeting prep agent to ask clarifying questions like “who are you meeting with” and “what’s the goal” before making recommendations. We instruct all our agents on how to access our APIs and to prefer our data over random web content. 

Build tools your agents can use to pull data, analyze it, and take action. Tools that scale and have built-in context are critical. 

Step 2: Add Tests, Evals, and Feedback Loops 

First, test manually. Take a few examples, run through them, and look at the results with your own eyes. 

Then add systematic tests across many runs. Come up with metrics that make sense. Determine where it’s important to ensure the agent works on reliable data rather than hallucinating. If your agent delivers answers that are provably false, you’ve destroyed trust in one interaction. 

For parts of the system that allow more LLM “creativity”—like inventive language describing analysis results—try to measure that too. 

Create a separate LLM that takes your defined standards as context and provides a “scorecard” to evaluate output. A great use of an LLM is to critically evaluate another LLM — let them compete. 

Step 3: Deploy, Get Feedback, Iterate 

This is a must. As models improve, outcomes will change. The same prompt, context, and data from Gemini will provide different outcomes than an OpenAI model. 

Refine, refine, refine. 

A Concrete Example: Market Share Tracker 

Let me walk through one agent we built to make this tangible. 

The manual alternative: An analyst spends 4-6 hours each month pulling traffic data for 10-15 competitors. They export to Excel, manually calculate share percentages, think of insights and build a PowerPoint to deliver a report 7-10 days after month-end. By the time leadership sees it, the data is already stale. 

The agent approach: Automated traffic aggregation across your competitive set-monthly visits, engagement metrics, channel mix. The agent calculates relative market share, flags any competitor gaining or losing ≥2% share, and identifies the channel driving the shift (paid advertising? organic search growth? referrals?). It then dives into each channel to explain the root cause. Results are delivered via Slack or email on the 3rd of each month with trend charts and executive summary. 

What it actually takes: 5-10 MCP data retrieval calls per run, depending on competitor count. A clear definition of the competitive set. Threshold logic for what constitutes a “significant” shift. A scheduled job that runs monthly. An agent platform to build: Copilot Studio or ChatGPT’s Agent builder. 

That’s not magic. That’s orchestration-and it frees your analyst to spend those 4-6 hours on the “what do we do about it” question instead of the “what happened” question. 

Refinement & optimization 

This deserves its own article, but here’s the core truth: you can’t build an agent and abandon it. Every LLM model update can dramatically change outcomes. Each tool, integration, and data point matters. 

Monitor execution time—sudden slowness signals problems. Alert on error rates above 10%. Review output quality weekly. Expect to adjust prompts 3-5 times in the first 60 days. 

You can’t “set and forget.” You have to keep watching and adjusting, continuously. 

A Bit of Humility 

Even the best engineers rarely get the right answer on the first try. The key is feedback loops, learning, and maintaining momentum. 

This isn’t rocket science, but it’s different from traditional software development. Bad design choices get magnified. Bad data feeding into an AI system makes everything worse. Asking the right questions won’t produce better answers if the underlying data is wrong. 

Focus on understanding the process, getting the right data, and taking time for lots of iterations. Those are my rules—at least until next quarter, when we’ll probably update everything again.