Build an AI Agent
for Website Automation Testing
A complete guide to building an autonomous website testing agent using Playwright, Ollama, local LLMs, browser automation and intelligent failure analysis.
What are we building?
The goal is to create an AI agent that can open a website, understand its UI, create test scenarios, interact with the browser, detect failures and explain the failures automatically.
AI Brain
The LLM understands requirements, creates test cases, decides the next action and analyzes failures.
Browser Hands
Playwright controls Chromium, Firefox or WebKit and performs real browser interactions.
Test Reporter
Collect screenshots, console errors, network failures, test results and AI-generated explanations.
AI Testing Agent Architecture
A good architecture separates the AI reasoning layer from the browser execution layer.
Recommended Tech Stack
| Component | Technology | Purpose |
|---|---|---|
| Browser Automation | Playwright | Control browser and execute tests |
| AI Runtime | Ollama | Run local AI models |
| LLM | Gemma / Qwen | Reasoning and test generation |
| Backend | Node.js | Agent application |
| Language | TypeScript | Application development |
| Database | SQLite | Store tests and results |
| Frontend | React | Testing dashboard |
Step 1 — Create the Project
Create a Node.js project and install Playwright.
mkdir ai-web-tester
cd ai-web-tester
npm init -y
npm init playwright@latest
npx playwright installStep 2 — Use Playwright
Playwright is the browser automation layer of your AI agent. It performs the actual actions that the AI decides.
import { chromium } from "playwright";
const browser = await chromium.launch({
headless: false
});
const page = await browser.newPage();
await page.goto("https://example.com");
console.log(await page.title());
await page.screenshot({
path: "homepage.png",
fullPage: true
});
await browser.close();Step 3 — Add a Free Local AI
Ollama allows you to run an LLM locally on your computer. This is useful when you want to develop without depending on paid API requests.
Install Ollama on your computer.
Download a supported local model.
Start the model through Ollama.
ollama pull gemma3
ollama run gemma3Step 4 — Build the AI Agent
The AI agent should not directly execute arbitrary code. Instead, give the model a controlled set of tools.
Example User Request
Test the login functionality of https://mywebsite.com. Check both valid and invalid credentials.
Give the Agent Browser Tools
Tools are the bridge between the AI model and Playwright.
open_page(url)
Open webpage
get_elements()
Inspect UI
click(target)
Click element
type(target, text)
Fill input
select(target, value)
Select option
screenshot()
Capture screen
get_console_errors()
Read JS errors
get_network_errors()
Read failed requests
Tool Action Example
{
"action": "click",
"target": "Login"
}Never allow an LLM to execute arbitrary JavaScript or shell commands directly. Validate every action through your own application layer before Playwright executes it.
Complete Testing Workflow
User Input
User provides website URL and testing requirements.
AI Discovery
Agent opens the website and identifies pages, forms, buttons and important UI elements.
Test Generation
LLM creates structured test cases from the requirements and discovered UI.
Browser Execution
Playwright executes each action and assertion.
Observation
Collect DOM state, screenshots, console messages, network failures and HTTP responses.
AI Analysis
The AI analyzes failed tests and determines possible root causes.
Report
Generate PASS/FAIL results and an explanation for every failure.
AI Failure & Root Cause Analysis
A major advantage of an AI testing agent is that it can explain why a test failed rather than simply saying "FAIL".
Test: Login Expected: Dashboard should appear Actual: User remained on login page Console: TypeError: Cannot read properties of undefined Network: POST /api/login → 500
Root Cause: Login API returned HTTP 500. Severity: Critical Likely Issue: Backend authentication service failure. Recommendation: Inspect /api/login server logs.
Capture Browser Errors
page.on("console", msg => {
if (msg.type() === "error") {
console.log(
"Console Error:",
msg.text()
);
}
});
page.on("requestfailed", request => {
console.log(
"Request Failed:",
request.url(),
request.failure()?.errorText
);
});Playwright MCP
For a more advanced agent architecture, you can use Playwright MCP to provide browser interaction capabilities to an AI agent.
npx @playwright/mcp@latestRecommended Project Structure
ai-web-tester/
│
├── agent/
│ ├── agent.ts
│ ├── planner.ts
│ ├── executor.ts
│ └── analyzer.ts
│
├── browser/
│ ├── browser.ts
│ └── actions.ts
│
├── tests/
│ └── generated/
│
├── screenshots/
│
├── reports/
│
├── database/
│ └── sqlite.db
│
├── frontend/
│ └── dashboard/
│
├── playwright.config.ts
├── package.json
└── README.mdGenerated Playwright Test
import { test, expect } from "@playwright/test";
test("valid login", async ({ page }) => {
await page.goto(
"https://example.com/login"
);
await page
.getByLabel("Email")
.fill("test@example.com");
await page
.getByLabel("Password")
.fill("password");
await page
.getByRole("button", {
name: "Sign in"
})
.click();
await expect(
page.getByText("Dashboard")
).toBeVisible();
});Testing Dashboard
Build It in Phases
Build Playwright browser controls.
Connect Ollama and a local LLM.
Add click, type, inspect and screenshot tools.
Implement plan → act → observe → reason loop.
Analyze screenshots, console and network errors.
Create a UI for running and monitoring tests.
Run the AI tester automatically in CI pipelines.
Your Complete AI Testing Agent
User
LLM
Agent
Tools
Playwright
Browser
Report
Start with a small MVP: Playwright + Ollama + a handful of browser tools. Once that works reliably, add autonomous planning, failure analysis, persistent test history and a dashboard.
Start Building ↑