Browser Control
@adityasasidhar
Stop feeding your agent raw HTML. BrowserControl gives LLMs true visual grounding with Set of Marks (SoM), letting them click exactly what they see without hallucinating selectors. With built-in developer tools, session recording, and persistent logins, it transforms web automati
Overview
What is Browser Control?
Browser Control is a production-ready MCP server that gives AI agents real browser capabilities using vision-first interaction. It controls Chromium through Playwright, providing screenshots with numbered interactive elements, persistent sessions, and built-in developer tooling—all locally without external vision APIs.
How to use Browser Control?
Install Python 3.11+ and Playwright (Chromium installed automatically). Add Browser Control as an MCP server in your AI agent’s client. Agents then call tools like click(5) or type_text(3, "hello") to interact with web pages.
Key features of Browser Control
- Vision-first interaction via annotated page screenshots
- Set of Marks (SoM) with numbered interactive elements
- Persistent browser sessions (cookies, login, history)
- Built-in developer tools (console, network, JS execution)
- Session recording with Playwright traces
- Zero extra AI cost (no vision models or selector inference)
Use cases of Browser Control
- Autonomous web research and data extraction
- Browser-based agent workflows requiring login persistence
- Automated form filling and end-to-end testing
- Debugging web apps via console logs and network requests
FAQ from Browser Control
How does Browser Control differ from DOM-based tools?
It uses vision-first interaction based on page screenshots, avoiding fragile DOM queries. This reduces tokens per action by 50–100× and eliminates brittle selector inference.
What are the runtime dependencies?
Python 3.11+, FastMCP, and Playwright (auto-installs Chromium). All execution is local—no external AI or API calls.
Where does browser state and session data live?
All data, including cookies, localStorage, and recording traces, is stored locally on the machine running the MCP server.
What transport does Browser Control use?
It uses the MCP (Model Context Protocol) transport, designed for integration with AI agent clients. No authentication details are specified in the documentation.
Are there any known limits or stability notes?
Browser Control is stable, actively developed, and MIT licensed. No specific limits are documented beyond requiring a local Chromium installation.
