Chapter 4: Web Automation
OpenClaw has two web access modes: lightweight (web_search and web_fetch via Brave API) and heavyweight (full browser control via Chromium). Use web_fetch when the content is in the HTML source or you need API data—it's 10x faster and doesn't require a browser. Use browser control when content loads via JavaScript, requires login, involves form submissions, or the site needs a real browser session. The Brave Search API (web_search) returns titles, URLs, and snippets with country-specific and freshness (pd/pw/pm/py) options. web_fetch returns markdown or plain text with optional maxChars` truncation. The pattern for competitive research is: search multiple queries, fetch promising URLs, extract and synthesize the data.
Browser control starts with browser: start, then snapshot to get the page accessibility tree with element references (e.g., e12), screenshot for visual capture, navigate for URL changes, and act for interactions (click, type, press, wait, evaluate). The evaluate action runs arbitrary JavaScript in the page context—extremely powerful for data extraction like Array.from(document.querySelectorAll('.price')).map(el => el.textContent). Waiting is critical for SPAs: wait with textGone waits for loading indicators to disappear; wait with text waits for specific content. Infinite scroll is triggered by evaluate with window.scrollTo(0, document.body.scrollHeight). Screenshots and PDFs are generated via browser: screenshot and browser: pdf with fullPage option.
Web automation is fragile because sites change. The most common failure is element reference drift after redesigns—mitigate by targeting by role/text content rather than position. Rate limiting (429s, CAPTCHAs) requires adding delays and being respectful. Login sessions expire and need cookie storage with re-authentication detection. Anti-bot JavaScript-heavy sites may require official APIs, realistic browser delays, alternative data sources, or accepting the automation isn't feasible. The architecture for a production web monitoring system combines web_fetch or browser control with a Python processing script, state file (JSON), change log, and alert dispatching—often wired into cron for scheduled runs.
Key Patterns
- **web_fetch vs browser:** Static content → web_fetch (fast); JS-rendered/login-required/forms → browser (heavyweight)
- **Browser snapshot → act loop:** snapshot gives element refs (e12); act uses those refs for click/type/press/evaluate
- **wait for content:** `wait` with `textGone` for loading spinners; `wait` with text for specific elements
- **evaluate for extraction:** Run arbitrary JS in page context for powerful, precise data extraction
- **Role-based targeting:** Click "Add to Cart" button rather than "third button in .product-actions div" for resilience to redesigns
Related Concepts
- [[kelly-handbook-ch6-cron]] for scheduling web monitoring tasks
- [[kelly-handbook-ch5-communication-automation]] for routing scraped data via WhatsApp alerts
- [[kelly-handbook-ch10-browser]] for deep-dive browser agent control patterns
- [[kelly-handbook-ch4-web-automation]] (this chapter covers the foundational web tools)