Scraping modern websites with Playwright
Half the web is rendered by JavaScript, and a plain HTTP request sees none of it. Playwright drives a real browser — here's how to use it to scrape, and how to do it responsibly.
Feeding an LLM good data often means collecting it from the web first — and the classic "fetch the HTML and parse it" approach falls apart on modern sites, where the content you want is drawn by JavaScript after the page loads. Playwright solves that by driving a real browser.
What it is
Playwright is a browser-automation library that controls Chromium, Firefox, and WebKit. It loads a page exactly as a user's browser would — running the JavaScript, waiting for content — and lets you interact with and read the fully-rendered result. That makes it far more capable than raw HTTP requests for dynamic pages.
import { chromium } from 'playwright';
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/products');
await page.waitForSelector('.product'); // wait for JS to render
const titles = await page.$$eval('.product h2',
(els) => els.map((e) => e.textContent?.trim()));
await browser.close();The patterns that make it reliable
- Wait for elements, not timers — waitForSelector beats arbitrary sleeps and cuts flakiness.
- Handle pagination and infinite scroll explicitly — click "next" or scroll, then wait for new content.
- Be resilient — pages change; select on stable attributes and fail loudly when structure shifts.
Scrape responsibly
Capability isn't permission. Respect robots.txt and terms of service, rate-limit your requests, avoid personal data, and prefer an official API when one exists. Getting the data isn't the hard part — getting it without being reckless is.
If a browser can see it, Playwright can read it. Whether you should is a separate question — ask it first.