Imagine you are working, spending hours configuring proxies, tweaking user agents, and, let’s say, finally bypassing that strict bot detection. You did it. The terminal is printing data, and everything feels great.
But when you deploy that script of yours to a server or try to run it on 10,000 URLs or more, it crawls at a slug’s pace, crashes randomly, and your RAM usage breaks the roof.
If that sounds familiar, chances are you are making one (or more) of these five Playwright mistakes
1. The time.sleep() Virus
This is a common problem people face when moving from older tools like Selenium. When a webpage takes time to load something, like a button or an image, some people just add a wait command to their code. They use a fixed time, like 5 or 10 seconds. Hoping that is going to be enough, and I also used to do the same not very long ago, but I realized and dropped this bad habit, and you should too
import time
# Stop using this
page.goto("URL of site you are working on")
time.sleep(5)
data = page.locator(".price-tag").text_content()
Why it breaks:
If a webpage loads in 1 second, then you have wasted 4 seconds. If you multiply that by 1000 webpages, then you have wasted over an hour. If the webpage takes six seconds to load, then your script will crash.
The Fix:
Playwright is really good because it has an auto-waiting feature that is built right into it. You can tell Playwright what you are waiting for, and that is really helpful. Playwright has this feature built in, which is very useful for waiting.
# The Professional Way
page.goto("URL of site you are working on")
page.locator(".price-tag").wait_for(state="visible", timeout=10000)
data = page.locator(".price-tag").text_content()
2. Downloading the Whole Internet
Now that your scraper is waiting efficiently, let’s look at the next bottleneck: downloading resources you don’t even use. When you scrape sites like massive e-commerce platforms or directories of real estate, the target pages are loaded with 4K property images, tracking scripts, auto-playing videos, and can also include custom fonts. If your goal is just to extract text data like prices and descriptions, letting Playwright download all those heavy assets is a massive waste of bandwidth and speed, and believe me, bandwidth doesn’t grow on trees.
The Fix:
The solution is pretty simple, all you have to do is use route.abort() to block any network requests that do not serve your pipeline.
# Block heavy media
async def intercept_routes(route):
if route.request.resource_type in ["image", "media", "font", "stylesheet"]:
await route.abort()
else:
await route.continue_()
await page.route("**/*", intercept_routes)
await page.goto("https://heavysite.com/listings")
Blocking out only these assets can easily cut your page load times by 60 to 70%.
3. The “New Browser per Request” Memory Leak
I once received a scraper from a client that launched a brand-new Chromium browser for every URL. It worked perfectly during testing. Then the client pointed it at 8,000 pages overnight. By morning, the server had run out of memory, and the scraper had stopped halfway through. It’s something I often see in freelance codebases. A developer wants to scrape 50 pages, so they put ‘browser = await p.chromium.launch()’ inside a for loop, and it’s not just beginners, many intermediate or even a few pro-level developers also make this mistake.
Launching Chromium can take hundreds of milliseconds every time. Multiply that by thousands of URLs, and you have successfully wasted minutes or even hours while putting unnecessary pressure on your poor CPU and memory. It can quickly drain your server’s resources, causing it to crash.
The Fix:
The solution is simple, launch the browser once. Then use Playwright’s Browser Contexts to create isolated sessions. These Contexts share the browser’s resources. Keep cookies and cache separate for each session. This way, you can scrape pages without overloading your server.
You should use browser contexts for launching a new browser for every URL. This will help you avoid crashing your server.
# Launch the browser once
browser = await p.chromium.launch(headless=True)
# Create contexts for concurrent tasks
context_1 = await browser.new_context()
page_1 = await context_1.new_page()
context_2 = await browser.new_context()
page_2 = await context_2.new_page()
4. Fragile CSS Chains
If your scraper relies on deeply nested DOM structures, then it’s not wrong to say that you are literally setting a trap for yourself.
Why:
Let’s say a front-end developer updates a class name, adds a new banner, or changes the UI framework; in this case, your entire scraper will break.
The Fix:
I would like to tell you a very important thing its “Scrape like a human.” A human doesn’t look for the 4th span inside a div; they would usually look for the button that says "Add to Cart" or "Contact Agent". Use Playwright’s semantic locators.
# Bulletproof Locators
page.get_by_role("button", name="Contact Agent")
page.get_by_text("Out of Stock")
5. UI Login Repetition
This final part is more of a suggestion. When you need to get to your target data, and it requires authentication, don’t make a script that puts in your username and password every time it is used. Not only is it very slow, but also logging in over and over again is the fastest way to trigger your biggest nightmare CAPTCHA and get your IP banned.
The Fix:
The way to make it work is to log in manually or use a script one time and save the authentication state (cookies and local storage) to a JSON file, and use it in all future sessions.
# Step 1: Saves the state after logging in
await context.storage_state(path="auth_state.json")
# Step 2: Inject the state on all future scraper runs
context = await browser.new_context(storage_state="auth_state.json")
page = await context.new_page()
await page.goto("https://targetsite.com/dashboard")
In the end, a piece of advice for my fellow freelancers and my readers: “Building a scraper that works once on your local machine is easy, but building a resilient data pipeline that runs reliably every single day without memory leaks, IP bans, or silent crashes requires treating your automation scripts like real production software.”
The difference between a beginner scraper and a production scraper is not how many pages it can scrape, but how gracefully it handles everything that goes wrong. Like websites change, networks fail, elements disappear, and sessions expire.
Your code should expect all of it. That’s when you stop writing scripts and start engineering data pipelines.
By dropping those old Selenium habits, optimizing your network requests, and writing smart locators, you will save yourself hours of debugging when the target website inevitably updates its UI. Your code will run faster, your servers will stay cool, and most importantly, your data will be perfectly clean.
(Dealing with a scraping edge case that’s currently driving you crazy? Drop it in the comments. If you want to see how I build these pipelines for clients, you can view my recent project history **her**e, or just connect with me right here on Medium.)
Comments
Loading comments…