When pure Scrapy isn't enough—when the website checks for a real browser, executes complex JavaScript, or has advanced anti-bot protection—it's time to bring in the heavy artillery: Scrapy + Playwright.
This guide shows you how to configure them together for maximum stealth, making your scraper look exactly like a real user browsing Chrome.
1. Why Playwright?
Pure Scrapy is just a script. It doesn't have a screen, a mouse, or a JavaScript engine. Playwright is a real browser (Chromium, Firefox, WebKit). It passes almost all "Are you a robot?" checks by default because it is the tool humans use.
2. Installation
First, you need to install the integration plugin and the browsers.
Run these commands in your terminal:
pip install scrapy-playwright
playwright install chromium
3. Basic Configuration
You need to tell Scrapy to use Playwright for downloading pages instead of its default downloader.
Open settings.py and add/update these lines:
DOWNLOAD_HANDLERS = {
"http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
4. The "Stealth" Configuration (The Secret Sauce)
Just using Playwright isn't always enough. Sophisticated sites check for "automation flags" (variables that say "Hey, I'm being controlled by a script"). We need to disable them.
Add this to your settings.py:
PLAYWRIGHT_LAUNCH_OPTIONS = {
"headless": True,
"args": [
"--disable-blink-features=AutomationControlled",
"--no-sandbox",
],
}
PLAYWRIGHT_CONTEXT_ARGS = {
"javaScriptEnabled": True,
"ignoreHTTPSErrors": True,
"viewport": {"width": 1920, "height": 1080},
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
}
5. How to Use It in Your Spider
Now that settings are configured, you need to tell your spider to use Playwright for specific requests.
In your spider file (e.g., spiders/myspider.py):
import scrapy
class StealthSpider(scrapy.Spider):
name = "stealth"
def start_requests(self):
yield scrapy.Request(
url="https://nowsecure.nl",
meta={
"playwright": True,
"playwright_include_page": True,
},
callback=self.parse
)
async def parse(self, response):
title = response.css('title::text').get()
print(f"Title: {title}")
page = response.meta["playwright_page"]
await page.close()
6. Advanced Stealth: Randomizing User-Agents
Using the same User-Agent for every request is suspicious. Let's randomize it for every request.
Update your Spider to pass context arguments dynamically:
import scrapy
import random
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
]
class RandomStealthSpider(scrapy.Spider):
name = "random_stealth"
def start_requests(self):
ua = random.choice(USER_AGENTS)
yield scrapy.Request(
url="https://bot.sannysoft.com",
meta={
"playwright": True,
"playwright_context_args": {
"user_agent": ua,
"viewport": {"width": 1920, "height": 1080},
}
},
callback=self.parse
)
def parse(self, response):
pass
7. Complete settings.py for Copy-Paste
Here is the full configuration block for settings.py.
DOWNLOAD_HANDLERS = {
"http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
PLAYWRIGHT_LAUNCH_OPTIONS = {
"headless": True,
"args": [
"--disable-blink-features=AutomationControlled",
"--no-sandbox",
],
}
PLAYWRIGHT_CONTEXT_ARGS = {
"javaScriptEnabled": True,
"ignoreHTTPSErrors": True,
"viewport": {"width": 1280, "height": 720},
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
}
DOWNLOAD_DELAY = 2
CONCURRENT_REQUESTS = 4
Summary
Install scrapy-playwright.
Configure DOWNLOAD_HANDLERS and TWISTED_REACTOR.
Add Stealth Args: --disable-blink-features=AutomationControlled is the most important line.
Use Meta: Pass meta={"playwright": True} in your requests.
With this setup, you are running a real Chrome browser that explicitly lies about being automated. This bypasses 99% of bot detection systems.