# How to Avoid Bot Detection Using Scrapy and Playwright

When pure Scrapy isn't enough—when the website checks for a real browser, executes complex JavaScript, or has advanced anti-bot protection—it's time to bring in the heavy artillery: **Scrapy + Playwright**.

This guide shows you how to configure them together for maximum stealth, making your scraper look exactly like a real user browsing Chrome.

## 1\. Why Playwright?

Pure Scrapy is just a script. It doesn't have a screen, a mouse, or a JavaScript engine. Playwright **is** a real browser (Chromium, Firefox, WebKit). It passes almost all "Are you a robot?" checks by default because it *is* the tool humans use.

---

## 2\. Installation

First, you need to install the integration plugin and the browsers.

**Run these commands in your terminal:**

```bash
pip install scrapy-playwright
playwright install chromium
```

---

## 3\. Basic Configuration

You need to tell Scrapy to use Playwright for downloading pages instead of its default downloader.

**Open** [`settings.py`](http://settings.py) and add/update these lines:

```python
# settings.py

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

# This is required for Playwright to work with Scrapy
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
```

---

## 4\. The "Stealth" Configuration (The Secret Sauce)

Just using Playwright isn't always enough. Sophisticated sites check for "automation flags" (variables that say "Hey, I'm being controlled by a script"). We need to disable them.

**Add this to your** [`settings.py`](http://settings.py):

```python
# settings.py

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,  # Set to False to see the browser pop up (good for debugging)
    "args": [
        "--disable-blink-features=AutomationControlled", # <--- THE KEY to stealth
        "--no-sandbox",
    ],
}

PLAYWRIGHT_CONTEXT_ARGS = {
    "javaScriptEnabled": True,
    "ignoreHTTPSErrors": True,
    # Set a real browser viewport size
    "viewport": {"width": 1920, "height": 1080},
    # Set a real User-Agent (very important!)
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
}
```

* `--disable-blink-features=AutomationControlled`: This removes the "I am a robot" flag that Chrome usually sends when controlled by code.
    
* `user_agent`: We manually set a modern Chrome user agent.
    

---

## 5\. How to Use It in Your Spider

Now that settings are configured, you need to tell your spider to use Playwright for specific requests.

**In your spider file (e.g.,** `spiders/`[`myspider.py`](http://myspider.py)):

```python
import scrapy

class StealthSpider(scrapy.Spider):
    name = "stealth"

    def start_requests(self):
        yield scrapy.Request(
            url="https://nowsecure.nl",  # A site to test security
            meta={
                "playwright": True,
                "playwright_include_page": True, # Optional: if you need to interact with the page
            },
            callback=self.parse
        )

    async def parse(self, response):
        # Extract data normally
        title = response.css('title::text').get()
        print(f"Title: {title}")
        
        # If you need to interact (click/scroll), you get the 'page' object
        page = response.meta["playwright_page"]
        await page.close()
```

---

## 6\. Advanced Stealth: Randomizing User-Agents

Using the same User-Agent for every request is suspicious. Let's randomize it for every request.

**Update your Spider to pass context arguments dynamically:**

```python
import scrapy
import random

# List of real User-Agents
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
]

class RandomStealthSpider(scrapy.Spider):
    name = "random_stealth"

    def start_requests(self):
        ua = random.choice(USER_AGENTS)
        yield scrapy.Request(
            url="https://bot.sannysoft.com", # A bot detection test site
            meta={
                "playwright": True,
                "playwright_context_args": {
                    "user_agent": ua,
                    "viewport": {"width": 1920, "height": 1080},
                }
            },
            callback=self.parse
        )

    def parse(self, response):
        # ... extraction logic
        pass
```

---

## 7\. Complete [`settings.py`](http://settings.py) for Copy-Paste

Here is the full configuration block for [`settings.py`](http://settings.py).

```python
# settings.py

# 1. Enable Playwright
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# 2. Launch Options (The Browser App)
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True, # Set False to watch it work
    "args": [
        "--disable-blink-features=AutomationControlled", # Hides the 'robot' flag
        "--no-sandbox",
    ],
}

# 3. Context Options (The Browser Tab)
PLAYWRIGHT_CONTEXT_ARGS = {
    "javaScriptEnabled": True,
    "ignoreHTTPSErrors": True,
    "viewport": {"width": 1280, "height": 720},
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
}

# 4. Standard Scrapy Politeness (Still applies!)
DOWNLOAD_DELAY = 2
CONCURRENT_REQUESTS = 4
```

## Summary

1. **Install** `scrapy-playwright`.
    
2. **Configure** `DOWNLOAD_HANDLERS` and `TWISTED_REACTOR`.
    
3. **Add Stealth Args:** `--disable-blink-features=AutomationControlled` is the most important line.
    
4. **Use Meta:** Pass `meta={"playwright": True}` in your requests.
    

With this setup, you are running a real Chrome browser that explicitly lies about being automated. This bypasses 99% of bot detection systems.

%%[google-add-ravi]