# How to Use Scrapy for Stealthy Web Scraping Without Getting Caught

Before you reach for heavy tools like Playwright or expensive proxies, you can do a LOT to avoid detection using just pure Scrapy. This guide covers every possible technique to make your standard Scrapy spider look more human.

## 1\. The Golden Rule: Don't Act Like a Robot

Robots are fast, precise, and repetitive. Humans are slow, random, and messy. To avoid detection, your spider must mimic human behavior.

---

## 2\. User-Agent Rotation (The Basics)

The `User-Agent` header tells the server what browser you are using. By default, Scrapy says "Scrapy/2.x". This is an instant ban on many sites.

**Solution:** Rotate through a list of real browser User-Agents.

**Step-by-Step Implementation:**

1. **Install the library:** Open your terminal and run:
    
    ```bash
    pip install scrapy-user-agents
    ```
    
2. **Edit** [`settings.py`](http://settings.py): Open the [`settings.py`](http://settings.py) file in your project folder. Find the `DOWNLOADER_MIDDLEWARES` section (or create it if it doesn't exist) and paste this:
    
    ```python
    # settings.py
    
    DOWNLOADER_MIDDLEWARES = {
        # Disable the default UserAgent middleware
        'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
        # Enable the random UserAgent middleware
        'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,
    }
    ```
    

---

## 3\. Headers: The "Fingerprint" of a Browser

Browsers send a specific set of headers with every request. If you only send a User-Agent, it looks suspicious.

**Solution:** Copy the full headers from a real browser request.

**How to get them:**

1. Open Chrome -> Network Tab.
    
2. Refresh the page.
    
3. Right-click the main request -> **Copy** -> **Copy as cURL (bash)**.
    
4. Use a tool (like [curlconverter.com](http://curlconverter.com)) to convert it to a Python dictionary.
    

**Where to put them:** You can put them in [`settings.py`](http://settings.py) to apply to *every* request, or in your spider for specific requests.

**Option A: Global Settings (In** [`settings.py`](http://settings.py))

```python
# settings.py

DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'none',
    'Sec-Fetch-User': '?1',
    'Cache-Control': 'max-age=0',
}
```

**Option B: Per Spider (In** `spiders/`[`myspider.py`](http://myspider.py))

```python
# spiders/myspider.py

class MySpider(scrapy.Spider):
    name = 'myspider'
    
    custom_settings = {
        'DEFAULT_REQUEST_HEADERS': {
            'Accept': 'text/html,...',
            # ... paste headers here
        }
    }
```

---

## 4\. Random Delays (Politeness)

Robots hit pages instantly. Humans take time to read.

**Solution:** Slow down your spider and make it random.

**Where to put it:** Open [`settings.py`](http://settings.py) and add/change these lines:

```python
# settings.py

# Enable Auto-Throttling (Scrapy adjusts speed based on server load)
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 2
AUTOTHROTTLE_MAX_DELAY = 60

# Add a random delay between requests
# If set to 2, Scrapy will wait between 1s and 3s randomly
DOWNLOAD_DELAY = 2 
RANDOMIZE_DOWNLOAD_DELAY = True
```

---

## 5\. Cookies and Sessions

Some sites track your "session". If you make 100 requests with no cookies (or the same cookie for too long), it looks weird.

**Scenario A: Disable Cookies (General Scraping)** If the site tracks users to ban them, disable cookies so every request looks like a new visitor.

**In** [`settings.py`](http://settings.py):

```python
COOKIES_ENABLED = False
```

**Scenario B: Maintain Session (Login/Complex Sites)** If the site requires a session, keep cookies enabled (default) but be careful not to make too many requests from one "user".

---

## 6\. Referer Spoofing

When you click a link from Google to a site, the `Referer` header says "[google.com](http://google.com)". If you go directly to a product page with no Referer, it looks like a bot.

**Solution:** Fake the `Referer` header.

**Where to put it:** Inside your spider code.

```python
# spiders/myspider.py

def start_requests(self):
    yield scrapy.Request(
        url="https://example.com/product/123",
        headers={'Referer': 'https://www.google.com/'}, # <--- Add this
        callback=self.parse
    )
```

---

## 7\. Concurrency Limits

Don't hammer the server.

**In** [`settings.py`](http://settings.py):

```python
CONCURRENT_REQUESTS = 8  # Default is 16, lower is safer
CONCURRENT_REQUESTS_PER_DOMAIN = 4
```

---

## Complete Example: Putting It All Together

Here is a complete [`settings.py`](http://settings.py) file optimized for stealth. You can copy-paste this into your project.

```python
# settings.py

BOT_NAME = 'myproject'
SPIDER_MODULES = ['myproject.spiders']
NEWSPIDER_MODULE = 'myproject.spiders'

# 1. Rotate User Agents
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,
}

# 2. Real Browser Headers
DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Upgrade-Insecure-Requests': '1',
}

# 3. Random Delays
DOWNLOAD_DELAY = 2
RANDOMIZE_DOWNLOAD_DELAY = True
AUTOTHROTTLE_ENABLED = True

# 4. Disable Cookies (Optional, depends on site)
COOKIES_ENABLED = False

# 5. Limit Concurrency
CONCURRENT_REQUESTS = 8

# Respect robots.txt (Good practice, but sometimes you need to disable it)
ROBOTSTXT_OBEY = True
```

By applying all these settings, you can scrape a surprising number of "protected" sites using just pure Scrapy, saving you the overhead of using a full browser.