# How to Set Up a Scrapy Project: A Beginner's Guide

## Creating a New Scrapy Project

Once Scrapy is installed, the first step is to set up a new project. Navigate to the directory where you want to store your code and run:

```bash
scrapy startproject myproject
```

This will create a `myproject` directory with the following structure:

```plaintext
myproject/
    scrapy.cfg            # deploy configuration file
    myproject/            # project's Python module, you'll import your code from here
        __init__.py
        items.py          # project items definition file
        middlewares.py    # project middlewares file
        pipelines.py      # project pipelines file
        settings.py       # project settings file
        spiders/          # a directory where you'll later put your spiders
            __init__.py
```

## Understanding the Project Structure

* `scrapy.cfg`: The project configuration file. It defines the project settings module.
    
* [`items.py`](http://items.py): Defines the data structures (containers) for the scraped data, similar to Django models.
    
* [`middlewares.py`](http://middlewares.py): Hooks to process requests and responses globally.
    
* [`pipelines.py`](http://pipelines.py): Processes the scraped items (e.g., cleaning data, saving to a database).
    
* [`settings.py`](http://settings.py): Contains project settings like user agent, download delay, and enabled pipelines.
    
* `spiders/`: This is where your "spiders" (the classes that define how to scrape a site) will live.
    

## Basic Scrapy Commands

Scrapy provides a command-line tool to control your project. Here are some common commands:

* `scrapy shell [url]`: Opens an interactive shell to try out selectors and debug.
    
* `scrapy crawl [spider_name]`: Runs a spider.
    
* `scrapy genspider [name] [domain]`: Generates a new spider file.
    

## Your First Spider

Let's create a simple spider to scrape quotes from [`quotes.toscrape.com`](http://quotes.toscrape.com).

1. Navigate into your project: `cd myproject`
    
2. Generate a spider: `scrapy genspider quotes` [`quotes.toscrape.com`](http://quotes.toscrape.com)
    

This creates `myproject/spiders/`[`quotes.py`](http://quotes.py). Let's edit it:

```python
import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
            }
```

## Running the Spider

To run the spider and save the output to a JSON file:

```bash
scrapy crawl quotes -O quotes.json
```

This command runs the `quotes` spider and outputs the results to `quotes.json`.

## Next Steps

In the next article, we will compare Scrapy with other tools like Selenium and Playwright to understand when to use which.