> For the complete documentation index, see [llms.txt](https://moluo.gitbook.io/notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://moluo.gitbook.io/notes/bian-cheng-xue-xi/python/scrapy/4.2-tiao-shi.md).

# 4.2调试

在命令行运行爬虫步骤

1.切换路径到项目根路径

2.使用如下命令运行指定爬虫

```bash
scrapy crawl {爬虫名称}
```

> 提示：爬虫名称与ArticleSpider/spiders/下的python文件名相同
>
> 如运行ArticleSpider/spiders/jobbole.py，则使用如下命令
>
> ```bash
> scrapy crawl jobbole
> ```

在pycharm中运行爬虫

在ArticleSpider/目录下新建main.py文件，内容如下

```bash
import os
import sys
from scrapy.cmdline import execute

sys.path.append(os.path.dirname(os.path.abspath(__file__)))
execute(["scrapy", "crawl", "jobbole"])
```

> 解析：
>
> `os.path.dirname(os.path.abspath(__file__))`获取ArticleSpider/的绝对路径
>
> execute(\["scrapy", "crawl", "jobbole"])运行jobbole爬虫

在ArticleSpider/settings.py中，设置不遵循robots.txt协议

```
ROBOTSTXT_OBEY = False
```

若ROBOTSTXT\_OBEY = True，scrapy会读取每个网站的robos.txt协议，将robos.txt协议中的url过滤掉，不进行爬取