实现分页

class HupuSpiderSpider(scrapy.Spider):
    name = 'hupu_spider'
    # 二次请求过滤的域名, 可以注释
    # allowed_domains = ['www']
    start_urls = []
    for i in range(1, 101):
        base_url = f"http://voice.hupu.com/news?category=all&page={i}"
        start_urls.append(base_url)

更改 settings.py

1. 设置请求头

   # Override the default request headers:
   DEFAULT_REQUEST_HEADERS = {
     'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
     'Accept-Language': 'en',
     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
                   'Chrome/86.0.4240.111 Safari/537.36',
   }

2. 允许 ROBOTS 协议

   # Obey robots.txt rules
   ROBOTSTXT_OBEY = False

在 parse 实现提取逻辑

1. scrapy 中 response 对象的属性
   1. response.text : 字符串响应内容
   2. response.content : 二进制响应内容
   3. response.xpath() : xpath 提取数据,不需要 lxml 模块解析
      • 返回值为 []
      • 提取数据方法
        1. ..extract() : 提取 selector 对象的 所有 data, 返回 list
        2. .extract_first() : 仅提取第一个数据
2. scrapy 发送二次请求
   • yield scrapy.Request(url, callback=self.parse_detail,encoding='utf-8')
   • 参数
     • Request() : 请求对象
     • url : 请求 request 对象对应的 url
     • callback : 请求scrapy下载好之后交给哪个方法处理下载好的响应

数据保存

1. 在 parse 方法内 yield item

   def parse_detail(self, response):
       news_title = response.xpath('//h1[@class="headline"]/text()').extract_first()
       news_source = response.xpath('//span[@id="source_baidu"]/a/text()').extract_first()
       news_data = response.xpath('//span[@id="pubtime_baidu"]/text()').extract_first()
       news_content = response.xpath('string(//div[@class="artical-main-content"])').extract_first()
       item = HupuNewsItem()
       item['news_url'] = response.url
       item['news_title'] = news_title
       item['news_source'] = news_source
       item['news_data'] = news_data
       item['news_content'] = news_content
       yield item

2. 更改 item 定义 : 使用 item.py 内重定义

   class HupuNewsItem(scrapy.Item):
       # define the fields for your item here like:
       # name = scrapy.Field()
       news_url = scrapy.Field()
       news_title = scrapy.Field()
       news_source = scrapy.Field()
       news_data = scrapy.Field()
       news_content = scrapy.Field()

3. 在 pipelines.py 内编写保存

   class HupuNewsPipeline:
   
       def __init__(self):
           self.client = pymongo.MongoClient()
           self.db = self.client['hupu_news']
   
       def process_item(self, item, spider):
           self.db['NBA'].update({'news_url': item['news_url']}, {'$set': dict(item)}, True)
           print(item)
           return item

4. 在 setting.py 内配置 pipelines 管道

   # Configure item pipelines
   # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
   ITEM_PIPELINES = {
      'hupu_news.pipelines.HupuNewsPipeline': 300,
   }

在 spiders 目录下 新建 main.py : 方便执行代码,只需要直接执行 main.py 即可执行爬虫

from scrapy import cmdline

cmdline.execute('scrapy crawl company_spider'.split())