### Scrapy 

#### Scrapy 

+ Scrapy 
+ Scrapy  **Twisted **,,,.

#### Scrapy 

##### Scrapy 

+ **Scrapy Engine () ** :  spiderItemPipelineDownloaderScheduler 
+ **Scheduler () ** :  Requests ,,,,
+ **Downloader () ** :  Scrapy Engine ()  Requests , Responses  Scrapy Engine () ,  spider 
+ **spider () ** :  Response,, Item , URL , Scheduler ()
+ **Item Pipeline () ** :  spider  Item,  ()
+ **Downloader Middlewares () ** : 
+ **SpiderMiddlewares (spider ) ** : ******** **spider**  ( spider  Response   spider  Requests)

##### Scrapy 

1.  spider 
2. spider  
3.  spider   URL
4. spider   URL
5.  request  
6. , request  
7.   request 
8. request 
9. ,;,,
10.    spider (responses  def parse() )
11.  spider ,  :  URL   Item 
12.   Item ,  URL  ,,
13.    requests , ( URL , Scrapy )

##### Scrapy 

1.  (Scrapy startproject project_name) : 
2.  ( item.py) : 
3.  (spiders/xxspider.py) : 
4.  (pipelines.py) : 

#### Scrapy 

##### spider

+ spider , : ()( item)
+ **class Scrapy.spider** : ,
+ ********
  1. `__init__()` :  start_urls( url) 
  2. `start_requests()` :  `make_requests_from_url()`  Requests  Scrapy  response
  3. `parse()` :  response,  item  Requests ().Item  Item pipline , Requests  Scrapy ,  ( parse()).,
+ ********
  1. `name` :  spider  ( mywebsite.com , spider  mywebsite)
  2. `allowed_domains` :  spider ,
  3. `start_urls` :  URL . URL ,spider 
  4. `start_requests(self)` :  (iterable) . spider  Request
  5. `parse(self, response)` :  URL , Request . response ,  Item  Request 

##### Parse() 

+ Parse()  **yield** ,,Scrapy  parse ,
  +  **request** : 
  +  **item** :  pipeline ,
  +  : 
+ Parse() 
  + Scrapy  request ,,
  +  request,  item,  item  pipeline 
  + parse()  (callback)  Request, parse()  `Scrapy.Request(url, callback=self.parse)`
  + Request , `Scrapy.http.response()` ,  parse() , Request ()
  +  parse() ,   pipelines 

##### Logging

+ Scrapy  **(log)** ,  logging  **settings.py** , 

  ```python
  LOG_FILE = "text.log"
  LOG_LEVEL = "INFO"
  ```

+ Log levels : Scrapy  5  logging 

  1. **CRITICAL** :  (critical)
  2. **ERROR** :  (regular errors)
  3. **WARNING** :  (warning messages)
  4. **INFO** :  (informational messages)
  5. **DEBUG** :  (debugging messages)

+ logging  :  **settings.py**  logging

  + **LOG_ENABLED** : : True,  logging
  + **LOG_ENCODING** : : "utf-8", logging 
  + **LOG_FILE** : : None,  logging 
  + **LOG_LEVEL** : : "DEBUG",log 
  + **LOG_STDOUT** : : False, True,() log 

##### Robots  ()

+ Robots  ( Robots Exclusion Protocal) :  robots ,

+ scrapy ,  **settings.py**  : ROBOTSTXT_OBER = False

+  robots : [https://www.hupu.com/robots.txt](https://www.hupu.com/robots.txt)

  ```
  User-agent: *
  Allow: /
  
  Sitemap: https://bbs.hupu.com/sitemap_index.xml
  Sitemap: https://bbs.hupu.com/sitemap/sitemap_boards.xml
  Sitemap: https://voice.hupu.com/sitemap_index.xml
  Sitemap: https://nba.hupu.com/players/index.xml
  ```


#### Scrapy 

##### Scrapy 	

1. 

   ```python
   class HupuSpiderSpider(scrapy.Spider):
       name = 'hupu_spider'
       # , 
       # allowed_domains = ['www']
       start_urls = []
       for i in range(1, 101):
           base_url = f"http://voice.hupu.com/news?category=all&page={i}"
           start_urls.append(base_url)
   ```

2.  **settings.py**

   1. 

      ```python
      # Override the default request headers:
      DEFAULT_REQUEST_HEADERS = {
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
        'Accept-Language': 'en',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
                      'Chrome/86.0.4240.111 Safari/537.36',
      }
      ```

   2.  ROBOTS 

      ```python
      # Obey robots.txt rules
      ROBOTSTXT_OBEY = False
      ```

3.  **parse** 

   1. scrapy  response 
      1. response.text : 
      2. response.content : 
      3. response.xpath() : xpath , lxml 
         +  *[]* 
         + 
           1. .`.extract()` :  selector   data,  list
           2. `.extract_first()` :  
   2. scrapy 
      + `yield scrapy.Request(url, callback=self.parse_detail,encoding='utf-8')`
      + 
        + **Request()** : 
        + **url** :  request  url
        + **callback** : scrapy

4. 

   1.  parse  `yield item`

      ```python
      def parse_detail(self, response):
          news_title = response.xpath('//h1[@class="headline"]/text()').extract_first()
          news_source = response.xpath('//span[@id="source_baidu"]/a/text()').extract_first()
          news_data = response.xpath('//span[@id="pubtime_baidu"]/text()').extract_first()
          news_content = response.xpath('string(//div[@class="artical-main-content"])').extract_first()
          item = HupuNewsItem()
          item['news_url'] = response.url
          item['news_title'] = news_title
          item['news_source'] = news_source
          item['news_data'] = news_data
          item['news_content'] = news_content
          yield item
      ```

   2.  item  :  item.py 

      ```python
      class HupuNewsItem(scrapy.Item):
          # define the fields for your item here like:
          # name = scrapy.Field()
          news_url = scrapy.Field()
          news_title = scrapy.Field()
          news_source = scrapy.Field()
          news_data = scrapy.Field()
          news_content = scrapy.Field()
      ```

   3.  **pipelines.py** 

      ```python
      class HupuNewsPipeline:
      
          def __init__(self):
              self.client = pymongo.MongoClient()
              self.db = self.client['hupu_news']
      
          def process_item(self, item, spider):
              self.db['NBA'].update({'news_url': item['news_url']}, {'$set': dict(item)}, True)
              print(item)
              return item
      ```

   4.  **setting.py**  pipelines 

      ```python
      # Configure item pipelines
      # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
      ITEM_PIPELINES = {
         'hupu_news.pipelines.HupuNewsPipeline': 300,
      }
      ```

5.  spiders   **main.py** : , main.py 

   ```python
   from scrapy import cmdline
   
   cmdline.execute('scrapy crawl company_spider'.split())
   ```


##### 

1.  Scrapy : `pip install scrapy`
2.  : `scrapy startproject `
3.  spider : `scrapy genspider (name) www` (www: )
4. 
5.  : `scrapy crawl (name )`