Viewing: raw.githubusercontent.com

[ Web Proxy ]

Viewing: https://raw.githubusercontent.com/We2one/Notes/master/python-basic-notes/python-days/Day_61.md

### Scrapy-redis #### Python ##### SMTP + **SMTP** (Simple Mail Transfer Protocol : ) : , + python smtplib , smtp ```python import smtplib smtpObj = smtplib.SMTP([host [, port [, local_hostname]]]) ``` + 1. **host** : SMTP , ip () 2. **port** : host SMTP , 25 3. **local_hostname** : SMTP , localhost + Python SMTP **sendmail** ,: ```python SMTP.sendmail(from_addr, to_addrs, msg [, mail_options, rcpt_options]) ``` + 1. **from_addr** : 2. **to_addrs** : , 3. **msg** : , () + msg smtp 1. 2. 3. 4. 5. ##### + ```python # import smtplib # smtpobj = smtplib.SMTP() # smtpobj.connect() # smtpobj.login() # smtpobj.sendmail() # smtpobj.quit() ``` 1. : `smtp.connect(self, host='localhost', port=0, source_address=None)` 2. : `smtplib.login(self, user, password, *, initial_response_ok=True)` 3. : `smtplib.sendmail(self, from_addr, to_addrs, msg, mail_options=(), rcpt_options=())` #### Redis 1.
1. redis 2. redis path 3. cmd `redis-server` , redis 2. redis + **redis.windows.conf** : redis + **redis-cli.exe** : + **redis-server.exe** : + : **6379** 3. reids redis 4. redis 1. , 2. ``` # Accept connections on the specified port, default is 6379 (IANA #815344). # If port 0 is specified Redis will not listen on a TCP socket. port 6379 ``` 3. , `redis-server []` redis 4. redis `redis-cli -p ` 5. redis 1. redis 2. redis , 6. [redis ](https://note.youdao.com/ynoteshare1/index.html?id=b25d0db9b99d173ad016f8d84184bd99&type=note) #### Scrapy-Redis ##### 1. + ,**** (: , ****,****url) + **** : ,,URL + 1. , 2. IP , IP + : **** URL 2. Scrapy Scrapy-Redis 1. Scrapy ,. Scrapy-redis. 2. Scrapy-redis ** (components) **, 1. Scheduler 2. Duplication Filter 3. Item Pipeline 4. Base Spider 3. Scrapy-redis : `pip intsall scrapy-redis` 3. Scrapy-redis 1. **Scheduler ** : request ( Scrapy queue), request ( Scrapy queue ) + Scrapy python **collection.deque () ** **Scrapy queue** + Scrapy spider Scrapy queue, Scrapy ,Scrapy-redis Scrapy queue **redis ** ( redis ), redis-server request, spider redis + Scrapy-redis ,,,. ``` { 0 : 0, 1 : 1, 2 : 2, ... } ``` 2. **Duplication Filter (DupeFilter)** : Scrapy-redis ,redis **set ** ,. + **Scrapy ** : request ,Scrapy request , request ,, request , ```python def request_seen(self, request): # self.request_fingerprints(request) fp = self.request_fingerprints(request) # if fp in self.fingerprints: return True self.fingerprints.add(fp) if self.file: self.file.write(fp + os.linesep) ``` + **Scrapy-redis ** : **request** , request redis set , request push redis request queue + **Scrapy-redis ** : request (Spider ) , redis request queue pop request , request spider 3. Item Pipeline + **Scrapy-redis ** : (spider ) Item spider ; Item Pipeline Item redis Items queue + Item Pipeline key item queue item, **items processes ** 4. Base Spider : RedisSpider **Spider** **RedisMixin ( redis url) ** + Spider RedisSpider , ** setup_redis **** redis **,** signals** ():
1. **spider ** signals : spider_idle , schedule_next_request , spider , **DontCloseSpider ** 2. ** item** signals : item-scraped , schedule_next_request , request 4. **** 1. **schedule ()** **spider ** : ******spider ****redis ** 2. **dupefilter ** **queue ** : **********redis **, redis , 3. spider scheduler , settings,** queue** ** dupefilter** 4. **Scrapy-redis ** : spider request ,Scrapy request spider scheduler , scheduler redis request , redis . , scheduler redis request spider , 5. **Scrapy-redis ** : spider url , scheduler spider redis , **spider_idle**, spider , redis start_urls url #### ##### + : , + ( : ****, 12) 1. URL ( : : ) 2. ( : ) 3. + **** : + url , redis set ,, url url set , + , redis set ,,, redis set

Web Proxy Viewer | New URL | Original Page