# [Python](https://github.com/lining0806/PythonSpiderNotes)

*** 

Python3************  

[Scrapy](http://scrapy.org/)    

[-](http://www.lining0806.com/category/spider/)  
***

url[http://www.lining0806.com/](http://www.lining0806.com/)



* IP
* IP
* 
* 

url

##   
HTMLJson  

#### 1.   

get  

Pythonurlliburllib2[requests](https://github.com/kennethreitz/requests)[httplib2](https://github.com/jcgregorio/httplib2)    

```
Requests
	import requests
	response = requests.get(url)
	content = requests.get(url).content
	print "response headers:", response.headers
	print "content:", content
Urllib2
	import urllib2
	response = urllib2.urlopen(url)
	content = urllib2.urlopen(url).read()
	print "response headers:", response.headers
	print "content:", content
Httplib2
	import httplib2
	http = httplib2.Http()
	response_headers, content = http.request(url, 'GET')
	print "response headers:", response_headers
	print "content:", content
```  

urlgeturl?url&  

```
data = {'data1':'XXXXX', 'data2':'XXXXX'}
Requestsdatadictjson
	import requests
	response = requests.get(url=url, params=data)
Urllib2datastring
	import urllib, urllib2    
	data = urllib.urlencode(data)
	full_url = url+'?'+data
	response = urllib2.urlopen(full_url)
```

[](http://www.lining0806.com/%E7%BD%91%E6%98%93%E6%96%B0%E9%97%BB%E6%8E%92%E8%A1%8C%E6%A6%9C%E6%8A%93%E5%8F%96%E5%9B%9E%E9%A1%BE/)

[](https://github.com/lining0806/PythonSpiderNotes/blob/master/NewsSpider)

### 2.   

**2.1 **  

postcookie  

```
data = {'data1':'XXXXX', 'data2':'XXXXX'}
Requestsdatadictjson
	import requests
	response = requests.post(url=url, data=data)
Urllib2datastring
	import urllib, urllib2    
	data = urllib.urlencode(data)
	req = urllib2.Request(url=url, data=data)
	response = urllib2.urlopen(req)
```  

**2.2 cookie**  

cookiecookie  

```
import requests			
requests_session = requests.session() 
response = requests_session.post(url=url_login, data=data) 
```

response = requests_session.post(url=url_login, data=data)  

```
response_captcha = requests_session.get(url=url_login, cookies=cookies)
response1 = requests.get(url_login) # 
response2 = requests_session.get(url_login) # Response Cookie
response3 = requests_session.get(url_results) # Response Cookie
```

[-](http://www.lining0806.com/6-%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB-%E9%AA%8C%E8%AF%81%E7%A0%81%E7%99%BB%E9%99%86/)  

[](https://github.com/lining0806/PythonSpiderNotes/blob/master/ZhihuSpider)  

### 3.  

**3.1 ** 

IP  

IPIP

```
proxies = {'http':'http://XX.XX.XX.XX:XXXX'}
Requests
	import requests
	response = requests.get(url=url, proxies=proxies)
Urllib2
	import urllib2
	proxy_support = urllib2.ProxyHandler(proxies)
	opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
	urllib2.install_opener(opener) # openerurlopen()opener
	response = urllib2.urlopen(url)
```

**3.2 ** 

 

RequestsUrllib2timesleep()

```
import time
time.sleep(1)
```

**3.3 **  

User-AgentRefererRefererReferer

```
headers = {'User-Agent':'XXXXX'} # 
headers = {'Referer':'XXXXX'}
headers = {'User-Agent':'XXXXX', 'Referer':'XXXXX'}
Requests
	response = requests.get(url=url, headers=headers)
Urllib2
	import urllib, urllib2   
	req = urllib2.Request(url=url, headers=headers)
	response = urllib2.urlopen(req)
```

### 4.   



```
def multi_session(session, *arg):
	retryTimes = 20
	while retryTimes>0:
		try:
			return session.post(*arg)
		except:
			print '.',
			retryTimes -= 1
```

  

```
def multi_open(opener, *arg):
	retryTimes = 20
	while retryTimes>0:
		try:
			return opener.open(*arg)
		except:
			print '.',
			retryTimes -= 1
```

multi_sessionmulti_opensessionopener    

### 5.   

[](http://live.wallstreetcn.com/ )[Python](https://github.com/lining0806/PythonSpiderNotes/blob/master/Spider_Python)  [Java](https://github.com/lining0806/PythonSpiderNotes/blob/master/Spider_Java)  

[PythonJava](http://www.lining0806.com/%E5%85%B3%E4%BA%8Epython%E5%92%8Cjava%E7%9A%84%E5%A4%9A%E8%BF%9B%E7%A8%8B%E5%A4%9A%E7%BA%BF%E7%A8%8B%E8%AE%A1%E7%AE%97%E6%96%B9%E6%B3%95%E5%AF%B9%E6%AF%94/)  

### 6. Ajax  

Ajax

urlJavaScripturl  

Google Chrome(NetworkGETTypetext/htmlgetRequest URL)  

* 1Ajax  
* json(str)json'\\uxxxx'unicode_escapeu'\uxxxx'unicode  

### 7. Selenium

SeleniumSelenium

selenium[](http://flight.qunar.com/)

[Selenium](https://github.com/lining0806/PythonSpiderNotes/blob/master/QunarSpider) 

### 8.   

  

* IP
* cookie
* 

cookie  

Tesseract-OCR  

[Captcha1](https://github.com/lining0806/PythonSpiderNotes/blob/master/Captcha1)

****

* 
* 

##   

  

[](http://deerchao.net/tutorials/regex/regex.htm)[BeautifulSoup](http://www.crummy.com/software/BeautifulSoup/)[lxml](http://lxml.de/)  

##   

  

[MySQL](http://www.mysql.com/)[MongoDB](https://www.mongodb.org/)  

****

* 
* 


## Scrapy  

ScrapyTwistedPython  

[Scrapy](http://www.lining0806.com/%E5%9F%BA%E4%BA%8Escrapy%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB%E7%9A%84%E6%90%AD%E5%BB%BA/)[](http://weixin.sogou.com/weixin)

[ScrapyRequests](https://github.com/lining0806/PythonSpiderNotes/blob/master/WechatSearchProjects)

## Robots  

**Robots**RobotsRobots Exclusion ProtocolRobots

robots.txt https://www.taobao.com/robots.txt robots.txt

### 1. Robots

	User-agent: 
	Disallow: 
	Allow: 

: "/"

### 2. Robots

	
		User-agent: *
		Disallow: /
	
		User-agent: *
		Disallow: 
	
		User-agent: BadBot
		Disallow: /
	
		User-agent: GoodBot
		Disallow: 
	
		User-agent: *
		Disallow: /images/
	
		User-agent: *
		Allow: /images/
		Disallow: /
	
		User-agent: *
		Disallow: /*.html$
	
		User-agent: *
		Allow: /*.html$
		Disallow: /