FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

use ReStructuredText for README instead of markdown · bocode/python-goose@2dff006 · GitHub

/ python-goose Public
forked from grangier/python-goose

Commit 2dff006

Browse files
grangier
committed
use ReStructuredText for README instead of markdown
1 parent 32f4dae commit 2dff006

2 files changed

Lines changed: 104 additions & 55 deletions

File tree

README.md renamed to README.rst

Lines changed: 103 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -1,46 +1,59 @@
1-
#Python-Goose - Article Extractor [![Build Status](https://www.travis-ci.org/xgdlm/python-goose.png?branch=master)](https://www.travis-ci.org/xgdlm/python-goose)
1+
Python-Goose - Article Extractor |Build Status|
2+
===============================================
23

3-
##Intro
4+
Intro
5+
-----
46

7+
Goose was originally an article extractor written in Java that has most
8+
recently (aug2011) converted to a scala project by Gravity.com
59

6-
Goose was originally an article extractor written in Java that has most recently (aug2011) converted to a scala project by Gravity.com
7-
8-
This is a complete rewrite in python. The aim of the software is is to take any news article or article type web page and not only extract what is the main body of the article but also all meta data and most probable image candidate.
10+
This is a complete rewrite in python. The aim of the software is is to
11+
take any news article or article type web page and not only extract what
12+
is the main body of the article but also all meta data and most probable
13+
image candidate.
914

1015
Goose will try to extract the following information:
1116

12-
- Main text of an article
13-
- Main image of article
14-
- Any Youtube/Vimeo movies embedded in article (TODO)
15-
- Meta Description
16-
- Meta tags
17-
17+
- Main text of an article
18+
- Main image of article
19+
- Any Youtube/Vimeo movies embedded in article (TODO)
20+
- Meta Description
21+
- Meta tags
1822

1923
Originally, Goose was open sourced by Gravity.com in 2011
2024

21-
- Lead Programmer: Jim Plush (Gravity.com)
22-
- Contributers: Robbie Coleman (Gravity.com)
25+
- Lead Programmer: Jim Plush (Gravity.com)
26+
- Contributers: Robbie Coleman (Gravity.com)
2327

2428
The python version was rewrite by:
2529

26-
- Xavier Grangier (Recrutae.com)
30+
- Xavier Grangier (Recrutae.com)
31+
32+
Licensing
33+
---------
34+
35+
If you find Goose useful or have issues please drop me a line, I'd love
36+
to hear how you're using it or what features should be improved
2737

28-
##Licensing
29-
If you find Goose useful or have issues please drop me a line, I'd love to hear how you're using it or what features should be improved
38+
Goose is licensed by Gravity.com under the Apache 2.0 license, see the
39+
LICENSE file for more details
3040

31-
Goose is licensed by Gravity.com under the Apache 2.0 license, see the LICENSE file for more details
41+
Setup
42+
-----
43+
44+
::
3245

33-
##Setup
3446
mkvirtualenv --no-site-packages goose
3547
git clone https://github.com/grangier/python-goose.git
3648
cd python-goose
3749
pip install -r requirements.txt
3850
python setup.py install
39-
40-
41-
4251

43-
##Take it for a spin
52+
Take it for a spin
53+
------------------
54+
55+
::
56+
4457
>>> from goose import Goose
4558
>>> url = 'http://edition.cnn.com/2012/02/22/world/europe/uk-occupy-london/index.html?hpt=ieu_c2'
4659
>>> g = Goose()
@@ -54,23 +67,35 @@ Goose is licensed by Gravity.com under the Apache 2.0 license, see the LICENSE f
5467
>>> article.top_image.src
5568
http://i2.cdn.turner.com/cnn/dam/assets/111017024308-occupy-london-st-paul-s-cathedral-story-top.jpg
5669

70+
Configuration
71+
-------------
5772

58-
##Configuration
59-
There is two way to pass configuration to goose. The first one is to pass to goose a Configuration() object. The second one is to pass a configuration dict
73+
There is two way to pass configuration to goose. The first one is to
74+
pass to goose a Configuration() object. The second one is to pass a
75+
configuration dict
6076

61-
For instance, if you want to change the userAgent used by Goose juste pass :
77+
For instance, if you want to change the userAgent used by Goose juste
78+
pass :
79+
80+
::
6281

6382
>>> g = Goose({'browser_user_agent': 'Mozilla'})
6483

84+
Switching parser : Goose can now be use with lxml html parser or lxml
85+
soup parser. By default the html parser is used. If you want to use the
86+
soup parser passe it in the configuration dict :
6587

66-
Switching parser :
67-
Goose can now be use with lxml html parser or lxml soup parser. By default the html parser is used. If you want to use the soup parser passe it in the configuration dict :
88+
::
6889

6990
>>> g = Goose({'browser_user_agent': 'Mozilla', 'parser_class':'soup'})
7091

92+
Goose is now language aware
93+
---------------------------
94+
95+
For exemple scrapping a spanish content page with correct meta language
96+
tags
7197

72-
##Goose is now language aware
73-
For exemple scrapping a spanish content page with correct meta language tags
98+
::
7499

75100
>>> from goose import Goose
76101
>>> url = 'http://sociedad.elpais.com/sociedad/2012/10/27/actualidad/1351332873_157836.html'
@@ -81,7 +106,10 @@ For exemple scrapping a spanish content page with correct meta language tags
81106
>>> article.cleaned_text[:150]
82107
u'Los recortes pasan factura a los pacientes. De diciembre de 2010 a junio de 2012 las listas de espera para operarse aumentaron un 125%. Hay m\xe1s ciudad'
83108

84-
Some pages don't have correct meta language tags, you can force it using configuration :
109+
Some pages don't have correct meta language tags, you can force it using
110+
configuration :
111+
112+
::
85113

86114
>>> from goose import Goose
87115
>>> url = 'http://www.elmundo.es/elmundo/2012/10/28/espana/1351388909.html'
@@ -90,14 +118,18 @@ Some pages don't have correct meta language tags, you can force it using configu
90118
>>> article.cleaned_text[:150]
91119
u'Importante golpe a la banda terrorista ETA en Francia. La Guardia Civil ha detenido en un hotel de Macon, a 70 kil\xf3metros de Lyon, a Izaskun Lesaka y '
92120

93-
Passing
94-
{'use_meta_language': False, 'target_language':'es'}
95-
will force as configuration will force the spanish language
121+
Passing {'use\_meta\_language': False, 'target\_language':'es'} will
122+
force as configuration will force the spanish language
96123

124+
Goose in Chinese
125+
----------------
97126

127+
Some users want to use Goose for chinese content. Chinese word
128+
segementation is way more difficult to deal with that occidental
129+
languages. Chinese needs a dedicated StopWord analyser that need to be
130+
passed to the config object
98131

99-
##Goose in Chinese
100-
Some users want to use Goose for chinese content. Chinese word segementation is way more difficult to deal with that occidental languages. Chinese needs a dedicated StopWord analyser that need to be passed to the config object
132+
::
101133

102134
>>> from goose import Goose
103135
>>> from goose.text import StopWordsChinese
@@ -111,10 +143,13 @@ Some users want to use Goose for chinese content. Chinese word segementation is
111143

112144
一些亲北京阵营议员欢迎梁振英道歉，且认为应能获得香港民众接受，但这些议员也质问梁振英有
113145

146+
Goose in Arabic
147+
---------------
114148

149+
In order to use Goose in Arabic you have to use the StopWordsArabic
150+
class.
115151

116-
##Goose in Arabic
117-
In order to use Goose in Arabic you have to use the StopWordsArabic class.
152+
::
118153

119154
>>> from goose import Goose
120155
>>> from goose.text import StopWordsArabic
@@ -124,38 +159,52 @@ In order to use Goose in Arabic you have to use the StopWordsArabic class.
124159
>>> print article.cleaned_text[:150]
125160
دمشق، سوريا (CNN) -- أكدت جهات سورية معارضة أن فصائل مسلحة معارضة لنظام الرئيس بشار الأسد وعلى صلة بـ"الجيش الحر" تمكنت من السيطرة على مستودعات للأسل
126161

162+
TODO
163+
----
164+
165+
- Video extraction
127166

128-
##TODO
129-
- Video extraction
167+
Known issues
168+
------------
130169

131-
##Known issues
132-
- There is some issue with unicode URLs.
170+
- There is some issue with unicode URLs.
133171

134-
##OS X 10.7 Install Instructions
172+
OS X 10.7 Install Instructions
173+
------------------------------
135174

136175
Installation Help:
137176

138177
1. Install python-devel if you don't have it
139-
2. Install libjpeg
140-
brew install libjpeg
178+
2. Install libjpeg brew install libjpeg
179+
180+
3. You need to install the python imaging library. We wont be using it,
181+
but its a dependency deep in the goose egg (fun!).
182+
183+
a. download
184+
185+
::
141186

142-
3. You need to install the python imaging library. We wont be using it, but its a dependency deep in the goose egg (fun!).
187+
curl -O -L http://effbot.org/downloads/Imaging-1.1.7.tar.gz
143188

144-
a. download
189+
b. extract
145190

146-
curl -O -L http://effbot.org/downloads/Imaging-1.1.7.tar.gz
191+
::
147192

148-
b. extract
193+
tar -xzf Imaging-1.1.7.tar.gz
194+
cd Imaging-1.1.7
149195

150-
tar -xzf Imaging-1.1.7.tar.gz
151-
cd Imaging-1.1.7
196+
c. build and install
152197

153-
c. build and install
198+
::
154199

155-
python setup.py build
156-
sudo python setup.py install
200+
python setup.py build
201+
sudo python setup.py install
157202

158203
4. Next up clone this repo and install the egg.
159204

160-
5. Once you install the egg you have to then copy the resources directory manually into the egg. There is something screwy about the way its setup.
205+
5. Once you install the egg you have to then copy the resources
206+
directory manually into the egg. There is something screwy about the
207+
way its setup.
161208

209+
.. |Build Status| image:: https://www.travis-ci.org/xgdlm/python-goose.png?branch=master
210+
:target: https://www.travis-ci.org/xgdlm/python-goose

‎setup.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@
4848

4949
# read long description
5050
try:
51-
with open(os.path.join(os.path.dirname(__file__), 'README.md')) as f:
51+
with open(os.path.join(os.path.dirname(__file__), 'README.rst')) as f:
5252
long_description = f.read()
5353
except:
5454
long_description = description

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL