FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

爬虫模块bug修改 · JavaCodeMood/SpringBootUnity@a8aba0a · GitHub

/ SpringBootUnity Public
forked from houko/SpringBootUnity

Commit a8aba0a

Browse files
houko
committed
爬虫模块bug修改
1 parent af47c5d commit a8aba0a

9 files changed

Lines changed: 100 additions & 33 deletions

File tree

‎README.md‎

Lines changed: 12 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -45,38 +45,23 @@
4545

4646

4747
### 更新日志
48+
由于更新日志占用篇幅太多，单独用一个文件存放。链接地址 [更新日志(详细)](/changeLog.md),这里只展示更新简介
49+
4850
- 2017-09-02 api模块: 添加swagger-bootstrap-ui,和原有ui并行存在。
4951
http://localhost:8080 默认UI
50-
http://localhost:808/doc.html bootstrap-ui
51-
52-
```
53-
1. spring boot版本从1.4.3更新到1.5.6
54-
2. 修复不配置数据库信息无法启动的bug
55-
3. 版本号更新到2017.1
56-
4. api模块(swagger)添加开源库swagger-bootstrap-ui，和swagger默认UI同时存在。
57-
5. web模块添加数据库sql文件,导入后一键启动可直接访问到web界面。
58-
```
59-
60-
- 2017-09-06 更新记录
61-
62-
```
63-
1. mybatis模块:添加USER.sql,启动后访问:http://localhost:8080 即可看到接口数据
64-
2. 所有模块： 添加 characterEncoding=utf8&useSSL=true 解决高版本mysql的sll警告
65-
3. 添加代码贡献者列表和支持者，赞助商链接。
66-
```
67-
68-
![support](screenshot/support.png)
69-
70-
#### swagger
71-
![默认](screenshot/swagger-ui.png)
72-
52+
http://localhost:808/doc.html bootstrap-ui
7353

74-
#### bootstrap-ui
75-
![bootstrap](screenshot/bootstrap.png)
54+
- 2017-09-02 spring boot版本从1.4.3更新到1.5.6
55+
- 2017-09-02 修复不配置数据库信息无法启动的bug
56+
- 2017-09-02 版本号更新到2017.1
57+
- 2017-09-02 api模块(swagger)添加开源库swagger-bootstrap-ui，和swagger默认UI同时存在。
58+
- 2017-09-02 web模块添加数据库sql文件,导入后一键启动可直接访问到web界面。
59+
- 2017-09-06 mybatis模块:添加USER.sql,启动后访问:http://localhost:8080 即可看到接口数据
60+
- 2017-09-06 所有模块： 添加 characterEncoding=utf8&useSSL=true 解决高版本mysql的sll警告
61+
- 2017-09-06 添加代码贡献者列表和支持者，赞助商链接。
62+
- 2017-09-08 crawler模块(网络爬虫):修复本地文件目录不存在会报错的bug。处理方式为：不存在则自动创建
7663

77-
![bootstrap](screenshot/interface.png)
7864

79-
![bootstrap](screenshot/api.png)
8065

8166

8267
### 项目说明

‎changeLog.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
- 2017-09-02 api模块: 添加swagger-bootstrap-ui,和原有ui并行存在。
2+
http://localhost:8080 默认UI
3+
http://localhost:808/doc.html bootstrap-ui
4+
5+
```
6+
1. spring boot版本从1.4.3更新到1.5.6
7+
2. 修复不配置数据库信息无法启动的bug
8+
3. 版本号更新到2017.1
9+
4. api模块(swagger)添加开源库swagger-bootstrap-ui，和swagger默认UI同时存在。
10+
5. web模块添加数据库sql文件,导入后一键启动可直接访问到web界面。
11+
```
12+
13+
14+
15+
![support](screenshot/support.png)
16+
17+
#### swagger
18+
![默认](screenshot/swagger-ui.png)
19+
20+
21+
#### bootstrap-ui
22+
![bootstrap](screenshot/bootstrap.png)
23+
24+
![bootstrap](screenshot/interface.png)
25+
26+
![bootstrap](screenshot/api.png)
27+
28+
29+
30+
- 2017-09-06 更新记录
31+
32+
```
33+
1. mybatis模块:添加USER.sql,启动后访问:http://localhost:8080 即可看到接口数据
34+
2. 所有模块： 添加 characterEncoding=utf8&useSSL=true 解决高版本mysql的sll警告
35+
3. 添加代码贡献者列表和支持者，赞助商链接。
36+
```
37+
38+
- 2017-09-08 更新记录
39+
40+
```
41+
1. crawler模块(网络爬虫):修复本地文件目录不存在会报错的bug。处理方式为：不存在则自动创建
42+
```
43+
44+
![crawler](screenshot/crawler.png)
45+
![yys](screenshot/yys.png)

‎core/src/main/java/info/xiaomo/core/untils/DownUtil.java‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
package info.xiaomo.core.untils;
22

3+
import lombok.extern.slf4j.Slf4j;
4+
35
import java.io.File;
46
import java.io.FileOutputStream;
57
import java.io.InputStream;
@@ -9,8 +11,10 @@
911

1012
/**
1113
* author 小莫 (https://xiaomo.info) (https://github.com/xiaomoinfo)
14+
*
1215
* @created : 2016/12/26 13:25
1316
*/
17+
@Slf4j
1418
public class DownUtil {
1519

1620
public static void download(String urlString) throws Exception {
@@ -40,10 +44,11 @@ public static void download(String urlString) throws Exception {
4044

4145
/**
4246
* 下载图片
47+
*
4348
* @param urlString url
44-
* @param filePath 存储路径 D:\MIR\config\data\
49+
* @param filePath 存储路径 D:\MIR\config\data\
4550
*/
46-
public static void download(String urlString,String filePath) throws Exception {
51+
public static void download(String urlString, String filePath) throws Exception {
4752
File file = new File(urlString);
4853
String filename = file.getName();
4954
// 构造URL
@@ -57,7 +62,16 @@ public static void download(String urlString,String filePath) throws Exception {
5762
// 读取到的数据长度
5863
int len;
5964
// 输出的文件流
60-
OutputStream os = new FileOutputStream(filePath+filename);
65+
66+
File output = new File(filePath);
67+
if (!output.exists()){
68+
boolean res = output.mkdir();
69+
if (res){
70+
log.debug("{} 目录创建成功", filePath);
71+
}
72+
}
73+
74+
OutputStream os = new FileOutputStream(filePath + filename);
6175
// 开始读取
6276
while ((len = is.read(bs)) != -1) {
6377
os.write(bs, 0, len);

‎crawler/src/main/java/info/xiaomo/crawler/CrawlerMain.java‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,25 +5,27 @@
55
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
66
import org.springframework.context.annotation.ComponentScan;
77
import org.springframework.context.annotation.Configuration;
8+
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
89
import org.springframework.scheduling.annotation.EnableScheduling;
910

1011
/**
1112
* 把今天最好的表现当作明天最新的起点．．～
1213
* いま 最高の表現 として 明日最新の始発．．～
1314
* Today the best performance as tomorrow newest starter!
1415
* Created by IntelliJ IDEA.
15-
*
16+
* <p>
1617
* author: xiaomo
1718
* github: https://github.com/xiaomoinfo
1819
* email: xiaomo@xiaomo.info
19-
20+
* <p>
2021
* Date: 2016/4/1 15:38
2122
* Copyright(©) 2015 by xiaomo.
2223
**/
2324
@Configuration
2425
@EnableAutoConfiguration
2526
@EnableScheduling
2627
@Configurable
28+
@EnableJpaRepositories("info.xiaomo.*.dao")
2729
@ComponentScan("info.xiaomo")
2830
public class CrawlerMain {
2931
public static void main(String[] args) throws Exception {

‎crawler/src/main/java/info/xiaomo/crawler/dao/ShikigamaDao.java‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,13 @@
22

33
import info.xiaomo.crawler.model.ShikigamiModel;
44
import org.springframework.data.jpa.repository.JpaRepository;
5+
import org.springframework.stereotype.Repository;
56

67
/**
78
* author 小莫 (https://xiaomo.info) (https://github.com/xiaomoinfo)
89
* @created : 2016/12/24 15:55
910
*/
11+
@Repository
1012
public interface ShikigamaDao extends JpaRepository<ShikigamiModel, Long> {
1113

1214
ShikigamiModel findByName(String name);

‎crawler/src/main/java/info/xiaomo/crawler/schedule/ScheduledTasks.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ public void downImage() throws Exception {
5252
List<ShikigamiModel> shikigamiModel = shikigamaService.findAll();
5353
for (ShikigamiModel aShikigamiModel : shikigamiModel) {
5454
String url = aShikigamiModel.getImage();
55-
DownUtil.download(url, "C:\\Users\\xiaomo\\Desktop\\yss\\");
55+
DownUtil.download(url, "D:\\yys\\");
5656
LOGGER.debug("开始下载图片:{}", url);
5757
}
5858
}

‎crawler/src/main/resources/config/application.properties‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
logging.config=classpath:config/logback-dev.xml
2+
server.port=8080
3+
server.session.timeout=1800
4+
server.max-http-header-size=20971520
5+
6+
#datasource
7+
spring.datasource.url=jdbc:mysql://ip/db?characterEncoding=utf8&useSSL=true
8+
# ?useUnicode=true&amp;characterEncoding=UTF-8
9+
spring.datasource.username=username
10+
spring.datasource.password=password
11+
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
12+
# \u914D\u7F6E\u8FD9\u4E2A\u503C\u5C31\u53EF\u4EE5\u683C\u5F0F\u5316\u65F6\u95F4
13+
#spring.jackson.date-format=yyyy-MM-dd HH:mm:ss
14+
#spring.jackson.time-zone=GMT+8
15+
16+
#jpa
17+
spring.jpa.database-platform=org.hibernate.dialect.MySQL5Dialect
18+
spring.jpa.hibernate.ddl-auto=update
19+
spring.jpa.show-sql=true

‎screenshot/crawler.png‎

228 KB
LoadingViewer requires iframe.

‎screenshot/yys.png‎

493 KB
LoadingViewer requires iframe.

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL