GithubStar/WeiboSpider
1
0
Fork 0
mirror of https://github.com/nghuyong/WeiboSpider.git synced 2026-10-17 05:21:11 +08:00
Code Issues Packages Projects Releases Wiki Activity

refactor

Browse Source
This commit is contained in:
rightyonghu
2022-10-27 19:41:43 +08:00
parent 80ebe77467
commit 200ed3423a
35 changed files with 677 additions and 841 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/cookie.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.7 MiB

.github/data_stracture.md
-85
View File
@@ -1,85 +0,0 @@
## 微博数据字段
### 用户数据
| 字段 | 说明 |
| :---: | :----: |
|_id | 用户的ID，可以作为用户的唯一标识 |
|nick_name|昵称|
|gender|性别|
|province | 所在省|
|city |所在市|
|brief_introduction|个人简介|
|birthday |生日|
|tweets_num | 微博发表数|
|fans_num| 粉丝数|
|followers_num|关注数|
|sex_orientation|性取向|
|sentiment|感情状况|
|vip_level| 会员等级|
|authentication|认证情况|
|education|教育经历|
|work|工作经历|
|person_url|用户首页链接|
|labels|用户标签，用逗号分割|
|crawl_time|抓取时间戳|
示例:
![](./images/information.png)
### 微博数据
| 字段 | 说明 |
| :---: | :----: |
| _id | 微博id |
|user_id| 这则微博作者的ID |
|content |微博的内容|
|created_at |微博发表时间|
|repost_num |转发数|
|comment_num |评论数|
|like_num| 点赞数|
|tool|发布微博的工具|
|image_url|微博中图片的URL，注意如果是组图会只抓取第一张|
|video_url|微博中视频的URL|
|location_map_info|定位的经纬度信息|
|origin_weibo|原始微博，只有转发的微博才有这个字段|
|crawl_time|抓取时间戳|
示例:
![](./images/tweet.png)
### 用户关系数据
| 字段 | 说明 |
| :---: | :----: |
| _id | 用户关系id |
|fan_id| 关注者的用户ID |
|follower_id|被关注者的用户ID|
|crawl_time|抓取时间戳|
示例:
![](./images/relationship.png)
### 评论数据
| 字段 | 说明 |
| :---: | :----: |
| _id | 评论的id |
|comment_user_id|评论的用户ID|
|weibo_id|weibo的ID|
|content|评论内容|
|created_at| 评论创建时间|
|crawl_time|抓取时间戳|
示例:
![](./images/comment.png)
### 转发数据
| 字段 | 说明 |
| :--------: | :-------------: |
| _id | null |
| crawl_time | 抓取时间戳 |
| weibo_id | 被转发weibo的ID |
| user_id | 转发用户的ID |
| content | 转发的评论内容 |
| created_at | 转发时间 |
![](./images/repost.png)
.github/images/comment-spider.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 262 KiB

.github/images/comment.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 874 KiB

.github/images/cookie_from_chrome.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 MiB

.github/images/fan-spider.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 297 KiB

.github/images/follow-spider.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 300 KiB

.github/images/information.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 222 KiB

.github/images/relationship.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 248 KiB

.github/images/repost-spider.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 257 KiB

.github/images/repost.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 34 KiB

.github/images/tweet-user-date.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 265 KiB

.github/images/tweet-user-spider.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 328 KiB

.github/images/tweet.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1022 KiB

.github/images/user-spider.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 283 KiB

.github/weibospider.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 586 KiB

README.md
+281 -60
View File
@@ -1,7 +1,18 @@
**中文说明** | [English](./README_EN.md)
# WeiboSpider
<a href="https://github.com/nghuyong/WeiboSpider/stargazers">
<p align="center">
<br>
<img src="./.github/weibospider.png" width="400"/>
<br>
<p>
<p align="center">
<a href="https://www.codacy.com/gh/nghuyong/WeiboSpider/dashboard?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=nghuyong/WeiboSpider&amp;utm_campaign=Badge_Grade">
<img src="https://app.codacy.com/project/badge/Grade/cf88a8b1e6e44c5d993d2cbea7d44c85"
alt="Codacy Badge">
</a>
<a href="https://scan.coverity.com/projects/nghuyong-weibospider">
<img alt="Coverity Scan Build Status"
src="https://scan.coverity.com/projects/26928/badge.svg"/>
</a>
<a href="https://github.com/nghuyong/WeiboSpider/stargazers">
<img src="https://img.shields.io/github/stars/nghuyong/WeiboSpider.svg?colorA=orange&colorB=orange&logo=github"
alt="GitHub stars">
</a>
@@ -9,112 +20,322 @@
<img src="https://img.shields.io/github/issues/nghuyong/WeiboSpider.svg"
alt="GitHub issues">
</a>
<a href="https://github.com/nghuyong/WeiboSpider/forks">
<img src="https://img.shields.io/github/forks/nghuyong/WeiboSpider.svg"
alt="GitHub forks">
</a>
<a href="https://github.com/nghuyong/WeiboSpider/">
<img src="https://img.shields.io/github/last-commit/nghuyong/WeiboSpider.svg">
</a>
<a href="https://github.com/nghuyong/WeiboSpider/blob/master/LICENSE">
<img src="https://img.shields.io/github/license/nghuyong/WeiboSpider.svg"
alt="GitHub license">
</a>
</a>
</p>
持续维护的新浪微博爬虫🚀🚀🚀
<h3 align="center">
<p>持续维护的新浪微博采集工具🚀🚀🚀</p>
</h3>
## 项目说明
## 项目特色
### 支持爬虫
- 用户信息抓取
- 用户微博抓取(全量/指定时间段)
- 用户社交关系抓取(粉丝/关注)
- 微博评论抓取
- 基于关键词和时间段(粒度到小时)的微博抓取
- 微博转发抓取
- 基于weibo.com的新版API构建，拥有最丰富的字段信息
- 多种采集模式，包含微博用户,推文,粉丝,关注,转发,评论,关键词搜索
- 核心代码仅100行，代码可读性高，可快速进行定制化改造
### 字段说明
项目基于[weibo.cn](https://weibo.cn)站点抓取，抓取的字段非常丰富。具体请移步:[数据字段说明](./.github/data_stracture.md)
## 快速开始
## 如何使用
### 拉取&&安装
### 拉取项目 && 安装依赖
本项目Python版本为Python3.6
```bash
git clone git@github.com:nghuyong/WeiboSpider.git --depth 1 --no-single-branch
cd WeiboSpider
pip install -r requirements.txt
```
除此之外，还需要安装mongodb.
### 替换Cookie
访问https://weibo.cn/
登陆账号，打开浏览器的开发者模式，再次刷新
访问[https://weibo.com/](https://weibo.com/)， 登陆账号，打开浏览器的开发者模式，再次刷新
![](./.github/images/cookie_from_chrome.png)
![](.github/cookie.png)
复制weibo.cn这个数据包，network中的cookie值
复制`weibo.com`数据包，network中的cookie值。编辑`weibospider/cookie.txt`并替换成刚刚复制的Cookie
将`weibospider/settings.py`中:
```python
DEFAULT_REQUEST_HEADERS = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0',
'Cookie':'SCF=AlvwCT3ltiVc36wsKpuvTV8uWF4V1tZ17ms9t-bZCAuiVJKpCsgvvmSdylNE6_4GbqwA_MWvxNgoc0Ks-qbZStc.; OUTFOX_SEARCH_USER_ID_NCOO=1258151803.428431; SUB=_2A25zjTjHDeRhGeBN6VUX9SvEzT-IHXVQjliPrDV6PUJbkdANLUvskW1NRJ24IEPNKfRaplNknl957NryzKEwBmhJ; SUHB=0ftpSdul-YZaMk; _T_WM=76982927613'
}
```
Cookie字段替换成你自己的Cookie
### 添加代理IP(可选)
**如果爬虫运行出现403/302，说明账号被封/cookie失效，请重新替换cookie**
## 添加代理IP(可选)
重写[fetch_proxy](./weibospider/middlewares.py#6L)方法，该方法需要返回一个代理ip，具体参考[这里](https://github.com/nghuyong/WeiboSpider/issues/124#issuecomment-654335439)
重写[fetch_proxy](./weibospider/middlewares.py#6L)
方法，该方法需要返回一个代理ip，具体参考[这里](https://github.com/nghuyong/WeiboSpider/issues/124#issuecomment-654335439)
## 运行程序
**可根据自己实际需要重写`./weibospider/spiders/*`中的`start_requests`函数**
根据自己实际需要重写`./weibospider/spiders/*`中的`start_requests`函数
### 抓取用户信息
采集的数据存在`output`文件中，命名为`{spider.name}_{datetime}.jsonl`
```
### 用户信息采集
```bash
cd weibospider
python run_spider.py user
```
![](./.github/images/user-spider.png)
### 抓取用户粉丝列表
```json
{
"crawl_time": 1666863485,
"_id": "1749127163",
"avatar_hd": "https://tvax4.sinaimg.cn/crop.0.0.1080.1080.1024/001Un9Srly8h3fpj11yjyj60u00u0q7f02.jpg?KID=imgbed,tva&Expires=1666874283&ssig=a%2FMfgFzvRo",
"nick_name": "雷军",
"verified": true,
"description": "小米董事长，金山软件董事长。业余爱好是天使投资。",
"followers_count": 22756103,
"friends_count": 1373,
"statuses_count": 14923,
"gender": "m",
"location": "北京 海淀区",
"mbrank": 7,
"mbtype": 12,
"verified_type": 0,
"verified_reason": "小米创办人，董事长兼CEO；金山软件董事长；天使投资人。",
"birthday": "",
"created_at": "2010-05-31 23:07:59",
"desc_text": "小米创办人，董事长兼CEO；金山软件董事长；天使投资人。",
"ip_location": "IP属地：北京",
"sunshine_credit": "信用极好",
"label_desc": [
"V指数 财经 75.30分",
"热门财经博主 数据飙升",
"昨日发博3，阅读数100万+，互动数1.9万",
"视频累计播放量9819.3万",
"群友 3132"
],
"company": "金山软件",
"education": {
"school": "武汉大学"
}
}
```
### 用户粉丝列表采集
```bash
python run_spider.py fan
```
![](./.github/images/fan-spider.png)
```json
{
"crawl_time": 1666863563,
"_id": "1087770692_5968044974",
"follower_id": "1087770692",
"fan_info": {
"_id": "5968044974",
"avatar_hd": "https://tvax1.sinaimg.cn/default/images/default_avatar_male_180.gif?KID=imgbed,tva&Expires=1666874363&ssig=UuzaeK437R",
"nick_name": "用户5968044974",
"verified": false,
"description": "",
"followers_count": 0,
"friends_count": 195,
"statuses_count": 9,
"gender": "m",
"location": "其他",
"mbrank": 0,
"mbtype": 0,
"credit_score": 80,
"created_at": "2016-06-25 22:30:13"
}
}
...
```
### 用户关注列表采集
### 抓取用户关注列表
```bash
python run_spider.py follow
```
![](./.github/images/follow-spider.png)
### 抓取微博评论
```json
{
"crawl_time": 1666863679,
"_id": "1087770692_7083568088",
"fan_id": "1087770692",
"follower_info": {
"_id": "7083568088",
"avatar_hd": "https://tvax4.sinaimg.cn/crop.0.0.1080.1080.1024/007JnVEcly8gyqd9jadjlj30u00u0gpn.jpg?KID=imgbed,tva&Expires=1666874479&ssig=9zhfeMPLzr",
"nick_name": "蒋昀霖",
"verified": true,
"description": "工作请联系：lijialun@kpictures.cn",
"followers_count": 329216,
"friends_count": 58,
"statuses_count": 342,
"gender": "m",
"location": "北京",
"mbrank": 6,
"mbtype": 12,
"credit_score": 80,
"created_at": "2019-04-17 16:25:43",
"verified_type": 0,
"verified_reason": "东申未来 演员"
}
}
...
```
### 用户的微博采集
```bash
python run_spider.py tweet
```
```json
{
"crawl_time": 1666864583,
"_id": "4762810834227120",
"mblogid": "LqlZNhJFm",
"created_at": "2022-04-27 10:20:54",
"geo": null,
"ip_location": null,
"reposts_count": 1907,
"comments_count": 1924,
"attitudes_count": 12169,
"source": "三星Galaxy S22 Ultra",
"content": "生于乱世纵横四海，义之所在不计生死，孤勇者陈恭一生当如是。#风起陇西今日开播# #风起陇西# 今晚，恭候你！",
"pic_urls": [],
"pic_num": 0,
"video": "http://f.video.weibocdn.com/o0/CmQEWK1ylx07VAm0nrxe01041200YDIc0E010.mp4?label=mp4_720p&template=1280x720.25.0&ori=0&ps=1CwnkDw1GXwCQx&Expires=1666868183&ssig=RlIeOt286i&KID=unistore,video",
"url": "https://weibo.com/1087770692/LqlZNhJFm"
}
...
```
### 微博评论采集
```bash
python run_spider.py comment
```
![](./.github/images/comment-spider.png)
### 抓取用户的微博(全量)
在`./weibospider/spiders/tweet.py`中`start_requests`, urls选择`init_url_by_user_id()`
```bash
python run_spider.py tweet
```json
{
"crawl_time": 1666863805,
"_id": 4826279188108038,
"created_at": "2022-10-19 13:41:29",
"like_counts": 1,
"ip_location": "来自河南",
"content": "五周年快乐呀，请坤哥哥继续保持这份热爱，奔赴下一场山海",
"comment_user": {
"_id": "2380967841",
"avatar_hd": "https://tvax4.sinaimg.cn/crop.0.0.888.888.1024/002B8iv7ly8gv647ipgxvj60oo0oojtk02.jpg?KID=imgbed,tva&Expires=1666874604&ssig=%2FdGaaIRkhf",
"nick_name": "流年执念的二瓜娇",
"verified": false,
"description": "蓝桉已遇释怀鸟，不爱万物唯爱你。",
"followers_count": 238,
"friends_count": 1655,
"statuses_count": 12546,
"gender": "f",
"location": "河南",
"mbrank": 6,
"mbtype": 11
}
}
...
```
![](./.github/images/tweet-user-spider.png)
### 抓取用户的微博(指定时间段)
在`./weibospider/spiders/tweet.py`中`start_requests`, urls选择`init_url_by_user_id_and_date()`
```bash
python run_spider.py tweet
```
![](./.github/images/tweet-user-date.png)
### 抓取微博转发
### 微博转发采集
```bash
python run_spider.py repost
```
![](./.github/images/repost-spider.png)
```json
{
"_id": "4826312651310475",
"mblogid": "Mb2vL5uUH",
"created_at": "2022-10-19 15:54:27",
"geo": null,
"ip_location": "发布于 德国",
"reposts_count": 0,
"comments_count": 0,
"attitudes_count": 0,
"source": "iPhone客户端",
"content": "共享[鼓掌][太开心][鼓掌]五周年快乐！//@陈坤:#山下学堂五周年# 五年， 感谢同行。",
"pic_urls": [],
"pic_num": 0,
"user": {
"_id": "2717869081",
"avatar_hd": "https://tvax1.sinaimg.cn/crop.0.0.160.160.1024/a1ff6419ly8gz1xoq9oolj204g04g745.jpg?KID=imgbed,tva&Expires=1666876939&ssig=Cl93CLjdB%2F",
"nick_name": "YuFeeC",
"verified": false,
"mbrank": 0,
"mbtype": 0
},
"url": "https://weibo.com/2717869081/Mb2vL5uUH",
"crawl_time": 1666866139
}
...
```
### 基于关键词的微博检索
```bash
python run_spider.py search
```
```json
{
"crawl_time": 1666869049,
"keyword": "丽江",
"_id": "4829255386537989",
"mblogid": "Mch46rqPr",
"created_at": "2022-10-27 18:47:50",
"geo": {
"type": "Point",
"coordinates": [
26.962427,
100.248299
],
"detail": {
"poiid": "B2094251D06FAAF44299",
"title": "山野文创旅拍圣地",
"type": "checkin",
"spot_type": "0"
}
},
"ip_location": "发布于 云南",
"reposts_count": 0,
"comments_count": 0,
"attitudes_count": 1,
"source": "iPhone1314iPhone客户端",
"content": "丽江小漾日出\n推出户外移动餐桌\n接受私人定制\n让美食融入美景心情自然美丽了！\n#小众宝藏旅行地##超出片的艺术街区# ",
"pic_urls": [
"https://wx1.sinaimg.cn/orj960/4b138405gy1h7k1a56c4oj234022onph",
"https://wx1.sinaimg.cn/orj960/4b138405gy1h7k19eb2kxj22ts1vvb2a",
"https://wx1.sinaimg.cn/orj960/4b138405gy1h7k1a0wzglj22ua1w7hdw",
"https://wx1.sinaimg.cn/orj960/4b138405gy1h7k19wsafnj231x21a7wj",
"https://wx1.sinaimg.cn/orj960/4b138405gy1h7k19jd1xkj22oh1sbkjo",
"https://wx1.sinaimg.cn/orj960/4b138405gy1h7k19mma74j22ru1ukx6q",
"https://wx1.sinaimg.cn/orj960/4b138405gy1h7k19tf1bfj234022oe85",
"https://wx1.sinaimg.cn/orj960/4b138405gy1h7k19pk37pj234022okjm",
"https://wx1.sinaimg.cn/orj960/4b138405gy1h7k19g6nzfj20wi0lo7my"
],
"pic_num": 9,
"user": {
"_id": "1259570181",
"avatar_hd": "https://tvax1.sinaimg.cn/crop.0.0.1080.1080.1024/4b138405ly8gzfkfikyqvj20u00u0ag1.jpg?KID=imgbed,tva&Expires=1666879848&ssig=6PUDG5RonQ",
"nick_name": "飞鸟与鱼",
"verified": true,
"mbrank": 7,
"mbtype": 12,
"verified_type": 0
},
"url": "https://weibo.com/1259570181/Mch46rqPr"
}
...
```
## 更新日志
- 2022.10: 添加IP归属地信息的采集，包括用户数据，微博数据和微博评论数据
- 2022.10: 基于weibo.com站点对项目进行重构
## 合作联系
- 已构建超大规模数据集WeiboCOV，可免费申请，包含2千万微博活跃用户以及6千万推文数据，具体参见[这里](https://github.com/nghuyong/weibo-public-opinion-datasets)
- 社交媒体相关的数据采集/分析，欢迎邮件交流 nghuyong@163.com 👏
README_EN.md
-120
View File
@@ -1,120 +0,0 @@
[中文说明](./README.md) | **English**
# WeiboSpider
<a href="https://github.com/nghuyong/WeiboSpider/stargazers">
<img src="https://img.shields.io/github/stars/nghuyong/WeiboSpider.svg?colorA=orange&colorB=orange&logo=github"
alt="GitHub stars">
</a>
<a href="https://github.com/nghuyong/WeiboSpider/issues">
<img src="https://img.shields.io/github/issues/nghuyong/WeiboSpider.svg"
alt="GitHub issues">
</a>
<a href="https://github.com/nghuyong/WeiboSpider/">
<img src="https://img.shields.io/github/last-commit/nghuyong/WeiboSpider.svg">
</a>
<a href="https://github.com/nghuyong/WeiboSpider/blob/master/LICENSE">
<img src="https://img.shields.io/github/license/nghuyong/WeiboSpider.svg"
alt="GitHub license">
</a>
Continuously maintained Sina Weibo crawler 🚀🚀🚀
## Introduction
### Supported crawling types
- User Information
- Tweets post by user(all / specific period)
- Users' social relationships (fans/followers)
- Comments of tweets
- Tweets based on keywords and time period
- Retweets following a tweet
### Data Structure
The spider based on [weibo.cn](https://weibo.cn), and the crawled fields are very rich. More detail:[Data Structure Description](./.github/data_stracture.md)
## Get Started
### Pull the project && Install dependencies
Note that the Python Version is Python3.6
```bash
git clone git@github.com:nghuyong/WeiboSpider.git --depth 1 --no-single-branch
cd WeiboSpider
pip install -r requirements.txt
```
In addition, you need to install mongodb.
### Replace Cookies
Vist https://weibo.cn/
Log in, open the developer mode of the browser, and refresh again
![](./.github/images/cookie_from_chrome.png)
Copy the cookie value in the network in the weibo.cn data packet.
Edit `weibospider/settings.py`中:
```python
DEFAULT_REQUEST_HEADERS = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0',
'Cookie':'SCF=AlvwCT3ltiVc36wsKpuvTV8uWF4V1tZ17ms9t-bZCAuiVJKpCsgvvmSdylNE6_4GbqwA_MWvxNgoc0Ks-qbZStc.; OUTFOX_SEARCH_USER_ID_NCOO=1258151803.428431; SUB=_2A25zjTjHDeRhGeBN6VUX9SvEzT-IHXVQjliPrDV6PUJbkdANLUvskW1NRJ24IEPNKfRaplNknl957NryzKEwBmhJ; SUHB=0ftpSdul-YZaMk; _T_WM=76982927613'
}
```
Replace the cookie field with your own cookie
**If 403/302 appears on the crawler, it means that the account is blocked or the cookie is invalid**
## Add proxy IP (optional)
Rewrite the function [fetch_proxy](./weibospider/middlewares.py#6L), [more details](https://github.com/nghuyong/WeiboSpider/issues/124#issuecomment-654335439)
## Run the program
You can rewrite functions of `start_requests` in `./weibospider/spiders/*`
### Crawl User Info
```
cd weibospider
python run_spider.py user
```
![](./.github/images/user-spider.png)
### Crawl Fans List
```bash
python run_spider.py fan
```
![](./.github/images/fan-spider.png)
### Crawl Followers List
```bash
python run_spider.py follow
```
![](./.github/images/follow-spider.png)
### Crawl Comments of tweets
```bash
python run_spider.py comment
```
![](./.github/images/comment-spider.png)
### Crawl Tweets of Users(ALL)
`urls` select `init_url_by_user_id()` in the function of `start_requests` in `./weibospider/spiders/tweet.py`
```bash
python run_spider.py tweet
```
![](./.github/images/tweet-user-spider.png)
### Crawl Tweets of Users(Specific period)
`urls` select `init_url_by_user_id_and_date()` in the function of `start_requests` in `./weibospider/spiders/tweet.py`
```bash
python run_spider.py tweet
```
![](./.github/images/tweet-user-date.png)
### Crawl Retweet/Repost
```
python run_spider.py repost
```
![](./.github/images/repost-spider.png)
output/.gitkeep
View File
requirements.txt
+1 -2
View File
@@ -1,3 +1,2 @@
pymongo
Scrapy
lxml
python_dateutil
weibospider/cookie.txt
+1
View File
@@ -0,0 +1 @@
OUTFOX_SEARCH_USER_ID_NCOO=1892334049.0611136; _T_WM=5d0d245ef51ec62db21a50916b1e384b; SCF=AkceVOKbLcdJ3i5GuWKRwRkI-sTF8ozuM9kLPqvaNmzIbMz84C_D9mrcfkUDZ5USvQLxxcNSHMeGPwtuCmRzkQA.; SUB=_2A25OUHI9DeRhGeBN6VUX9SvEzT-IHXVtux51rDV6PUJbktAKLVCmkW1NRJ24IFM3w0MsIksejOBZJEcAgJZZobMv; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WWYACoMUZFHDoS6U9MYf.vu5NHD95Qce0zNSo-f1hq0Ws4DqcjzdJUQUPLadJMt; SSOLoginState=1666450029
weibospider/items.py
-76
View File
@@ -1,76 +0,0 @@
# -*- coding: utf-8 -*-
from scrapy import Item, Field
class TweetItem(Item):
"""Tweet information """
_id = Field() # 微博id
weibo_url = Field() # 微博URL
created_at = Field() # 微博发表时间
like_num = Field() # 点赞数
repost_num = Field() # 转发数
comment_num = Field() # 评论数
content = Field() # 微博内容
user_id = Field() # 发表该微博用户的id
tool = Field() # 发布微博的工具
image_url = Field() # 图片
video_url = Field() # 视频
origin_weibo = Field() # 原始微博，只有转发的微博才有这个字段
location_map_info = Field() # 定位的经纬度信息
crawl_time = Field() # 抓取时间戳
class UserItem(Item):
""" User Information"""
_id = Field() # 用户ID
nick_name = Field() # 昵称
gender = Field() # 性别
province = Field() # 所在省
city = Field() # 所在城市
brief_introduction = Field() # 简介
birthday = Field() # 生日
tweets_num = Field() # 微博数
follows_num = Field() # 关注数
fans_num = Field() # 粉丝数
sex_orientation = Field() # 性取向
sentiment = Field() # 感情状况
vip_level = Field() # 会员等级
authentication = Field() # 认证
education = Field() # 学习经历
work = Field() # 工作经历
person_url = Field() # 首页链接
labels = Field() # 标签
crawl_time = Field() # 抓取时间戳
class RelationshipItem(Item):
""" 用户关系，只保留与关注的关系 """
_id = Field()
fan_id = Field() # 关注者,即粉丝的id
followed_id = Field() # 被关注者的id
crawl_time = Field() # 抓取时间戳
class CommentItem(Item):
"""
微博评论信息
"""
_id = Field()
comment_user_id = Field() # 评论用户的id
content = Field() # 评论的内容
weibo_id = Field() # 评论的微博的id
created_at = Field() # 评论发表时间
like_num = Field() # 点赞数
crawl_time = Field() # 抓取时间戳
class RepostItem(Item):
"""
微博转发信息
"""
_id = Field()
user_id = Field() # 转发用户的id
content = Field() # 转发的内容
weibo_id = Field() # 转发的微博的id
created_at = Field() # 转发时间
crawl_time = Field() # 抓取时间戳
weibospider/middlewares.py
+12 -2
View File
@@ -2,13 +2,23 @@
class IPProxyMiddleware(object):
"""
代理IP中间件
"""
def fetch_proxy(self):
@staticmethod
def fetch_proxy():
"""
获取一个代理IP
"""
# You need to rewrite this function if you want to add proxy pool
# the function should return a ip in the format of "ip:port" like "12.34.1.4:9090"
# the function should return an ip in the format of "ip:port" like "12.34.1.4:9090"
return None
def process_request(self, request, spider):
"""
将代理IP添加到request请求中
"""
proxy_data = self.fetch_proxy()
if proxy_data:
current_proxy = f'http://{proxy_data}'
weibospider/pipelines.py
+20 -30
View File
@@ -1,37 +1,27 @@
# -*- coding: utf-8 -*-
import pymongo
from pymongo.errors import DuplicateKeyError
from settings import MONGO_HOST, MONGO_PORT
import datetime
import json
import time
class MongoDBPipeline(object):
class JsonWriterPipeline(object):
"""
写入json文件的pipline
"""
def __init__(self):
client = pymongo.MongoClient(MONGO_HOST, MONGO_PORT)
db = client['weibo']
self.Users = db["Users"]
self.Tweets = db["Tweets"]
self.Comments = db["Comments"]
self.Relationships = db["Relationships"]
self.Reposts = db["Reposts"]
self.file = None
def process_item(self, item, spider):
if spider.name == 'comment_spider':
self.insert_item(self.Comments, item)
elif spider.name == 'fan_spider':
self.insert_item(self.Relationships, item)
elif spider.name == 'follower_spider':
self.insert_item(self.Relationships, item)
elif spider.name == 'user_spider':
self.insert_item(self.Users, item)
elif spider.name == 'tweet_spider':
self.insert_item(self.Tweets, item)
elif spider.name == 'repost_spider':
self.insert_item(self.Reposts, item)
"""
处理item
"""
if not self.file:
now = datetime.datetime.now()
file_name = spider.name + "_" + now.strftime("%Y%m%d%H%M%S") + '.jsonl'
self.file = open(f'../output/{file_name}', 'wt', encoding='utf-8')
item['crawl_time'] = int(time.time())
line = json.dumps(dict(item), ensure_ascii=False) + "\n"
print(json.dumps(dict(item), ensure_ascii=False, indent=4))
self.file.write(line)
return item
@staticmethod
def insert_item(collection, item):
try:
collection.insert_one(dict(item))
except DuplicateKeyError:
pass
weibospider/run_spider.py
+3 -2
View File
@@ -1,7 +1,6 @@
#!/usr/bin/env python
# encoding: utf-8
"""
File Description:
Author: nghuyong
Mail: nghuyong@163.com
Created Time: 2019-12-07 21:27
@@ -16,10 +15,11 @@ from spiders.follower import FollowerSpider
from spiders.user import UserSpider
from spiders.fan import FanSpider
from spiders.repost import RepostSpider
from spiders.search import SearchSpider
if __name__ == '__main__':
mode = sys.argv[1]
os.environ['SCRAPY_SETTINGS_MODULE'] = f'settings'
os.environ['SCRAPY_SETTINGS_MODULE'] = 'settings'
settings = get_project_settings()
process = CrawlerProcess(settings)
mode_to_spider = {
@@ -29,6 +29,7 @@ if __name__ == '__main__':
'tweet': TweetSpider,
'user': UserSpider,
'repost': RepostSpider,
'search': SearchSpider
}
process.crawl(mode_to_spider[mode])
# the script will block here until the crawling is finished
weibospider/settings.py
+6 -7
View File
@@ -7,14 +7,16 @@ NEWSPIDER_MODULE = 'spiders'
ROBOTSTXT_OBEY = False
# change cookie to yours
with open('cookie.txt', 'rt', encoding='utf-8') as f:
cookie = f.read().strip()
DEFAULT_REQUEST_HEADERS = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0',
'Cookie': 'OUTFOX_SEARCH_USER_ID_NCOO=1521427420.3020706; SCF=AlvwCT3ltiVc36wsKpuvTV8uWF4V1tZ17ms9t-bZCAuiV-ZsmkYB9TEWXW7sd-gDxW34UesUalWH5lMgGsFIVAY.; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WWYACoMUZFHDoS6U9MYf.vu5NHD95Qce0zNSo-f1hq0Ws4DqcjzdJUQUPLadJMt; SUB=_2A25NMZbHDeRhGeBN6VUX9SvEzT-IHXVu3TqPrDV6PUJbkdAKLXnikW1NRJ24IENfNQhgKlnrWsrJzXjq5AY_x0Mb; _T_WM=391c3b6aaf54f1ca5a8688cdd9cfec2c'}
'Cookie': cookie
}
CONCURRENT_REQUESTS = 16
DOWNLOAD_DELAY = 3
DOWNLOAD_DELAY = 1
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': None,
@@ -24,8 +26,5 @@ DOWNLOADER_MIDDLEWARES = {
}
ITEM_PIPELINES = {
'pipelines.MongoDBPipeline': 300,
'pipelines.JsonWriterPipeline': 300,
}
MONGO_HOST = '127.0.0.1'
MONGO_PORT = 27017
weibospider/spiders/comment.py
+41 -39
View File
@@ -1,54 +1,56 @@
#!/usr/bin/env python
# encoding: utf-8
"""
File Description:
Author: nghuyong
Mail: nghuyong@163.com
Created Time: 2020/4/14
"""
import re
from lxml import etree
import json
from scrapy import Spider
from scrapy.http import Request
import time
from items import CommentItem
from spiders.utils import extract_comment_content, time_fix
from spiders.common import parse_user_info, parse_time, url_to_mid
class CommentSpider(Spider):
name = "comment_spider"
base_url = "https://weibo.cn"
"""
微博评论数据采集
"""
name = "comment"
def start_requests(self):
tweet_ids = ['IDl56i8av', 'IDkNerVCG', 'IDkJ83QaY']
urls = [f"{self.base_url}/comment/hot/{tweet_id}?rl=1&page=1" for tweet_id in tweet_ids]
for url in urls:
yield Request(url, callback=self.parse)
"""
爬虫入口
"""
# 这里tweet_ids可替换成实际待采集的数据
tweet_ids = ['Mb15BDYR0']
for tweet_id in tweet_ids:
mid = url_to_mid(tweet_id)
url = f"https://weibo.com/ajax/statuses/buildComments?" \
f"is_reload=1&id={mid}&is_show_bulletin=2&is_mix=0&count=20"
yield Request(url, callback=self.parse, meta={'source_url': url})
def parse(self, response):
if response.url.endswith('page=1'):
all_page = re.search(r'/>&nbsp;1/(\d+)页</div>', response.text)
if all_page:
all_page = all_page.group(1)
all_page = int(all_page)
all_page = all_page if all_page <= 50 else 50
for page_num in range(2, all_page + 1):
page_url = response.url.replace('page=1', 'page={}'.format(page_num))
yield Request(page_url, self.parse, dont_filter=True, meta=response.meta)
tree_node = etree.HTML(response.body)
comment_nodes = tree_node.xpath('//div[@class="c" and contains(@id,"C_")]')
for comment_node in comment_nodes:
comment_user_url = comment_node.xpath('.//a[contains(@href,"/u/")]/@href')
if not comment_user_url:
continue
comment_item = CommentItem()
comment_item['crawl_time'] = int(time.time())
comment_item['weibo_id'] = response.url.split('/')[-1].split('?')[0]
comment_item['comment_user_id'] = re.search(r'/u/(\d+)', comment_user_url[0]).group(1)
comment_item['content'] = extract_comment_content(etree.tostring(comment_node, encoding='unicode'))
comment_item['_id'] = comment_node.xpath('./@id')[0]
created_at_info = comment_node.xpath('.//span[@class="ct"]/text()')[0]
like_num = comment_node.xpath('.//a[contains(text(),"赞[")]/text()')[-1]
comment_item['like_num'] = int(re.search('\d+', like_num).group())
comment_item['created_at'] = time_fix(created_at_info.split('\xa0')[0])
yield comment_item
def parse(self, response, **kwargs):
"""
网页解析
"""
data = json.loads(response.text)
for comment_info in data['data']:
item = self.parse_comment(comment_info)
yield item
if data.get('max_id', 0) != 0:
url = response.meta['source_url'] + '&max_id=' + str(data['max_id'])
yield Request(url, callback=self.parse, meta=response.meta)
@staticmethod
def parse_comment(data):
"""
解析comment
"""
item = dict()
item['created_at'] = parse_time(data['created_at'])
item['_id'] = data['id']
item['like_counts'] = data['like_counts']
item['ip_location'] = data['source']
item['content'] = data['text_raw']
item['comment_user'] = parse_user_info(data['user'])
return item
weibospider/spiders/common.py
+105
View File
@@ -0,0 +1,105 @@
#!/usr/bin/env python
# encoding: utf-8
"""
Author: rightyonghu
Created Time: 2022/10/24
"""
import dateutil.parser
def base62_decode(string):
"""
base
"""
alphabet = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
string = str(string)
num = 0
idx = 0
for char in string:
power = (len(string) - (idx + 1))
num += alphabet.index(char) * (len(alphabet) ** power)
idx += 1
return num
def reverse_cut_to_length(content, code_func, cut_num=4, fill_num=7):
"""
url to mid
"""
content = str(content)
cut_list = [content[i - cut_num if i >= cut_num else 0:i] for i in range(len(content), 0, (-1 * cut_num))]
cut_list.reverse()
result = []
for i, item in enumerate(cut_list):
s = str(code_func(item))
if i > 0 and len(s) < fill_num:
s = (fill_num - len(s)) * '0' + s
result.append(s)
return ''.join(result)
def url_to_mid(url: str):
""">>> url_to_mid('z0JH2lOMb')
3501756485200075
"""
result = reverse_cut_to_length(url, base62_decode)
return int(result)
def parse_time(s):
"""
Wed Oct 19 23:44:36 +0800 2022 => 2022-10-19 23:44:36
"""
return dateutil.parser.parse(s).strftime('%Y-%m-%d %H:%M:%S')
def parse_user_info(data):
"""
解析用户信息
"""
# 基础信息
user = {
"_id": str(data['id']),
"avatar_hd": data['avatar_hd'],
"nick_name": data['screen_name'],
"verified": data['verified'],
}
# 额外的信息
keys = ['description', 'followers_count', 'friends_count', 'statuses_count',
'gender', 'location', 'mbrank', 'mbtype', 'credit_score']
for key in keys:
if key in data:
user[key] = data[key]
if 'created_at' in data:
user['created_at'] = parse_time(data.get('created_at'))
if user['verified']:
user['verified_type'] = data['verified_type']
if 'verified_reason' in data:
user['verified_reason'] = data['verified_reason']
return user
def parse_tweet_info(data):
"""
解析推文数据
"""
tweet = {
"_id": str(data['mid']),
"mblogid": data['mblogid'],
"created_at": parse_time(data['created_at']),
"geo": data['geo'],
"ip_location": data.get('region_name', None),
"reposts_count": data['reposts_count'],
"comments_count": data['comments_count'],
"attitudes_count": data['attitudes_count'],
"source": data['source'],
"content": data['text_raw'].replace('\u200b', ''),
"pic_urls": ["https://wx1.sinaimg.cn/orj960/" + pic_id for pic_id in data.get('pic_ids', [])],
"pic_num": data['pic_num'],
"user": parse_user_info(data['user']),
}
if 'page_info' in data and data['page_info'].get('object_type', '') == 'video':
tweet['video'] = data['page_info']['media_info']['mp4_720p_mp4']
tweet['url'] = f"https://weibo.com/{tweet['user']['_id']}/{tweet['mblogid']}"
return tweet
weibospider/spiders/fan.py
+30 -31
View File
@@ -1,46 +1,45 @@
#!/usr/bin/env python
# encoding: utf-8
"""
File Description:
Author: nghuyong
Mail: nghuyong@163.com
Created Time: 2020/4/14
"""
import re
import json
from scrapy import Spider
from scrapy.selector import Selector
from scrapy.http import Request
from items import RelationshipItem
import time
from spiders.comment import parse_user_info
class FanSpider(Spider):
name = "fan_spider"
base_url = "https://weibo.cn"
"""
微博粉丝数据采集
"""
name = "fan"
base_url = 'https://weibo.com/ajax/friendships/friends'
def start_requests(self):
user_ids = ['1087770692', '1699432410', '1266321801']
urls = [f"{self.base_url}/{user_id}/fans?page=1" for user_id in user_ids]
for url in urls:
yield Request(url, callback=self.parse)
"""
爬虫入口
"""
# 这里user_ids可替换成实际待采集的数据
user_ids = ['1087770692']
for user_id in user_ids:
url = self.base_url + f"?relate=fans&page=1&uid={user_id}&type=fans"
yield Request(url, callback=self.parse, meta={'user': user_id, 'page_num': 1})
def parse(self, response):
if response.url.endswith('page=1'):
all_page = re.search(r'/>&nbsp;1/(\d+)页</div>', response.text)
if all_page:
all_page = all_page.group(1)
all_page = int(all_page)
for page_num in range(2, all_page + 1):
page_url = response.url.replace('page=1', 'page={}'.format(page_num))
yield Request(page_url, self.parse, dont_filter=True, meta=response.meta)
selector = Selector(response)
urls = selector.xpath('//a[text()="关注他" or text()="关注她" or text()="移除"]/@href').extract()
uids = re.findall('uid=(\d+)', ";".join(urls), re.S)
ID = re.findall('(\d+)/fans', response.url)[0]
for uid in uids:
relationships_item = RelationshipItem()
relationships_item['crawl_time'] = int(time.time())
relationships_item["fan_id"] = uid
relationships_item["followed_id"] = ID
relationships_item["_id"] = 'fans' + '-' + uid + '-' + ID
yield relationships_item
def parse(self, response, **kwargs):
"""
网页解析
"""
data = json.loads(response.text)
for user in data['users']:
item = dict()
item['follower_id'] = response.meta['user']
item['fan_info'] = parse_user_info(user)
item['_id'] = response.meta['user'] + '_' + item['fan_info']['_id']
yield item
if data['users']:
response.meta['page_num'] += 1
url = self.base_url + f"?relate=fans&page={response.meta['page_num']}&uid={response.meta['user']}&type=fans"
yield Request(url, callback=self.parse, meta=response.meta)
weibospider/spiders/follower.py
+30 -32
View File
@@ -1,47 +1,45 @@
#!/usr/bin/env python
# encoding: utf-8
"""
File Description:
Author: nghuyong
Mail: nghuyong@163.com
Created Time: 2020/4/14
"""
import re
import json
from scrapy import Spider
from scrapy.selector import Selector
from scrapy.http import Request
from items import RelationshipItem
import time
from spiders.comment import parse_user_info
class FollowerSpider(Spider):
name = "follower_spider"
base_url = "https://weibo.cn"
"""
微博关注数据采集
"""
name = "follower"
base_url = 'https://weibo.com/ajax/friendships/friends'
def start_requests(self):
user_ids = ['1087770692', '1699432410', '1266321801']
urls = [f"{self.base_url}/{user_id}/follow?page=1" for user_id in user_ids]
for url in urls:
yield Request(url, callback=self.parse)
"""
爬虫入口
"""
# 这里user_ids可替换成实际待采集的数据
user_ids = ['1087770692']
for user_id in user_ids:
url = self.base_url + f"?page=1&uid={user_id}"
yield Request(url, callback=self.parse, meta={'user': user_id, 'page_num': 1})
def parse(self, response):
if response.url.endswith('page=1'):
all_page = re.search(r'/>&nbsp;1/(\d+)页</div>', response.text)
if all_page:
all_page = all_page.group(1)
all_page = int(all_page)
for page_num in range(2, all_page + 1):
page_url = response.url.replace('page=1', 'page={}'.format(page_num))
yield Request(page_url, self.parse, dont_filter=True, meta=response.meta)
selector = Selector(response)
urls = selector.xpath('//a[text()="关注他" or text()="关注她" or text()="取消关注"]/@href').extract()
uids = re.findall('uid=(\d+)', ";".join(urls), re.S)
ID = re.findall('(\d+)/follow', response.url)[0]
for uid in uids:
relationships_item = RelationshipItem()
relationships_item['crawl_time'] = int(time.time())
relationships_item["fan_id"] = ID
relationships_item["followed_id"] = uid
relationships_item["_id"] = ID + '-' + uid
yield relationships_item
def parse(self, response, **kwargs):
"""
网页解析
"""
data = json.loads(response.text)
for user in data['users']:
item = dict()
item['fan_id'] = response.meta['user']
item['follower_info'] = parse_user_info(user)
item['_id'] = response.meta['user'] + '_' + item['follower_info']['_id']
yield item
if data['users']:
response.meta['page_num'] += 1
url = self.base_url + f"?page={response.meta['page_num']}&uid={response.meta['user']}"
yield Request(url, callback=self.parse, meta=response.meta)
weibospider/spiders/repost.py
+28 -40
View File
@@ -1,55 +1,43 @@
#!/usr/bin/env python
# encoding: utf-8
"""
File Description:
Author: nghuyong
Mail: nghuyong@163.com
Created Time: 2020/4/14
"""
import re
from lxml import etree
import json
from scrapy import Spider
from scrapy.http import Request
import time
from items import RepostItem
from spiders.utils import extract_repost_content, time_fix
from spiders.common import parse_tweet_info, url_to_mid
class RepostSpider(Spider):
name = "repost_spider"
base_url = "https://weibo.cn"
"""
微博转发数据采集
"""
name = "repost"
def start_requests(self):
tweet_ids = ['JtSFZ4eGg']
urls = [f"{self.base_url}/repost/{tweet_id}?page=1" for tweet_id in tweet_ids]
for url in urls:
yield Request(url, callback=self.parse)
"""
爬虫入口
"""
# 这里tweet_ids可替换成实际待采集的数据
tweet_ids = ['Mb15BDYR0']
for tweet_id in tweet_ids:
mid = url_to_mid(tweet_id)
url = f"https://weibo.com/ajax/statuses/repostTimeline?id={mid}&page=1&moduleID=feed&count=10"
yield Request(url, callback=self.parse, meta={'page_num': 1, 'mid': mid})
def parse(self, response):
if response.url.endswith('page=1'):
all_page = re.search(r'/>&nbsp;1/(\d+)页</div>', response.text)
if all_page:
all_page = all_page.group(1)
all_page = int(all_page)
all_page = all_page if all_page <= 50 else 50
for page_num in range(2, all_page + 1):
page_url = response.url.replace('page=1', 'page={}'.format(page_num))
yield Request(page_url, self.parse, dont_filter=True, meta=response.meta)
tree_node = etree.HTML(response.body)
repo_nodes = tree_node.xpath('//div[@class="c" and not(contains(@id,"M_"))]')
for repo_node in repo_nodes:
try:
repo_user_url = repo_node.xpath('.//a[contains(@href,"/")]/@href')
if not repo_user_url:
continue
repo_item = RepostItem()
repo_item['crawl_time'] = int(time.time())
repo_item['weibo_id'] = response.url.split('/')[-1].split('?')[0]
repo_item['user_id'] = repo_user_url[0].strip('/')
content = extract_repost_content(etree.tostring(repo_node, encoding='unicode'))
repo_item['content'] = content.split(':', maxsplit=1)[1]
created_at_info = repo_node.xpath('.//span[@class="ct"]/text()')[0].split('来自')[0].strip()
repo_item['created_at'] = time_fix(created_at_info)
yield repo_item
except:
continue
def parse(self, response, **kwargs):
"""
网页解析
"""
data = json.loads(response.text)
for tweet in data['data']:
item = parse_tweet_info(tweet)
yield item
if data['data']:
mid, page_num = response.meta['mid'], response.meta['page_num']
page_num += 1
url = f"https://weibo.com/ajax/statuses/repostTimeline?id={mid}&page={page_num}&moduleID=feed&count=10"
yield Request(url, callback=self.parse, meta={'page_num': page_num, 'mid': mid})
weibospider/spiders/search.py
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env python
# encoding: utf-8
"""
Author: rightyonghu
Created Time: 2022/10/22
"""
import json
import re
from scrapy import Spider, Request
from spiders.common import parse_tweet_info
class SearchSpider(Spider):
"""
关键词搜索采集
"""
name = "search_spider"
base_url = "https://s.weibo.com/"
def start_requests(self):
"""
爬虫入口
"""
# 这里keywords可替换成实际待采集的数据
keywords = ['丽江']
for keyword in keywords:
url = f"https://s.weibo.com/weibo?q={keyword}&page=1"
yield Request(url, callback=self.parse, meta={'keyword': keyword})
def parse(self, response, **kwargs):
"""
网页解析
"""
html = response.text
tweet_ids = re.findall(r'\d+/(.*?)\?refer_flag=1001030103_\'\)">复制微博地址</a>', html)
for tweet_id in tweet_ids:
url = f"https://weibo.com/ajax/statuses/show?id={tweet_id}"
yield Request(url, callback=self.parse_tweet, meta=response.meta)
next_page = re.search('<a href="(.*?)" class="next">下一页</a>', html)
if next_page:
url = "https://s.weibo.com" + next_page.group(1)
yield Request(url, callback=self.parse, meta=response.meta)
@staticmethod
def parse_tweet(response):
"""
解析推文
"""
data = json.loads(response.text)
item = parse_tweet_info(data)
item['keyword'] = response.meta['keyword']
yield item
weibospider/spiders/tweet.py
+28 -147
View File
@@ -1,164 +1,45 @@
#!/usr/bin/env python
# encoding: utf-8
"""
File Description:
Author: nghuyong
Mail: nghuyong@163.com
Created Time: 2020/4/14
"""
import datetime
import re
from lxml import etree
import json
from scrapy import Spider
from scrapy.http import Request
import time
from items import TweetItem
from urllib.parse import unquote
from spiders.utils import time_fix, extract_weibo_content
from spiders.common import parse_tweet_info
class TweetSpider(Spider):
"""
用户推文数据采集
"""
name = "tweet_spider"
base_url = "https://weibo.cn"
def start_requests(self):
"""
爬虫入口
"""
# 这里user_ids可替换成实际待采集的数据
user_ids = ['1087770692']
for user_id in user_ids:
url = f"https://weibo.com/ajax/statuses/mymblog?uid={user_id}&page=1"
yield Request(url, callback=self.parse, meta={'user_id': user_id, 'page_num': 1})
def init_url_by_user_id():
# crawl tweets post by users
# === change the following config ===
user_ids = ['1087770692', '1699432410', '1266321801']
# === change the above config ===
urls = [f'{self.base_url}/{user_id}/profile?page=1' for user_id in user_ids]
return urls
def init_url_by_user_id_and_date():
# crawl specific users' tweets in a specific date
# === change the following config ===
user_ids = ['1087770692', '1699432410', '1266321801']
start_date = datetime.datetime.strptime("2020-01-01", '%Y-%m-%d')
end_date = datetime.datetime.strptime("2020-12-31", '%Y-%m-%d')
# === change the above config ===
time_spread = datetime.timedelta(days=20)
url_format = "https://weibo.cn/{}/profile?hasori=0&haspic=0&starttime={}&endtime={}&advancedfilter=1&page=1"
urls = []
while start_date < end_date:
for user_id in user_ids:
start_date_string = start_date.strftime("%Y%m%d")
tmp_end_date = start_date + time_spread
if tmp_end_date >= end_date:
tmp_end_date = end_date
end_date_string = tmp_end_date.strftime("%Y%m%d")
urls.append(url_format.format(user_id, start_date_string, end_date_string))
start_date = start_date + time_spread
return urls
# select urls generation by the following code
# urls = init_url_by_user_id()
urls = init_url_by_user_id_and_date()
for url in urls:
yield Request(url, callback=self.parse)
def parse(self, response):
if response.url.endswith('page=1'):
all_page = re.search(r'/>&nbsp;1/(\d+)页</div>', response.text)
if all_page:
all_page = all_page.group(1)
all_page = int(all_page)
for page_num in range(2, all_page + 1):
page_url = response.url.replace('page=1', 'page={}'.format(page_num))
yield Request(page_url, self.parse, dont_filter=True, meta=response.meta)
# 如果是搜索接口，按照天的粒度结果已经是100页，那继续按照小时的粒度进行切分
if 'search/mblog' in response.url and all_page == 100 and '-' not in response.url:
start_time_string = re.search(r'starttime=(\d+)&', unquote(response.url, "utf-8")).group(1)
keyword = re.search(r'keyword=(.*?)&', unquote(response.url, "utf-8")).group(1)
self.logger.info(f'split by hour,{start_time_string},{keyword}, {unquote(response.url, "utf-8")}')
date_start = datetime.datetime.strptime(start_time_string, "%Y%m%d")
time_spread = datetime.timedelta(days=1)
url_format_by_hour = "https://weibo.cn/search/mblog?hideSearchFrame=&keyword={}&advancedfilter=1&starttime={}&endtime={}&sort=time&atten=1&page=1"
one_day_back = date_start - time_spread
# from today's 7:00-8:00am to 23:00-24:00am
for hour in range(7, 24):
# calculation rule of starting time: start_date 8:00am + offset:16
begin_hour = one_day_back.strftime("%Y%m%d") + "-" + str(hour + 16)
# calculation rule of ending time: (end_date+1) 8:00am + offset:-7
end_hour = one_day_back.strftime("%Y%m%d") + "-" + str(hour - 7)
page_url = url_format_by_hour.format(keyword, begin_hour, end_hour)
yield Request(page_url, self.parse, dont_filter=True, meta=response.meta)
two_day_back = one_day_back - time_spread
# from today's 0:00-1:00am to 6:00-7:00am
for hour in range(0, 7):
# note the offset change bc we are two-days back now
begin_hour = two_day_back.strftime("%Y%m%d") + "-" + str(hour + 40)
end_hour = two_day_back.strftime("%Y%m%d") + "-" + str(hour + 17)
page_url = url_format_by_hour.format(keyword, begin_hour, end_hour)
yield Request(page_url, self.parse, dont_filter=True, meta=response.meta)
tree_node = etree.HTML(response.body)
tweet_nodes = tree_node.xpath('//div[@class="c" and @id]')
for tweet_node in tweet_nodes:
try:
tweet_item = TweetItem()
tweet_item['crawl_time'] = int(time.time())
tweet_repost_url = tweet_node.xpath('.//a[contains(text(),"转发[")]/@href')[0]
user_tweet_id = re.search(r'/repost/(.*?)\?uid=(\d+)', tweet_repost_url)
tweet_item['weibo_url'] = 'https://weibo.com/{}/{}'.format(user_tweet_id.group(2),
user_tweet_id.group(1))
tweet_item['user_id'] = user_tweet_id.group(2)
tweet_item['_id'] = user_tweet_id.group(1)
create_time_info_node = tweet_node.xpath('.//span[@class="ct"]')[-1]
create_time_info = create_time_info_node.xpath('string(.)')
if "来自" in create_time_info:
tweet_item['created_at'] = time_fix(create_time_info.split('来自')[0].strip())
tweet_item['tool'] = create_time_info.split('来自')[1].strip()
else:
tweet_item['created_at'] = time_fix(create_time_info.strip())
like_num = tweet_node.xpath('.//a[contains(text(),"赞[")]/text()')[-1]
tweet_item['like_num'] = int(re.search('\d+', like_num).group())
repost_num = tweet_node.xpath('.//a[contains(text(),"转发[")]/text()')[-1]
tweet_item['repost_num'] = int(re.search('\d+', repost_num).group())
comment_num = tweet_node.xpath(
'.//a[contains(text(),"评论[") and not(contains(text(),"原文"))]/text()')[-1]
tweet_item['comment_num'] = int(re.search('\d+', comment_num).group())
images = tweet_node.xpath('.//img[@alt="图片"]/@src')
if images:
tweet_item['image_url'] = images
videos = tweet_node.xpath('.//a[contains(@href,"https://m.weibo.cn/s/video/show?object_id=")]/@href')
if videos:
tweet_item['video_url'] = videos
map_node = tweet_node.xpath('.//a[contains(text(),"显示地图")]')
if map_node:
map_node = map_node[0]
map_node_url = map_node.xpath('./@href')[0]
map_info = re.search(r'xy=(.*?)&', map_node_url).group(1)
tweet_item['location_map_info'] = map_info
repost_node = tweet_node.xpath('.//a[contains(text(),"原文评论[")]/@href')
if repost_node:
tweet_item['origin_weibo'] = repost_node[0]
all_content_link = tweet_node.xpath('.//a[text()="全文" and contains(@href,"ckAll=1")]')
if all_content_link:
all_content_url = self.base_url + all_content_link[0].xpath('./@href')[0]
yield Request(all_content_url, callback=self.parse_all_content, meta={'item': tweet_item},
priority=1)
else:
tweet_html = etree.tostring(tweet_node, encoding='unicode')
tweet_item['content'] = extract_weibo_content(tweet_html)
yield tweet_item
except Exception as e:
self.logger.error(e)
def parse_all_content(self, response):
tree_node = etree.HTML(response.body)
tweet_item = response.meta['item']
content_node = tree_node.xpath('//*[@id="M_"]/div[1]')[0]
tweet_html = etree.tostring(content_node, encoding='unicode')
tweet_item['content'] = extract_weibo_content(tweet_html)
yield tweet_item
def parse(self, response, **kwargs):
"""
网页解析
"""
data = json.loads(response.text)
tweets = data['data']['list']
for tweet in tweets:
item = parse_tweet_info(tweet)
del item['user']
yield item
if tweets:
user_id, page_num = response.meta['user_id'], response.meta['page_num']
page_num += 1
url = f"https://weibo.com/ajax/statuses/mymblog?uid={user_id}&page={page_num}"
yield Request(url, callback=self.parse, meta={'user_id': user_id, 'page_num': page_num})
weibospider/spiders/user.py
+39 -75
View File
@@ -1,94 +1,58 @@
#!/usr/bin/env python
# encoding: utf-8
"""
File Description:
Author: nghuyong
Mail: nghuyong@163.com
Created Time: 2020/4/14
"""
import re
from scrapy import Selector, Spider
import json
from scrapy import Spider
from scrapy.http import Request
import time
from items import UserItem
from spiders.common import parse_user_info
class UserSpider(Spider):
"""
微博用户信息爬虫
"""
name = "user_spider"
base_url = "https://weibo.cn"
def start_requests(self):
user_ids = ['1087770692', '1699432410', '1266321801','1749127163']
urls = [f'{self.base_url}/{user_id}/info' for user_id in user_ids]
"""
爬虫入口
"""
# 这里user_ids可替换成实际待采集的数据
user_ids = ['1749127163']
urls = [f'https://weibo.com/ajax/profile/info?uid={user_id}' for user_id in user_ids]
for url in urls:
yield Request(url, callback=self.parse)
def parse(self, response):
user_item = UserItem()
user_item['crawl_time'] = int(time.time())
selector = Selector(response)
user_item['_id'] = re.findall('(\d+)/info', response.url)[0]
user_info_text = ";".join(selector.xpath('body/div[@class="c"]//text()').extract())
nick_name = re.findall('昵称;?:?(.*?);', user_info_text)
gender = re.findall('性别;?:?(.*?);', user_info_text)
place = re.findall('地区;?:?(.*?);', user_info_text)
brief_introduction = re.findall('简介;?:?(.*?);', user_info_text)
birthday = re.findall('生日;?:?(.*?);', user_info_text)
sex_orientation = re.findall('性取向;?:?(.*?);', user_info_text)
sentiment = re.findall('感情状况;?:?(.*?);', user_info_text)
vip_level = re.findall('会员等级;?:?(.*?);', user_info_text)
authentication = re.findall('认证;?:?(.*?);', user_info_text)
labels = re.findall('标签;?:?(.*?)更多>>', user_info_text)
if nick_name and nick_name[0]:
user_item["nick_name"] = nick_name[0].replace(u"\xa0", "")
if gender and gender[0]:
user_item["gender"] = gender[0].replace(u"\xa0", "")
if place and place[0]:
place = place[0].replace(u"\xa0", "").split(" ")
user_item["province"] = place[0]
if len(place) > 1:
user_item["city"] = place[1]
if brief_introduction and brief_introduction[0]:
user_item["brief_introduction"] = brief_introduction[0].replace(u"\xa0", "")
if birthday and birthday[0]:
user_item['birthday'] = birthday[0]
if sex_orientation and sex_orientation[0]:
if sex_orientation[0].replace(u"\xa0", "") == gender[0]:
user_item["sex_orientation"] = "同性恋"
else:
user_item["sex_orientation"] = "异性恋"
if sentiment and sentiment[0]:
user_item["sentiment"] = sentiment[0].replace(u"\xa0", "")
if vip_level and vip_level[0]:
user_item["vip_level"] = vip_level[0].replace(u"\xa0", "")
if authentication and authentication[0]:
user_item["authentication"] = authentication[0].replace(u"\xa0", "")
if labels and labels[0]:
user_item["labels"] = labels[0].replace(u"\xa0", ",").replace(';', '').strip(',')
education_info = selector.xpath('//div[contains(text(),"学习经历")]/following-sibling::div[1]'). \
xpath('string(.)').extract()
if education_info:
user_item['education'] = education_info[0].replace(u"\xa0", "")
work_info = selector.xpath('//div[contains(text(),"工作经历")]/following-sibling::div[1]'). \
xpath('string(.)').extract()
if work_info:
user_item['work'] = work_info[0].replace(u"\xa0", "")
request_meta = response.meta
request_meta['item'] = user_item
yield Request(self.base_url + '/u/{}'.format(user_item['_id']),
callback=self.parse_further_information,
meta=request_meta, dont_filter=True, priority=1)
def parse(self, response, **kwargs):
"""
网页解析
"""
data = json.loads(response.text)
item = parse_user_info(data['data']['user'])
url = f"https://weibo.com/ajax/profile/detail?uid={item['_id']}"
yield Request(url, callback=self.parse_detail, meta={'item': item})
def parse_further_information(self, response):
text = response.text
user_item = response.meta['item']
tweets_num = re.findall('微博\[(\d+)\]', text)
if tweets_num:
user_item['tweets_num'] = int(tweets_num[0])
follows_num = re.findall('关注\[(\d+)\]', text)
if follows_num:
user_item['follows_num'] = int(follows_num[0])
fans_num = re.findall('粉丝\[(\d+)\]', text)
if fans_num:
user_item['fans_num'] = int(fans_num[0])
yield user_item
@staticmethod
def parse_detail(response):
"""
解析详细数据
"""
item = response.meta['item']
data = json.loads(response.text)['data']
item['birthday'] = data.get('birthday', '')
if 'created_at' not in item:
item['created_at'] = data.get('created_at', '')
item['desc_text'] = data.get('desc_text', '')
item['ip_location'] = data.get('ip_location', '')
item['sunshine_credit'] = data.get('sunshine_credit', {}).get('level', '')
item['label_desc'] = [label['name'] for label in data.get('label_desc', [])]
if 'company' in data:
item['company'] = data['company']
if 'education' in data:
item['education'] = data['education']
yield item
weibospider/spiders/utils.py
-93
View File
@@ -1,93 +0,0 @@
#!/usr/bin/env python
# encoding: utf-8
import re
import datetime
def time_fix(time_string):
now_time = datetime.datetime.now()
if '分钟前' in time_string:
minutes = re.search(r'^(\d+)分钟', time_string).group(1)
created_at = now_time - datetime.timedelta(minutes=int(minutes))
return created_at.strftime('%Y-%m-%d %H:%M')
if '小时前' in time_string:
minutes = re.search(r'^(\d+)小时', time_string).group(1)
created_at = now_time - datetime.timedelta(hours=int(minutes))
return created_at.strftime('%Y-%m-%d %H:%M')
if '今天' in time_string:
return time_string.replace('今天', now_time.strftime('%Y-%m-%d'))
if '月' in time_string:
time_string = time_string.replace('月', '-').replace('日', '')
time_string = str(now_time.year) + '-' + time_string
return time_string
return time_string
keyword_re = re.compile('<span class="kt">|</span>|原图|<!-- 是否进行翻译 -->|<span class="cmt">|\[组图共.+张\]')
emoji_re = re.compile('<img alt="|" src="(https:)?//h5\.sinaimg(.*?)/>| style="width:1em; height:1em;" />')
white_space_re = re.compile('<br />')
div_re = re.compile('</div>|<div>')
image_re = re.compile('<img(.*?)/>|<span class="url-icon">')
url_re = re.compile('<a href=(.*?)>|</a>')
def extract_weibo_content(weibo_html):
s = weibo_html
if 'class="ctt">' in s:
s = s.split('class="ctt">', maxsplit=1)[1]
s = emoji_re.sub('', s)
s = url_re.sub('', s)
s = div_re.sub('', s)
s = image_re.sub('', s)
if '<span class="ct">' in s:
s = s.split('<span class="ct">')[0]
splits = s.split('赞[')
if len(splits) == 2:
s = splits[0]
if len(splits) == 3:
origin_text = splits[0]
retweet_text = splits[1].split('转发理由:')[1]
s = origin_text + '转发理由:' + retweet_text
s = white_space_re.sub(' ', s)
s = keyword_re.sub('', s)
s = s.replace('\xa0', '')
s = s.strip(':')
s = s.strip()
return s
def extract_comment_content(comment_html):
s = comment_html
if 'class="ctt">' in s:
s = s.split('class="ctt">', maxsplit=1)[1]
s = s.split('举报', maxsplit=1)[0]
s = emoji_re.sub('', s)
s = keyword_re.sub('', s)
s = url_re.sub('', s)
s = div_re.sub('', s)
s = image_re.sub('', s)
s = white_space_re.sub(' ', s)
s = s.replace('\xa0', '')
s = s.strip(':')
s = s.strip()
return s
def extract_repost_content(repost_html):
s = repost_html
if 'class="cc">' in s:
s = s.split('<span class="cc">', maxsplit=1)[0]
s = emoji_re.sub('', s)
s = keyword_re.sub('', s)
s = url_re.sub('', s)
s = div_re.sub('', s)
s = image_re.sub('', s)
s = white_space_re.sub(' ', s)
s = s.replace('\xa0', '')
s = s.replace('<div class="c">', '')
s = s.strip(':')
s = s.strip()
return s