85fc6f0776
defect fix
2018-07-10 23:40:20 +08:00
2703e6d174
refactor zookeeper config provider
...
Also use a more elegent way to exit in memory Nscrapy Crawl mode
2018-07-09 23:42:03 +08:00
46086ffaf9
ConfigProvider update
...
Create a ConifgProviderFactory so thay we can get ConfigProvider from a single point
Also enhenced the logic of refresh config, right now when zookeeper watcher dected the config is changed
NScrapy will update its context, and during that time, NScrapy will lock the access to the Connection of Redis
2018-07-04 23:31:44 +08:00
5a0ff77c8c
ZKConfigProvider v0.1
2018-07-04 00:11:40 +08:00
c98236cf7d
Merge branch 'ConfigManagement' of github.com:xboxeer/NScrapy into ConfigManagement
2018-06-29 01:36:49 +08:00
e4351fa803
Optimized ZKHelper
...
make create to be able to create node recursively from the path
2018-06-29 01:35:26 +08:00
6db9cc376a
Merge pull request #25 from xboxeer/master
...
Keep this latest brunch align with master
2018-06-28 00:27:09 +08:00
66275a5598
POC of Zookeeper .net Client
...
also intergrate with ZookeeperConfigProvider
to be honest Zookeeper api is so hard to understand, why it is designed in such way?
2018-06-28 00:24:12 +08:00
723a51ea62
Init ConfigManagement Component
...
Will use Zookeeper to manager the config for every node
Also changed some methed name in old design for the purpose of align naming practice
2018-06-27 01:31:28 +08:00
6f53104ecc
Update README.md
2018-06-21 22:07:32 +08:00
ec787fd5e7
Update README.md
2018-06-21 18:16:35 +08:00
4d2e127724
Update README.md
2018-06-21 18:16:15 +08:00
4577536ea9
Update README.md
2018-06-20 22:26:16 +08:00
135770a8ba
Merge branch 'master' of github.com:xboxeer/NScrapy
2018-06-19 00:18:38 +08:00
1c8d81f283
Add somemore details for StatusUpdateMiddleware
...
The key point is LastUpdate, we can use this field to tell if the Downloader is still alive
In some cases, when Redis is doing some GC or the memory of the Host is fullfilled, DownloaderShell will exit unexcepted
2018-06-19 00:18:01 +08:00
82a9cc5d26
Update README.md
2018-06-12 22:02:09 +08:00
9219f689ab
Update README.md
2018-06-07 00:28:24 +08:00
871b7a0a67
Update README.md
2018-06-07 00:26:45 +08:00
121fc50bef
Update README.md
2018-06-07 00:13:59 +08:00
f58f038afb
Update README.md
2018-06-07 00:10:11 +08:00
88db9026d3
Update README.md
2018-06-07 00:08:37 +08:00
8aea43d7fd
Merge branch 'master' of github.com:xboxeer/NScrapy
2018-06-05 22:47:23 +08:00
8d471c72b0
Fix some bugs in sample project and increase timeout limit to 10sec while Spider is getting data from Redis
...
The timeout should be configable in the future
Signed-off-by: Elendil <zy4182@sina.com >
2018-06-05 22:46:31 +08:00
621ac66efd
Set theme jekyll-theme-cayman
2018-06-04 15:48:33 +08:00
3622c13fea
individual downloader shell will update its downloader status to Redis
...
This is for the purpose of web console displaying distributed downloader status in the new future
2018-05-31 23:42:04 +08:00
2d7be4f77d
enhence the RedisScheduler
...
RedisScheduler will now register all the proper method in its registedCallback set at first, no need to wait until the callback is actually called
2018-05-25 01:02:08 +08:00
32a1bf82f1
make the call back as Async, and fix bugs for ItemLoader while we are calling callback function in async way
2018-05-24 00:20:23 +08:00
20e0982e88
ItemLoader Bug fix,CssSelector enhencement, ItemPipeline added
2018-05-22 23:50:25 +08:00
0e36fea14d
Bug Fix of ItemLoader and Add a MongoDBPipeline as Pipeline sample
...
Intergrate with MongoDB with MongoDB pipeline to store the information captured by NScrapy is AWESOME!
far more easy then Scrapy
2018-05-17 22:31:44 +08:00
2219a5f87d
Enhenced the ItemPipeline Field Mapping Logic
...
we can use Lambda to pick a property for mapping, instead of type the Field name manually
2018-05-17 00:52:00 +08:00
7c15c702c6
ItemPipeLine first version
...
Enable ItemPipeline function by setting the below appsetting.json setting
"Pipelines": [
{ "Pipeline": "NScrapy.Infra.Pipeline.MockPipeline" }
],
there is also a mock test case for ItemPipeline
2018-05-17 00:11:59 +08:00
8dbd6a1e2a
forget to delete unused class results to build failed in Jenkins
2018-05-16 21:24:23 +08:00
c3c2d6e56b
ItemLoader support field mapping for xpath selector
2018-05-16 01:41:35 +08:00
f92343e0d3
Merge pull request #16 from xboxeer/DistributedVersion
...
Add support for xPathSelector
2018-05-16 01:33:58 +08:00
4afa454ebb
Add support for xPathSelector
...
HtmlAgilityPack does not support extract attr directly, made some enhencement to xPathSelector so it can now directly extract attribute of html element like this
/html/body/div/div/div/div/ul/li[i/b="企"]/div/div/h3/a@href
2018-05-16 01:32:47 +08:00
1be0823f08
Update README.md
2018-05-13 12:37:52 +08:00
3d7afe7212
Update README.md
2018-05-11 18:44:20 +08:00
a38063293d
Update README.md
2018-05-11 14:02:47 +08:00
7b02f1e840
Merge pull request #15 from xboxeer/DistributedVersion
...
Distributed version to master
2018-05-11 01:13:36 +08:00
1c018a39a4
Bug Fix and Add HttpPost,cookies Support
2018-05-11 01:12:33 +08:00
245aefa502
bug fix
2018-05-10 21:33:54 +08:00
4e61597193
Optimize the Lock/Release structure
...
release the lock as long as we fetched the item, instead of waiting for the call back finish
2018-05-10 21:31:07 +08:00
00f90d9492
Change the response from Topic to Queue so that multiple Spider won't process the same response duplicate
...
Add lock to response pick logic
2018-05-10 02:04:54 +08:00
267fef7b79
Merge branch 'DistributedVersion' of github.com:xboxeer/NScrapy into DistributedVersion
2018-05-09 23:01:42 +08:00
39723a6822
Fix bug that spider in different platform unable to grap together
...
The issue is each request is bind to a callback fingerprint before, in different platform
say Mac and Windows, the callback fingerprint for the same method is different beacuse they have different MSIL stream
so this time in registered call back list of each spider, using callback name as key instead of callback body msil steam MD5
2018-05-09 23:01:07 +08:00
cb0804a049
Fix bug that spider in different platform unable to grap together
...
The issue is each request is bind to a callback fingerprint before, in different platform
say Mac and Windows, the callback fingerprint for the same method is different beacuse they have different MSIL stream
so this time in registered call back list of each spider, using callback name as key instead of callback body msil steam MD5
2018-05-09 23:00:31 +08:00
c73765d4e4
Redis Lock Added
...
Add a locker before getting url from receiverqueue, so that multiple downloader won't confilct
2018-05-09 22:45:49 +08:00
542393b03f
Update README.md
2018-05-09 16:55:11 +08:00
7f365aabe5
Merge branch 'DistributedVersion' of github.com:xboxeer/NScrapy into DistributedVersion
2018-05-09 01:32:07 +08:00
2ed341c009
Add Distributed Support for Spider
...
Still need some investigate against that if the spider will crawl the same page while there are multiple spider running
2018-05-09 01:30:43 +08:00
Elendil