Create a ConifgProviderFactory so thay we can get ConfigProvider from a single point
Also enhenced the logic of refresh config, right now when zookeeper watcher dected the config is changed
NScrapy will update its context, and during that time, NScrapy will lock the access to the Connection of Redis
The key point is LastUpdate, we can use this field to tell if the Downloader is still alive
In some cases, when Redis is doing some GC or the memory of the Host is fullfilled, DownloaderShell will exit unexcepted
HtmlAgilityPack does not support extract attr directly, made some enhencement to xPathSelector so it can now directly extract attribute of html element like this
/html/body/div/div/div/div/ul/li[i/b="企"]/div/div/h3/a@href
A big step for NScrayp, it can now utilize multiple Downloaders
Downloaders are now distributed to different process/machine, the connection
between Downloader and Spider is based on Scheduler, the scheduler distribute
Downloader task to different Downloader through Redis MQ
Details can be configed in appsetting.json
NScrapy still support single process mode, as long as we specify the run mode to InMemory
After Compressed the Response, tie the compressed str with url then put that pair in Specific Topic
Spider on the other hand will consume the message that published to the specific topic and do analysis
Make a wrap of Downloader named DownloaderShell, so that it can handle
request sent from Redis, which is actually pushed in to the specific Queue defined in appsetting.json
Not finished yet
Elendil