The key point is LastUpdate, we can use this field to tell if the Downloader is still alive
In some cases, when Redis is doing some GC or the memory of the Host is fullfilled, DownloaderShell will exit unexcepted
A big step for NScrayp, it can now utilize multiple Downloaders
Downloaders are now distributed to different process/machine, the connection
between Downloader and Spider is based on Scheduler, the scheduler distribute
Downloader task to different Downloader through Redis MQ
Details can be configed in appsetting.json
NScrapy still support single process mode, as long as we specify the run mode to InMemory
After Compressed the Response, tie the compressed str with url then put that pair in Specific Topic
Spider on the other hand will consume the message that published to the specific topic and do analysis
Make a wrap of Downloader named DownloaderShell, so that it can handle
request sent from Redis, which is actually pushed in to the specific Queue defined in appsetting.json
Not finished yet
Elendil