Create a ConifgProviderFactory so thay we can get ConfigProvider from a single point
Also enhenced the logic of refresh config, right now when zookeeper watcher dected the config is changed
NScrapy will update its context, and during that time, NScrapy will lock the access to the Connection of Redis
A big step for NScrayp, it can now utilize multiple Downloaders
Downloaders are now distributed to different process/machine, the connection
between Downloader and Spider is based on Scheduler, the scheduler distribute
Downloader task to different Downloader through Redis MQ
Details can be configed in appsetting.json
NScrapy still support single process mode, as long as we specify the run mode to InMemory
After Compressed the Response, tie the compressed str with url then put that pair in Specific Topic
Spider on the other hand will consume the message that published to the specific topic and do analysis
Make a wrap of Downloader named DownloaderShell, so that it can handle
request sent from Redis, which is actually pushed in to the specific Queue defined in appsetting.json
Not finished yet
Sam (OpenClaw)