Create a ConifgProviderFactory so thay we can get ConfigProvider from a single point
Also enhenced the logic of refresh config, right now when zookeeper watcher dected the config is changed
NScrapy will update its context, and during that time, NScrapy will lock the access to the Connection of Redis
Enable ItemPipeline function by setting the below appsetting.json setting
"Pipelines": [
{ "Pipeline": "NScrapy.Infra.Pipeline.MockPipeline" }
],
there is also a mock test case for ItemPipeline
A big step for NScrayp, it can now utilize multiple Downloaders
Downloaders are now distributed to different process/machine, the connection
between Downloader and Spider is based on Scheduler, the scheduler distribute
Downloader task to different Downloader through Redis MQ
Details can be configed in appsetting.json
NScrapy still support single process mode, as long as we specify the run mode to InMemory
Added log4net at the bottom of the project, in NScrapyContext
Enhenced the Response.CssSelector so that we can select element based on a set of selector
This is very userful when the web page have tiny different between each page
HttpHeaderMiddleware as a Default DownloaderMiddleware will set HttpRequest Header
based on the config in appsetting.json (AppSettings:HttpHeader section)
If specified node is not set in config file, Middleware will use its default value to fullfill the request header
Engine:
1.Configurable in appsetting.json to choice a real engine or a mock engine for test
2.Listen to Item queue to handle the request sent by Scheduler
Shell:
1.Init Context for NScrapy to run
1).Read Config
2.Get Spider through Spider Factory
Scheduler:
1.Send request to Queue which the engine is listening
Downloader
1.Init DownloaderPool
2.Asyncly Process all the incoming request by picking a Downloader from DownloaderPool
*Test Case update
EngineTest:Test if the message can be sent to MockEngine through Shell->Spider->Scheduler->Engine
ConstructDownloader:Test if DownloaderPool can work, if the Downloader can actually get response from Website
Elendil