Engine:
1.Configurable in appsetting.json to choice a real engine or a mock engine for test
2.Listen to Item queue to handle the request sent by Scheduler
Shell:
1.Init Context for NScrapy to run
1).Read Config
2.Get Spider through Spider Factory
Scheduler:
1.Send request to Queue which the engine is listening
Downloader
1.Init DownloaderPool
2.Asyncly Process all the incoming request by picking a Downloader from DownloaderPool
*Test Case update
EngineTest:Test if the message can be sent to MockEngine through Shell->Spider->Scheduler->Engine
ConstructDownloader:Test if DownloaderPool can work, if the Downloader can actually get response from Website
Elendil