Create a ConifgProviderFactory so thay we can get ConfigProvider from a single point
Also enhenced the logic of refresh config, right now when zookeeper watcher dected the config is changed
NScrapy will update its context, and during that time, NScrapy will lock the access to the Connection of Redis
Enable ItemPipeline function by setting the below appsetting.json setting
"Pipelines": [
{ "Pipeline": "NScrapy.Infra.Pipeline.MockPipeline" }
],
there is also a mock test case for ItemPipeline
HtmlAgilityPack does not support extract attr directly, made some enhencement to xPathSelector so it can now directly extract attribute of html element like this
/html/body/div/div/div/div/ul/li[i/b="企"]/div/div/h3/a@href
A big step for NScrayp, it can now utilize multiple Downloaders
Downloaders are now distributed to different process/machine, the connection
between Downloader and Spider is based on Scheduler, the scheduler distribute
Downloader task to different Downloader through Redis MQ
Details can be configed in appsetting.json
NScrapy still support single process mode, as long as we specify the run mode to InMemory
After Compressed the Response, tie the compressed str with url then put that pair in Specific Topic
Spider on the other hand will consume the message that published to the specific topic and do analysis
Make a wrap of Downloader named DownloaderShell, so that it can handle
request sent from Redis, which is actually pushed in to the specific Queue defined in appsetting.json
Not finished yet
using Redis as messagequeue service to distribute request to different distributed Downloader
the first commit of this branch finishes the Config part of RedisScheduler
Rewrite the Scheduler, now it is called InMemoryScheduler
IOC to get the Actual Scheduler from appsettings.json
Fix a bug of refresh the config file
this ItemLoader is more like a draft version of ItemLoader
It can do basic mappings from selector to object by defining mappings between selector and object field
later on will add pre processor, post processor while getting values from response and setting values to object
besides make a factory method to get ItemLoader, since for a particular Type, its ItemLoader barely changes, no need to new a ItemLoader for a same type everytime
By checking if there is anymore Pending Request/Response in Queue and Running Downloader
NScrapy.Crawl would exit if no more pending Request/Response/RunningDownloader
Also add test case for this mechanism
Fix the HttpHeaderMiddleware bug
Added UrlFilter with MD5 enhenced
Improved the sample Spider "JobSpider" so that it can now crawl all the .net postions posted at liepin.com
Added log4net at the bottom of the project, in NScrapyContext
Enhenced the Response.CssSelector so that we can select element based on a set of selector
This is very userful when the web page have tiny different between each page
we can now use ::attr(href) syntax to get values from html attribute
example:
SourceHTML:
<div class="job-info">
<h3 title="招聘销售总监">
<a href="https://www.liepin.com/job/1913549323.shtml" data-promid="imscid=R000000075&ckid=b6ab9f62c5abd657&headckid=b6ab9f62c5abd657&pageNo=0&pageIdx0&totalIdx=0&sup=1&d_pageSize=40&siTag=aEPqOr56EDcL1HyFlajr3w%7EfA9rXquZc5IkJpXC-Ycixw&d_headId=549281be67f4c28a83bd3d340965b270&d_ckId=549281be67f4c28a83bd3d340965b270&d_sfrom=search_fp&d_curPage=0&d_posi=0" target="_blank" onclick="tlog=window.tlog||[];tlog.push('c:w_sojob_jobclick_9')">
销售总监 </a>
</h3>
</div>
Selector:
Response.CssSelector(".job-info h3 a::attr(href)").Extract();
The above will get a IEnumerable<String> that only contains the href of the link in above SourceHTML
Response.CssSelector(".job-info h3 a::attr(href)").ExtractFirst();
The above code will get the first element as string from the CssSelector, here the value is
https://www.liepin.com/job/1913549323.shtml
::attr(text) will get the InnerText of the element
Engine:
1.Configurable in appsetting.json to choice a real engine or a mock engine for test
2.Listen to Item queue to handle the request sent by Scheduler
Shell:
1.Init Context for NScrapy to run
1).Read Config
2.Get Spider through Spider Factory
Scheduler:
1.Send request to Queue which the engine is listening
Downloader
1.Init DownloaderPool
2.Asyncly Process all the incoming request by picking a Downloader from DownloaderPool
*Test Case update
EngineTest:Test if the message can be sent to MockEngine through Shell->Spider->Scheduler->Engine
ConstructDownloader:Test if DownloaderPool can work, if the Downloader can actually get response from Website
Elendil