- Changed NScrapy.Core target framework from net8.0 back to net10.0
- Added NScrapy.Core project reference to NScrapy.Test
- Created FluentApiTests.cs with the following test cases:
- TestSpiderBuilder_CanCreateSpiderWithName
- TestSpiderBuilder_BuildCreatesSpider
- TestSpiderBuilder_SetsStartUrls
- TestSpiderBuilder_AddsPipeline
- TestDistributedBuilder_SetsRedisConfig
- TestSpiderOptions_DefaultValues
- TestDistributedBuilder_Chaining
- TestSpiderBuilder_ConfiguresOptions
- Added helper classes TestItem and TestPipeline for testing pipelines
Create a ConifgProviderFactory so thay we can get ConfigProvider from a single point
Also enhenced the logic of refresh config, right now when zookeeper watcher dected the config is changed
NScrapy will update its context, and during that time, NScrapy will lock the access to the Connection of Redis
Enable ItemPipeline function by setting the below appsetting.json setting
"Pipelines": [
{ "Pipeline": "NScrapy.Infra.Pipeline.MockPipeline" }
],
there is also a mock test case for ItemPipeline
HtmlAgilityPack does not support extract attr directly, made some enhencement to xPathSelector so it can now directly extract attribute of html element like this
/html/body/div/div/div/div/ul/li[i/b="企"]/div/div/h3/a@href
A big step for NScrayp, it can now utilize multiple Downloaders
Downloaders are now distributed to different process/machine, the connection
between Downloader and Spider is based on Scheduler, the scheduler distribute
Downloader task to different Downloader through Redis MQ
Details can be configed in appsetting.json
NScrapy still support single process mode, as long as we specify the run mode to InMemory
Make a wrap of Downloader named DownloaderShell, so that it can handle
request sent from Redis, which is actually pushed in to the specific Queue defined in appsetting.json
Not finished yet
using Redis as messagequeue service to distribute request to different distributed Downloader
the first commit of this branch finishes the Config part of RedisScheduler
Rewrite the Scheduler, now it is called InMemoryScheduler
IOC to get the Actual Scheduler from appsettings.json
Fix a bug of refresh the config file
By checking if there is anymore Pending Request/Response in Queue and Running Downloader
NScrapy.Crawl would exit if no more pending Request/Response/RunningDownloader
Also add test case for this mechanism
Fix the HttpHeaderMiddleware bug
Added UrlFilter with MD5 enhenced
Improved the sample Spider "JobSpider" so that it can now crawl all the .net postions posted at liepin.com
Added log4net at the bottom of the project, in NScrapyContext
Enhenced the Response.CssSelector so that we can select element based on a set of selector
This is very userful when the web page have tiny different between each page
we can now use ::attr(href) syntax to get values from html attribute
example:
SourceHTML:
<div class="job-info">
<h3 title="招聘销售总监">
<a href="https://www.liepin.com/job/1913549323.shtml" data-promid="imscid=R000000075&ckid=b6ab9f62c5abd657&headckid=b6ab9f62c5abd657&pageNo=0&pageIdx0&totalIdx=0&sup=1&d_pageSize=40&siTag=aEPqOr56EDcL1HyFlajr3w%7EfA9rXquZc5IkJpXC-Ycixw&d_headId=549281be67f4c28a83bd3d340965b270&d_ckId=549281be67f4c28a83bd3d340965b270&d_sfrom=search_fp&d_curPage=0&d_posi=0" target="_blank" onclick="tlog=window.tlog||[];tlog.push('c:w_sojob_jobclick_9')">
销售总监 </a>
</h3>
</div>
Selector:
Response.CssSelector(".job-info h3 a::attr(href)").Extract();
The above will get a IEnumerable<String> that only contains the href of the link in above SourceHTML
Response.CssSelector(".job-info h3 a::attr(href)").ExtractFirst();
The above code will get the first element as string from the CssSelector, here the value is
https://www.liepin.com/job/1913549323.shtml
::attr(text) will get the InnerText of the element
Wrap the Css selected node into a IResponse again so that we can process in chain
NScrapy can now get url from response through regex and HtmlAgilityPack.CssSelectors
By getting the url, we can again send request and write call back function to get more data
HttpHeaderMiddleware as a Default DownloaderMiddleware will set HttpRequest Header
based on the config in appsetting.json (AppSettings:HttpHeader section)
If specified node is not set in config file, Middleware will use its default value to fullfill the request header
Engine:
1.Configurable in appsetting.json to choice a real engine or a mock engine for test
2.Listen to Item queue to handle the request sent by Scheduler
Shell:
1.Init Context for NScrapy to run
1).Read Config
2.Get Spider through Spider Factory
Scheduler:
1.Send request to Queue which the engine is listening
Downloader
1.Init DownloaderPool
2.Asyncly Process all the incoming request by picking a Downloader from DownloaderPool
*Test Case update
EngineTest:Test if the message can be sent to MockEngine through Shell->Spider->Scheduler->Engine
ConstructDownloader:Test if DownloaderPool can work, if the Downloader can actually get response from Website
zheng elendil