we can now use ::attr(href) syntax to get values from html attribute
example:
SourceHTML:
<div class="job-info">
<h3 title="招聘销售总监">
<a href="https://www.liepin.com/job/1913549323.shtml" data-promid="imscid=R000000075&ckid=b6ab9f62c5abd657&headckid=b6ab9f62c5abd657&pageNo=0&pageIdx0&totalIdx=0&sup=1&d_pageSize=40&siTag=aEPqOr56EDcL1HyFlajr3w%7EfA9rXquZc5IkJpXC-Ycixw&d_headId=549281be67f4c28a83bd3d340965b270&d_ckId=549281be67f4c28a83bd3d340965b270&d_sfrom=search_fp&d_curPage=0&d_posi=0" target="_blank" onclick="tlog=window.tlog||[];tlog.push('c:w_sojob_jobclick_9')">
销售总监 </a>
</h3>
</div>
Selector:
Response.CssSelector(".job-info h3 a::attr(href)").Extract();
The above will get a IEnumerable<String> that only contains the href of the link in above SourceHTML
Response.CssSelector(".job-info h3 a::attr(href)").ExtractFirst();
The above code will get the first element as string from the CssSelector, here the value is
https://www.liepin.com/job/1913549323.shtml
::attr(text) will get the InnerText of the element
Wrap the Css selected node into a IResponse again so that we can process in chain
NScrapy can now get url from response through regex and HtmlAgilityPack.CssSelectors
By getting the url, we can again send request and write call back function to get more data
HttpHeaderMiddleware as a Default DownloaderMiddleware will set HttpRequest Header
based on the config in appsetting.json (AppSettings:HttpHeader section)
If specified node is not set in config file, Middleware will use its default value to fullfill the request header
Engine:
1.Configurable in appsetting.json to choice a real engine or a mock engine for test
2.Listen to Item queue to handle the request sent by Scheduler
Shell:
1.Init Context for NScrapy to run
1).Read Config
2.Get Spider through Spider Factory
Scheduler:
1.Send request to Queue which the engine is listening
Downloader
1.Init DownloaderPool
2.Asyncly Process all the incoming request by picking a Downloader from DownloaderPool
*Test Case update
EngineTest:Test if the message can be sent to MockEngine through Shell->Spider->Scheduler->Engine
ConstructDownloader:Test if DownloaderPool can work, if the Downloader can actually get response from Website
Elendil