HtmlAgilityPack does not support extract attr directly, made some enhencement to xPathSelector so it can now directly extract attribute of html element like this
/html/body/div/div/div/div/ul/li[i/b="企"]/div/div/h3/a@href
A big step for NScrayp, it can now utilize multiple Downloaders
Downloaders are now distributed to different process/machine, the connection
between Downloader and Spider is based on Scheduler, the scheduler distribute
Downloader task to different Downloader through Redis MQ
Details can be configed in appsetting.json
NScrapy still support single process mode, as long as we specify the run mode to InMemory
Make a wrap of Downloader named DownloaderShell, so that it can handle
request sent from Redis, which is actually pushed in to the specific Queue defined in appsetting.json
Not finished yet
this ItemLoader is more like a draft version of ItemLoader
It can do basic mappings from selector to object by defining mappings between selector and object field
later on will add pre processor, post processor while getting values from response and setting values to object
besides make a factory method to get ItemLoader, since for a particular Type, its ItemLoader barely changes, no need to new a ItemLoader for a same type everytime
By checking if there is anymore Pending Request/Response in Queue and Running Downloader
NScrapy.Crawl would exit if no more pending Request/Response/RunningDownloader
Also add test case for this mechanism
Fix the HttpHeaderMiddleware bug
Added UrlFilter with MD5 enhenced
Improved the sample Spider "JobSpider" so that it can now crawl all the .net postions posted at liepin.com
Added log4net at the bottom of the project, in NScrapyContext
Enhenced the Response.CssSelector so that we can select element based on a set of selector
This is very userful when the web page have tiny different between each page
we can now use ::attr(href) syntax to get values from html attribute
example:
SourceHTML:
<div class="job-info">
<h3 title="招聘销售总监">
<a href="https://www.liepin.com/job/1913549323.shtml" data-promid="imscid=R000000075&ckid=b6ab9f62c5abd657&headckid=b6ab9f62c5abd657&pageNo=0&pageIdx0&totalIdx=0&sup=1&d_pageSize=40&siTag=aEPqOr56EDcL1HyFlajr3w%7EfA9rXquZc5IkJpXC-Ycixw&d_headId=549281be67f4c28a83bd3d340965b270&d_ckId=549281be67f4c28a83bd3d340965b270&d_sfrom=search_fp&d_curPage=0&d_posi=0" target="_blank" onclick="tlog=window.tlog||[];tlog.push('c:w_sojob_jobclick_9')">
销售总监 </a>
</h3>
</div>
Selector:
Response.CssSelector(".job-info h3 a::attr(href)").Extract();
The above will get a IEnumerable<String> that only contains the href of the link in above SourceHTML
Response.CssSelector(".job-info h3 a::attr(href)").ExtractFirst();
The above code will get the first element as string from the CssSelector, here the value is
https://www.liepin.com/job/1913549323.shtml
::attr(text) will get the InnerText of the element
Elendil