Commit Graph

Select branches

Hide Pull Requests

ConfigManagement

DistributedVersion

azure-pipelines

dependabot/nuget/NScrapy.DownloaderShell/Newtonsoft.Json-13.0.2

dependabot/nuget/NScrapy.Infra/Newtonsoft.Json-13.0.2

dependabot/nuget/NScrapy.Project/MongoDB.Driver-2.19.0

dependabot/nuget/NScrapy.Scheduler/Newtonsoft.Json-13.0.2

feature/cli-tool

feature/docker-compose

feature/fluent-api

feature/readme-restructuring

feature/upgrade-dotnet-10

master

#10

#15

#16

#22

#25

#26

#28

#33

#33

#34

#35

#58

#59

#60

#61

#61

#62

#62

#63

#63

#64

#64

#65

#66

#67

#68

#69

#70

#71

#72

#9

v0.1

Mono Color

• f021117b89 Update README.md Elendil 2018-04-20 01:27:34 +08:00
• af5edcb955 Merge branch 'master' of github.com:xboxeer/NScrapy Elendil 2018-04-20 01:24:11 +08:00
• 2d9542a252 Add advanced CssSelector we can now use ::attr(href) syntax to get values from html attribute example: SourceHTML: <div class="job-info"> <h3 title="招聘销售总监"> <a href="https://www.liepin.com/job/1913549323.shtml" data-promid="imscid=R000000075&amp;ckid=b6ab9f62c5abd657&amp;headckid=b6ab9f62c5abd657&amp;pageNo=0&amp;pageIdx0&amp;totalIdx=0&amp;sup=1&amp;d_pageSize=40&amp;siTag=aEPqOr56EDcL1HyFlajr3w%7EfA9rXquZc5IkJpXC-Ycixw&amp;d_headId=549281be67f4c28a83bd3d340965b270&amp;d_ckId=549281be67f4c28a83bd3d340965b270&amp;d_sfrom=search_fp&amp;d_curPage=0&amp;d_posi=0" target="_blank" onclick="tlog=window.tlog||[];tlog.push('c:w_sojob_jobclick_9')"> 销售总监 </a> </h3> </div> Elendil 2018-04-20 01:23:22 +08:00
• 9f51b28516 Create README.md Elendil 2018-04-19 02:08:09 +08:00
• 307dad4622 Delete README.md Elendil 2018-04-19 02:04:51 +08:00
• 400460530e Update README.md Elendil 2018-04-19 02:01:47 +08:00
• d25499879c Intergarated with HtmlAgilityPack.CssSelectors Wrap the Css selected node into a IResponse again so that we can process in chain NScrapy can now get url from response through regex and HtmlAgilityPack.CssSelectors By getting the url, we can again send request and write call back function to get more data Elendil 2018-04-19 01:58:58 +08:00
• 120b6173a1 corrected one typo Elendil 2018-04-18 22:16:03 +08:00
• 05368bee59 Optimized the Structure of the Entire Project Involve HtmlAgilityPack and HtmlAgilityPack.CssSelector.NetCore in Infra To help Getting element through CSS Selector Elendil 2018-04-18 22:12:37 +08:00
• aaaefc49e9 Add HttpCompressionMiddleware to Infra in case the returned Content is Compressed Elendil 2018-04-15 22:09:38 +08:00
• adc1044009 Add DownloaderMiddleware process logic and added a default DownloaderMiddleware HttpHeaderMiddleware as a Default DownloaderMiddleware will set HttpRequest Header based on the config in appsetting.json (AppSettings:HttpHeader section) If specified node is not set in config file, Middleware will use its default value to fullfill the request header Elendil 2018-04-13 01:12:15 +08:00
• 18c4dca3ab Created base structure of NScrapy Shell, Engine, Scheduler, Downloader Engine: 1.Configurable in appsetting.json to choice a real engine or a mock engine for test 2.Listen to Item queue to handle the request sent by Scheduler Shell: 1.Init Context for NScrapy to run 1).Read Config 2.Get Spider through Spider Factory Scheduler: 1.Send request to Queue which the engine is listening Downloader 1.Init DownloaderPool 2.Asyncly Process all the incoming request by picking a Downloader from DownloaderPool *Test Case update EngineTest:Test if the message can be sent to MockEngine through Shell->Spider->Scheduler->Engine ConstructDownloader:Test if DownloaderPool can work, if the Downloader can actually get response from Website Elendil 2018-04-12 00:27:39 +08:00
• 4096ac1abf Framework setup Elendil 2018-04-09 01:03:14 +08:00
• fb3dbbbe0a rename shell name class to scrapy Elendil 2018-04-05 01:47:12 +08:00
• 5a88d438ea Merge branch 'master' of github.com:xboxeer/NScrapy Elendil 2018-04-05 00:29:44 +08:00
• 318ae792bd Init checkin of NScrapy Elendil 2018-04-05 00:20:49 +08:00
• 10be9aec50 Initial commit Elendil 2018-04-05 00:10:30 +08:00