GithubStar/NScrapy
1
0
Fork 0
mirror of https://github.com/xboxeer/NScrapy.git synced 2026-10-18 21:57:48 +08:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • eeeca0531c Merge pull request #72 from xboxeer/feature/cli-tool master Elendil 2026-03-29 23:46:18 +08:00
  • 230bbeb5d8 docs(readme): add CLI usage documentation and update Docker section feature/cli-tool zheng elendil 2026-03-29 23:44:12 +08:00
  • 66ca5e8bbb Merge pull request #71 from xboxeer/feature/cli-tool Elendil 2026-03-29 23:36:25 +08:00
  • f53d6b78dc feat(cli): add nscrapy CLI tool with new and run commands zheng elendil 2026-03-29 23:34:07 +08:00
  • 9ed01f4b40 Merge pull request #70 from xboxeer/feature/readme-restructuring Elendil 2026-03-29 10:34:41 +08:00
  • f2e89a3367 Restructure README: Fluent API first, Docker deployment, legacy in separate doc feature/readme-restructuring zheng elendil 2026-03-29 10:30:23 +08:00
  • 8d5bc2456e Merge pull request #69 from xboxeer/feature/docker-compose Elendil 2026-03-29 10:25:30 +08:00
  • ef96e3aa45 feat(docker): add Docker Compose deployment for NScrapy feature/docker-compose zheng elendil 2026-03-29 10:21:12 +08:00
  • 087d2e613e Add JsRenderMiddleware with Playwright support zheng elendil 2026-03-29 09:58:50 +08:00
  • c6cea481f4 Update README with Fluent API documentation zheng elendil 2026-03-29 09:52:46 +08:00
  • f04233c177 Merge pull request #68 from xboxeer/feature/fluent-api Elendil 2026-03-29 09:47:44 +08:00
  • 1d0743da36 Fix Spider base class null context issue and remove deprecated UnitTest1 feature/fluent-api zheng elendil 2026-03-29 09:26:38 +08:00
  • eb15a3afb5 Add Fluent API unit tests and restore .NET 10.0 target zheng elendil 2026-03-28 22:30:44 +08:00
  • af3343fa03 Fix Spider.cs: use config dict instead of reflection for RedisSchedulerContext zheng elendil 2026-03-28 22:25:54 +08:00
  • cc95cf9ddc Add ResponseHandler override to satisfy abstract base class zheng elendil 2026-03-28 22:25:24 +08:00
  • 9f0210ccdc Add NScrapy.Core project with Fluent API architecture zheng elendil 2026-03-28 22:16:34 +08:00
  • 3cc56c36e1 Merge pull request #66 from xboxeer/feature/upgrade-dotnet-10 Elendil 2026-03-28 21:48:51 +08:00
  • 47d31ce73a Add NScrapy-framework-style LianjiaRentSpider demo feature/upgrade-dotnet-10 Sam (OpenClaw) 2026-03-28 21:42:35 +08:00
  • 730a6b0429 Merge pull request #65 from xboxeer/feature/upgrade-dotnet-10 Elendil 2026-03-28 21:12:11 +08:00
  • 44a6df9804 Upgrade .NET Core 2.0 → .NET 10, add LianjiaHouseSpider demo Sam (OpenClaw) 2026-03-28 20:56:20 +08:00
  • 6e410d02d7 Bump MongoDB.Driver from 2.4.0 to 2.19.0 in /NScrapy.Project dependabot/nuget/NScrapy.Project/MongoDB.Driver-2.19.0 dependabot[bot] 2023-03-04 00:31:57 +00:00
  • ae9669760a Bump Newtonsoft.Json from 11.0.2 to 13.0.2 in /NScrapy.Infra dependabot/nuget/NScrapy.Infra/Newtonsoft.Json-13.0.2 dependabot[bot] 2022-12-08 00:57:51 +00:00
  • d1a4765bcc Bump Newtonsoft.Json from 11.0.2 to 13.0.2 in /NScrapy.DownloaderShell dependabot/nuget/NScrapy.DownloaderShell/Newtonsoft.Json-13.0.2 dependabot[bot] 2022-12-08 00:53:31 +00:00
  • 2fe35bf3b6 Bump Newtonsoft.Json from 11.0.2 to 13.0.2 in /NScrapy.Scheduler dependabot/nuget/NScrapy.Scheduler/Newtonsoft.Json-13.0.2 dependabot[bot] 2022-12-08 00:52:39 +00:00
  • ecbbaf2372 Merge pull request #34 from xboxeer/dependabot/nuget/NScrapy.Scheduler/log4net-2.0.10 Elendil 2022-01-05 12:08:23 +08:00
  • c200bfa41c Merge pull request #35 from xboxeer/dependabot/nuget/NScrapy.Infra/log4net-2.0.10 Elendil 2022-01-05 12:08:12 +08:00
  • a932be6795 Bump log4net from 2.0.8 to 2.0.10 in /NScrapy.Infra dependabot[bot] 2021-01-29 19:49:07 +00:00
  • d6bd68b4ca Bump log4net from 2.0.8 to 2.0.10 in /NScrapy.Scheduler dependabot[bot] 2021-01-29 19:49:03 +00:00
  • 5b4d09c534 Update README.md Elendil 2019-07-05 17:54:12 +08:00
  • d52d9784f0 Set up CI with Azure Pipelines Elendil 2019-07-05 17:47:42 +08:00
  • 242b1b0372 Update _config.yml for Azure Pipelines azure-pipelines Elendil 2019-07-05 17:07:59 +08:00
  • ee6cd729c7 Update README.md Elendil 2018-09-14 00:11:30 +08:00
  • 85fc6f0776 defect fix ConfigManagement Elendil 2018-07-10 23:40:20 +08:00
  • 40bce81838 Merge pull request #28 from xboxeer/ConfigManagement Elendil 2018-07-09 23:47:53 +08:00
  • 2703e6d174 refactor zookeeper config provider Also use a more elegent way to exit in memory Nscrapy Crawl mode Elendil 2018-07-09 23:42:03 +08:00
  • 46086ffaf9 ConfigProvider update Create a ConifgProviderFactory so thay we can get ConfigProvider from a single point Also enhenced the logic of refresh config, right now when zookeeper watcher dected the config is changed NScrapy will update its context, and during that time, NScrapy will lock the access to the Connection of Redis Elendil 2018-07-04 23:31:44 +08:00
  • 6a58d76d36 Merge pull request #26 from xboxeer/ConfigManagement Elendil 2018-07-04 00:13:59 +08:00
  • 5a0ff77c8c ZKConfigProvider v0.1 Elendil 2018-07-04 00:11:40 +08:00
  • c98236cf7d Merge branch 'ConfigManagement' of github.com:xboxeer/NScrapy into ConfigManagement Elendil 2018-06-29 01:36:49 +08:00
  • e4351fa803 Optimized ZKHelper make create to be able to create node recursively from the path Elendil 2018-06-29 01:35:26 +08:00
  • 6db9cc376a Merge pull request #25 from xboxeer/master Elendil 2018-06-28 00:27:09 +08:00
  • 66275a5598 POC of Zookeeper .net Client also intergrate with ZookeeperConfigProvider to be honest Zookeeper api is so hard to understand, why it is designed in such way? Elendil 2018-06-28 00:24:12 +08:00
  • 723a51ea62 Init ConfigManagement Component Will use Zookeeper to manager the config for every node Also changed some methed name in old design for the purpose of align naming practice Elendil 2018-06-27 01:31:28 +08:00
  • 6f53104ecc Update README.md Elendil 2018-06-21 22:07:32 +08:00
  • ec787fd5e7 Update README.md Elendil 2018-06-21 18:16:35 +08:00
  • 4d2e127724 Update README.md Elendil 2018-06-21 18:16:15 +08:00
  • 4577536ea9 Update README.md DistributedVersion Elendil 2018-06-20 22:26:16 +08:00
  • 135770a8ba Merge branch 'master' of github.com:xboxeer/NScrapy Elendil 2018-06-19 00:18:38 +08:00
  • 1c8d81f283 Add somemore details for StatusUpdateMiddleware The key point is LastUpdate, we can use this field to tell if the Downloader is still alive In some cases, when Redis is doing some GC or the memory of the Host is fullfilled, DownloaderShell will exit unexcepted Elendil 2018-06-19 00:18:01 +08:00
  • 82a9cc5d26 Update README.md Elendil 2018-06-12 22:02:09 +08:00
  • 9219f689ab Update README.md Elendil 2018-06-07 00:28:24 +08:00
  • 871b7a0a67 Update README.md Elendil 2018-06-07 00:26:45 +08:00
  • 121fc50bef Update README.md Elendil 2018-06-07 00:13:59 +08:00
  • f58f038afb Update README.md Elendil 2018-06-07 00:10:11 +08:00
  • 88db9026d3 Update README.md Elendil 2018-06-07 00:08:37 +08:00
  • 8aea43d7fd Merge branch 'master' of github.com:xboxeer/NScrapy v0.1 Elendil 2018-06-05 22:47:23 +08:00
  • 8d471c72b0 Fix some bugs in sample project and increase timeout limit to 10sec while Spider is getting data from Redis The timeout should be configable in the future Elendil 2018-06-05 22:46:31 +08:00
  • 621ac66efd Set theme jekyll-theme-cayman Elendil 2018-06-04 15:48:33 +08:00
  • 3622c13fea individual downloader shell will update its downloader status to Redis This is for the purpose of web console displaying distributed downloader status in the new future Elendil 2018-05-31 23:42:04 +08:00
  • 2d7be4f77d enhence the RedisScheduler RedisScheduler will now register all the proper method in its registedCallback set at first, no need to wait until the callback is actually called Elendil 2018-05-25 01:02:08 +08:00
  • 32a1bf82f1 make the call back as Async, and fix bugs for ItemLoader while we are calling callback function in async way Elendil 2018-05-24 00:20:23 +08:00
  • 20e0982e88 ItemLoader Bug fix,CssSelector enhencement, ItemPipeline added Elendil 2018-05-22 23:50:25 +08:00
  • 0e36fea14d Bug Fix of ItemLoader and Add a MongoDBPipeline as Pipeline sample Intergrate with MongoDB with MongoDB pipeline to store the information captured by NScrapy is AWESOME! far more easy then Scrapy Elendil 2018-05-17 22:31:44 +08:00
  • 2219a5f87d Enhenced the ItemPipeline Field Mapping Logic we can use Lambda to pick a property for mapping, instead of type the Field name manually Elendil 2018-05-17 00:52:00 +08:00
  • 7c15c702c6 ItemPipeLine first version Enable ItemPipeline function by setting the below appsetting.json setting "Pipelines": [ { "Pipeline": "NScrapy.Infra.Pipeline.MockPipeline" } ], there is also a mock test case for ItemPipeline Elendil 2018-05-17 00:11:59 +08:00
  • 8dbd6a1e2a forget to delete unused class results to build failed in Jenkins Elendil 2018-05-16 21:24:23 +08:00
  • c3c2d6e56b ItemLoader support field mapping for xpath selector Elendil 2018-05-16 01:41:35 +08:00
  • f92343e0d3 Merge pull request #16 from xboxeer/DistributedVersion Elendil 2018-05-16 01:33:58 +08:00
  • 4afa454ebb Add support for xPathSelector HtmlAgilityPack does not support extract attr directly, made some enhencement to xPathSelector so it can now directly extract attribute of html element like this /html/body/div/div/div/div/ul/li[i/b="企"]/div/div/h3/a@href Elendil 2018-05-16 01:32:47 +08:00
  • 1be0823f08 Update README.md Elendil 2018-05-13 12:37:52 +08:00
  • 3d7afe7212 Update README.md Elendil 2018-05-11 18:44:20 +08:00
  • a38063293d Update README.md Elendil 2018-05-11 14:02:47 +08:00
  • 7b02f1e840 Merge pull request #15 from xboxeer/DistributedVersion Elendil 2018-05-11 01:13:36 +08:00
  • 1c018a39a4 Bug Fix and Add HttpPost,cookies Support Elendil 2018-05-11 01:12:33 +08:00
  • 245aefa502 bug fix Elendil 2018-05-10 21:33:54 +08:00
  • 4e61597193 Optimize the Lock/Release structure release the lock as long as we fetched the item, instead of waiting for the call back finish Elendil 2018-05-10 21:31:07 +08:00
  • 00f90d9492 Change the response from Topic to Queue so that multiple Spider won't process the same response duplicate Add lock to response pick logic Elendil 2018-05-10 02:04:54 +08:00
  • 267fef7b79 Merge branch 'DistributedVersion' of github.com:xboxeer/NScrapy into DistributedVersion Elendil 2018-05-09 23:01:42 +08:00
  • 39723a6822 Fix bug that spider in different platform unable to grap together The issue is each request is bind to a callback fingerprint before, in different platform say Mac and Windows, the callback fingerprint for the same method is different beacuse they have different MSIL stream so this time in registered call back list of each spider, using callback name as key instead of callback body msil steam MD5 Elendil 2018-05-09 23:00:31 +08:00
  • cb0804a049 Fix bug that spider in different platform unable to grap together The issue is each request is bind to a callback fingerprint before, in different platform say Mac and Windows, the callback fingerprint for the same method is different beacuse they have different MSIL stream so this time in registered call back list of each spider, using callback name as key instead of callback body msil steam MD5 Elendil 2018-05-09 23:00:31 +08:00
  • c73765d4e4 Redis Lock Added Add a locker before getting url from receiverqueue, so that multiple downloader won't confilct Elendil 2018-05-09 22:45:49 +08:00
  • 542393b03f Update README.md Elendil 2018-05-09 16:55:11 +08:00
  • 7f365aabe5 Merge branch 'DistributedVersion' of github.com:xboxeer/NScrapy into DistributedVersion Elendil 2018-05-09 01:32:07 +08:00
  • 2ed341c009 Add Distributed Support for Spider Still need some investigate against that if the spider will crawl the same page while there are multiple spider running Elendil 2018-05-09 01:30:43 +08:00
  • ea36e6b5d9 Merge pull request #9 from xboxeer/DistributedVersion Elendil 2018-05-06 23:02:07 +08:00
  • 9ecf506323 Merge pull request #10 from xboxeer/master Elendil 2018-05-06 23:00:50 +08:00
  • a47b109a2e Distributed Downloader Version 0.1 A big step for NScrayp, it can now utilize multiple Downloaders Downloaders are now distributed to different process/machine, the connection between Downloader and Spider is based on Scheduler, the scheduler distribute Downloader task to different Downloader through Redis MQ Details can be configed in appsetting.json NScrapy still support single process mode, as long as we specify the run mode to InMemory Elendil 2018-05-06 22:57:39 +08:00
  • f7514b07cb Json Serialize Response from Downloader and Compress the Serialized Json to GZip After Compressed the Response, tie the compressed str with url then put that pair in Specific Topic Elendil 2018-05-04 01:27:53 +08:00
  • 46cf06b664 Create DownloaderShell Project to make Downloader Distributeable Make a wrap of Downloader named DownloaderShell, so that it can handle request sent from Redis, which is actually pushed in to the specific Queue defined in appsetting.json Elendil 2018-05-04 00:26:46 +08:00
  • 2850fbbfd7 Update README.md Elendil 2018-05-03 00:50:23 +08:00
  • aa2cd8c5ba New Branch of Distributed Spider using Redis as messagequeue service to distribute request to different distributed Downloader the first commit of this branch finishes the Config part of RedisScheduler Rewrite the Scheduler, now it is called InMemoryScheduler IOC to get the Actual Scheduler from appsettings.json Fix a bug of refresh the config file Elendil 2018-05-03 00:18:38 +08:00
  • ebc8a738ef Created the very first verion of ItemLoader this ItemLoader is more like a draft version of ItemLoader It can do basic mappings from selector to object by defining mappings between selector and object field later on will add pre processor, post processor while getting values from response and setting values to object besides make a factory method to get ItemLoader, since for a particular Type, its ItemLoader barely changes, no need to new a ItemLoader for a same type everytime Elendil 2018-05-01 23:43:38 +08:00
  • d36d75558e Add mechanism to Exit a Crawl of a Spider and a Test Case for it By checking if there is anymore Pending Request/Response in Queue and Running Downloader NScrapy.Crawl would exit if no more pending Request/Response/RunningDownloader Elendil 2018-05-01 21:38:42 +08:00
  • 290d150b14 Bug Fix and UrlFilter Added Fix the HttpHeaderMiddleware bug Added UrlFilter with MD5 enhenced Improved the sample Spider "JobSpider" so that it can now crawl all the .net postions posted at liepin.com Elendil 2018-04-28 01:17:08 +08:00
  • 572f86634c Commit suo file Elendil 2018-04-26 01:44:46 +08:00
  • d299ff60f3 Bug Fix and Log Function Added log4net at the bottom of the project, in NScrapyContext Enhenced the Response.CssSelector so that we can select element based on a set of selector This is very userful when the web page have tiny different between each page Elendil 2018-04-26 01:44:14 +08:00
  • 27f63d5040 Bug fix and add Test case for DownloaderMiddleware Bug fix of DownloaderMiddleware loading Add Test case of HttpUserAgentPoolMiddleware Elendil 2018-04-23 22:41:50 +08:00
  • 2f3aa3b5f3 Merge branch 'master' of github.com:xboxeer/NScrapy Elendil 2018-04-23 20:23:47 +08:00
  • 941750c5dc * HttpHeaderMiddleware.cs: Add HttpUserAgentPool middleware Elendil 2018-04-23 20:22:24 +08:00
  • 5b1d0fa549 Update README.md Elendil 2018-04-20 15:30:49 +08:00