GithubStar/NScrapy
1
0
Fork 0
mirror of https://github.com/xboxeer/NScrapy.git synced 2026-10-24 09:28:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
100 Commits 13 Branches 1 Tag
feature/fluent-api
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
zheng elendil 1d0743da36 Fix Spider base class null context issue and remove deprecated UnitTest1 2026-03-29 09:26:38 +08:00
zheng elendil eb15a3afb5 Add Fluent API unit tests and restore .NET 10.0 target 
- Changed NScrapy.Core target framework from net8.0 back to net10.0
- Added NScrapy.Core project reference to NScrapy.Test
- Created FluentApiTests.cs with the following test cases:
  - TestSpiderBuilder_CanCreateSpiderWithName
  - TestSpiderBuilder_BuildCreatesSpider
  - TestSpiderBuilder_SetsStartUrls
  - TestSpiderBuilder_AddsPipeline
  - TestDistributedBuilder_SetsRedisConfig
  - TestSpiderOptions_DefaultValues
  - TestDistributedBuilder_Chaining
  - TestSpiderBuilder_ConfiguresOptions
- Added helper classes TestItem and TestPipeline for testing pipelines
 2026-03-28 22:30:44 +08:00
zheng elendil af3343fa03 Fix Spider.cs: use config dict instead of reflection for RedisSchedulerContext 
- Replace broken reflection approach with direct config dictionary manipulation
- RedisSchedulerContext.Connect() reads config at runtime from NScrapyContext.CurrentContext.CurrentConfig
- Set config values BEFORE RedisScheduler singleton is created
- Also fix TargetFramework from net10.0 to net8.0
 2026-03-28 22:25:54 +08:00
zheng elendil cc95cf9ddc Add ResponseHandler override to satisfy abstract base class 2026-03-28 22:25:24 +08:00
zheng elendil 9f0210ccdc Add NScrapy.Core project with Fluent API architecture 
- Add new NScrapy.Core project with Fluent API for spider creation
- Add Fluent/NScrapy.cs - Static entry class with CreateSpider(name)
- Add Fluent/ISpiderBuilder.cs - Builder interface for fluent configuration
- Add Fluent/SpiderBuilder.cs - Builder implementation
- Add Fluent/IDistributedBuilder.cs - Distributed config interface
- Add Fluent/DistributedBuilder.cs - Distributed config implementation
- Add Fluent/ISpider.cs - New spider runtime interface
- Add Fluent/Spider.cs - Spider implementation inheriting from base Spider
- Add Fluent/SpiderOptions.cs - Configuration options class
- Add Middleware/JsRenderMiddleware.cs - Stub middleware (not implemented)
- Update solution file to include NScrapy.Core project
 2026-03-28 22:16:34 +08:00
Elendil 3cc56c36e1 Merge pull request #66 from xboxeer/feature/upgrade-dotnet-10 
Add NScrapy-framework-style LianjiaRentSpider demo
 2026-03-28 21:48:51 +08:00
Sam (OpenClaw) 47d31ce73a Add NScrapy-framework-style LianjiaRentSpider demo 
Rewrite demo following NScrapy framework conventions:
- LianjiaRentSpider inherits Spider.Spider with [Name]/[URL] attributes
- Override ResponseHandler to process HTTP responses
- Use NScrapy.GetInstance().Follow() to handle pagination
- Use ItemLoaderFactory + IPipeline for structured data output
- LianjiaRentCSVItemPipeline implements IPipeline<LianjiaRentItem>
- All config via appsetting.json (InMemory scheduler, middleware, pipelines)
- Regex-based field extraction from list page HTML
  (Lianjia detail pages require JS rendering)
 2026-03-28 21:42:35 +08:00
Elendil 730a6b0429 Merge pull request #65 from xboxeer/feature/upgrade-dotnet-10 
feat: Upgrade .NET Core 2.0 → .NET 10 + add LianjiaHouseSpider demo
 2026-03-28 21:12:11 +08:00
Sam (OpenClaw) 44a6df9804 Upgrade .NET Core 2.0 → .NET 10, add LianjiaHouseSpider demo 
## Major Changes

- Upgrade all 9 projects from netcoreapp2.0 to net10.0
- Update NuGet packages to latest compatible versions:
  - HtmlAgilityPack → 1.11.61
  - log4net → 2.0.17
  - Microsoft.Extensions.Configuration* → 8.0.0
  - Newtonsoft.Json → 13.0.3
  - StackExchange.Redis → 2.7.33
  - MongoDB.Driver → 2.25.0
  - Microsoft.NET.Test.Sdk → 17.9.0, MSTest → 3.2.0
- Remove ZooKeeperNetEx (no .NET 8/10 compatible version)
- Remove HtmlAgilityPack.CssSelectors.NetCore → replaced with Fizzler.Systems.HtmlAgilityPack
- Exclude ZooKeeper-dependent files: ZookeeperConfigProvider, ConfigProviderFactory, DefaultConfigProvider
- Add .gitignore
- Fix source code API changes from package upgrades (QuerySelectorAll, etc.)

## New Demo Project

- NScrapy.Demo: LianjiaHouseSpider - crawls Beijing rental listings
  - Runs in InMemory mode (no ZooKeeper/Redis required)
  - Outputs results to CSV
  - Demonstrates NScrapy framework usage
 2026-03-28 20:56:20 +08:00
Elendil ecbbaf2372 Merge pull request #34 from xboxeer/dependabot/nuget/NScrapy.Scheduler/log4net-2.0.10 
Bump log4net from 2.0.8 to 2.0.10 in /NScrapy.Scheduler
 2022-01-05 12:08:23 +08:00
Elendil c200bfa41c Merge pull request #35 from xboxeer/dependabot/nuget/NScrapy.Infra/log4net-2.0.10 
Bump log4net from 2.0.8 to 2.0.10 in /NScrapy.Infra
 2022-01-05 12:08:12 +08:00
dependabot[bot] a932be6795 Bump log4net from 2.0.8 to 2.0.10 in /NScrapy.Infra 
Bumps [log4net](https://github.com/apache/logging-log4net) from 2.0.8 to 2.0.10.
- [Release notes](https://github.com/apache/logging-log4net/releases)
- [Changelog](https://github.com/apache/logging-log4net/blob/master/ReleaseInstructions.txt)
- [Commits](https://github.com/apache/logging-log4net/compare/rel/2.0.8...rel/2.0.10)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-01-29 19:49:07 +00:00
dependabot[bot] d6bd68b4ca Bump log4net from 2.0.8 to 2.0.10 in /NScrapy.Scheduler 
Bumps [log4net](https://github.com/apache/logging-log4net) from 2.0.8 to 2.0.10.
- [Release notes](https://github.com/apache/logging-log4net/releases)
- [Changelog](https://github.com/apache/logging-log4net/blob/master/ReleaseInstructions.txt)
- [Commits](https://github.com/apache/logging-log4net/compare/rel/2.0.8...rel/2.0.10)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-01-29 19:49:03 +00:00
Elendil 5b4d09c534 Update README.md 
Fix typo
 2019-07-05 17:54:12 +08:00
Elendil d52d9784f0 Set up CI with Azure Pipelines 
[skip ci]
 2019-07-05 17:47:42 +08:00
Elendil ee6cd729c7 Update README.md 2018-09-14 00:11:30 +08:00
Elendil 40bce81838 Merge pull request #28 from xboxeer/ConfigManagement 
Config management
 2018-07-09 23:47:53 +08:00
Elendil 2703e6d174 refactor zookeeper config provider 
Also use a more elegent way to exit in memory Nscrapy Crawl mode
 2018-07-09 23:42:03 +08:00
Elendil 46086ffaf9 ConfigProvider update 
Create a ConifgProviderFactory so thay we can get ConfigProvider from a single point
Also enhenced the logic of refresh config, right now when zookeeper watcher dected the config is changed
NScrapy will update its context, and during that time, NScrapy will lock the access to the Connection of Redis
 2018-07-04 23:31:44 +08:00
Elendil 6a58d76d36 Merge pull request #26 from xboxeer/ConfigManagement 
Config management
 2018-07-04 00:13:59 +08:00
Elendil 5a0ff77c8c ZKConfigProvider v0.1 2018-07-04 00:11:40 +08:00
Elendil c98236cf7d Merge branch 'ConfigManagement' of github.com:xboxeer/NScrapy into ConfigManagement 2018-06-29 01:36:49 +08:00
Elendil e4351fa803 Optimized ZKHelper 
make create to be able to create node recursively from the path
 2018-06-29 01:35:26 +08:00
Elendil 6db9cc376a Merge pull request #25 from xboxeer/master 
Keep this latest brunch align with master
 2018-06-28 00:27:09 +08:00
Elendil 66275a5598 POC of Zookeeper .net Client 
also intergrate with ZookeeperConfigProvider
to be honest Zookeeper api is so hard to understand, why it is designed in such way?
 2018-06-28 00:24:12 +08:00
Elendil 723a51ea62 Init ConfigManagement Component 
Will use Zookeeper to manager the config for every node
Also changed some methed name in old design for the purpose of align naming practice
 2018-06-27 01:31:28 +08:00
Elendil 6f53104ecc Update README.md 2018-06-21 22:07:32 +08:00
Elendil ec787fd5e7 Update README.md 2018-06-21 18:16:35 +08:00
Elendil 4d2e127724 Update README.md 2018-06-21 18:16:15 +08:00
Elendil 4577536ea9 Update README.md 2018-06-20 22:26:16 +08:00
Elendil 135770a8ba Merge branch 'master' of github.com:xboxeer/NScrapy 2018-06-19 00:18:38 +08:00
Elendil 1c8d81f283 Add somemore details for StatusUpdateMiddleware 
The key point is LastUpdate, we can use this field to tell if the Downloader is still alive
In some cases, when Redis is doing some GC or the memory of the Host is fullfilled, DownloaderShell will exit unexcepted
 2018-06-19 00:18:01 +08:00
Elendil 82a9cc5d26 Update README.md 2018-06-12 22:02:09 +08:00
Elendil 9219f689ab Update README.md 2018-06-07 00:28:24 +08:00
Elendil 871b7a0a67 Update README.md 2018-06-07 00:26:45 +08:00
Elendil 121fc50bef Update README.md 2018-06-07 00:13:59 +08:00
Elendil f58f038afb Update README.md 2018-06-07 00:10:11 +08:00
Elendil 88db9026d3 Update README.md 2018-06-07 00:08:37 +08:00
Elendil 8aea43d7fd Merge branch 'master' of github.com:xboxeer/NScrapy v0.1 2018-06-05 22:47:23 +08:00
Elendil 8d471c72b0 Fix some bugs in sample project and increase timeout limit to 10sec while Spider is getting data from Redis 
The timeout should be configable in the future

Signed-off-by: Elendil <zy4182@sina.com>
 2018-06-05 22:46:31 +08:00
Elendil 621ac66efd Set theme jekyll-theme-cayman 2018-06-04 15:48:33 +08:00
Elendil 3622c13fea individual downloader shell will update its downloader status to Redis 
This is for the purpose of web console displaying distributed downloader status in the new future
 2018-05-31 23:42:04 +08:00
Elendil 2d7be4f77d enhence the RedisScheduler 
RedisScheduler will now register all the proper method in its registedCallback set at first, no need to wait until the callback is actually called
 2018-05-25 01:02:08 +08:00
Elendil 32a1bf82f1 make the call back as Async, and fix bugs for ItemLoader while we are calling callback function in async way 2018-05-24 00:20:23 +08:00
Elendil 20e0982e88 ItemLoader Bug fix,CssSelector enhencement, ItemPipeline added 2018-05-22 23:50:25 +08:00
Elendil 0e36fea14d Bug Fix of ItemLoader and Add a MongoDBPipeline as Pipeline sample 
Intergrate with MongoDB with MongoDB pipeline to store the information captured by NScrapy is AWESOME!
far more easy then Scrapy
 2018-05-17 22:31:44 +08:00
Elendil 2219a5f87d Enhenced the ItemPipeline Field Mapping Logic 
we can use Lambda to pick a property for mapping, instead of type the Field name manually
 2018-05-17 00:52:00 +08:00
Elendil 7c15c702c6 ItemPipeLine first version 
Enable ItemPipeline function by setting the below appsetting.json setting
"Pipelines": [
      { "Pipeline": "NScrapy.Infra.Pipeline.MockPipeline" }
    ],
there is also a mock test case for ItemPipeline
 2018-05-17 00:11:59 +08:00
Elendil 8dbd6a1e2a forget to delete unused class results to build failed in Jenkins 2018-05-16 21:24:23 +08:00
Elendil c3c2d6e56b ItemLoader support field mapping for xpath selector 2018-05-16 01:41:35 +08:00