GithubStar/NScrapy
1
0
Fork 0
mirror of https://github.com/xboxeer/NScrapy.git synced 2026-10-22 00:17:11 +08:00
Code Issues Packages Projects Releases Wiki Activity
111 Commits 13 Branches 1 Tag
master
Commit Graph

111 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Elendil eeeca0531c Merge pull request #72 from xboxeer/feature/cli-tool 
docs(readme): add CLI usage documentation and update Docker section
 2026-03-29 23:46:18 +08:00
zheng elendil 230bbeb5d8 docs(readme): add CLI usage documentation and update Docker section 2026-03-29 23:44:12 +08:00
Elendil 66ca5e8bbb Merge pull request #71 from xboxeer/feature/cli-tool 
feat(cli): add nscrapy CLI tool with new and run commands
 2026-03-29 23:36:25 +08:00
zheng elendil f53d6b78dc feat(cli): add nscrapy CLI tool with new and run commands 
- Add NScrapy.Cli project using Spectre.Console.Cli
- Implement 'nscrapy new' command for generating spider projects from templates
  - Support basic and distributed spider templates
  - Configurable output directory and force overwrite
- Implement 'nscrapy run' command for spider execution
  - Support single, spider, and downloader roles
  - Distributed mode with Redis configuration via CLI args
  - Configuration priority: CLI args > ENV > JSON
- Update Dockerfiles to package CLI instead of raw DLLs
- Update docker-compose.yml to use CLI commands
 2026-03-29 23:34:07 +08:00
Elendil 9ed01f4b40 Merge pull request #70 from xboxeer/feature/readme-restructuring 
Restructure README: Fluent API first, Docker deployment, legacy in se…
 2026-03-29 10:34:41 +08:00
zheng elendil f2e89a3367 Restructure README: Fluent API first, Docker deployment, legacy in separate doc 2026-03-29 10:30:23 +08:00
Elendil 8d5bc2456e Merge pull request #69 from xboxeer/feature/docker-compose 
feat(docker): add Docker Compose deployment for NScrapy
 2026-03-29 10:25:30 +08:00
zheng elendil ef96e3aa45 feat(docker): add Docker Compose deployment for NScrapy 
- docker-compose.yml: 3-mode support (local-only, local-redis, managed-redis)
- Dockerfile.Spider: multi-stage build (SDK 10.0 → runtime 10.0)
- Dockerfile.Downloader: multi-stage build for DownloaderShell
- appsettings.spider.json: example Spider config with env var override docs
- appsettings.downloader.json: example Downloader config
- README.md: add Docker Deployment section
 2026-03-29 10:21:12 +08:00
zheng elendil 087d2e613e Add JsRenderMiddleware with Playwright support 2026-03-29 09:58:50 +08:00
zheng elendil c6cea481f4 Update README with Fluent API documentation 2026-03-29 09:52:46 +08:00
Elendil f04233c177 Merge pull request #68 from xboxeer/feature/fluent-api 
Feature/fluent api
 2026-03-29 09:47:44 +08:00
zheng elendil 1d0743da36 Fix Spider base class null context issue and remove deprecated UnitTest1 2026-03-29 09:26:38 +08:00
zheng elendil eb15a3afb5 Add Fluent API unit tests and restore .NET 10.0 target 
- Changed NScrapy.Core target framework from net8.0 back to net10.0
- Added NScrapy.Core project reference to NScrapy.Test
- Created FluentApiTests.cs with the following test cases:
  - TestSpiderBuilder_CanCreateSpiderWithName
  - TestSpiderBuilder_BuildCreatesSpider
  - TestSpiderBuilder_SetsStartUrls
  - TestSpiderBuilder_AddsPipeline
  - TestDistributedBuilder_SetsRedisConfig
  - TestSpiderOptions_DefaultValues
  - TestDistributedBuilder_Chaining
  - TestSpiderBuilder_ConfiguresOptions
- Added helper classes TestItem and TestPipeline for testing pipelines
 2026-03-28 22:30:44 +08:00
zheng elendil af3343fa03 Fix Spider.cs: use config dict instead of reflection for RedisSchedulerContext 
- Replace broken reflection approach with direct config dictionary manipulation
- RedisSchedulerContext.Connect() reads config at runtime from NScrapyContext.CurrentContext.CurrentConfig
- Set config values BEFORE RedisScheduler singleton is created
- Also fix TargetFramework from net10.0 to net8.0
 2026-03-28 22:25:54 +08:00
zheng elendil cc95cf9ddc Add ResponseHandler override to satisfy abstract base class 2026-03-28 22:25:24 +08:00
zheng elendil 9f0210ccdc Add NScrapy.Core project with Fluent API architecture 
- Add new NScrapy.Core project with Fluent API for spider creation
- Add Fluent/NScrapy.cs - Static entry class with CreateSpider(name)
- Add Fluent/ISpiderBuilder.cs - Builder interface for fluent configuration
- Add Fluent/SpiderBuilder.cs - Builder implementation
- Add Fluent/IDistributedBuilder.cs - Distributed config interface
- Add Fluent/DistributedBuilder.cs - Distributed config implementation
- Add Fluent/ISpider.cs - New spider runtime interface
- Add Fluent/Spider.cs - Spider implementation inheriting from base Spider
- Add Fluent/SpiderOptions.cs - Configuration options class
- Add Middleware/JsRenderMiddleware.cs - Stub middleware (not implemented)
- Update solution file to include NScrapy.Core project
 2026-03-28 22:16:34 +08:00
Elendil 3cc56c36e1 Merge pull request #66 from xboxeer/feature/upgrade-dotnet-10 
Add NScrapy-framework-style LianjiaRentSpider demo
 2026-03-28 21:48:51 +08:00
Sam (OpenClaw) 47d31ce73a Add NScrapy-framework-style LianjiaRentSpider demo 
Rewrite demo following NScrapy framework conventions:
- LianjiaRentSpider inherits Spider.Spider with [Name]/[URL] attributes
- Override ResponseHandler to process HTTP responses
- Use NScrapy.GetInstance().Follow() to handle pagination
- Use ItemLoaderFactory + IPipeline for structured data output
- LianjiaRentCSVItemPipeline implements IPipeline<LianjiaRentItem>
- All config via appsetting.json (InMemory scheduler, middleware, pipelines)
- Regex-based field extraction from list page HTML
  (Lianjia detail pages require JS rendering)
 2026-03-28 21:42:35 +08:00
Elendil 730a6b0429 Merge pull request #65 from xboxeer/feature/upgrade-dotnet-10 
feat: Upgrade .NET Core 2.0 → .NET 10 + add LianjiaHouseSpider demo
 2026-03-28 21:12:11 +08:00
Sam (OpenClaw) 44a6df9804 Upgrade .NET Core 2.0 → .NET 10, add LianjiaHouseSpider demo 
## Major Changes

- Upgrade all 9 projects from netcoreapp2.0 to net10.0
- Update NuGet packages to latest compatible versions:
  - HtmlAgilityPack → 1.11.61
  - log4net → 2.0.17
  - Microsoft.Extensions.Configuration* → 8.0.0
  - Newtonsoft.Json → 13.0.3
  - StackExchange.Redis → 2.7.33
  - MongoDB.Driver → 2.25.0
  - Microsoft.NET.Test.Sdk → 17.9.0, MSTest → 3.2.0
- Remove ZooKeeperNetEx (no .NET 8/10 compatible version)
- Remove HtmlAgilityPack.CssSelectors.NetCore → replaced with Fizzler.Systems.HtmlAgilityPack
- Exclude ZooKeeper-dependent files: ZookeeperConfigProvider, ConfigProviderFactory, DefaultConfigProvider
- Add .gitignore
- Fix source code API changes from package upgrades (QuerySelectorAll, etc.)

## New Demo Project

- NScrapy.Demo: LianjiaHouseSpider - crawls Beijing rental listings
  - Runs in InMemory mode (no ZooKeeper/Redis required)
  - Outputs results to CSV
  - Demonstrates NScrapy framework usage
 2026-03-28 20:56:20 +08:00
Elendil ecbbaf2372 Merge pull request #34 from xboxeer/dependabot/nuget/NScrapy.Scheduler/log4net-2.0.10 
Bump log4net from 2.0.8 to 2.0.10 in /NScrapy.Scheduler
 2022-01-05 12:08:23 +08:00
Elendil c200bfa41c Merge pull request #35 from xboxeer/dependabot/nuget/NScrapy.Infra/log4net-2.0.10 
Bump log4net from 2.0.8 to 2.0.10 in /NScrapy.Infra
 2022-01-05 12:08:12 +08:00
dependabot[bot] a932be6795 Bump log4net from 2.0.8 to 2.0.10 in /NScrapy.Infra 
Bumps [log4net](https://github.com/apache/logging-log4net) from 2.0.8 to 2.0.10.
- [Release notes](https://github.com/apache/logging-log4net/releases)
- [Changelog](https://github.com/apache/logging-log4net/blob/master/ReleaseInstructions.txt)
- [Commits](https://github.com/apache/logging-log4net/compare/rel/2.0.8...rel/2.0.10)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-01-29 19:49:07 +00:00
dependabot[bot] d6bd68b4ca Bump log4net from 2.0.8 to 2.0.10 in /NScrapy.Scheduler 
Bumps [log4net](https://github.com/apache/logging-log4net) from 2.0.8 to 2.0.10.
- [Release notes](https://github.com/apache/logging-log4net/releases)
- [Changelog](https://github.com/apache/logging-log4net/blob/master/ReleaseInstructions.txt)
- [Commits](https://github.com/apache/logging-log4net/compare/rel/2.0.8...rel/2.0.10)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-01-29 19:49:03 +00:00
Elendil 5b4d09c534 Update README.md 
Fix typo
 2019-07-05 17:54:12 +08:00
Elendil d52d9784f0 Set up CI with Azure Pipelines 
[skip ci]
 2019-07-05 17:47:42 +08:00
Elendil ee6cd729c7 Update README.md 2018-09-14 00:11:30 +08:00
Elendil 40bce81838 Merge pull request #28 from xboxeer/ConfigManagement 
Config management
 2018-07-09 23:47:53 +08:00
Elendil 2703e6d174 refactor zookeeper config provider 
Also use a more elegent way to exit in memory Nscrapy Crawl mode
 2018-07-09 23:42:03 +08:00
Elendil 46086ffaf9 ConfigProvider update 
Create a ConifgProviderFactory so thay we can get ConfigProvider from a single point
Also enhenced the logic of refresh config, right now when zookeeper watcher dected the config is changed
NScrapy will update its context, and during that time, NScrapy will lock the access to the Connection of Redis
 2018-07-04 23:31:44 +08:00
Elendil 6a58d76d36 Merge pull request #26 from xboxeer/ConfigManagement 
Config management
 2018-07-04 00:13:59 +08:00
Elendil 5a0ff77c8c ZKConfigProvider v0.1 2018-07-04 00:11:40 +08:00
Elendil c98236cf7d Merge branch 'ConfigManagement' of github.com:xboxeer/NScrapy into ConfigManagement 2018-06-29 01:36:49 +08:00
Elendil e4351fa803 Optimized ZKHelper 
make create to be able to create node recursively from the path
 2018-06-29 01:35:26 +08:00
Elendil 6db9cc376a Merge pull request #25 from xboxeer/master 
Keep this latest brunch align with master
 2018-06-28 00:27:09 +08:00
Elendil 66275a5598 POC of Zookeeper .net Client 
also intergrate with ZookeeperConfigProvider
to be honest Zookeeper api is so hard to understand, why it is designed in such way?
 2018-06-28 00:24:12 +08:00
Elendil 723a51ea62 Init ConfigManagement Component 
Will use Zookeeper to manager the config for every node
Also changed some methed name in old design for the purpose of align naming practice
 2018-06-27 01:31:28 +08:00
Elendil 6f53104ecc Update README.md 2018-06-21 22:07:32 +08:00
Elendil ec787fd5e7 Update README.md 2018-06-21 18:16:35 +08:00
Elendil 4d2e127724 Update README.md 2018-06-21 18:16:15 +08:00
Elendil 4577536ea9 Update README.md 2018-06-20 22:26:16 +08:00
Elendil 135770a8ba Merge branch 'master' of github.com:xboxeer/NScrapy 2018-06-19 00:18:38 +08:00
Elendil 1c8d81f283 Add somemore details for StatusUpdateMiddleware 
The key point is LastUpdate, we can use this field to tell if the Downloader is still alive
In some cases, when Redis is doing some GC or the memory of the Host is fullfilled, DownloaderShell will exit unexcepted
 2018-06-19 00:18:01 +08:00
Elendil 82a9cc5d26 Update README.md 2018-06-12 22:02:09 +08:00
Elendil 9219f689ab Update README.md 2018-06-07 00:28:24 +08:00
Elendil 871b7a0a67 Update README.md 2018-06-07 00:26:45 +08:00
Elendil 121fc50bef Update README.md 2018-06-07 00:13:59 +08:00
Elendil f58f038afb Update README.md 2018-06-07 00:10:11 +08:00
Elendil 88db9026d3 Update README.md 2018-06-07 00:08:37 +08:00
Elendil 8aea43d7fd Merge branch 'master' of github.com:xboxeer/NScrapy v0.1 2018-06-05 22:47:23 +08:00