refactor: integrate BGP classifier into main repository

.github/workflows/build.yml

@@ -29,6 +29,7 @@ jobs:
           restore-keys: ${{ runner.os }}-cargo-bin-
           path: ~/.cargo/bin
       - run: just dependency
+      - run: cargo test --locked
       - uses: nick-fields/retry@v3
         with:
           timeout_minutes: 60

.gitignore

@@ -6,4 +6,5 @@ rib6.*
 rib-*
 result/
 analysis/
+/target/
 *.bin

Cargo.toml

@@ -0,0 +1,21 @@
+[package]
+name = "china-operator-ip"
+version = "0.1.0"
+edition = "2024"
+authors = ["Yifan Gao <git@yfgao.com>"]
+description = "Generate China operator IP lists from BGP RIB data"
+repository = "https://github.com/gaoyifan/china-operator-ip"
+license = "MIT"
+publish = false
+
+[dependencies]
+clap = { version = "4.5", features = ["derive"] }
+bgpkit-parser = "0.12"
+iprange = { version = "0.6.7", features = ["serde"] }
+ipnet = { version = "2.11.0", features = ["serde"] }
+prefix-trie = { version = "0.8", features = ["ipnet"] }
+vec-collections = "0.4.3"
+serde = { version = "1", features = ["derive"] }
+bincode = "1"
+rayon = "1.10"
+smallvec = "1.13"

README.md

@@ -79,7 +79,6 @@ git clone -b ip-lists https://github.com/gaoyifan/china-operator-ip.git
 * [just](https://github.com/casey/just?tab=readme-ov-file#installation)
 * [Rust Toolchain](https://www.rust-lang.org/tools/install)
 * [bgpkit-broker](https://github.com/bgpkit/bgpkit-broker) (`cargo install bgpkit-broker@0.7.0`)
-* [bgptools](https://github.com/gaoyifan/bgptools) (`cargo install bgptools@0.3.4`)
 * [aria2](https://github.com/aria2/aria2)
 * [Ruby](https://www.ruby-lang.org)
 
@@ -105,7 +104,7 @@ just
 just trusted_transit_asn china
 ```
 
-该规则依赖支持 `--trusted-cn-transit-file` 的 `bgptools` 版本，并直接读取本地 `rib-*` 快照。
+BGP 分类程序的 Rust 源码位于仓库顶层的 `src/`，执行 `just dependency` 会在当前仓库构建。它直接读取本地 `rib-*` 快照，与生成规则在同一个提交和 CI 中同步修改、验证。
 
 ## 社区关联项目
 

docs/algorithm.md

@@ -0,0 +1,55 @@
+## Algorithm Overview
+
+The built-in BGP classifier reads one or more MRT/RIB files and outputs the IPv4/IPv6 prefixes originated by the ASNs provided on the command line.
+
+- Inputs: MRT file paths (`--mrt-file`), target ASNs (positional), `--ignore-private-asn`, `--origin-only`, `--cache`, an optional low-priority CIDR fallback list (`--fallback-prefix-file`), an optional foreign-upstream filter (`--exclude-foreign-upstream-only` + `--asn-country-file`), and an optional trusted-CN-transit filter (`--trusted-cn-transit-file` + `--asn-country-file`, requires `--origin-only`).
+- Output: Simplified, sorted list of CIDR prefixes (v4 then v6) for the requested ASNs.
+
+## Processing Steps
+
+1) **Parse MRT files (parallel)**
+   Each MRT file is parsed with `BgpkitParser`, keeping only ANNOUNCE records. For every prefix the code collects:
+   - Origin ASNs (skipping private ASNs when requested).
+   - AS path (truncated to last 4 hops).
+   - Split points: prefix network address and the next address after the broadcast. These points mark boundaries for later interval construction.
+   Results are stored separately for v4 and v6:
+   - `announced_*`: every non-default announced prefix, collected before origin, private-ASN, country, or transit filtering. IPv4/IPv6 default routes are excluded because they do not establish a globally routed address assignment.
+   - `prefix_map_*`: longest-prefix-match map of prefix → set of origin ASNs.
+   - `as_paths_*`: prefix → origin ASN → list of truncated AS paths.
+   - `direct_upstreams`: origin ASN → set of observed direct upstream ASNs.
+   - `split_points_*`: ordered set of addresses that delimit intervals.
+
+2) **Merge per-file data**
+   Parsed structures are merged across files. Split points are deduped and sorted (via `BTreeSet`).
+
+3) **Optional trusted-CN-transit filtering**
+   With `--origin-only`, inspect every observed AS path from the origin side. An origin ASN is retained, along with all of its prefixes, when at least one path has a contiguous CN suffix containing an ASN from `--trusted-cn-transit-file`. Inspection stops at the first non-CN or unknown ASN, so a trusted CN network beyond foreign transit does not make the origin domestic. The CLI rejects this filter without `--origin-only` because shared-upstream attribution is a different mode.
+
+4) **Add shared upstream ASNs**
+   Unless `--origin-only` is set, the algorithm computes the longest common suffix of the collected AS paths (capped to 4 ASNs). These shared upstream ASNs are added to the prefix map so they are treated like origin ASNs for interval attribution.
+
+5) **Build ASN → IP ranges**
+   Consecutive split points define half-open intervals `[start, end)`. For each interval, a /32 (v4) or /128 (v6) lookup finds the longest covering prefix and its ASNs. Each ASN receives the interval, converted to a minimal set of CIDRs via `interval_to_cidrs_v4/v6`. The per-AS ranges are stored as `IpRange` structures to allow merging.
+
+6) **Optional foreign-upstream filtering**
+   When `--exclude-foreign-upstream-only <COUNTRY>` is enabled, the classifier loads ASN → country data from `--asn-country-file` and removes any requested ASN whose observed direct upstream ASNs are all known and all outside `<COUNTRY>`. A hidden debug flag can print this matched ASN list directly.
+
+7) **Apply registration fallbacks**
+   Each CIDR from `--fallback-prefix-file` contributes only its difference from the complete observed announcement set. This gives BGP higher priority than fallback data: `result = classified ∪ (fallback − announced)`. Any announcement, regardless of ASN or classification, blocks fallback coverage for that space.
+
+8) **Finalize result**
+   For the remaining requested ASNs, the collected ranges and unannounced fallbacks are merged and simplified, then emitted in sorted order (v4 then v6).
+
+## Caching
+
+When `--cache` is enabled, the computed ASN→range maps, complete observed announcement sets, and origin→direct-upstream map are serialized to a bincode file keyed by input file list, `ignore_private_asn`, `origin_only`, and the trusted-transit policy fingerprint. The fallback file is read after cache loading and therefore does not affect the key. Subsequent runs reuse the cache when the key matches.
+
+## Key Functions (in `src/main.rs`)
+
+- `process_mrt_file`: Parses one MRT file and extracts prefix/ASN/path data plus split points.
+- `longest_common_suffix`: Finds shared tail of AS paths (≤4 hops).
+- `interval_to_cidrs_v4/v6`: Converts `[start, end)` intervals to minimal CIDR cover.
+- `foreign_upstream_only_asns`: Computes which target ASNs have only foreign direct upstreams.
+- `apply_fallback_prefixes`: Adds only the unannounced part of configured IPv4/IPv6 fallback ranges.
+- `has_domestic_suffix`: Checks whether an AS path has a contiguous CN origin-side suffix containing trusted transit.
+- `build_asn_data`: Orchestrates merging, shared-upstream attribution, interval slicing, and ASN range construction.

docs/issue-45-fallback-prefixes.md

@@ -55,7 +55,7 @@ china:
 
 生成器拥有 RIR 外部格式的实现，负责下载五个 delegated 文件、筛选记录、转换非 CIDR IPv4 范围，并生成可审计的 `result/.china.registered.txt`。
 
-`bgptools` 不理解 RIR 格式，只通过一个稳定的通用接口读取规范 CIDR：
+内置的 BGP 分类器不理解 RIR 格式，只通过一个稳定的通用接口读取规范 CIDR：
 
 ```text
 --fallback-prefix-file result/.china.registered.txt
@@ -72,7 +72,7 @@ china:
 
 ## 验证
 
-`bgptools` 行为测试覆盖：
+BGP 分类器的行为测试覆盖：
 
 1. 已分类半段与未宣告半段能合并为完整 fallback；
 2. fallback 内已宣告的更具体前缀保持为空洞；
@@ -89,11 +89,8 @@ china:
 - `china` 的集合差仅包含登记为 CN 且未宣告的地址；
 - Issue #45 的结果包含 `121.46.0.0/18`，具体运营商集合不因此扩张。
 
-## 发布顺序
+## 代码归属
 
-该改动跨越 `bgptools` 和本仓库，按以下顺序发布：
+BGP 分类器只有本项目一个使用者，其 Rust crate 已直接合并到仓库顶层的 `Cargo.toml`、`Cargo.lock` 和 `src/`，由 `just dependency` 从当前仓库构建。生成规则和 BGP 集合算法在同一个提交与 CI 中修改、验证，不再依赖跨仓库发布顺序。
 
-1. 合并并发布 `bgptools 0.3.4`；
-2. 本仓库固定并使用 `0.3.4`，同时启用 RIR 数据准备和文件透传；
-3. 在 pull request 构建中检查完整集合差和隐藏登记输入；
-4. 合并后手动触发一次 `update_ip_lists`，检查产物再恢复每日自动更新。
+crates.io 上旧的 `bgptools 0.3.4` 是最后一个独立发布版本，仅为已有安装保留；当前顶层 crate 名为 `china-operator-ip` 并设置了 `publish = false`。旧 crate 后续不再更新，原独立 GitHub 仓库已归档。

justfile

@@ -1,5 +1,4 @@
 set unstable
-bgptools_version := "0.3.4"
 
 default: prepare all stat
 
@@ -8,15 +7,13 @@ dependency:
   #!/usr/bin/env bash
   set -euo pipefail
 
-  if ! bgptools --version 2>/dev/null | grep -F "{{bgptools_version}}" >/dev/null; then
-    cargo install --force --version "{{bgptools_version}}" bgptools
-  fi
+  cargo build --release --locked
 
   if ! bgpkit-broker --version >/dev/null 2>&1; then
     cargo binstall --secure --no-confirm bgpkit-broker@0.7.0
   fi
 
-  bgptools --version
+  target/release/china-operator-ip --version
   bgpkit-broker --version
 
 # Download and normalize latest autnums list
@@ -73,7 +70,7 @@ prepare_rib collector:
   rm -f "${outfile}"
   aria2c -s 4 -x 4 -q -o "${outfile}" "${url}"
   stat "${outfile}"
-  echo "INFO> ${outfile} ready for bgptools" >&2
+  echo "INFO> ${outfile} ready for BGP classification" >&2
 
 # Download the latest RIB snapshots (rrc21, rrc12, route-views6)
 [parallel]
@@ -201,33 +198,33 @@ gen operator:
 
   ribs = Dir["rib-*.{gz,bz2}"].sort
   abort("No rib-*.gz or rib-*.bz2 files found. Run 'just prepare_ribs' first.") if ribs.empty?
-  bgptools = ["bgptools", "--ignore-private-asn", "--cache"]
-  bgptools << "--origin-only" if origin_only
+  classifier = ["target/release/china-operator-ip", "--ignore-private-asn", "--cache"]
+  classifier << "--origin-only" if origin_only
   if country = cfg["registry_fallback_country"]
     registered = IO.popen(["just", "registry_prefixes", country], &:read)
     abort("Failed to get registered prefixes for #{country}") unless $?.success?
     registered_path = "result/.#{operator}.registered.txt"
     File.write(registered_path, registered)
-    bgptools += ["--fallback-prefix-file", registered_path]
+    classifier += ["--fallback-prefix-file", registered_path]
   end
   if cfg.fetch("trusted_transit_operators", []).any?
     trusted_transit = IO.popen(["just", "trusted_transit_asn", operator], &:read)
     abort("Failed to get trusted transit ASNs for #{operator}") unless $?.success?
     trusted_transit_path = "result/.#{operator}.trusted-transit.txt"
     File.write(trusted_transit_path, trusted_transit)
-    bgptools += [
+    classifier += [
       "--trusted-cn-transit-file",
       trusted_transit_path,
       "--asn-country-file",
       "asnames.txt",
     ]
   end
-  bgptools += ribs.flat_map { |r| ["--mrt-file", r] }
+  classifier += ribs.flat_map { |r| ["--mrt-file", r] }
 
   warn "INFO> #{operator} start"
   asns = IO.popen(["just", "get_asn_candidates", operator], &:read)
   abort("Failed to get ASN list for #{operator}") unless $?.success?
-  abort("Failed to run bgptools for #{operator}") unless system(*bgptools, *asns.split, out: out)
+  abort("Failed to run BGP classifier for #{operator}") unless system(*classifier, *asns.split, out: out)
 
   v6_lines, v4_lines = File.readlines(out).partition { |line| line.include?(":") }
   File.write(v4, v4_lines.join)