mirror of
https://github.com/Tencent/flare.git
synced 2026-10-17 04:41:22 +08:00
master
21 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 CHEN Feng
|
8396477144 |
ci: add ASan/TSan fiber regression lane (#134, #196) (#198)
* ci: add ASan/TSan fiber regression lane (#134, #196) The fiber-switch sanitizer fixes (#134 ASan/TSan SEGV, #196 detect_stack_use_after_return) had no automated coverage -- flare's CI is blade-based and never built under sanitizers, so these bugs sat undetected. Add a Bazel + clang job that builds and runs a curated set of fiber tests under both `--config=asan` and `--config=tsan` (matrix). Also add a `tsan` config to .bazelrc (mirroring the existing `asan` one). Scope: a curated list of fiber tests that pass cleanly under both sanitizers, which is enough to catch any regression of the fiber-switch fixes. The full `//flare/fiber/...` is intentionally NOT used -- it has unrelated sanitizer problems out of scope here: curl fails to foreign_cc-build under -fsanitize; a few sync-primitive tests (scheduling_group/latch/waitable) have their own ASan/TSan failures; and some tests exceed even a raised timeout under TSan. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: download bazelisk to a temp path (repo has a bazel/ dir) The install step did `curl -o bazel`, but flare tracks a `bazel/` directory, so curl failed with exit 23 (can't write over a directory) before any test ran. Download to $RUNNER_TEMP instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: narrow sanitizer_fiber to a fast core set, tighten timeout The 15-test set was fine under ASan (~4 min) but pathologically slow under TSan on shared 2-4 core runners (the heavier sync-primitive tests). Narrow to a focused set that exercises fiber create/resume/switch -- enough to guard the switch fixes, since a broken switch crashes every fiber test on first resume. Drop --test_timeout to 300s so a straggler fails fast instead of stalling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: pin sanitizer_fiber to clang-17 (clang-18 TSan header clash) flare's annotation.h hand-declares __tsan_* interface symbols (mutex flags, fiber no_sync) that collide with clang-18's <sanitizer/tsan_interface.h> ('non-static declaration follows static declaration'), breaking the TSan build. That's a separate pre-existing issue; pin the lane to clang-17 (the toolchain the fiber-switch fixes were verified on) so the regression guard is green. clang-18 TSan-header compatibility tracked separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: force clang-17 via bazel --repo_env (update-alternatives ignored) The previous attempt used update-alternatives to point /usr/bin/clang at clang-17, but bazel's autodetected toolchain still resolved clang-18 (/usr/lib/llvm-18/bin/clang) from the runner's PATH, so the TSan build kept hitting the clang-18 tsan_interface.h clash. Configure the toolchain compiler explicitly with --repo_env=CC=/usr/bin/clang-17 (and CXX), and drop --config=llvm (which forced the generic 'clang'). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: make clang-17 install non-interactive (apt hang) A TSan-leg runner got stuck in the apt install step (the ASan leg's identical step passed) -- a needrestart/debconf interactive hang. Set DEBIAN_FRONTEND=noninteractive + NEEDRESTART_MODE=a, sudo -E, and --no-install-recommends so the install can't block. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: use preinstalled clang-17, stop apt-installing it (apt hangs) Multiple runner instances hung in 'apt-get install clang-17'. ubuntu-24.04 ships clang 16/17/18 preinstalled, so locate clang-17 directly (probe /usr/bin/clang-17 and /usr/lib/llvm-17/bin/clang) and point bazel at it via --repo_env. No apt install -> no hang; if clang-17 were ever absent the step fails fast instead of stalling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fiber/ci: build TSan via <sanitizer/tsan_interface.h>; use default clang in CI annotation.h hand-declared the __tsan_* symbols (fiber API, mutex API + flag constants, acquire/release). Those collide with compiler-rt's <sanitizer/tsan_interface.h> ("non-static declaration follows static declaration") whenever that header is in the translation unit under -fsanitize=thread, which broke the TSan build on the CI runners (both the default clang-18 and the runner's clang-17 hit it). It is NOT version-specific. Include the canonical header directly instead of redeclaring. FLARE_INTERNAL_ USE_TSAN is only defined under -fsanitize=thread, and any toolchain providing that flag ships this header (verified: clang-17, and gcc-13 which also defines __SANITIZE_THREAD__ and exposes the fiber/mutex API), so it's always available. With the build no longer tied to a clang version, drop the CI lane's clang-17 pinning (which needed an apt install that intermittently hung) and just use the runner's default clang via --config=llvm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
CHEN Feng
|
ed29968f5a |
thirdparty: migrate openssl/nghttp2/curl to vcpkg (#179 curl cluster) (#190)
* thirdparty: migrate openssl/nghttp2/curl to vcpkg (#179 curl cluster) vcpkg has no openssl 1.1.1, so this upgrades the cluster to 3.x and migrates all three off foreign builds. Consumers' //thirdparty/{openssl,nghttp2,curl} targets are unchanged (thin vcpkg wrappers). - openssl -> vcpkg 3.x. flare's crypto/init code adapted for 3.x: - flare/io/util/openssl.cc: gut the obsolete OpenSSL-1.0.x ceremony (SSL_library_init, ERR_load_*, ENGINE_*, CRYPTO_set_locking_callback / CRYPTO_num_locks). OpenSSL 1.1+ auto-initializes and is internally thread-safe, so both entry points are now empty no-ops. - flare/base/crypto/{md5,sha}: -Wno-deprecated-declarations -- the low-level MD5/SHA/HMAC APIs are deprecated in 3.x (provider/EVP migration) but still valid until 4.0, so suppress rather than rewrite to EVP. - nghttp2, curl -> vcpkg (API-stable version bumps). curl built with the openssl + http2 features; its macOS frameworks (SystemConfiguration/Security/ CoreFoundation/CoreServices) link via blade's new pkg-config -framework support. Wrapper declares the sibling vcpkg deps (openssl/nghttp2/zlib). - thirdparty/blade/blade.zip: rebuilt with that -framework support. - Drop the vendored tarballs and patches (openssl/curl/nghttp2, curl dyld patch). Verified: full `blade test flare/...` green on macOS (crypto/ssl/curl/cos all pass). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * thirdparty/openssl: drop redundant #dl/#pthread blade now auto-propagates libcrypto's pkg-config system libs (Libs.private: -ldl) via the #1322 mechanism, and flare links pthread globally on Linux (cc_config). So the wrapper's explicit #dl/#pthread are redundant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: checkout the pinned vcpkg baseline (not just fetch) Fetching only the baseline commit let 'git show <baseline>:versions/baseline.json' resolve the baseline version, but vcpkg reads the per-port versions/*.json from the working tree -- still the runner's stale HEAD. For ports whose baseline version is newer than that HEAD (openssl 3.6.3, nghttp2 1.69, curl 8.20) vcpkg errors 'no version database entry'. Checkout the baseline so the whole version DB / ports tree is consistent (matching local dev). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
CHEN Feng
|
55d97852f7 |
thirdparty: migrate fmt/gflags/glog/zlib/lz4/zstd/snappy/xxhash to vcpkg (#180)
Migrate eight third-party libraries (fmt, gflags, glog, zlib, lz4, zstd, snappy, xxhash) from in-tree foreign builds to vcpkg, via blade native vcpkg support. Bundles the matching blade.zip (proto import-root + vcpkg protoc + transitive Requires linking + textual_hdrs) and the CI/Bazel adjustments (pinned vcpkg baseline; xxhash externalized for Bazel). |
||
|
CHEN Feng
|
7fb4c8f0d8 |
docs(flare): verify and fix small issues across doc/ (#175)
* docs(flare): verify and fix small issues across docs/ Pure-mechanical cleanup pass through the remaining flare/doc/ files. Every change is verified against the actual code referenced. * `profiler.md`: title typo ``Profiller -> Profiler``; ``flag::Server`` -> ``flare::Server`` in the CPU profiler section. * `async.md`: ``Futrue`` typo -> ``Future``; link ``Async`` to its real header ``flare/fiber/async.h`` instead of leaving it as bare text. * `controller.md`: replace "see source code comments" with concrete header links (``rpc_client_controller.h``, ``rpc_server_controller.h``, ``rpc_controller.h``); link the "tracing" reference to ``tracing.md``. * `scheduling-group.md`: replace the ``...`` placeholder in the per-scheduling-group resources list with two concrete entries that exist in the code (object pool's ``memory_node_shared`` bucketing per NUMA node, monitoring's local-accumulate + cross-group aggregate reporting). * `gflags.md` / `buffer.md`: add the missing ``[返回目录]`` footer that every other doc has. ``buffer.md`` also had a stray triple ``)))`` in the last code sample -- removed. * `performance-guide.md`: fix broken header path ``../base/spinlock.h`` -> ``../base/thread/spinlock.h``. The file was moved into the ``thread/`` subdirectory at some point and the doc never followed. * `rpc-log-and-dry-run.md`: fix two weird relative paths that contained nested ``../`` segments and a literal ``binlog/`` prefix that doesn't exist (``binlog/../../rpc/binlog/dumper.h`` -> ``../rpc/binlog/dumper.h``, ``../binlog/../rpc/binlog/dry_runner.h`` -> ``../rpc/binlog/dry_runner.h``). No content rewrites; only typos, broken refs, and missing footers. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(controller) + ci(link-check): fix #175 CI failures Two issues link-check surfaced on this PR: * controller.md referenced a non-existent ``flare/rpc/rpc_controller.h``. Actual hierarchy: ``RpcClientController`` and ``RpcServerController`` both inherit from ``protobuf::RpcControllerCommon`` (no shared ``RpcController`` header). Replace the wrong link with a plain text mention of the common base class. * performance-guide.md links to cppreference / stackoverflow / linux.die.net all 403'd against the link-check user agent. These URLs aren't broken in any normal browser; the docs were untouched in this PR but ``check-modified-files-only=yes`` re-scanned the file. Add the three domains to ``ignorePatterns`` with comments documenting the reason. The existing pattern for bazel.build set the precedent. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * ci(link-check): add dev.mysql.com to UA-filter ignore list Same shape as cppreference / stackoverflow / linux.die.net: the docs URL returns 403 to the action's User-Agent but works in any browser. Surfaced by rpc-log-and-dry-run.md re-scanning under this PR's edits. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
CHEN Feng
|
356df6bb30 |
ci: skip heavy workflows on markdown-only changes (#173)
Adds `paths-ignore: ['**/*.md']` to the push/pull_request triggers of: - build-and-test.yml (blade build + test, longest job at ~25 min) - bazel-build.yml (bazel build + test, second-longest at ~15 min) - clang-format-check.yml (only inspects C/C++/Protobuf sources) GitHub Actions skips the workflow only when *every* changed file matches the ignore pattern, so mixed PRs (e.g. a code fix that also touches a README) still get full coverage. Pure docs/README PRs -- like the README rewrite -- now only run markdown-link-check, which is what actually verifies them. Saves ~25 min of CI wall time per pure-docs PR. The markdown-link-check workflow keeps its unconstrained trigger; documentation correctness still gets verified end-to-end. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
CHEN Feng
|
362e00a180 |
docs(readme): update for C++20 + macOS support, retranslate zh/en (#172)
* docs(readme): update for C++20 + macOS support, retranslate zh/en Bring both READMEs in line with the project's current state: * C++ badge upgraded from a generic ``C++`` to ``C++20`` to match the actual standard the project compiles with (``-std=gnu++2a``). * Platform badge upgraded from ``linux`` to ``linux | macOS`` -- macOS port landed in #158, CI matrix in #167 onwards. * Features section: drop the historical ``C++11/14/17/2a`` mix in favor of just ``C++20`` (the standard is committed; the staged adoption is no longer informative for a new reader). * System Requirements section split into ``Linux`` and ``macOS`` subsections, each listing the supported kernel/OS and the minimum toolchain that ships ``std::atomic<T>::wait`` / ``notify_*``: - Linux: GCC 11+ (libstdc++ added the operations in 11.1, per P1135 -- exposed locally on flare/fiber/detail/scheduling_group after the futex path went away in #158). - macOS: 13 (Ventura) or later with Apple Clang from Xcode 14+ (its libc++ has the matching runtime symbols). Also calls out aarch64 / arm64 / x86_64 explicitly. * Retranslated the zh README to read more like native Chinese -- the previous version had several literal-translation-isms ("吸收先前服务 框架"... etc.). Cross-checked the en against the zh and rewrote several en sentences that drifted out of step over the years (badge alt text, "out of the box" phrasing, the third-party tarball blurb, etc.). Both files now describe the same project in the same shape. * Added a back-link from the en README to the zh one (was one-way before). * Updated the blade-build link from chen3feng/blade-build to blade-build/blade-build (org moved). * Fixed bazel/support/README.md link typo (was REAMD.md). No code changes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs: fix two markdown-link-check failures surfaced by the README PR * Rename `bazel/support/REAMD.md` -> `bazel/support/README.md`. The file has carried the swapped-letter spelling since 2023; the previous README link pointed at the typo'd name, so it was working by accident. The README rewrite in this PR pointed at the canonical spelling and surfaced that the file on disk was wrong. * Add `.github/markdown-link-check-config.json` to suppress the spurious "Status: 0" failure for `https://bazel.build`. The site returns a valid 200 to a normal browser and to `curl`, but apparently UA-filters the markdown-link-check action -- the action reads Status 0 and counts it as broken. Listed it in `ignorePatterns`; wired the config into the existing `gaurav-nelson/github-action-markdown-link-check@v1` invocation via the documented `config-file` input. Also bumps `retryOn429` (small reliability win when the action hits rate limits) and pins the success set to 200/206. Both surfaced by CI on this PR; no other changes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
CHEN Feng
|
802aa0b0b3 |
ci(test): tighten per-test timeout via --test-timeout-multiplier (#169)
* ci(test): tighten per-test timeout via blade --test-timeout-multiplier Three coupled changes: 1. **Bump thirdparty/blade.zip** from a76de22 to 9306687, picking up three new blade features merged today: - #1241 live test output streamed to <run_dir>/blade-test.log - #1244 blade.getenv() for opt-in env-driven config (config phase) - #1246 --test-timeout-multiplier CLI flag (run-time test-timeout scaling) 2. **Lower global_config.test_timeout from 600s to 60s.** The old 600s was a "fits all environments" compromise that meant a hung fast unit test took 10+ minutes to fail (witness: dumper_integration_test recently stuck 60+ min on a flaky ubuntu_22 runner). Profiling on M2 release/-O2 shows the slowest legitimate tests today are CPU-heavy ones around 30-40s -- and those are all already exclusive=True so they don't contend with the parallel pool. 60s gives ~50% headroom over the slowest legit case; anything blowing past it is almost certainly hung and worth failing fast. 3. **Scale up via --test-timeout-multiplier=5 in CI.** GHA runners are slower and contended; 5x = 300s effective ceiling. Tight enough that a real hang fails inside the GH step timeout (~6 hours), loose enough to absorb routine CI variance. Net effect: dev-machine timeouts catch hangs in 60s, CI absorbs runner slowness, and the cap-on-slowness lives in the CI workflow (where it belongs -- it's a property of the runner) rather than in BLADE_ROOT (which describes the project's intrinsic test cost). Verified locally: ./blade test ... --full-test passes all 235 tests under the 60s baseline. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(build): drop unconditional -lpthread from gtest_libs on macOS Every cc_test that pulled gtest got '-lpthread' appended to the link line regardless of platform. On macOS Apple's ld then warns: ld: warning: ignoring duplicate libraries: '-lpthread' (pthread is part of libSystem and auto-linked; the explicit -lpthread is redundant and triggers the duplicate-library diagnostic on the new Xcode 16 linker). Wrap gtest_libs in a lambda that adds '#pthread' only when targeting Linux, matching the pattern already used for chrono / time_keeper / thread_pool and the thirdparty glog/gflags/etc. cc_test entries. Verified locally: every cc_test ninja file regenerated after this change has '-lpthread' gone on macOS; nothing changes on Linux (blade.cc_toolchain.target_os == 'linux' branch still emits it). Note: -lrt warnings observed in CI still appear, but those come from foreign_cc subbuilds (jemalloc / lz4 configure scripts) honoring the host environment via autoconf detection; they're internal to those packages, not from flare's link lines. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(build): guard unconditional '-ldl' / '-lm' with Linux check Three places listed POSIX libraries as deps without a target_os check: - flare/base/internal:cpu -> '#dl' (uses dlsym for symbol lookup) - flare/base/internal:curl -> '#dl' (curl bindings) - thirdparty/openssl:crypto -> '#dl' (openssl needs dlopen at runtime) - thirdparty/protobuf-3.4.1 -> '#m' (math symbols) On macOS these symbols all live in libSystem and are auto-linked; the explicit '-ldl' / '-lm' is redundant. They don't typically trigger ld's "ignoring duplicate libraries" warning the way unconditional '#pthread' in BLADE_ROOT.gtest_libs did (since the .tbd stubs are designed for this redundancy), but the inconsistency was confusing and a single "-l<x>" arriving from two places anywhere downstream would tip into a real duplicate warning. Wrap each in the existing `target_os == 'linux'` / `_is_linux` pattern used elsewhere in the file. macOS link lines no longer contain '-ldl'/'-lm'; Linux is unchanged. Note: gperftools' BUILD still has unconditional '#pthread'/'#rt', but gperftools is excluded on darwin entirely (the macOS port routes through jemalloc instead), so those don't matter on macOS. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
CHEN Feng
|
ed1e854280 |
ci: bump oldest-supported gcc to 11; make matrix.cc actually pick the matrix version (#168)
* ci: make matrix.cc actually pick the matrix gcc version
The Build and Test workflow's matrix promises gcc-10 on ubuntu_22 and
gcc-13 on ubuntu_24, but until now both jobs were actually compiling
with the distro default (gcc-11 / gcc-13) for the bulk of the build.
Cause: ``apt-get install gcc-10`` only installs the versioned binaries
(``/usr/bin/gcc-10`` etc.); ``/usr/bin/gcc`` keeps pointing at the
system default. Blade resolves the compiler via ``shutil.which('gcc')``
and reads no environment variable, so the ``CC=gcc-10 CXX=g++-10`` env
prefix on the Build step was decorative for the main build. Only
thirdparty foreign_cc subbuilds, which read $CC, actually saw gcc-10.
Effect on ubuntu_22 (the more important case -- gcc-10 is the
oldest-supported compiler the matrix claims to cover): the matrix
labelled "gcc-10" was silently testing gcc-11 for everything blade
compiled directly. Real gcc-10 compatibility was not under test.
Fix: register the matrix version with update-alternatives in Install
dependencies. After this, ``which gcc`` resolves to the matrix
version; both blade's compiler discovery and ccache's dispatch see
the right one. Drop the decorative ``CC=...`` prefixes from the Build
and Run tests steps now that they are no longer load-bearing (and
would have been misleading to future readers).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* thirdparty/glog: patch self-referencing deprecated overload for gcc-10
PR #168 switched ubuntu_22_04 from the distro-default gcc-11 (which the
prior workflow was silently using) to the actually-promised gcc-10. That
flipped the lid on a long-standing latent issue in glog 0.7.1:
glog/logging.h:494-505 declares a [[deprecated]] type alias
`CustomPrefixCallback` inside a `#pragma GCC diagnostic
push/ignored "-Wdeprecated-declarations"` block, then closes the
pragma block, then declares a [[deprecated]] overload of
`InitGoogleLogging` whose parameter type is `CustomPrefixCallback`.
gcc 11+ has a heuristic that suppresses deprecation warnings inside a
deprecated declaration's signature, so the overload's self-reference
stays silent. gcc 10 lacks that heuristic and fires
`-Werror=deprecated-declarations` on every TU pulling in <glog/logging.h>.
Patch (silence_self_deprecation.patch) moves the overload inside the
existing pragma block. No behavior change for callers; only the
internal self-reference stops warning. macOS clang and gcc 13 are
unaffected -- the diagnostic-suppression region simply covers slightly
more code on those toolchains too.
Verified by dry-run + real apply against a fresh extraction of
glog-0.7.1.tar.gz; the patched header preserves the original semantics.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* ci: bump oldest-supported gcc from 10 to 11
PR #168 made the matrix actually pick gcc-${{ matrix.cc }} instead of the
distro default (which was silently gcc-11 on ubuntu_22). With gcc-10 truly
exercised, two real incompatibilities surfaced:
1. flare/fiber/detail/scheduling_group.cc uses std::atomic<int>::wait,
notify_one, notify_all (C++20 P1135). libstdc++ implements these
starting in 11.1 only; gcc-10's <atomic> has no such members. PR #158
introduced the dependency when it replaced the Linux-only futex(2)
path with the portable C++20 primitives to make the macOS port work.
2. Several .cc files reference POSIX `unlink` without
`#include <unistd.h>`. Newer libstdc++ stopped transitively pulling
unistd through other headers.
Both are gcc-10 blockers we don't want to walk back -- atomic wait is
materially better than the futex path (portable, no syscall asymmetry).
Bump the matrix to gcc-11 and update the comment.
gcc-11 is the ubuntu-22.04 default, so the update-alternatives plumbing
becomes a no-op for this version but stays in place to keep future bumps
straightforward.
Keep the glog self-deprecation patch -- on gcc-11+ it is a no-op (the
heuristic suppresses the warning anyway), but it shields a local gcc-10
attempt from breaking on glog before the user even gets to the real
flare-side incompatibilities.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
CHEN Feng
|
6df0c8d6c8 |
ci: use ccache to make incremental CI builds actually incremental (#167)
Background: PR #164 added actions/cache for build64_release/, but incremental builds still run nearly cold (~15-18 min on Linux, ~14 min on macOS) -- log shows ninja starts from [1/2956], not a partial. Root cause: actions/checkout sets all source mtimes to "now", which is newer than the cached .o files, so ninja decides everything has changed and recompiles. ccache fixes this with content-based hashing, ignoring mtime entirely. Hooked in via the PATH-injection mechanism that ccache was designed for: /usr/lib/ccache/{gcc,g++,gcc-10,...} -> ccache binary echo "/usr/lib/ccache" >> $GITHUB_PATH Blade resolves the compiler via shutil.which('gcc'), so prepending /usr/lib/ccache to PATH redirects to the ccache shim with no change to blade itself. ccache dispatches to the real compiler internally. The existing build64_release cache is kept -- ccache cannot cache thirdparty/foreign_cc subbuilds (autotools/cmake produce many compile + link invocations that ccache treats individually but the foreign_cc driver itself is the slow part). The two caches are complementary: build64_release -> thirdparty; ccache -> regular cc_library/cc_test. Expected effect on incremental PRs (master cache hits): Build step on Linux: ~15 min -> ~3-5 min (only changed .cc recompile) Build step on macOS: ~14 min -> ~3-5 min Cold cache (e.g. thirdparty bump): no regression The "ccache stats" steps print hit/miss counts so we can watch the ratio over the first few PRs and tune if needed. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
CHEN Feng
|
8a156b686e |
ci: cache bazel output + blade build64_release across runs (#164)
Both bazel and blade CI workflows currently start from scratch on every push. Big wastes: foreign_cc thirdparty libs (curl/openssl/jemalloc/ nghttp2) each take minutes to build, and all .cc files always recompile even when the PR only touches a few of them. Add ``actions/cache@v4`` to both: * ``bazel-build.yml``: cache ~/.cache/bazel (the output base; covers external repos, all build artifacts, and test result cache). * ``build-and-test.yml``: cache build64_release. Key per (OS, gcc) since binaries aren't portable across either. Cache key for both is hashed over WORKSPACE / BUILD files / thirdparty source tarballs so any structural change invalidates. restore-keys falls back to the most recent same-OS cache on miss so non-master branches still benefit from master's cache. Estimated effect: full cold-cache build still ~24 min; subsequent incremental builds (most PRs) drop to ~5-10 min depending on which .cc files changed. README/doc-only PRs become near-instant. Caveats: * GHA cache is capped at 10 GB/repo; older keys evicted LRU. With 3-4 OS x compiler combinations and bazel + blade entries we may push that ceiling -- watch for thrash. * PRs only read the base-branch (master) cache; cross-PR sharing doesn't happen. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
CHEN Feng
|
d5e44603db |
test: shrink over-scaled loops + exclude *_benchmark from bazel CI
Six commits' worth of test-step CI wall-time reduction.
* Loop reductions in five tests that were stress-testing far harder than
their assertions required. Local arm64 fastbuild measurements:
semaphore_test 115.6s -> 1.1s (105x; cut 1M threads to 10K)
id_alloc_test 50.7s -> 0.9s (56x; cut 1B allocs to 10M)
hash_map_test 42.0s -> 1.8s (23x; cut 5M iter to 500K)
timer_worker_test 31.0s -> 5.8s (5x; cut 4M ops to 100K)
writing_buffer_list_test 22.0s -> 4.5s (5x; cut Torture configs)
* Tag *_benchmark cc_test with tags = ["benchmark"] and exclude from CI
via --test_tag_filters=-benchmark. 30 BUILD.bazel targets tagged.
Bazel CI was running 30 benchmark cc_test targets including
hash_map_benchmark (86s), case_insensitive_hash_map_benchmark (70s),
dpc_benchmark (54s). Matches blade's semantics where cc_benchmark is
a cc_binary (not picked up by ./blade test).
Estimated CI wall savings ~3 min on bazel job. All reduced tests still
exercise the same race / overflow / stress patterns -- just at sensible
iteration counts.
|
||
|
CHEN Feng
|
51cf5a5587 |
ci: unflake timing tests + bump deprecated actions + blade.zip (3 fixes)
Three CI hygiene fixes bundled in one PR.
* **Flaky timing assertions widened**. Two tests had ASSERT_NEAR / EXPECT_NEAR
with budgets too tight for shared-VM CI runners:
- flare/net/http/binlog_dry_runner_integration_test:220 -- 20ms -> 50ms
(observed 136ms vs expected 100ms on GHA).
- flare/io/native/stream_connection_test:411 and :491 (WriteBandwidthLimit
/ ReadBandwidthLimit) -- 1s -> 2s (observed 8.4s vs expected 7s).
Both tests still validate order-of-magnitude correctness; just give
shared-runner jitter the headroom it needs.
* **Deprecated GitHub Actions bumped** to silence the "Node.js 20 actions
are deprecated" warning (forced upgrade scheduled for Sept 2026):
- actions/checkout: @v2 / @v3 / @master -> @v6 (Node 24)
- actions/setup-python: @v2 -> @v6
* **thirdparty/blade: bump blade.zip to master a76de22**. Picks up three
upstream blade-build fixes that surfaced after master a8d12b2 wired in
the new cc_check_undefined static check:
- #1238: add __tls_get_addr / __tls_get_offset to undefined-check
baseline (ld-linux.so injects these); unbreak stale CheckUndefinedDiff
unit tests; defer PrebuiltCcLibrary._emit_archive_syms to generate
phase (was tripping the __build_code-None invariant in
Target.before_generate).
- #1239: treat uppercase 'V' / 'W' nm symbol types as defined, not
ambient -- they're weak DEFINED, not weak UNDEFINED.
- #1240: treat lowercase 'u' as defined (GNU "unique global"
extension) -- fmt's basic_data<void>::*_padding_shifts and ~40 other
template-static-data symbols were silently dropped from libfmt.a's
syms set, breaking any cc_library that transitively used fmt.
|
||
|
CHEN Feng
|
86f175c3c4 |
Add macOS (darwin/arm64) build support (#158)
* feat(macos): build flare on macOS (darwin/arm64) + thirdparty alignment & bug fixes
Make flare build with both blade and bazel on macOS (Apple Silicon,
darwin/arm64); previously Linux-only. Verified: `blade build flare/...` and
`bazel build //flare/...` (653 targets) both green on macOS arm64.
Third-party / build system:
- Align bazel deps with the verified blade build: glog 0.4.0->0.7.1,
zlib 1.2.13->1.3.2, openssl 1.0.2u->1.1.1w (1.1 required by HMAC_CTX usage).
- glog: extend the export-guard patch to every header gating <glog/export.h>
behind GLOG_USE_GLOG_EXPORT, and update CHECK_OP to use
google::logging::internal::{Make,}CheckOpString.
- foreign_cc macOS fixes: force a real `ar` (rules_foreign_cc defaults AR to
Apple's libtool, incompatible with autotools' ar-style calls); gate -lrt to
Linux; pick .dylib vs .so; build nghttp2 static-only (its sandbox dylib
install_name breaks curl's configure run-time check).
- Register the host preinstalled make toolchain (rules_foreign_cc's bundled
make segfaults building jemalloc on macOS; blade also uses the system make).
- Track patch files as unpack-rule inputs (foreign_build.bld).
- Silence third-party warnings (external/ and in-repo thirdparty/) via
per_file_copt.
flare portability:
- IO: new Poller abstraction with epoll (Linux) and kqueue (macOS) backends;
event_loop/descriptor rewired off raw epoll; eventfd -> pipe notifier.
- net/thread/cpu/rlimit platform guards; OpenSSL 1.1 HMAC_CTX API.
- Add missing `gflags/gflags.h` / `<unistd.h>` includes (were pulled in
transitively by the old glog).
- gperftools/tcmalloc is Linux-only: excluded on darwin (mirrors blade);
macOS uses jemalloc.
Bug fixes (also affect Linux):
- base/expected.h: Unexpected::swap referenced a nonexistent member
(other.unex -> other.unex_); never compiled once instantiated.
- io/detail/writing_buffer_list.cc: FlushTo crashed (CHECK) on a benign drain
race (head_ stale while tail_ == nullptr); now treats the list as empty.
- base/chrono.{h,cc}: clocks delegated to a hand-rolled clock_gettime that
mixed clock epochs (broke TimerWorker::wait_until); now use
std::chrono::*::now(); ReadCoarseSystemClock duration_casts to the clock's
duration.
- fiber/detail/scheduling_group.cc: spin-wait bound was a hardcoded TSC-tick
count assuming a GHz-scale TSC; now derived from the measured TSC frequency.
- base/internal/dpc.cc: "Run N DPCs" logged cbs.size() (always 0 at loop exit)
instead of the accumulated dpcs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fiber: defeat Clang's TLS-base CSE across returns_twice on AArch64
On AArch64, Clang caches the resolved thread_local slot address in a
callee-saved register and reuses it across `[[gnu::returns_twice]]`
calls. `jump_context` may resume the same C++ frame on a different OS
thread, so the cached slot points to the previous worker's TLS region
on resume, and accesses go to the wrong memory.
GCC handles `returns_twice` more conservatively (it re-issues
`mrs tpidr_el0` after the call), which is why Linux flare worked
inlined for years -- Linux + GCC, not "Linux".
Centralize the canonical `jump_context` declaration in a new
`context.h` (so every TU sees the `returns_twice` attribute) and
force the three TLS accessors out-of-line: `[[gnu::noinline]]` makes
each access an ABI call, and ABI rules force the caller to redo the
TLS resolution -- Clang can't CSE across a real function boundary.
Discovered while porting flare to macOS arm64 (Apple Clang), but the
underlying hazard exists on any Clang aarch64 build; GCC users were
just lucky.
* io/kqueue: fix Modify to disable removed filters; stop fanning EV_EOF to opposite direction
Two macOS-port correctness fixes in the kqueue poller, both surfaced
by a flare RPC client that died with STATUS_IO_ERROR on the first
write to a freshly-opened connection.
1. `Modify` now explicitly EV_DISABLEs filters that aren't in the
new event mask. Unlike Linux epoll (EPOLL_CTL_MOD replaces the
whole event bitmask in one shot), kqueue registers EVFILT_READ
and EVFILT_WRITE independently -- a bare EV_ADD on one filter
does NOT affect the other. Without explicit disable, the kernel
keeps firing the "removed" filter under level-triggered mode,
and `OnWritable()` runs on an empty writing buffer, which the
connection layer mistakes for "remote closed" and kills the
connection. EV_DISABLE (not EV_DELETE) is what we want -- it
pauses delivery while preserving the filter's internal readiness
state, mirroring epoll's behaviour. (EV_DELETE + later EV_ADD on
a socket that is already readable produces no edge with EV_CLEAR,
and the reader stalls forever.)
2. `Wait` no longer fans an EV_EOF flag out into both kPollerRead
and kPollerWrite. EV_EOF on EVFILT_READ just means the peer
half-closed for reading; the write direction is still usable
until the next write returns EPIPE. The old fan-out manufactured
a fake write event whenever the read side EOF'd, kicking
`OnWritable()` on an empty writing buffer -- same false-disconnect
chain as #1.
* io/util/socket: SetCloseOnExec must use F_SETFD; rename helper to SetStatusFlags
FD_CLOEXEC is a *file-descriptor* flag (F_GETFD / F_SETFD), not a
file-status flag (F_GETFL / F_SETFL). The existing `SetCloseOnExec`
routed through the generic `SetFlags` helper, which goes through
F_SETFL, and so passed FD_CLOEXEC = 0x01 into the status-flag space.
On Linux glibc, the unsupported bit is silently discarded by F_SETFL,
so this was invisible. On Darwin the kernel actually interprets the
0x01 bit inside the F_SETFL flag space: O_NONBLOCK gets wiped off the
listening socket and O_APPEND gets set. The downstream effect is that
`accept(2)` on the listening socket reverts to blocking and the
server's accept loop hangs forever after the first connection.
Fix `SetCloseOnExec` to use F_GETFD / F_SETFD directly, and rename
the generic helper from `SetFlags` to `SetStatusFlags` so the
descriptor-flag-vs-status-flag distinction is hard to miss next time.
The only remaining caller of the helper is `SetNonBlocking`, which
is correct (O_NONBLOCK *is* a status flag).
* io/native: treat ENOTCONN as transient in FlushWritingBuffer
When a non-blocking TCP socket is in the middle of its `connect(2)`,
Linux returns `EAGAIN` on `write(2)`/`sendmsg(2)`, which flare maps
to `FlushStatus::SystemBufferSaturated` and retries via the write
event. Darwin instead returns `ENOTCONN` ("Socket is not connected")
in the same situation -- the kernel distinguishes "would block on a
connected socket" (EAGAIN) from "not yet connected" (ENOTCONN).
The existing code treated ENOTCONN as a fatal write error, returned
`NothingWritten`, and `Kill(CleanupReason::Disconnect)` immediately
fired. The first RPC on every channel died with STATUS_IO_ERROR.
Fold ENOTCONN into the EAGAIN / EWOULDBLOCK branch so the write is
deferred until the kernel signals writability (which happens when
the connect completes), matching Linux semantics.
* base/write_mostly: order members so mutex outlives TLS wrappers
`WriteMostly` had `tls_buffer_` declared before `mutex`, which means
`mutex` was destroyed *first* (members are destroyed in reverse
declaration order). When `tls_buffer_` was then torn down, each
per-thread `WriteBufferWrapper` ran its destructor and tried to
`std::unique_lock(parent_->mutex)` -- locking an already-destroyed
mutex.
libstdc++ on Linux silently allowed this (the underlying pthread
mutex didn't actually reject), so the bug was invisible.
libc++ on Darwin returns EINVAL from `pthread_mutex_lock`, std::mutex
throws `system_error("mutex lock failed: Invalid argument")`, and
since destructors are noexcept by default the unhandled exception
aborts via std::terminate.
Reorder the members so `tls_buffer_` is the last declared field
(thus the first destroyed): both `mutex` and `exited_thread_combined_`
are still alive while the wrappers merge into them.
* base/deferred_test: initialise bool to avoid UB read
`Deferred.All` declared `bool f4;` and then `ASSERT_FALSE(f4)`
before any store. Reading an uninitialised bool is undefined
behaviour: the value is "indeterminate" and a load of an
indeterminate value is itself UB regardless of which bit pattern
happens to be in the stack slot.
Linux happened to observe zero on the stack and slipped through.
macOS Clang traps on the indeterminate read (SIGTRAP, exit 133),
exposing the bug.
Initialise `bool f4 = false`.
* macOS port: platform-specific test adjustments
A bag of test-only changes that fall out of the macOS port.
* attribute_test.cc -- SetCurrentAffinity skipped on Darwin.
Darwin has no pthread_setaffinity_np equivalent and the legacy
thread_policy_set THREAD_AFFINITY_POLICY is hint-only; the
`expected == GetCurrentProcessorId()` assertion cannot hold.
* descriptor_test.cc -- ConcurrentRestartRead skipped on Darwin.
The test attaches a PipeDesc with Event::Write to the read end
of a closed-write pipe. Linux epoll never fires EPOLLOUT on
that fd; macOS kqueue fires EVFILT_WRITE (no write buffer ==
"has space"). The fire trips a pre-existing race in
`Descriptor::QueueCleanupCallbackCheck`: FireXxxEvent does not
consult `cleanup_pending_` before fetch_add, so an event between
cleanup-task queueing and execution makes the strict CHECK at
descriptor.cc abort. Fixing that race needs Dekker-style
ordering between FireXxxEvent and Kill -- bigger scope than
this batch.
* writing_datagram_list_test.cc -- shrink the test datagram from
16384 to 4096 bytes. Darwin's `net.inet.udp.maxdgram` defaults
to 9216, and sendmsg() rejects larger UDP payloads with
EMSGSIZE; Linux is happy up to ~64KB.
* datagram_transceiver_test.cc -- cap the fragment count from
60'000 to 8192 on Darwin. Still well above IOV_MAX (1024 on
both libc's) so the "flatten on iovec overflow" path is still
exercised; total payload now fits under net.inet.udp.maxdgram.
* uri_matcher_test.cc -- regex pattern `"\\abc.*efg"` used the
`\a` escape, which is not a valid ECMAScript escape. libstdc++
accepted it silently (treating it as literal `a`); libc++ on
Darwin throws regex_error("invalid escaped character") and the
test aborts. The intended pattern was `abc.*efg`.
* dirty_hook_test.cc, hooking_mock_test.cc -- skipped on Darwin.
Both rely on `mprotect`-ing the text segment to writable to
overwrite a function prologue. macOS arm64 enforces W^X on
signed code pages: mprotect returns EPERM, the CHECK aborts.
Toggling JIT-style write-execute would require
`pthread_jit_write_protect_np`, the
`com.apple.security.cs.allow-jit` entitlement, and code placed
in a JIT region -- none of which apply to the regular text
segment that DirtyHook patches. (The single non-hook subtest
in hooking_mock_test, `NotEnabled`, still runs.)
* fixup: satisfy clang-format-15 (LLVM style) on the macOS port commits
CI's clang-format-15 (LLVM style) flagged two spots from the macOS
port batch:
* flare/fiber/detail/context.h: `} // extern "C"` -> `} // extern "C"`
(LLVM trailing-comment style wants two spaces before //).
* flare/io/detail/kqueue_poller.cc: the `read_flags`/`write_flags`
ternaries were hand-aligned to a column width that clang-format
didn't pick. Drop the explicit alignment and pull the disabled
case into a named `disable_flags` local so each assignment fits
on a single line that clang-format leaves alone.
No behaviour change.
* thirdparty/blade: bump blade.zip for target_arch cache fix
Picks up the toolchain.target_arch caching from
https://github.com/blade-build/blade-build/pull/1224.
BUILD-file load on this workspace drops from 13.2s to 0.25s.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Luo Bo
|
34996ed323 | ci: try to fix CI failures (#153) | ||
|
Luo Bo
|
bb511f9b85 | fix: clang does not always work well with libstdc++, using GCC instead (#138) | ||
|
kang jinci
|
1d60bd710b |
feat: support bazel build (#121)
* feat: suppport bazel build * feat(bazel): add rpc plugin * code change * fix comment |
||
|
kang jinci
|
5f24093d1c | chore: format code and add clang-format check (#112) | ||
|
kang jinci
|
bf9b82edc2 |
fix: GCC13 compile failed (#108)
* fix: remove redundant std::move * fix: null-compare * ci: add gcc-13 build * fix: add header file <cstdint> * fix: add compile option -Wself-move when gcc-13 compile maybe_owning_test.cc |
||
|
kang jinci
|
d90635ddf0 |
Adjust CI (#78)
* CI: add gcc-8 and gcc-10 * CI: add gcc-11 and gcc-12 |
||
|
Luo Bo
|
4feff801fe | Use Ubuntu 20.04 for running CI. Remove __const__ hack. (#80) | ||
|
Luo Bo
|
647cb0288b | Adding CI. (#35) |