Files
Evil0ctal e72bc75b42 fix(scheduler): SET NX for the lock, per-call tie-break, isolated policy registry
Three changes, all found by measuring rather than reasoning.

Rotation was not rotating. The tie-break seed came from the retry count, so
every concurrent first attempt ranked the candidate list identically and callers
walked it in lockstep; and recency was compared exactly, so no two candidates
ever tied and the tie-break was never consulted at all. Over 600 concurrent
requests across 12 identities that gave a 26..74 split. With a per-call seed and
recency quantized to half a second the median spread is 40% against a uniform
random baseline of 46% - the ordering now earns its place by beating chance,
which is the only honest yardstick once perfect round-robin is unreachable.

The lock now uses SET NX instead of EXISTS-then-SET. Both are safe inside a Lua
script, but NX states the requirement in one indivisible operation rather than
two statements that happen to add up to it. The no-token path releases the lock
it just took, which the two-statement form did not need.

The policy table is process-global mutable state and tests that registered an
endpoint left it there, where the next test read it as configuration for an
endpoint that does not exist. snapshot/restore plus an autouse fixture.

Honest note on an unresolved observation: while three other workloads were
saturating this machine, the exclusivity assertion intermittently reported more
leases than identities. Redis is single-threaded and SET NX is indivisible, so
that should be impossible without the key disappearing - yet expired_keys and
evicted_keys stay at zero across a failing run, and it does not reproduce on an
idle machine or outside pytest. Four separate probes were built to chase it and
three of them turned out to have bugs of their own (in-process ordering mistaken
for server ordering, an off-by-one in a Lua key slice, a single-sample threshold
on a wide distribution), which is most of why this took as long as it did. The
test is left asserting the real invariant rather than relaxed into something
that always passes.
2026-09-07 05:47:01 -07:00

23 lines
600 B
Python

"""Shared fixtures.
The scheduler's policy table is process-global mutable state. A test that
registers an endpoint leaves it there, where the next test sees it as
configuration for an endpoint that does not exist. Snapshotting around every
test keeps that from turning into a false failure somewhere else.
"""
from __future__ import annotations
import pytest
from dtk.scheduler.policies import restore_policies, snapshot_policies
@pytest.fixture(autouse=True)
def _isolated_policy_registry():
saved = snapshot_policies()
try:
yield
finally:
restore_policies(saved)