@eslint/js 10 adds no-useless-assignment to the recommended set. Two of
the three were initial values every branch overwrites; the third was a
post-increment on the last key the function hands out.
Typing the global value back could not undo an override that already
equalled it: the row was unchanged, so nothing was saved, and the
platform stayed pinned to a number the scheduler page no longer moved.
Each platform row now has a "Restore default" button, shown only while
it has a number of its own, which writes -1 to whichever of its two
marks are overridden. A switched-off platform is an override too, so
this is also how it goes back to following the global marks.
The low-water mark and top-up target were one pair shared by every
platform. A deployment that only serves Douyin, on a server with no
route to TikTok, kept minting TikTok identities every minute, failing,
and raising pool_empty for a pool it never wanted (#763).
pool.<platform>.min_size / target_size now override the global pair.
-1 (the default) follows it, so existing instances behave exactly as
before. A mark of 0 turns that platform's automatic minting off, along
with its pool_empty / pool_below_min alerts. Manual imports and manual
mints are unaffected.
The worker, GET /admin/identities/pool and the diagnostic all resolve
the marks through dtk.identity.marks, so the console cannot show a
threshold the filler is not using. The refill card on the Identities
page now has a row per platform with its own Auto switch and numbers,
and marks which of them still follow the global value.
Version bumped in the four declaring files plus uv.lock, release notes in
.github/RELEASE_v5.1.0.md.
Also adds the hygiene test 01-release.md has been asking for since it was
written: the four version strings must agree, must be bare semver with no
leading v, and uv.lock must have been regenerated. Nothing enforced any of
that, and a missed file does not fail a build - it surfaces as
`dtk --version` disagreeing with the image tag on a server somebody is
already trying to debug. Verified it fails when one file is left behind.
Wires collects/list/ and collects/video/list/, which had URL constants in
endpoints.py and nothing else. Closes#759, and corrects the premise that
issue was filed on.
I wrote there that Douyin has no public half - that its web client only
ever shows you your own folders, so everything here would need an
imported identity. That is wrong. Douyin lets each folder be public or
private exactly as TikTok does, and a public folder's contents are
readable with a minted guest identity and no cookie at all. Measured on
one account holding two folders set either way:
guest owner
public folder 1 post 1 post
private folder refused 1 post
So collection_posts is now offered on both platforms and means the same
thing on each.
The folder LIST is where the platforms genuinely differ, and it is worth
stating plainly because it shapes the API. collects/list/ carries no user
id: it has no subject but the session holding it and cannot be pointed at
anyone else, where TikTok's takes a secUid and will answer about a
stranger's public folders. So on Douyin an author is refused rather than
accepted and ignored - a dropped sec_user_id would return the operator's
own folders under the stranger's name, and nothing in the response would
say so. The route decides this by asking the registry whether the
endpoint accepts an author, not by branching on the platform name, and
the Playground hides the two controls the same way it already hides ones
the session's role cannot use.
Two traps recorded because both fail quietly:
`status` means opposite things. Douyin calls a folder public with 1;
TikTok calls it private with 1 and public with 3. The constants are
defined separately in each platform's parser and neither is exported,
because sharing one would report a private folder as public.
collection_posts cannot reuse parse_author_posts on Douyin. That parser
demands max_cursor when hasMore is set, and this endpoint pages by
cursor with no max_cursor at all - so sharing it would have worked on
every single-page folder and raised on the first one needing page two.
It has its own parser and a test that pins exactly that.
/api/collection/detail/ answers about a folder without naming its owner.
That is the gap it fills: author_collections has to be asked by way of a
secUid, so given only a collection id - which is all a shared link
carries - there was no way to find out what it was.
The response side is measured, not inferred. #753 shipped cover: null
because its envelope was guessed, so this one was probed against the live
endpoint first and written against what came back: collectionInfo wraps
the same node author_collections returns a list of, down to the camelCase
urlList cover container and the numeric-string total. One builder answers
for both, which is asserted rather than assumed.
Two fields the model had nowhere to put, both stated on every folder node
by both endpoints:
owner_id / owner_name - whose folder this is
is_public - whether anyone can open it
author_collections gains both for free, which matters more than the new
endpoint does: the public/private distinction is the thing every caller
has been guessing at, and the list now simply states it per folder.
is_public is three-state on purpose. Measured on one account holding two
folders set either way, public came back status 3 and private status 1,
on both endpoints. Two samples name 3 and do not rule out a fourth value,
so an unrecognized status reports None rather than False - "I do not
know" and "no" are different answers, and it is the false "no" that would
stop someone bothering to ask.
Verified end to end with a guest identity: the public folder returns
name, cover, count, owner and is_public true; the private one is refused.
Closes#758.
TikTok gives reposts their own profile tab and their own endpoint,
/api/repost/item_list/. Nothing in the tree referenced it; the only
mention of reposts anywhere was repostCount in a few fixtures.
The request shape is author_posts' exactly, needPinnedItemIds and
post_item_list_request_type included, so a builder copied from
author_likes - the nearest-looking neighbour - would have been missing
both.
Not owner-only, which is the part worth knowing before spending an
imported identity on it. The capture this was built from came from a
signed-in browser looking at somebody else's profile, and a minted guest
identity with no session cookie reads the same tab: 10 items, then 6
more, no overlap, hasMore false.
One difference from author_posts that is invisible until you page: the
cursor is an item offset here, not the millisecond timestamp author_posts
uses. Both are opaque to callers and both pass through _cursor unchanged,
so nothing downstream cares - but the two builders sit next to each other
and would otherwise read as meaning the same thing.
Closes#757.
Closes the last of the Saved tab's four paths. `author_collections` lists
the folders, `author_bookmarks` returns every saved post across all of
them, and `/api/collection/item_list/` returns the contents of one named
folder - three distinct paths, not one path with different parameters,
which is what #754 asked to be settled.
The endpoint is keyed by the folder rather than by the account, and that
turns out to be the interesting part: the request names a thing, not a
person, so a folder its owner has set to public is readable with no
imported identity at all. Verified against the live endpoint with a
minted guest identity and no session cookie.
Also corrects what #753 merged with. It landed saying an author's
collections are private and that a guest gets an empty page. Both are
wrong, and wrong in the direction that stops anyone from trying: measured
against an account holding one public folder and one private one, a guest
identity got the public folder complete with name, cover and item count,
and no trace of the private one. Visibility is per folder. The V4 crawler
needed a cookie because it was asked about private folders, not because
the path is gated.
`author_bookmarks` really is owner-only, and its docstring was wrong in
the other direction: a guest is not handed an empty page, it gets an
explicit refusal - HTTP 200 with a 104-byte status envelope that
classifies as a business error and surfaces as NOT_FOUND. That is the
good case, so it is now written down as measured rather than guessed.
Playground labels now name the four as a set, per #754: an author's own
series against bookmark folders, saved posts across every folder, and
saved posts in one folder.
#753 landed the folders. TikTok's Saved tab has two halves and this is
the other one: everything the account saved, regardless of which folder
it was filed in. The URL constant has been in the tree since the port,
registered by nothing; v4 served it and v5 never did.
Every request parameter is from a capture of the signed-in page rather
than copied from a sibling, which turned out to matter:
`needPinnedItemIds` and `post_item_list_request_type` are sent for this
path and are not part of the shared base set, so a builder that started
from author_likes would have been missing both.
The response side is the itemList/hasMore/cursor envelope every other
item_list endpoint on this host returns, and the items are ordinary posts
rather than a new object - so `parse_author_posts` answers for this too,
joining the case it already answers for author_posts, author_likes and
mix_posts. That is still an assumption, a weaker one than the envelope
guess that put `cover: null` into #753 for a day, and the endpoint spec
says so and says what would settle it: one response from an account that
has saved something.
Only ever answers the account that owns the list. A guest gets an empty
page, which is indistinguishable from an account that has saved nothing,
so the route documents `identity` as the point of the endpoint rather
than an option on it. Douyin refuses by name - its equivalent is a
different shape and is still unwired.
Lists the bookmark folders a TikTok author has filed posts into, so a caller
can discover a collection's id without opening devtools. TikTok only; Douyin's
equivalent is still unwired and #740 stays open for it.
`/api/user/collection_list/` keyed by `secUid`, registered as
`tiktok.author_collections` with `risk_weight=1.8` like the other list
endpoints, its own `EndpointPolicy`, and a `Collection` model carrying id,
name, cover and item count. These folders are private the same way an
author's likes are: a guest identity gets an empty page rather than an error,
so reading them needs `?identity=` pointed at an identity imported from that
account's session.
The response envelope was confirmed against a live capture rather than
inferred: `cover` arrives as a `{"urlList": [...]}` container and `total` as a
numeric string. The first version read `cover` as a bare URL string, which
made it parse as `null` on every real response while the test - written to the
same wrong shape - stayed green. The replay payload is the real envelope now,
with the user fields redacted.
Co-authored-by: BennoCrafter <BennoCrafter@users.noreply.github.com>
Version in the four places that carry it plus the lock file, the example
tag in .env.example, and the release notes. The two fixes are already on
main and each explains itself in its own commit.
Version in the four places that carry it plus the lock file, the example
tag in .env.example, and the release notes. Nothing else moves here - the
four fixes are already on main and each explains itself in its own
commit.
Reported against the live demo: on Downloads, the author-posts mode shows
douyin and the platform menu will not open.
It was disabled for the whole of that mode, on the reasoning that an
author is named by a sec_user_id rather than by digits, so the menu had
nothing to decide. But resolveTarget reads the menu for a bare author id
too, and has to: Douyin's sec_user_id and TikTok's secUid are the same
MS4wLjABAAAA… shape, and nothing in the string says which platform it
came from. The comment on SEC_UID says exactly that, three lines above
the code that assumed otherwise.
So the menu sat greyed out at its default and every author download went
to Douyin. TikTok was unreachable, and the one control that could have
said otherwise could not be opened.
The condition now mirrors resolveTarget rather than paraphrasing it -
enabled precisely when its value is the one that gets used. Empty box
counts as using it, since it is the platform whatever is typed next will
be read as; a link does not, because the link settles the platform and
the API refuses one that disagrees rather than guessing. Post mode
behaves exactly as before.
The hygiene test pins the shape, not the condition: what must not come
back is disabling the menu on the mode alone. Checked it fails with the
old line restored.
Version bumped in the four places that carry it plus the lockfile.
The English README is now `README.md` and the Chinese moved to
`README.zh-CN.md`. Renamed with `git mv` so the history follows, and the three
things that referenced the old names - the language switcher in both files and
the wallet-address test - are updated. Nothing else in the tree pointed at
them, which the link checker confirms.
Two new sections, both above the fold:
* The live demo at douyin.wtf, with what it is for and what it is not. It says
the rate limit (30 requests per 10 seconds, then a 10-second cooldown) and
says plainly that it can go down, be reset or be switched off without notice -
somebody deciding whether to self-host should not first build against it.
* REER, the reverse-engineering forum, under the sponsors where it was asked
for. Its logo is in `logo/reer.png`, fetched from the site's own
apple-touch-icon at 512x512 and checked to be a real PNG of that size rather
than a 404 page with a .png on the end.
Release notes follow `.github/RELEASE_TEMPLATE.md`: English first, the same five
sections in both languages, upgrade steps above the feature list. Written from
`git log v5.0.0..HEAD` rather than from memory.
The cat already sat on the entry points. It now also sits on the four files
people actually come here to read - the pure-Python a_bogus, X-Bogus, the two
TikTok signatures, and the decoder that proved all of them - each with a line of
its own rather than the same comment copied four times.
Two in the console:
* Tap the version on About five times. It reveals the mark and six shields, and
it is behind a tap on purpose: those are images from shields.io, so loading
one tells shields.io that a browser opened this page. Every other screen here
makes no outbound request at all, and the README says exactly that about the
update check. A row of badges on a page everybody opens would have quietly
made it untrue. Open About and nothing is fetched; go looking and you have
asked. Verified: zero requests to shields.io before the fifth tap.
* The Konami code, anywhere, for two seconds of stars. Not a game - a game in an
operations console is something to maintain, and something that will one day
be open on the screen somebody is using to find out why the pool is empty.
The overlay is `pointer-events: none` so it cannot swallow a click, it does
nothing while focus is in a field, and it respects prefers-reduced-motion.
Two real bugs found while testing, both mine:
* The tap counter was state, so React batched a quick burst of clicks and every
one of them read the same stale zero - the fifth tap never arrived for anyone
who taps fast, which is what a person does when they suspect something is
there. It is a ref now; the count is never rendered.
* This repository has THREE no-CJK checks, not one: the Python sources, the
console sources checked from Python, and an eslint rule. Only the first had a
mark exemption. All three do now, all by shape rather than by filename, and
`tests/support/marks.py` splits the alphabet into the drawing - identical in
both languages, asserted by a new test against `web/eslint.config.js` - and
the syntax each language wraps it in, which differs. The drawing stays written
as escapes: a literal cat in the file that defines the exemption is CJK in a
Python source, and `ruff format` joins the lines into one that no longer looks
like a cat. That happened.
READMEs go from seven badges to twelve: forks, CodeQL, last commit, Python
version, MCP. Every one fetched and checked for a real answer rather than
assumed - a Docker `version` badge was dropped after it came back reading
`sha-a763431bfe2c…`, which is true and useless.
Three things, all found at 360px in a real narrow viewport rather than by
resizing a desktop window.
**The navigation drawer took the whole screen.** `.drawerStart` sets
`width: min(300px, 88vw)`, and the breakpoint below it sets `.drawer { width:
100vw }` for the details drawer. Same specificity, later rule, so the menu won
nothing and lost everything: it covered the page you were navigating away from
and left a column of dead space beside labels that are never more than a few
words. It is 300px or 80vw now, whichever is smaller. Measured: 426px viewport
leaves a 126px strip, 360px leaves 72px, 320px leaves 64px - enough to see
where you are, and enough to tap to close, which is the gesture people try
first.
**About and the sponsor were not in the mobile markup at all.** The pinned foot
was wrapped in `{!embedded ? … : null}` and the drawer renders the sidebar
embedded, so they were not cut off - they were absent, and the only route to
either was typing the URL. They render now, and `.embedded` gives the nav the
column layout `.sidebar` was carrying, so the foot sits at the bottom of the
drawer instead of floating up under the last nav group.
**The Tools page overflowed by 62px.** Five tabs in a `.u-row`, which does not
wrap - correct for a label beside its control, wrong for a strip of tabs. The
fifth sat past the right edge, unreachable, dragging the page into a horizontal
scroll. New `.u-row-wrap`, applied to the four tablists that share the pattern
(Tools, Library, Playground, Logs). Wrapping rather than `overflow-x: auto`
because these are navigation: a tab behind a scroll nobody can see is a tab
that does not exist.
Swept all nineteen console pages at 360px afterwards: no horizontal overflow
anywhere. Desktop re-checked at 1440px - sidebar still sticky at 240px, foot
still pinned, no drawer, no overflow.
Closes#743, which could not land on its own: `@vitejs/plugin-react@4` peer-caps
at vite 7, so bumping vite alone failed `npm ci` before any test ran. The plugin
goes 4.3 to 6.1 in the same change, which is what vite 8 requires.
Vite 8 is the rolldown build, so this is a bundler swap and `npm run build`
passing proves very little on its own. Verified by serving the production
`dist/` against a running API and walking every page in both languages -
playground, tools, library, downloads, logs, system, api keys, docs, mcp guide,
about - with an error listener attached: no runtime errors, titles and copy
translate, the pages that hold data render it. The dev pipeline is a separate
path through the same plugin and was checked too.
Two measurements worth writing down, because they point opposite ways:
* The build got roughly twice as fast: 1.15s to 0.53s.
* The main chunk got bigger: 612 kB to 664 kB raw, 206 kB to 219 kB gzipped.
That is an 8% payload increase on a console people self-host, and rolldown's
chunking is the only thing that changed. Worth revisiting if it does not
narrow in a later 8.x; it is under `chunkSizeWarningLimit` either way.
Merges #745 and #744, which had to land together: eslint-plugin-react-hooks 5
caps at eslint 9, so bumping either alone fails `npm ci` on a peer conflict.
The plugin's version 7 is the React Compiler rule set, and it found twelve
things. Two were worth fixing:
* `react-hooks/purity` on ApiKeys: `const now = Date.now()` in the render body,
listed as a dependency of two memos. A clock read during render is a new value
every render, so those memos never memoised anything. Replaced with the
query's `dataUpdatedAt`, which moves when the list is refetched and is also
the honest answer to "expired as of when".
* `react-hooks/immutability` on Downloads: `saveComments` and `saveAuthorPosts`
called `resolveTarget` two hundred lines before it was declared. It works
because the callbacks run after render, but it is a temporal dead zone with
nothing enforcing the deferral. Moved the declaration above its callers; every
variable it closes over is declared above it.
The other ten are `set-state-in-effect`, and all ten are the same shape: reset
local state when the thing it belongs to changes, or mark an async load in
flight. Closing a drawer on navigation, dropping a revealed cookie jar when the
drawer moves to another identity, clearing a session-check result that belongs
to one identity. None is a bug. Rewriting ten effects to satisfy a rule that has
not found a defect is a worse trade than saying so, so each site carries a
disable naming its reason - and the rule stays on, which is the part that
matters: a probe with a fresh violation is still rejected.
Also fixes a dev-server trap found while checking the pages above. Vite matches
a proxy prefix as a plain string, so `/api` swallowed the console's own
`/api-keys` route: opening it directly on :5173 proxied to the API, which
answered with the built console it serves, and the page came up blank with no
error anywhere. Everything the console calls is under `/api/v1/`, so the
trailing slash costs nothing.
Verified against a running instance: the API keys table renders, the status
column classifies, and the status filter returns 1 active and 0 expired; the
downloads page renders its index and totals.
Reported as "ticking explain on /playground says permission denied". That part
is the API behaving correctly and stays: an explanation is the signed upstream
URL, which carries the identity's msToken, plus the cookie jar itself as a
paste-ready header. A redacted version was considered and is not worth having -
strip the URL and the jar and what remains is the method, the endpoint and the
identity id, every one of which the demo console already reads off the logs
page. Redacting would also mean assembling a credential-bearing object and then
filtering it, and the filter is what somebody forgets the day a field is added
to Explanation.
So the fix is that the control is never offered. The Playground now disables
any parameter whose minRole outranks the session, keeps its value out of the
query, and puts the required role where the description was. `identity` gets
the same treatment: it is gated identically in the API and had the same latent
403 nobody had hit yet.
Found while confirming it: /auth/me sat behind `authenticated`, whose floor is
viewer, so a demo session was refused its own principal. That protected nothing
- /auth/demo publishes the account and its scopes in plaintext while the switch
is on - but it made the console unable to tell it was a demo, and navItemsFor
filters on exactly that. A demo visitor was being shown the whole sidebar:
Identities, Proxies, Watchlist, Endpoint access, Users, Diagnose, Backup,
Notifications, Settings, each answering 403 when pressed. Verified on the local
instance before and after: twenty nav items, then eleven.
Three integration tests cover it, including that the refusal names which field
was rejected. The /auth/me one was checked against the old guard first and
fails there.
Version goes from 5.0.0.dev0 to 5.0.0 in the four places that carry it.
`system.check_updates` now defaults to on. It was off with the note
"the user opts in rather than being opted in", which is the right
instinct aimed at the wrong thing: the check is made by the operator's
browser and never by the server, so it carries nothing about the
deployment. What it opts you into is finding out you are out of date,
and an instance that never learns that is the more common harm on a
self-hosted tool. Air-gapped boxes turn it off.
Added the notice that goes with it: after signing in, once per 24 hours
per browser, a modal if a newer release exists. Never for the demo
account — a visitor cannot upgrade anything, and telling them how the
operator runs their box is a small leak for no benefit.
The comparison moved to lib/version.ts and got fixed on the way. The old
one was an equality check that treated *anything* different as newer, so
an instance built from main at 5.1.0.dev0 was told to upgrade to the
older 5.0.0. It now understands that a pre-release is behind the release
of the same numbers, which is exactly the 5.0.0.dev0 → 5.0.0 case this
release creates. Seven cases checked, including that one.
Both READMEs get an Updating section: pull, migrate, up, and the fact
that named volumes mean the identity pool and archive stay put.
Also two files for publishing releases — a bilingual template and the
5.0.0 notes written against it. English first, breaking changes above
the feature list, and every entry says what the reader should do.
Not committed here: nothing is tagged or published yet.
A demo whose password lives only in a README is a demo most visitors
give up on. So the login page now fills the form in and they click, and
the API keys page shows the demo key in full so they can try the API.
Both need the pair to be readable, which is the one place this schema
stores a credential it can recover. Two nullable columns, encrypted with
the instance key like a cookie jar or a proxy URL, NULL for every other
account and every other key. Authentication is untouched — argon2 still
verifies the login and sha256 still resolves the key — so these are for
display and nothing else.
That NULL is the interesting part. An operator's key cannot be revealed
on the keys page because there is no plaintext to reveal, not because a
check says no, and the note under the table says exactly that. It is a
stronger promise than a permission would be.
GET /auth/demo is unauthenticated, because the login page is. It can
name one account — the one that exists to be published — and only while
the switch is on; off, it answers `enabled: false` and an instance that
never ran a demo is indistinguishable from one that stopped.
Also opened /api-keys to the demo account, which contradicts what I
listed earlier as closed. Leaving it shut meant a visitor could never
get the key, which defeats the API half of the demo. It is opened and
then filtered in the query to that one row, so the operator's names,
scopes, rate limits and last-used times are never loaded, let alone
drawn. Creating and revoking are still refused.
Plus the banner: on the login page, and on every console page while the
demo account is signed in. Keyed on the session's role rather than on
whether a demo is running, so the operator on their own instance is not
told they are looking at a demo.
Local stack rebuilt and migrated to 0009.
An instance on a public server needs a login somebody can publish, and
the role ladder alone does not give one: a viewer reads the identity
pool, the proxies, the keys and the settings, which on a machine anyone
can sign into is the whole instance.
So `demo` is a fourth role, ranked BELOW viewer. That ordering is the
safety argument rather than a detail. Every guard already defaults to
"viewer or better", so a role numbered underneath is refused by all of
them without one call site changing, and a route added next year is shut
to a demo box until somebody opens it deliberately. Two new guards,
demo_read and read_admin_demo, are what opening it looks like.
Turning on demo.enabled mints one account and one key and shows the
password and the key once — nothing stores either. Turning it off is
enforced on every request, not at login: a cookie is good until it
expires, so a check at the door would leave whoever was already inside
still inside. Nothing is deleted, so turning it back on works with the
same credentials.
Two rules, not one, decide what the pair can do. Scopes stop the
published key: it holds douyin:read and tiktok:read, so the request log
refuses it on its own. That is not enough for the session, which is
unscoped like every console session and would sail through the scope
check on POST /downloads — so every write is refused unless its route is
on a short allowlist. Deny-by-method with an allowlist, rather than a
gate per route, because there are 55 write endpoints today and the one
that gets forgotten is the one that matters.
What a demo request leaves behind is almost nothing. No request_log row
— the scheduler is unaffected, health and the circuit fold in through
Redis, so a demo caller still cools a burnt identity. No archive row and
no snapshot. A tasks row, which cannot be skipped because a task id has
to refer to something, so it is marked and swept in 30 minutes instead
of 90 days. Demo work also never coalesces with real work: sharing an
in-flight task would break the guarantee in both directions.
Two things the integration tests found that unit tests could not:
users.role carries a CHECK constraint that rejected the new value
outright, and the Users API would happily have hand-made a second demo
account with a chosen password and no key. Both fixed, both covered.
Rate limiting is deliberately not here — that is Cloudflare's job on the
box this is for.
**`?wait=` was a one-line summary for the one thing people hit first.** The data
endpoints are asynchronous by default, and nothing a caller reads said so, said
how to opt out, or said what a 202 after waiting means. So the front page of the
document now explains the three ways to get a result - poll, `?wait=`, or a
callback - and what each status means, and the parameter's own text says the
same in one paragraph rather than one clause. In particular: a 202 after the
wait elapses is not an error and loses nothing, and a value above the ceiling is
a 400 rather than a quiet clamp, because a caller that asked to block for five
minutes has to learn it cannot.
Both texts live in two places by necessity - `_translate` declines on the
default language, so `app.DESCRIPTION` and `WAIT_QUERY.description` are the
English document and the catalogue holds their translations. Tests pin the pairs
together, since drift between them is the obvious failure.
**Donations.** Five addresses on the About page and in both READMEs, with the
network first because that is the part that has to be read: an address pasted
into the wrong chain is unrecoverable, which the note under the list says
plainly. Ethereum and BNB Smart Chain sharing one address is said out loud too -
it reads as a copy-paste slip otherwise.
Each address is pinned in `test_wallet_addresses.py` and checked against its own
format's checksum: bech32 for Bitcoin, base58check for Tron, a 32-byte decode
for Solana, and EIP-55 for the EVM pair. The last needs Keccak-256, which is not
`hashlib.sha3_256`, so it is implemented there and self-tested against the
published digest of the empty string - a wallet address is the rare typo with no
error message and no recovery.
Also fixes a flaky test of my own from the previous commit. It failed once in a
few hundred runs, and the cause is documented on `test_signing.AB_NOW_MS`: the
SDK's 3->4 expansion drops a trailing zero byte when the body length leaves a
remainder of two, so one signature in 690 has an unverifiable checksum. That is
a property of the format - the recovered bundle shows `b` is a byte array, so
`if (b[i+1])` is falsy on zero in JavaScript exactly as in Python - and the
generator is faithful. The tests now pin the clock the way the signing tests
already did, and the one path that cannot pin it re-signs once, which is what
SignerRegistry does before believing a mismatch.
Two gaps, both invisible from the code.
**Request bodies were undocumented.** `schemas.py` carried zero `description=`,
so every field a caller has to fill in reached Swagger as a bare name beside an
input box - 85 of them. The knowledge was already written, as `#:` comments,
which readers of the source get and readers of Swagger do not; most of what is
added here is that text moved somewhere it can be seen. The envelope schemas
every response is described by had the same problem in five fields.
**Only part of the document was translatable.** Operations, tags and parameters
went through the catalogue; request-body fields had no lookup at all, so the
Chinese document explained what an endpoint did and then asked for
`skip_existing` in English. Added `openapi.field.<Schema>.<field>`, keyed by
model rather than by field alone because a field name is not global - `url` is a
share link on ParseRequest and a proxy address on ProxyCreate.
Of the 53 parameters in the document, 36 had no catalogue entry, `explain`
among them. All 53 now do. A parameter may also be worded per operation
(`openapi.param.<operation>.<name>`) where its name says nothing about its role:
`identity_id` filters the log on one route and names the identity being retired
on another, and the English source is already two different sentences there.
The envelope localisation had to move after `_type_responses`, which is where
those schemas are added - running it earlier translated every request body and
left DtkResponse and DtkError in English, which is what the new test caught.
`test_openapi_completeness.py` pins both halves: nothing unlabelled in English,
nothing without a CJK character in Chinese. The second is checked by looking for
Chinese rather than by diffing against the English, because the failure being
guarded produces two identical strings.
Also: the About page now carries the author's contact - the issue tracker and an
email - with a line on each saying which to use, since most of what arrives by
email would have been answered faster in the tracker.
Four defects behind one report, each measured against the links in it.
**Every two-hop Douyin share link was refused as a redirect loop.** `expand`
deduped hops by canonical form, and Douyin's chain is `v.douyin.com/X` ->
`iesdouyin.com/share/video/123?...` -> `douyin.com/video/123?previous_page=...`
- the last two normalize to the same canonical URL, so the second hop looked
like a repeat of the first. Two URLs with one canonical form ARE the same
resource by this package's own definition, so that is arrival, not a loop: it
now returns and a real loop terminates the same way, reported by `resolve` as
`unresolved_short_link`, which says what happened.
**The batch box split pastes on whitespace.** The input people have is a share
caption - `2.84 nqe:/ <title> https://v.douyin.com/L4FJNR3/ <sentence>` - which
is one link wrapped in prose, and splitting it produced four rows, three
nonsense and the fourth carrying the caption glued to the URL. Extraction needs
the host allowlist, which is a security control and lives on the server, so the
console asks `/tools/parse-batch` instead of keeping a second copy of it in
TypeScript. That endpoint now extracts every URL in a line rather than taking
the first, and a line it does not recognise is named in the console before
anything is submitted rather than after a task has failed.
**A deleted TikTok post cooled the identity that asked for it.** TikTok answers
one with HTTP 200 and both `status_code: 0` and `statusCode: 10204`;
`envelope_status` returned the first key present, found the zero, and the
response fell through to the rule that reads an envelope-with-no-payload as risk
control. So walking a list of old links counted toward the endpoint's risk rate
and could trip the breaker for the whole pool - the exact failure that module's
docstring opens by warning about. A non-zero sibling now wins, and 10204 is
named from the capture.
**A short link that dies is not an unrecognisable URL.** `www.tiktok.com/t/<slug>`
answers a server-side client with a 302 to TikTok's own front page. That was
reported as `unknown_resource`, which sends the reader off to check the one
thing that is not wrong; it is now `short_link_dead_end` with the URL it landed
on, and the console shows the reason beside the code.
Also: minting collects `navigator.hardwareConcurrency` and
`navigator.deviceMemory`, so `cpu_core_num` and `device_memory` follow the
identity like the rest of the profile. An identity that reports neither - every
one minted before this, and any Firefox - keeps the default rather than claiming
hardware nobody measured. `StaticFingerprint.of` replaces two copies of the same
derivation, one of which was the copy that dropped the screen; and the ops path
now passes a fingerprint-derived profile, like the fetch path.
Two things, both about a request describing itself honestly.
**A failure can now be explained.** Which is the case the feature exists for:
"it worked" rarely needs the request spelled out, and "it was refused" always
does. The explanation could not travel inside the error - errors are serialized
into the task row and rendered to every reader, while this contains a cookie
jar gated on identity:manage - so `fetch` writes it onto the context the caller
already owns, on the way past, before any raise. The worker stores it beside
the error and `_view_payload` emits it through the same `_visible_meta` gate it
uses on success. That gate is the part worth pinning: nothing else about a
failure is privileged, so a `result_meta` emitted beside an error without it
would read as entirely unremarkable. Verified live both ways - a read key sees
the failure and not the jar.
**The query string agreed with nothing.** Douyin and TikTok both echo the
screen, language, OS and browser version back in every query, right beside a
User-Agent carrying the same facts, and ours was a constant: every request
claimed Chrome 130 on a zh-CN Windows box while its own header said Chrome 146
and its Accept-Language said en-US. No browser produces that. `ClientProfile`
has documented itself as "derived from the identity's fingerprint by the
caller" since it was written; nobody derived it.
Now each adapter owns a `profile_for`, because the two platforms report the
same facts in different vocabularies - Douyin the marketing name and the real
version, TikTok the navigator product strings. Nothing is invented: the OS
table is lifted from Douyin's own chunk-43693 bundle, the browser version is
the substring its detector takes out of the User-Agent, and TikTok's
`5.0 (Windows)` / `os=windows` are reproduced exactly from a live capture. A
field the User-Agent does not state keeps its default - a value nobody can
vouch for is worth no more than one that is simply wrong.
The signer had the same bug one layer down: it was handed only a User-Agent, so
A-Bogus carried the fallback 1920x1080 Win32 geometry inside every signature
whatever the identity was - contradicting the screen_width the same request
sent in plain sight. For this pool the derived value is byte-identical, so
nothing changes today and it is correct for any identity minted later.
Measured after the change: Douyin returns the video, TikTok returns the
profile, and the signing self-check still reports douyin: match, tiktok: match
against the real browser.
The signature panel offered a calculator, which was an honest admission of a
gap rather than a fix for it: it signed a URL you typed, while the signature
that mattered was built in the worker against the platform's own URL and never
came back. That URL comes back now, so the panel shows this request's own
signature parameters, with a button to read them back into plaintext through
/tools/decode. The calculator is still on /tools, where signing a URL you type
is the whole point.
The pool card gained the other half. Which identity served a call was already
here on a failure and answers "who" and nothing else - a jar that has gone
stale and one that was never logged in look identical from an id. Beside it
now: the cookie jar that call actually sent, behind a click rather than behind
a mask, and the upstream request as a curl line.
That curl is the point of the whole thing. Reproducing a platform request by
hand is what anybody debugging one ends up doing, and it is where it goes
wrong: the query has to travel byte for byte because the signature covers it,
the User-Agent has to be the one it was hashed with, and the jar has to be the
identity's own. Verified by running the generated line against Douyin: 200,
87KB, the right aweme_id, title and author.
`explain` reported only the signer's headers, which would have produced a curl
missing the one header both platforms hash into the signature - it now reports
what build_headers is about to send, User-Agent and client hints included.
Client-side, a task's `result_meta` was being thrown away by `waitForTask`; it
is now returned beside `meta` rather than merged into it, so the worker's
report and this HTTP round trip cannot shadow each other's `duration_ms`.
The brand in the sidebar was the one thing on every screen that looked
like a link and was not. It goes to the overview now, which matters most
in the collapsed sidebar where it is the only thing left in that row. It
stays quiet when you are already there rather than looking like a dead
link.
The sign-in screen had a wordmark and nothing else. It is the one page
in this console anybody can reach, and the first one a stranger sees, so
it now carries the mark and a line saying where the software came from:
the source and the licence, plus the copyright.
Both footer links leave the console, and that is deliberate. The About
page inside it reads /system/status, which is not public, so a signed-out
reader who clicked it would be bounced straight back to the screen they
were already on. A link that returns you to where you started is worse
than no link.
Not verified in a browser: seeing the sign-in screen requires signing
out, and this session holds the operator's. Types, lint, the i18n
catalogue check and the build all pass over it.
`authenticated` on an identity was a statement about the paste - a
sessionid was in it - and not about the account. The cookie is there
whether or not the platform still honours it, and the two only ever
diverged silently: logged-in-only data quietly missing from otherwise
successful responses.
Both platforms publish an endpoint that answers about the cookies that
asked. Both are registered and both are checkable from the identity
drawer. Only one of them is a login check, and finding that out is most
of what this commit is.
**TikTok's /passport/token/beat/web/ works.** Measured both ways: a
logged-in jar answers error_code 0 with the account in user_id_str; a
guest answers 401, session_expired, user_id_str "0". A lapsed login
answers exactly like the guest, so this reports "signed out" rather than
guessing which - the platform does not distinguish them and neither
should we. Registered unsigned, because the call the page makes carries
no msToken, no X-Bogus and no _signature; appending them would make this
the one probe whose failure could mean the session is dead or the signer
is.
**Douyin's /aweme/v1/web/query/user/ does not.** It was proposed as the
Douyin equivalent and it looks like one, but a freshly minted guest gets
status_code 0 and a 16-digit uid of its own where a logged-in browser
gets a 19-digit account uid - same fields, same user_uid_type, nothing
between them. /passport/token/beat/ answers {"message":"success"} for
both as well. So Douyin reports "indeterminate" and says why. It still
returns the uid, which establishes the jar reached the platform and was
recognised; what it will not do is send somebody to re-import a working
jar over a verdict this endpoint cannot give.
The unit test holds all four real payloads side by side, because the
rule here was wrong when it was first written and only a measurement
said so.
Neither probe records an outcome against the identity it measures - a
check that cooled what it was diagnosing would move the state the
operator is reading - and neither goes through the worker's endpoint
registry, which is keyed by capability and needs an argument map for an
endpoint that takes no arguments.
Three asks, one shape: a jar could go into this instance and never come
back out of it, in any form.
**Reveal.** The drawer showed masked values, and a mask cannot be
compared against a browser's cookies - which is what somebody debugging
a jar is doing. Values are now readable through a route of its own:
operator role rather than any admin reader, an audit line per read, and
still masked on the view the drawer loads on every open. The disclosure
is a decision somebody made and a record somebody can read, which the
shell-and-hand-rolled-decrypt it replaces was not. Both copy shapes are
there because both are the next step: the Cookie header goes into curl,
the JSON goes into another tool.
**Export.** Select identities, get a document. An operator who had built
a pool that works could not move it to a second instance, keep a copy
before a risky change, or hand one identity to somebody debugging - the
only route out was the database. The result is a credential file and
says so in a field the importer ignores, so whoever finds it later knows
what they are holding.
**Restore.** The same document back in, through a route that is not the
paste route. Re-pasting an exported jar re-infers the browser from a
user agent no longer beside it, so the restored identity signs with a
different fingerprint from the one the platform first saw - which is
precisely the difference risk control looks for. Entries are judged one
at a time: a jar that expired since the export does not stop the rest,
and a file from a newer build is refused rather than read on a guess,
because dropping a field it does not know about is invisible until it
is not.
The drawer opened on a request history, which answers "what has this been
doing" and never "what is this". When an identity stopped working the
second question was the one people had, and answering it meant opening a
shell and decrypting the cookie column by hand - so the console said the
session was spent and the operator went elsewhere to find out why.
It now leads with the jar: every cookie in it, what each one is for, how
long its value is, and its first and last four characters. Plus the
things that were already known and not shown together - platform,
source, whether it carries a login, the browser and time zone it claims.
Values stay masked, and that is the design rather than a gap. A cookie
jar is the credential and a logged-in one is somebody's account; this
module's rule is that no response carries one. What the endpoint returns
instead is what makes a jar diagnosable. `length` in particular: a
truncated token and a wrong token look identical from outside, and the
length is the only thing that separates them.
Two accuracy fixes fell out of writing it.
`UIFID_TEMP` came back "other", so the console rendered "not something
this build reads" over the value the Douyin web signature SDK signs
with - the importer does not judge a paste on it, so nothing classified
it. The role now also consults the signer's own cookie names.
The glossary is grounded in this repository rather than in recollection:
UIFID_TEMP and s_v_web_id from websign.py, the fingerprint hints from
params.py. A cookie this build has never heard of gets its role's
sentence rather than a guess.
Tested against the raw response body, not a parsed field: no value
escapes anywhere, and no mask keeps more than eight characters.
CopyableId cut every value to twelve characters and put the rest in a
tooltip. That is a strange trade: the only thing anybody does with an id
is compare it to another one, and a prefix cannot settle that - two
uuids issued in the same second differ at the end. The prop that did it
is gone rather than defaulted off, because a component that can truncate
is one that will be asked to.
Fitting is CSS's job now. The text ellipsises at whatever width it
actually has, so it grows with the window instead of being cut to a
fixed character count. Where a column could not give a uuid its 36
characters, it now has a floor and the table scrolls - a cost the reader
can see, rather than one hidden in an ellipsis. `width` was not enough
on its own: auto layout treats it as a suggestion and overrode it, which
is why Column gained `minWidth`.
The drawer that started this showed `video.mp4` beside a button that
saves `dtk-douyin-…-video.mp4`, and a sha256 cut to twelve characters
with nothing saying which algorithm made it. It now shows the name the
reader will get - derived by the API from the same policy that writes
the Content-Disposition, so the two cannot disagree - and the whole
digest, wrapped, labelled SHA-256. A truncated digest verifies nothing,
which is the only thing a digest is for.
Also whole: the git commit on the overview (git's short form has to be
expanded before it is any use to someone who was not standing at this
checkout) and the identity id in the playground's picker, where an
<option> has neither hover nor a copy button to fall back on.
API keys are untouched. They go through MaskedSecret, which is a
different component for a different reason, and `dtk_847132c94367_...`
is what that page should keep showing.
Evil0ctal