- Keep aiExtract (spec and scraped sample rows) out of the robot-created
telemetry event, in the app and SDK routes.
- Send OLLAMA_API_KEY only to the operator's Ollama endpoint, not to a
base URL chosen per request; a key given with the request is still sent.
- Reset the common-class cache with the path cache per list, so field
XPaths aren't built from another list's classes.
- ENVEXAMPLE: use host.docker.internal for Ollama, as Docker Compose does.
- AI Extract tests fail when Chromium can't launch; AI_EXTRACT_SKIP_BROWSER=1
skips them explicitly.
- SDK extract/llm: generation failures return 400 again, as before.
Signed-off-by: amhsirak <karishmashuklaa@gmail.com>
"Extract 15 products" still returned 12: the generic pagination detector
found nothing. Its patterns miss common markups, e.g. "Next →" / "Next »"
(the arrow breaks the /^next$/ match) and numbered pages without a Next
link, and it never looks at rel="next".
AI Extract now tries its own finder first (findNextPageControl in
aiExtractPage.js), then the generic detector:
- rel="next" links
- Next / Load more controls, ignoring arrows, preferring ones inside a
pagination container; disabled controls are skipped
- the link after the current page number (aria-current="page" or an
active/current marker)
Selectors are comma-free (the runtime splits on commas), checked to match
the chosen control, and must stay on the same site. Every decision is
logged, including the pagination labels seen when nothing matches.
Tests cover "Next →", numbered-only and arrow-only rel=next markup, each
replayed across pages in maxun-core's interpreter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LCfkReTCeeFhy3mGCQYNDj
"Extract 15 products" on a page showing 12 returned 12: list robots only
allowed none/scrollDown pagination.
When the prompt asks for a specific count and the page shows fewer items
(and the page does not load more as it scrolls), the robot now uses the
page's own next-page or load-more control, capped at the requested count.
- Detection only reads the page; creating the robot still never leaves it.
- A control that links to another site is refused.
- No count, or a count the first page satisfies: no pagination.
- assertWorkflowInScope allows clickNext/clickLoadMore only when the
scrapeList limit equals the requested count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LCfkReTCeeFhy3mGCQYNDj
AI Extract never sent an API key to Ollama and fell back to the local
default model, so Ollama Cloud answered 404 and the user only saw
"Request failed with status code 404".
- Send OLLAMA_API_KEY (or the configured key) as a Bearer token.
- Use OLLAMA_MODEL when no model is configured.
- Report the provider's own error, the URL called, and what to check.
- Document OLLAMA_BASE_URL / OLLAMA_API_KEY / OLLAMA_MODEL in ENVEXAMPLE.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LCfkReTCeeFhy3mGCQYNDj
Port the Cloud AI Extract pipeline (getmaxun/maxun-cloud-recording#656,
#657) to OSS, adapted to the OSS provider config.
- Website URL is now required for AI Extract (UI, /storage/recordings/llm
and /api/sdk/extract/llm); the search and multi-site fallbacks are no
longer used.
- Extraction is driven by the prompt: prompt -> spec -> id-tagged page
snapshot -> grounded citations -> compiled selectors verified with
maxun-core's scraper. No fallback to group selection.
- Page scope is enforced in the browser: navigation requests to other
documents are aborted, window.open is disabled, route changes are
undone, cross-site redirects are refused, and generated robots may only
use none/scrollDown pagination.
- Overlays, lazy loading and collapsed sections are revealed on the same
page with fixed, non-model steps.
- Spec, sample rows and missing fields saved in recording_meta.aiExtract.
- Tests: npm run test:ai-extract (unit + browser integration).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LCfkReTCeeFhy3mGCQYNDj
Adds a Monitoring switch under Options for Scrape and for Extract in
both recorder and AI mode. Scrape and AI robots get compareRuns set
right after they are created; recorder robots carry it to the save in
the recording tab, which stores it on the new robot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hskm8VdJhy1krbeJZMXCcS
A page over 20 segments put the rest of the page in the last one, which
the prompt then cut, so a large CSV or sheet silently lost its tail
(a 6,000-row CSV came back with 2,302 rows). Pages are now split into as
many segments as they need, as multi-page documents already are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hskm8VdJhy1krbeJZMXCcS
- buildChunks counts each unit's label and separator against the budget;
the prompt's chunk cap is now a safety net above it, so packed chunks
no longer lose their last rows.
- Retry halves of a cut-off chunk run one at a time, keeping requests
within EXTRACTION_CONCURRENCY.
- salvageTruncatedJson cuts only between array elements or top-level
fields, so a record cut mid-way is dropped instead of kept with
missing fields.
- extractBalancedJson skips brackets in prose ("[see notes]") and only
treats an unclosed JSON-looking value as cut-off output.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hskm8VdJhy1krbeJZMXCcS
Completes the port of the document extraction fixes: the helpers were
added but not used yet.
- Chunking: split oversized pages (CSV/DOCX are one page), repeat table
headers across chunks, label sheets, size record chunks from the
document's own table rows.
- Providers: report hitting the output limit (Ollama done_reason, OpenAI
finish_reason, Anthropic stop_reason) and salvage the complete JSON.
- Extraction: chunks run 3 at a time, unwrap envelopes, re-extract
cut-off chunks in halves (up to 3 levels), tables only as context for
single-chunk documents, part N of M prompts with the JSON shape.
- PDF: per-page tables from pdf-parse laid out in place.
- XLSX/CSV: one Markdown table per sheet. DOCX: Markdown with tables.
- Markdown/HTML output rendered from page blocks.
- Schema prompt models repeated records as an array of objects.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hskm8VdJhy1krbeJZMXCcS
updateGoogleSheet passed the screenshot rows wrapped in another array,
so processOutputType flattened the array itself: the Screenshot sheet
got headers "0", "1", ... and a single row of JSON strings instead of
"Screenshot Key" / "Screenshot URL" rows. Because the column count
follows the number of screenshots, the header check also failed on
later runs and re-appended a header row.
Pass the rows array directly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
API, SDK and scheduled runs have hung since socket auth was added.
`authenticateSocket` is applied to every namespace via `io.on('new_namespace')`,
and it requires a token in `handshake.auth` or a `token` cookie. Browser clients
carry the httpOnly cookie, so the UI is unaffected. But the backend also joins
its own `/<browserId>` namespace as a socket.io *client* when a run is started
by something other than a browser, and that connection has neither a cookie nor
an auth token. It is refused.
The refusal is not what makes this hard to spot. `connect_error` is logged and
the socket cleaned up, but the run is never failed - so the browser eventually
falls back to a dummy socket, `ready-for-run` never reaches the caller, and the
run sits there. It reads as "slow", not as "broken".
Both server-opened connections are affected and they are the three paths people
report: `api/record.ts` (API and SDK) and `workflow-management/scheduler`
(scheduled runs).
Fixed by minting a short-lived token with the same `id` claim the HTTP login
issues, so the gate needs no special case and `socketOwns` keeps working: the
connection authenticates as the user the run already belongs to, not as a
privileged internal caller. A 60-second expiry because it is used once,
immediately - a login token is deliberately long-lived, which is right for a
session and wrong for a handshake.
Verified by A/B on one stack, same database, only the image changing:
before run hangs, 120s deadline exceeded
"Socket connection error for API run ...: Unauthorized"
"proceeding with dummy socket" -> "[SDK] Error executing robot: Run not found"
after run completes in 18.6s
0 Unauthorized socket errors, 0 dummy socket fallbacks
And the gate itself, against `/queued-run`: no auth is refused, a minted token
connects, a token signed with the wrong secret is refused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Called downscaleIfOversized immediately after writing the temp file.
Also switched the resized dimensions from Math.round to Math.floor so the integer result cannot exceed MAX_OCR_PIXELS.
amhsirak