317 Commits

Author SHA1 Message Date
amhsirak 72fd752083 chore: core v0.0.38 2026-08-24 23:03:12 +05:30
Santosh-Prasad-Verma 12d931e5e7 fix(maxun-core): check active execution before resetting isAborted
Move this.stopper guard to top of run() before resetting isAborted.
This ensures an in-flight execution's abort flag cannot be accidentally
cleared if run() is invoked while the interpreter is already active.
2026-08-19 00:48:07 +05:30
Santosh-Prasad-Verma 10922d114f fix(maxun-core): prevent orphaned crawl loops after timeout
Move isAborted flag reset from cleanup() to run() so that in-flight
crawl loops see the abort flag during teardown and exit cleanly,
preventing CPU/RAM leaks from timed-out runs.

Previously, cleanup() reset isAborted = false during teardown, which
accidentally un-aborted active async crawl loops. The loops would then
continue running against a closed Playwright browser context, consuming
CPU/RAM and flooding server logs with errors.

Now, isAborted is only reset at the start of a new run() call, which is
the only time it should be cleared. This covers all three timeout
triggers: task-runner, scheduler, and API record handler.

Refs: #1183
2026-08-16 03:30:35 +05:30
Karishma Shukla 79b1e995c8 Merge pull request #1130 from getmaxun/strip-hidden
fix: strip hidden base64-encoded data blobs
2026-07-15 23:06:42 +05:30
hmhngx 0c0033216f chore: drop jest test infra to keep PR scoped to persistence fix
Per maintainer review (RohitR311 on #1138): the repo has no existing
test infra or CI to run tests, so introducing jest here is out of
scope. Removes jest configs, test files, tsconfig.test.json, the
test:server script, and jest/ts-jest/@types/jest/jest-environment-jsdom
dependencies from both maxun-core and server.

output-validation.ts stays - it's a real production module (not test
code) extracted to fix the duplicated isPartial bug CodeRabbit flagged;
only its test file is dropped.
2026-07-13 21:16:32 +07:00
hmhngx 675dc94b76 fix(maxun-core): exclude tests from the production build
Discovered while actually building maxun-core to verify the fix end-to-end:
running `npm run build` compiled with type errors (test files pulled into
the production tsconfig with no jest types, plus a pre-existing implicit-any
in concurrency.ts) and still silently emitted output anyway. Exclude
src/__tests__ from the shipped build and fix the implicit-any so `npm run
build` is actually clean, not just non-fatal.
2026-07-10 10:45:22 +07:00
hmhngx 616d3bf8d4 chore: core v0.0.37 2026-07-10 10:45:22 +07:00
hmhngx d8d42b9da6 test(maxun-core): add per-page persistence tests for crawl/search
maxun-core had no test infra at all (jest was declared in package.json
but never installed, no config, no test files). Add jest/ts-jest and a
jsdom-backed mock Page, and drive the real Interpreter through crawl
and search-scrape workflows to prove serializableCallback fires once
per page with a strictly growing snapshot - not once at the end - and
that a mid-run failure or abort still leaves prior pages recorded.
2026-07-10 10:45:22 +07:00
hmhngx 728d03ece7 fix(maxun-core): persist crawl and search results after every page
The crawl and search-scrape actions accumulated every visited page into
a local array and only handed it to the persistence callback once,
after the whole page-by-page loop finished. If the run was killed,
timed out, or errored mid-loop, everything scraped so far was lost
even though it had already been extracted.

Persist the running snapshot after every page (success or failure),
matching the pattern scrapeList pagination already uses. Also stop the
search scrape loop promptly on abort, matching crawl's existing check.
2026-07-10 10:45:22 +07:00
asherdrake 0fc11a694e fix: address CodeRabbit review on single-pass screenshot capture
- validate formats in PUT /runs/:id before persisting, so a malformed
  value can't reach the interpreter and break formats.includes(...)
- re-throw on failed search-result navigation so it records an error
  entry instead of scraping/screenshotting the stale page (matches the
  crawl loop's behavior)
- correct stale getNavigationWaitStrategy JSDoc (networkidle is driven
  by visualRenderRequired, not screenshot formats)
- fix duplicated phrase in processRobotOutputFormats JSDoc
2026-07-05 16:03:03 -07:00
asherdrake b7efd26e39 don't block crawl/search navigation on networkidle when screenshots are enabled 2026-07-05 15:31:27 -07:00
emperor a7283d0c31 fix: capture crawl/search screenshots in single scraping pass reducing time from 2N pages just down to N (#1105) 2026-07-04 16:05:51 -07:00
Rohit Rajan bbc91ce52e fix: rm hidden input search scrape 2026-06-29 21:54:19 +05:30
amhsirak afc5c54258 fix: strip hidden blob data 2026-06-29 16:33:29 +05:30
Rohit 42f97deab1 Merge pull request #1125 from getmaxun/pre-release-43
chore: pre-release v0.0.43
2026-06-24 22:50:04 +05:30
amhsirak eda52a5d49 chore: core v0.0.36 2026-06-24 22:22:26 +05:30
Rohit Rajan ac23aeb0b7 fix: rm list fallback 2026-06-24 22:20:05 +05:30
Rohit Rajan c02573254c fix: add selector waiting mechanism 2026-06-24 22:17:56 +05:30
Rohit Rajan 1080c77081 fix: add visited signature check 2026-06-20 12:20:38 +05:30
Rohit Rajan a3d49312cf chore: rename to wait for page ready 2026-06-20 12:13:42 +05:30
Rohit Rajan 711f25d333 fix: extract engine speed, accuracy enhancement 2026-06-20 11:39:23 +05:30
amhsirak 13442adb30 chore: core v0.0.35 2026-06-10 23:39:01 +05:30
Karishma Shukla 478f759e57 Merge pull request #1094 from getmaxun/hybrid-pag
fix: support for hybrid pagination
2026-06-10 22:31:42 +05:30
Rohit Rajan 82c5564d7e chore: rm working selector condition 2026-06-10 17:04:05 +05:30
Rohit Rajan f404ed8835 fix: support for hybrid pagination 2026-06-10 17:00:42 +05:30
Rohit Rajan 2c1fc9182b fix: centralize data formats 2026-06-08 18:51:14 +05:30
amhsirak 393eeb6810 chore: core v0.0.34 2026-05-17 23:27:34 +05:30
Rohit Rajan 8b1329f01d feat: add support for document robot 2026-05-14 20:16:01 +05:30
amhsirak 83d45302e7 chore: v0.0.33 2026-04-20 22:19:40 +05:30
Rohit Rajan 78af841576 fix: enforce dynamic wait for load state 2026-04-20 22:13:10 +05:30
Rohit Rajan ec48d29b3d feat: fix type errors, add custom network wait check 2026-04-19 18:55:49 +05:30
Rutetid 2443007ce9 applied all fixes 2026-03-29 20:29:02 +05:30
Rutetid bcfcaad2da Added Output Format Selection to Crawl and Search Robots 2026-03-29 19:17:21 +05:30
amhsirak 5861834906 chore: core v0.0.32 2026-02-25 21:03:23 +05:30
Karishma Shukla 1441caf45f fix: normalize scrape schema 2026-02-24 22:54:20 +05:30
amhsirak ca94044628 chore: core v0.0.31 2026-02-15 02:25:02 +05:30
Karishma Shukla d2441e9e90 Merge pull request #964 from getmaxun/fix-page
fix(core): scroll down pagination
2026-02-15 02:18:14 +05:30
Rohit Rajan c31fdfd41f fix: add incremental scrolling 2026-02-05 21:40:44 +05:30
amhsirak 83f28f4746 chore(ts): set skipLibCheck true 2026-01-23 18:18:49 +05:30
Karishma Shukla 3db4f918ff chore: core v0.0.30 2026-01-23 14:43:22 +05:30
Rohit Rajan 19f2df87fc fix: allow robots.txt, max depth config 2026-01-19 00:38:06 +05:30
amhsirak 2c795af858 temp: remove adblocker 2026-01-04 22:42:22 +05:30
amhsirak fc98f536cf chore: core v0.0.29 2026-01-04 22:35:23 +05:30
Rohit 8ab2defdeb Merge branch 'develop' into crawl-search 2026-01-04 18:21:47 +05:30
Rohit Rajan 9898dc410d feat: add server side crawl and search logic 2026-01-02 15:46:10 +05:30
amhsirak 36b80da631 feat(core): real time progress update for runs 2026-01-02 11:18:23 +05:30
amhsirak c963febbe6 chore: core v0.0.28 2025-11-30 20:52:26 +05:30
Rohit 8c653f113d Merge branch 'develop' into browser-service 2025-11-30 19:26:42 +05:30
Rohit Rajan cf19a72dc0 feat: add separate browser service 2025-11-30 17:41:44 +05:30
amhsirak bf1a6f1aa7 fix(core): clean socket handlers causing memory leaks 2025-11-28 15:03:43 +05:30