While looking into #6009 I noticed the Bing fallback in -g never
returns anything, even when Bing itself serves a normal results page.
Two things changed on Bing's side. The result anchor now carries
target="_blank" before href, so '<h2><a href="..." h=' no longer
matches at all. And the href is no longer the target URL but a
tracking redirect (https://www.bing.com/ck/a?...&u=a1<base64url>),
where the real link sits base64url-encoded after the "a1" prefix
in the u parameter.
The regex now allows extra attributes on both tags and captures
either the u payload of a ck/a redirect or a plain href. The payload
is decoded with decodeBase64(), which already deals with the
URL-safe alphabet and the missing padding. Plain hrefs still work
the same as before (with & unescaped), in case Bing serves the
old markup to some clients.
Checked against a saved live results page (0 -> 10 links) and the
old markup, and --smoke-test passes. Google still can't be scraped
since it only serves a JS page now, so this doesn't fully close
#6009, but Bing is usable again as a fallback when DuckDuckGo blocks
the IP.
A scan covering multiple targets inside a single API task (crawl,
'-m' bulk file, forms) ran every target through one StdDbOut keyed by
the single taskid, so _storeData()'s COMPLETE branch deleted each
earlier target's TARGET/TECHNIQUES rows - /scan/<id>/data returned
only the last target's results.
Mirror the --report-json fix for #6123: controller.start() now sets
kb.reportTaskId per target for API scans too, StdDbOut.write() keys
data rows '<taskid>-<n>' per target, and scan_data() folds the
per-target rows back into the task's flat result in target order.
Single-target scans keep the exact same response shape.
Regression tests in tests/test_api.py (TestStdDbOutMultiTarget,
TestScanDataMultiTarget).
Every tests/test_*.py file provides an
if __name__ == "__main__":
unittest.main(...)
entrypoint so it can be run directly, except test_ssti.py and
test_xpath.py. Running either of those directly produced no output and
executed none of its tests; they only ran under `unittest discover`.
Add the missing entrypoint to both so they behave like the rest of the
suite. No test logic changed (62 and 48 tests respectively still pass).
Miroslav Štampar