While looking into #6009 I noticed the Bing fallback in -g never
returns anything, even when Bing itself serves a normal results page.
Two things changed on Bing's side. The result anchor now carries
target="_blank" before href, so '<h2><a href="..." h=' no longer
matches at all. And the href is no longer the target URL but a
tracking redirect (https://www.bing.com/ck/a?...&u=a1<base64url>),
where the real link sits base64url-encoded after the "a1" prefix
in the u parameter.
The regex now allows extra attributes on both tags and captures
either the u payload of a ck/a redirect or a plain href. The payload
is decoded with decodeBase64(), which already deals with the
URL-safe alphabet and the missing padding. Plain hrefs still work
the same as before (with & unescaped), in case Bing serves the
old markup to some clients.
Checked against a saved live results page (0 -> 10 links) and the
old markup, and --smoke-test passes. Google still can't be scraped
since it only serves a JS page now, so this doesn't fully close
#6009, but Bing is usable again as a fallback when DuckDuckGo blocks
the IP.
A scan covering multiple targets inside a single API task (crawl,
'-m' bulk file, forms) ran every target through one StdDbOut keyed by
the single taskid, so _storeData()'s COMPLETE branch deleted each
earlier target's TARGET/TECHNIQUES rows - /scan/<id>/data returned
only the last target's results.
Mirror the --report-json fix for #6123: controller.start() now sets
kb.reportTaskId per target for API scans too, StdDbOut.write() keys
data rows '<taskid>-<n>' per target, and scan_data() folds the
per-target rows back into the task's flat result in target order.
Single-target scans keep the exact same response shape.
Regression tests in tests/test_api.py (TestStdDbOutMultiTarget,
TestScanDataMultiTarget).
Every tests/test_*.py file provides an
if __name__ == "__main__":
unittest.main(...)
entrypoint so it can be run directly, except test_ssti.py and
test_xpath.py. Running either of those directly produced no output and
executed none of its tests; they only ran under `unittest discover`.
Add the missing entrypoint to both so they behave like the rest of the
suite. No test logic changed (62 and 48 tests respectively still pass).
JWT_REGEX accepts a signature segment of any length and parseJWT only
validates the header/payload, so a value that matches the JWT pattern but
carries a malformed/truncated signature (e.g. 41 base64url characters,
which is length % 4 == 1 and thus impossible base64) reached crackHMAC.
There decodeBase64(signature) let binascii.Error propagate, aborting the
whole run with an unhandled exception during checkJWT().
Guard the signature decode: a signature that is not valid base64url cannot
be an HMAC we could verify, so crackHMAC now treats it as not crackable
(returns None) instead of raising. The other auditJWT findings are
unaffected. Added a regression doctest.
Co-authored-by: chan770 <tsramachandran770@gmail.com>
Raju Ranjan