Error
ValueError: Not many councillors found (0)
Root cause
www.rbkc.gov.uk is Cloudflare-protected (confirmed: response headers include server: cloudflare and set-cookie: __cf_bm=...). The wreq Haskell HTTP client does not pass the Cloudflare Bot Management JS challenge, so instead of XML, the ASMX endpoint https://www.rbkc.gov.uk/Committees/mgWebService.asmx/GetCouncillorsByWard returns the Cloudflare "Attention Required!" HTML challenge page.
Unlike most Cloudflare-blocked councils, this one does not return HTTP 403 — instead the challenge page returns 200, so raise_for_status() doesn't fire. The HTML is then parsed as XML (or fails gracefully) and yields 0 wards/councillors, triggering the "Not many councillors found" guard.
Fix patterns ruled out
- HTTPS migration:
base_url is already https://www.rbkc.gov.uk/Committees ✓
- Cert errors: no cert error, the site is accessible
- URL changes: endpoint is reachable (returns Cloudflare HTML, not 404)
Suggested fix
Add http_lib = "playwright" to the scraper class. Headless Chromium will solve the Cloudflare JS challenge and obtain the real XML from the ASMX endpoint. This is the same pattern as PR #407 (BPC) and the five PRs opened on 2026-08-02 for other Cloudflare-blocked ModernGov councils.
class Scraper(ModGovCouncillorScraper):
http_lib = "playwright"This fix couldn't be applied and verified in the current run because this session had already reached the 5-PR limit for the day.
Error
Root cause
www.rbkc.gov.ukis Cloudflare-protected (confirmed: response headers includeserver: cloudflareandset-cookie: __cf_bm=...). The wreq Haskell HTTP client does not pass the Cloudflare Bot Management JS challenge, so instead of XML, the ASMX endpointhttps://www.rbkc.gov.uk/Committees/mgWebService.asmx/GetCouncillorsByWardreturns the Cloudflare "Attention Required!" HTML challenge page.Unlike most Cloudflare-blocked councils, this one does not return HTTP 403 — instead the challenge page returns 200, so
raise_for_status()doesn't fire. The HTML is then parsed as XML (or fails gracefully) and yields 0 wards/councillors, triggering the "Not many councillors found" guard.Fix patterns ruled out
base_urlis alreadyhttps://www.rbkc.gov.uk/Committees✓Suggested fix
Add
http_lib = "playwright"to the scraper class. Headless Chromium will solve the Cloudflare JS challenge and obtain the real XML from the ASMX endpoint. This is the same pattern as PR #407 (BPC) and the five PRs opened on 2026-08-02 for other Cloudflare-blocked ModernGov councils.This fix couldn't be applied and verified in the current run because this session had already reached the 5-PR limit for the day.