LDS (Leeds) scraper failing — IP-level block from Lambda, both HTTP and HTTPS return 403/timeout

[symroe]

Error (2 consecutive failures: 2026-08-10, 2026-08-11)

is_timeout error: wreq::Error { kind: Request, uri: http://democracy.leeds.gov.uk/mgWebService.asmx/GetCouncillorsByWard, source: TimedOut }

Root cause

democracy.leeds.gov.uk (hosted on AWS ELB at 18.134.217.164) is blocking Lambda's IP range at the load balancer level:

• HTTP (http://democracy.leeds.gov.uk): wreq in Lambda → TCP timeout (30s); direct curl without proxy → 403 Forbidden from awselb/2.0
• HTTPS (https://democracy.leeds.gov.uk): curl → 403 Forbidden from awselb/2.0

Both HTTP and HTTPS return 403 from this environment, and the Lambda scraper times out (rather than receiving the 403 quickly), suggesting the ELB drops the connection rather than responding to Lambda IPs.

The scraper ran successfully every day from 2026-08-02 through 2026-08-09 then failed on Aug 10–11 — this may be a temporary block or a recent firewall change.

Fix patterns ruled out

1. HTTPS migration: HTTP (http://democracy.leeds.gov.uk is the current base_url) times out from Lambda; HTTPS also returns 403 from other IPs — no improvement possible
2. Cert error / verify_requests = False: No cert error; this is an IP-level 403/timeout
3. Playwright / http_lib = "playwright": Would not help — block is at ELB/network layer before TLS, not a JS challenge
4. URL change: ASMX endpoint URL is correct; block is network-level
5. use_proxy = True: Could help if the proxy's egress IP is not blocked, but this needs testing

What's needed

• Re-run the scraper in a few days — the block started only 2 days ago and may self-resolve
• If it persists, try adding use_proxy = True to the scraper to route traffic through a static proxy IP
• Or contact Leeds City Council to ask whether they intentionally blocked Lambda egress IPs from democracy.leeds.gov.uk