DUR (County Durham) scraper failing — TCP timeout from Lambda, TLS reset from proxy

[symroe]

Error (2026-07-22)

is_connect error: wreq::Error { kind: Request, uri: https://democracy.durham.gov.uk/mgWebService.asmx/GetCouncillorsByWard, source: Error { kind: Connect, source: Some(ConnectError("tcp connect error", 217.23.233.70:443, Os { code: 110, kind: TimedOut, message: "Connection timed out" })), connect_info: None } }

The scraper has been failing for 10+ consecutive days.

Investigation

From the automated run environment:

• GET https://democracy.durham.gov.uk/mgWebService.asmx/GetCouncillorsByWard from Lambda → TCP connection timeout to 217.23.233.70:443
• From a non-Lambda proxy environment, the TCP connection establishes through CONNECT tunnel but the TLS handshake is immediately reset by the server (Recv failure: Connection reset by peer during SSL_connect), even with --insecure (so it is not a certificate verification issue)

This means the server is actively rejecting connections at the TLS level from automated environments — Lambda gets an IP-level firewall block, while other IPs reach the TLS layer but are still reset (likely TLS fingerprint-based filtering).

Fix patterns ruled out

1. HTTPS migration — already HTTPS (base_url = "https://democracy.durham.gov.uk")
2. verify_requests = False — the failure is a TLS connection reset by the server, not a client-side certificate verification error; --insecure made no difference
3. http_lib = "playwright" — would help with Lambda IP block, but the TLS reset from non-Lambda IPs suggests a deeper network block that playwright cannot bypass
4. URL changes — DNS resolves correctly to 217.23.233.70; the server is reachable at the network layer but blocking at TLS/IP level

What needs to happen

The Durham County Council democracy portal appears to be filtering automated requests at a network/TLS level. Options:

• Check whether democracy.durham.gov.uk has moved to a new hostname or IP (the council may have reconfigured their firewall)
• Contact Durham County Council to request whitelist access for the scraper
• Monitor whether access is restored (could be a temporary firewall misconfiguration)