NKE (North Kesteven) scraper failing — DNS NXDOMAIN for democracy.n-kesteven.gov.uk from Lambda VPC

[symroe]

Error (2026-07-19, 10+ consecutive failures)

exceptions.ConnectionError: is_connect error: wreq::Error { kind: Request,
uri: https://democracy.n-kesteven.gov.uk//mgWebService.asmx/GetCouncillorsByWard,
source: Error { kind: Connect, source: Some(ConnectError("dns error",
ResolveError { kind: Proto(ProtoError { kind: NoRecordsFound { query: Query {
name: Name("democracy.n-kesteven.gov.uk.eu-west-2.compute.internal."), ...
NXDomain ... } }) })) } } }

Root cause

democracy.n-kesteven.gov.uk does not resolve from Lambda's eu-west-2 VPC. The wreq DNS resolver appends the VPC search domain (.eu-west-2.compute.internal.) and gets NXDOMAIN.

However, the domain resolves fine from the public internet: www.n-kesteven.gov.uk/councillors redirects to https://democracy.n-kesteven.gov.uk/mgMemberIndex.aspx?bcr=1, confirming the democracy subdomain is valid and live.

From automated environments (proxy, this build), the domain resolves but the F5 BIG-IP load balancer resets the TCP connection (ConnectionReset), suggesting IP-based bot protection.

What was ruled out

• HTTPS migration: already on HTTPS; trailing slash in base_url causes a harmless double-slash (//mgWebService.asmx) in the URI but is not the root cause
• Cert error: connection fails before TLS
• playwright: also gets net::ERR_CONNECTION_RESET from automated environments
• URL change: domain still active; the ModGov member index page loads correctly from a browser

Observations

• Note: base_url in metadata.json has a trailing slash (https://democracy.n-kesteven.gov.uk/) which causes a double slash in the constructed ASMX URL. Worth fixing if/when the connectivity issue is resolved.
• The council's Drupal site at www.n-kesteven.gov.uk explicitly whitelists democracy.n-kesteven.gov.uk in its external link exclude list, confirming the domain is intentional.

Suggested next steps

1. Try adding use_proxy = True — the proxy may resolve DNS externally and reach the democracy subdomain through a non-Lambda IP
2. If the proxy also gets a connection reset, this site is actively blocking cloud/proxy IPs and may need an IP allow-list arrangement with the council or ModGov
3. Fix the trailing slash in base_url as a housekeeping improvement

Opened by LGSF maintenance bot (2026-07-21)