SFK (Suffolk) scraper returning 0 councillors — possible WAF block or scraper exception

[symroe]

Error (2026-07-20)

Scraper runs without a hard crash but returns 0 councillors.

Investigation

From our proxy environment (curl):

• https://www.suffolk.gov.uk/council-and-democracy/councillors-and-committees/find-your-councillor → 200 OK, 460KB HTML
• The page contains 70 .result elements matching the councillor_css_selector — data is present
• No obvious Cloudflare headers; x-origin-server: backend-coordinator suggests a custom WAF/proxy layer

Possible causes:

1. wreq TLS fingerprint blocked: The server may return a WAF challenge page (not valid HTML) to wreq, while curl succeeds because it presents a standard TLS fingerprint. The challenge page would have no .results container, producing 0 results with no error.
2. Silent exception in get_single_councillor(): An exception on a required field (e.g. councillor_html.select_one(".division .value") returning None) could silently skip all councillors if the CSS selectors have changed.
3. JavaScript-gated content: The councillor list may require JS rendering not present in the wreq response.

Current scraper (councillors.py)

class Scraper(HTMLCouncillorScraper):
    list_page = {
        "container_css_selector": ".results",
        "councillor_css_selector": ".result",
    }
    def get_single_councillor(self, councillor_html):
        url = urljoin(self.base_url, councillor_html.select_one("a")["href"])
        identifier = url.rstrip("/").split("/")[-1]
        name = councillor_html.find_all("a")[0].text.strip()
        division = councillor_html.select_one(".division .value").get_text(strip=True)
        party = councillor_html.select_one(".party .value").get_text(strip=True)
        councillor = self.add_councillor(url, identifier=identifier, name=name, party=party, division=division)
        text = self.get_text(url)
        soup = BeautifulSoup(text, "lxml")
        email_el = soup.select_one("a[href^=mailto]")
        if email_el:
            councillor.email = email_el["href"].replace("mailto:", "")
        councillor.photo_url = soup.select_one(".councillor__profile img")["src"]
        return councillor

What needs to happen

• Try adding http_lib = "playwright" to see if a real browser fingerprint bypasses the WAF
• If playwright also returns 0 results, add logging to identify which CSS selector is failing
• If playwright returns results, commit that as the fix
• Check whether .division .value and .party .value selectors still exist on the live page