Merge pull request #27 from rusq/26

fix 406 error and migrate to new Auckland API address

Author: rusq

.gitignore

@@ -6,3 +6,4 @@ aklrubbish
 
 # binary files
 aklapi
+.env

AGENTS.md

@@ -1,34 +1,215 @@
-# AGENTS.md
-
-## Project Snapshot
-- Module: `github.com/rusq/aklapi`
-- Language: Go
-- Purpose: unofficial Auckland Council API wrapper/service for address lookup and rubbish/recycling collection dates.
-
-## Repository Map
-- `cmd/`: executable entrypoints.
-- `addr.go`: address lookup logic.
-- `rubbish.go`: rubbish/recycling API logic and response shaping.
-- `caches.go`: cache helpers.
-- `time.go`: date/time helpers.
-- `*_test.go`: unit tests.
-- `test_assets/`: fixtures used by tests.
-
-## Development Commands
-- Run tests: `go test ./...`
-- Run focused tests: `go test ./... -run <Name>`
-- Tidy dependencies: `go mod tidy`
-- Build all packages: `go build ./...`
-
-## Working Conventions
-- Prefer small, targeted changes over broad refactors.
-- Keep API behavior backward compatible unless explicitly requested.
-- Add or update tests for behavioral changes.
-- Keep exported identifiers and package-level docs concise.
-
-## Validation Checklist
-Before finishing a code change, run:
-1. `go test ./...`
-2. `go build ./...`
-
-If a change only affects docs or comments, note that tests/build were not required.
+# AGENTS.md — Coding Agent Instructions for `aklapi`
+
+This document provides guidance for agentic coding assistants operating in this repository.
+
+---
+
+## Project Overview
+
+`aklapi` is a Go library and HTTP server that exposes Auckland Council APIs
+(rubbish collection schedules, property address lookup) as a simple REST service.
+
+- **Module:** `github.com/rusq/aklapi` (`go 1.25`)
+- **Library package:** root (`aklapi`)
+- **Binary:** `cmd/aklapi/` — standard HTTP server on port 8080
+- **Language:** Go only — no TypeScript, JavaScript, or Node tooling
+
+---
+
+## Build, Run & Test Commands
+
+```sh
+# Build the server binary
+go build -o server ./cmd/aklapi
+
+# Run the server (port defaults to 8080)
+./server
+
+# Run all tests
+go test ./...
+
+# Run all tests with verbose output
+go test -v ./...
+
+# Run a single test by name (supports regex)
+go test -v -run TestFunctionName ./...
+
+# Run a single test in a specific package
+go test -v -run TestCollectionDayDetail ./cmd/aklapi/
+
+# Run tests with race detector
+go test -race ./...
+
+# Build all packages (verify compilation)
+go build -v ./...
+
+# Format code (use goimports, not gofmt)
+goimports -w .
+
+# Lint (golangci-lint with default config)
+golangci-lint run ./...
+
+# Docker build
+docker build -t aklapi .
+
+# Make targets
+make server   # go build -o server ./cmd/aklapi
+make test     # go test ./... -race
+make docker   # docker build -t aklapi .
+```
+
+> **To run a single test:** use `go test -v -run <TestName> <./package/path>`
+> Example: `go test -v -run TestNextRubbish .`
+
+---
+
+## Code Style Guidelines
+
+### Formatting
+
+- Use **`goimports`** (not plain `gofmt`) — it manages imports automatically.
+- Indentation: **tabs** (Go standard).
+- VS Code devcontainer is configured with `"editor.formatOnSave": true` using `goimports`.
+- No trailing whitespace; no blank lines at end of file.
+
+### Imports
+
+Group imports in two blocks separated by a blank line:
+1. Standard library
+2. Third-party packages
+
+```go
+import (
+    "context"
+    "encoding/json"
+    "net/http"
+
+    "github.com/PuerkitoBio/goquery"
+)
+```
+
+- Use blank imports only where required: `_ "time/tzdata"`, `_ "embed"`.
+- Never use dot imports (`.`).
+- Alias imports only when disambiguation is genuinely needed.
+
+### Naming Conventions
+
+| Element | Convention | Example |
+|---|---|---|
+| Exported types | PascalCase | `AddrRequest`, `RubbishCollection` |
+| Unexported types | camelCase | `refuseParser`, `lruCache` |
+| Exported functions | PascalCase | `AddressLookup`, `CollectionDayDetail` |
+| Unexported functions | camelCase | `fetchandparse`, `oneAddress` |
+| Receiver names | Short (1–2 chars) | `(r *RubbishCollection)`, `(c *lruCache[K,V])` |
+| Package-level vars | camelCase | `addrCache`, `defaultLoc` |
+| Unexported constants | camelCase | `defCacheSz`, `dateLayout` |
+| Acronyms | Go convention | `addrURI` (not `addrUrl`), `ID` (not `Id`) |
+
+### Types & Structs
+
+- Add JSON struct tags to all exported response types: `json:"field,omitempty"`.
+- Prefer pointer receivers for types that may mutate state or are large.
+- Use **generics** for reusable containers (see `lruCache[K comparable, V any]`).
+- Use a stateful parser type (struct with fields for state, error, and results) when
+  parsing multi-step data (see `refuseParser`).
+
+### Error Handling
+
+- Always check errors: `if err != nil { return nil, err }`.
+- No `panic` in production code.
+- Use `errors.New("...")` for static error messages.
+- Use string concatenation (not `fmt.Sprintf`) for simple dynamic error strings:
+  ```go
+  errors.New("address API returned status code: " + strconv.Itoa(resp.StatusCode))
+  ```
+- Prefer `fmt.Errorf("context: %w", err)` for wrapping errors that need context.
+- Use package-level sentinel errors for flow control:
+  ```go
+  var errSkip = errors.New("skip this date")
+  ```
+- Use `errors.Is` for sentinel error comparisons.
+- HTTP handlers: use `http.Error(w, msg, code)` or a typed `respond(w, body, code)` helper.
+
+### HTTP & Networking
+
+- Use **standard library `net/http` only** — no external router (no Gin, Echo, Chi).
+- Register routes with `http.HandleFunc` on the default mux.
+- Always pass context to outgoing HTTP requests:
+  ```go
+  req, err := http.NewRequestWithContext(ctx, http.MethodGet, uri, nil)
+  ```
+- Always `defer resp.Body.Close()` immediately after a successful response.
+- Decode JSON responses with `json.NewDecoder(resp.Body).Decode(&v)`.
+
+### Logging
+
+- Use **`log/slog`** for all logging — not `log.Printf`, `fmt.Println`, etc.
+- Prefer context-aware variants: `slog.DebugContext(ctx, ...)`, `slog.InfoContext(ctx, ...)`.
+- Add structured key-value pairs for observability:
+  ```go
+  start := time.Now()
+  // ... operation ...
+  slog.DebugContext(ctx, "fetched addresses", "count", len(results), "duration", time.Since(start))
+  ```
+
+### Dependency Injection & Testability
+
+- Declare external URLs as **package-level `var`** (not `const`) so tests can override them:
+  ```go
+  var addrURI = `https://example.com/api/addresses`
+  ```
+- Inject time via a replaceable variable: `var now = time.Now`.
+- Injectable function-type variables enable handler testing without real upstream calls:
+  ```go
+  var addressLookup = aklapi.AddressLookup
+  ```
+- Restore overridden vars with `defer`:
+  ```go
+  old := addrURI
+  addrURI = ts.URL
+  defer func() { addrURI = old }()
+  ```
+
+---
+
+## Testing Guidelines
+
+### Style
+
+- Use **table-driven tests** for all non-trivial functions.
+- Table entry struct fields: `name string`, `args`, `want`, `wantErr bool`.
+- Field names may be omitted for the `name` field in composite literals.
+- Prefer `github.com/stretchr/testify/assert` for assertions in new tests (avoid raw
+  `reflect.DeepEqual` + `t.Errorf` patterns from older tests).
+- Use `t.Context()` (Go 1.24+) for context in subtests.
+- Use `t.Cleanup(func() {...})` for teardown instead of `defer` in the test function body
+  when working with subtests.
+
+### HTTP Testing
+
+- Use `net/http/httptest.NewServer` to mock upstream APIs.
+- Use `httptest.NewRequest` + `httptest.NewRecorder` for handler unit tests.
+
+### Test Fixtures
+
+- Embed HTML fixture files with `//go:embed`:
+  ```go
+  //go:embed test_assets/some-page.html
+  var fixtureHTML []byte
+  ```
+- Fixtures are refreshed by `//go:generate` directives that `curl` the live page.
+
+### Subtests
+
+- Always run subtests with `t.Run(tt.name, func(t *testing.T) { ... })`.
+- Use `t.Helper()` in assertion helper functions.
+
+---
+
+## Repository Conventions
+
+- **One concern per file:** `addr.go`, `rubbish.go`, `caches.go`, `time.go`.
+- **Library in root, binary in `cmd/`:** follows standard Go project layout.
+- CI runs on `push` and `pull_request` to `master` (see `.github/workflows/go.yml`):
+  `go build -v ./...` then `go test -v ./...`.
+- Docker images are published to `ffffuuu/aklapi` on GitHub Release events.

addr.go

@@ -12,9 +12,9 @@ import (
 
 var (
 	// defined as a variable so it can be overridden in tests.
-	addrURI = `https://www.aucklandcouncil.govt.nz/nextapi/property`
+	addrURI = `https://experience.aucklandcouncil.govt.nz/nextapi/property`
 	// defined as a variable so tests can replace it.
-	addrHTTPClient = &http.Client{Timeout: 15 * time.Second}
+	addrHTTPClient = &http.Client{Timeout: 15 * time.Second, Transport: &browserTransport{wrapped: http.DefaultTransport}}
 )
 
 // AddrRequest is the address request.
@@ -25,8 +25,8 @@ type AddrRequest struct {
 
 // Address is the address and its unique identifier (rate account key).
 type Address struct {
-	ID      string `json:"ID"`
-	Address string `json:"Address"`
+	ID      string `json:"id"`
+	Address string `json:"address"`
 }
 
 // AddrResponse is the address response.

aklapi.go

@@ -1,9 +1,33 @@
 package aklapi
 
 import (
+	"net/http"
 	"time"
 )
 
 var (
 	defaultLoc, _ = time.LoadLocation("Pacific/Auckland") // Auckland is in NZ.
 )
+
+// userAgent is sent with all outgoing HTTP requests. The Auckland Council
+// website CDN (Fastly) returns 406 for requests that identify as Go's default
+// http client, so we send a browser-compatible value instead.
+const userAgent = "Mozilla/5.0 (compatible; aklapi/1.0)"
+
+// browserTransport is an http.RoundTripper that adds browser-like headers to
+// every request before forwarding it to the underlying transport.
+type browserTransport struct {
+	wrapped http.RoundTripper
+}
+
+func (t *browserTransport) RoundTrip(req *http.Request) (*http.Response, error) {
+	// Clone the request so we don't mutate the caller's copy.
+	r := req.Clone(req.Context())
+	if r.Header.Get("User-Agent") == "" {
+		r.Header.Set("User-Agent", userAgent)
+	}
+	if r.Header.Get("Accept") == "" {
+		r.Header.Set("Accept", "application/json, text/html, */*")
+	}
+	return t.wrapped.RoundTrip(r)
+}

cmd/aklapi/handlers.go

@@ -10,13 +10,14 @@ import (
 	"github.com/rusq/aklapi"
 )
 
-const dttmLayout = "2006-01-02"
-
+// injectable for testing
 var (
 	addressLookup       = aklapi.AddressLookup
 	collectionDayDetail = aklapi.CollectionDayDetail
 )
 
+const dttmLayout = "2006-01-02"
+
 type rrResponse struct {
 	Rubbish    string `json:"rubbish,omitempty"`
 	Recycle    string `json:"recycle,omitempty"`
@@ -53,7 +54,7 @@ func addrHandler(w http.ResponseWriter, r *http.Request) {
 	resp, err := addressLookup(r.Context(), addr)
 	if err != nil {
 		slog.Error("address lookup failed", "error", err)
-		http.Error(w, http.StatusText(http.StatusBadGateway), http.StatusBadGateway)
+		http.Error(w, err.Error(), http.StatusBadGateway)
 		return
 	}
 	respond(w, resp, http.StatusOK)

rubbish.go

@@ -22,9 +22,9 @@ const (
 
 var (
 	// defined as a variable so it can be overridden in tests.
-	collectionDayURI = `https://new.aucklandcouncil.govt.nz/en/rubbish-recycling/rubbish-recycling-collections/rubbish-recycling-collection-days/%s.html`
+	collectionDayURI = `https://www.aucklandcouncil.govt.nz/en/rubbish-recycling/rubbish-recycling-collections/rubbish-recycling-collection-days/%s.html`
 	// defined as a variable so tests can replace it.
-	collectionHTTPClient = &http.Client{Timeout: 15 * time.Second}
+	collectionHTTPClient = &http.Client{Timeout: 15 * time.Second, Transport: &browserTransport{wrapped: http.DefaultTransport}}
 )
 
 var errSkip = errors.New("skip this date")

rubbish_test.go

@@ -14,8 +14,8 @@ import (
 	"github.com/stretchr/testify/assert"
 )
 
-//go:generate curl -L https://new.aucklandcouncil.govt.nz/en/rubbish-recycling/rubbish-recycling-collections/rubbish-recycling-collection-days/12342478585.html -o test_assets/500-queen-street.html
-//go:generate curl -L https://new.aucklandcouncil.govt.nz/en/rubbish-recycling/rubbish-recycling-collections/rubbish-recycling-collection-days/12341511281.html -o test_assets/1-luanda-drive.html
+//go:generate curl -L https://www.aucklandcouncil.govt.nz/en/rubbish-recycling/rubbish-recycling-collections/rubbish-recycling-collection-days/12342478585.html -o test_assets/500-queen-street.html
+//go:generate curl -L https://www.aucklandcouncil.govt.nz/en/rubbish-recycling/rubbish-recycling-collections/rubbish-recycling-collection-days/12341511281.html -o test_assets/1-luanda-drive.html
 
 // Test data, run go:generate to update, then update dates in tests
 // accordingly.