Retrieval with zero connected peers returns 404 "address not found", indistinguishable from missing content

[crtahlin]

Context

Bee 2.8.1-7cf53193, API version 8.1.0, beeMode: light, macOS, home network behind NAT.

Node state at time of failure:

{"beeMode":"light","connectedPeers":0,"neighborhoodSize":1,"reserveSize":0,"isReachable":false}
{"depth":9,"connected":0,"population":7181,"reachability":"Private","networkAvailability":"Unavailable"}

Chequebook was funded (totalBalance: 19584745999995000), so this is not a payment issue.

Summary

When a node has no connected peers, GET /bzz/{ref}/ returns 404 {"code":404,"message":"address not found or incorrect"}. That is the same response the node gives for content that genuinely does not exist, so an operator cannot tell "my node is disconnected" from "this content is gone".

This cost a full benchmarking session: the conclusion was first that content had failed to upload, then that the network was slow, before discovering the node had silently dropped from 75 peers to 0 partway through. Every measurement taken in between was invalid, and nothing in the API response hinted at the real cause.

Expected behavior

Retrieval failing because the node has no peers to ask should be distinguishable from retrieval failing because the chunk was not found. A 503 Service Unavailable with a message such as no peers available to retrieve from; node may be disconnected or still connecting would communicate that the fault is local and transient.

Actual behavior

All retrievals return the same 404, in a consistent ~1.8 s:

$ curl -s -o /dev/null -w '%{http_code} %{time_starttransfer}s\n' \
    http://localhost:1633/bzz/<ref>/
404 1.803649s

Against 11 references known to exist — all but one served successfully by a public gateway at the same moment — this node returned 404 for 9. The two that succeeded were already in its local cache.

Steps to reproduce

1. Run a light node and confirm GET /status reports connectedPeers: 0 and /topology reports networkAvailability: "Unavailable" (on this node it happened on its own after several hours).
2. GET /bzz/{ref}/ for any reference known to exist on the network.
3. Observe 404 address not found or incorrect, identical to the response for a nonexistent reference.

Possible solution

The distinction already exists internally and is discarded at the HTTP boundary:

• pkg/topology/topology.go:19 — ErrNotFound = errors.New("no peer found")
• pkg/retrieval/retrieval.go:214-219 — returns that error when all peers are depleted, distinct from storage.ErrNotFound returned at line 287 for genuine not-found
• pkg/api/bzz.go:517-521 — collapses any manifest-load error into jsonhttp.NotFound(w, nil)

So the API layer could check errors.Is(err, topology.ErrNotFound) and return 503 with a distinct message, rather than 404. The same treatment likely applies to the other download handlers (/bytes, /chunks).

A lighter-touch alternative, if changing the status code is considered breaking: keep 404 but make the message distinguish the two cases.

AI Disclosure

• This issue contains suggestions and text generated by an LLM.
• I have reviewed the AI generated content thoroughly.
• I possess the technical expertise to responsibly review the AI generated content mentioned in this issue.

[aloknerurkar]

Important nuance

topology.ErrNotFound means “no peer found,” not strictly “zero connected peers.”
It may occur when:

• the node has zero peers;
• all peers have been exhausted or skipped;
• no peer is currently eligible for the request;
• peers are temporarily unavailable due to accounting or retry state.

A more accurate message would be:
content could not be retrieved because no usable peer was available

Returning 503 Service Unavailable is still fine because the failure is transient and local/network-related.