Add proposal for proxy cache referrer's API

[stonezdj]

No description provided.

[wy65701436]

lgtm

[bupd]

@stonezdj added suggestions.

Thanks

[Vad1mo]

The proposal should be compliant with the OCI spec. The proposal should support OCI registries and not only proxying from Harbor.

[reasonerjt]

@Vad1mo
The intention of the proposal is to enhance the proxy cache feature, which proxies data from upstream registries, so it doesn't break OCI compliance Harbor.

I'm not against the idea of proxying referrers from other upstream registries. I also believe the implementation supports other upstream registries that are OCI-compliant, but I don't think we have the bandwidth to test other registries.

[stonezdj]

The proposal should be compliant with the OCI spec. The proposal should support OCI registries and not only proxying from Harbor.

@Vad1mo For bandwith limitation, we only test/verify it on Harbor, it doesn't mean it is not comply with OCI spec. any registry follow the OCI spec should work this proxy cache feature.

[Vad1mo]

I find the proposal incomplete, and from what I understand, it is contradictory to OCI spec.

1. Missing: Referrers Tag Schema Fallback

OCI Spec Requirement:

"If the referrers API returns a 404, the client MUST fallback to pulling the referrers tag schema."

The tag schema format is <alg>-<ref> (e.g., sha256-aaaa... with first 64 chars of digest).

Proposal Gap: The proposal doesn't address this fallback mechanism. When proxying from an upstream registry that doesn't support the referrers API (returns 404), Harbor should:

1. Attempt the referrers tag schema fallback on the upstream
2. Return appropriate response to the client

Impact: OCI 1.0 registries and 1.1 registries without referrers API support won't work correctly. This includes many production registries (Docker Hub, older Harbor versions, ECR, GCR, etc.).

2. Missing: Pagination Support

OCI Spec Requirement:

"A Link header MUST be included in the response when the descriptor list cannot be returned in a single manifest."

Proposal Gap: No mention of pagination handling. Questions:

• How will Harbor handle paginated responses from upstream?
• Will Harbor aggregate all pages before caching?
• Will Harbor pass through Link headers to clients?

Impact: Large referrer lists may be truncated or cause errors.

3. Missing: artifactType Filtering

OCI Spec Requirement:

"The registry SHOULD support filtering on artifactType."

Request: GET /v2/<name>/referrers/<digest>?artifactType=<mediaType>
Response header: OCI-Filters-Applied: artifactType

Proposal Gap: The rawQuery parameter in the proposed interface hints at filter support, but no explicit handling is described:

ListReferrers(repository, ref string, rawQuery string) (*v1.Index, error)

[OrlinVasilev]

@stonezdj @wy65701436 Before merging that we should have agreement from all parties not only Broadcom maintainers.
Also we have merged the goharbor/harbor#22746 which is the actual implementation, before the proposal is approved.
Please follow the governance and respect the maintainers team! Let's reverse the goharbor/harbor#22746

[reasonerjt]

@OrlinVasilev
I agree with you that the code change should be reverted.

@bupd @stonezdj
Let's defer this to 2.16 to make sure merging the referrer list is incorporated.

[Vad1mo]

Thanks for putting the work into improving the proposal, @stonezdj. The structure is much better than the initial version, and the compatibility matrix was a good addition.

A few observations from reviewing the full thread and the implementation attempt (#22746):

What works well in the current proposal:

• The opt-in toggle approach avoids forcing behavior changes on existing users
• The rawQuery passthrough for pagination and artifactType filtering is a reasonable simplification
• The clarification that referrers tag schema fallback is the client's responsibility (Harbor acts as server) is correct per the OCI spec

What I think is still not clear:

1. The merge gap remains the central issue. When the proxy toggle is enabled, locally-created referrers (signatures, SBOMs attached in Harbor) become invisible.
   For anyone who both proxies and signs locally. The proposal should either implement merge or explicitly document this as a known limitation with a mitigation path.
2. Cross-registry validation. The proposal motivates itself with OCI spec compliance but was only tested against Harbor.
3. Caching specifics. The 7-day TTL and JSON-comparison invalidation are a start, but there's no cache size bound or eviction policy. For registries with many artifacts, unbounded Redis keys could become a problem.

I'm not blocking this, but I don't think it is the right approach for the future of OCI/Harbor.
The deferral to v2.16 with merge support seems like the right call

[stonezdj]

@Vad1mo Thank your comments, here is my answer to your comments:

The merge gap remains the central issue. When the proxy toggle is enabled, locally-created referrers (signatures, SBOMs attached in Harbor) become invisible.
For anyone who both proxies and signs locally. The proposal should either implement merge or explicitly document this as a known limitation with a mitigation path.

The proposal is updated after discussed with Prasanth, the locally-created referrers are merged with the upstream referrer list, it is visible to the user whenever it the proxy toggle is enabled/disabled.

Cross-registry validation. The proposal motivates itself with OCI spec compliance but was only tested against Harbor.
Caching specifics.

For resource limitation, we can only verify it with Harbor and dockerhub.

The 7-day TTL and JSON-comparison invalidation are a start, but there's no cache size bound or eviction policy. For registries with many artifacts, unbounded Redis keys could become a problem.

Will limit per each cached referrer list size to 1MB

[Vad1mo]

ok

abstained from voting

[chlins]

lgtm

[reasonerjt]

Overall LGTM.

Please merge after resolving my misc comments.