Skip to content

[BUG] clusteridentityoperator logs HTTP error response body as decimal byte array instead of decoded JSON #452

Description

@Karuch

I'm not sure if this is the correct repository, since the issue is related to Azure Arc-enabled Kubernetes running on an external RKE2 cluster rather than AKS itself.

If this doesn't belong in this repository, please let me know where the appropriate place to report it would be. Thank you!

Describe the bug

When clusteridentityoperator receives an HTTP error response from HIS, the response body is logged as a decimal byte array instead of decoded text.

For example, HIS returned the following error:

{
  "error": {
    "code": "HCRP403",
    "message": "Access to the requested resource has been rejected. The connection used a Private Endpoint but no Private Link Scope has been assigned to the resource.",
    "target": "/type/ConnectedClusters/identity"
  }
}

However, the operator logged:

Get Cluster Certificate failed failed retrieving certificate from HIS with a non-successful status code 403, error:
[123 34 101 114 114 111 114 34 58 123 34 99 111 100 101 ...]

Instead of the decoded JSON response.

This makes troubleshooting difficult because the actual Azure error code and message are hidden unless the byte array is manually decoded.

To Reproduce

  1. Have clusteridentityoperator send a request to HIS that returns an HTTP error response.
  2. View the clusteridentityoperator controller logs.
  3. Observe that the HTTP response body is logged as a decimal byte array instead of readable text.

Expected behavior

The operator should log the decoded HTTP response body as readable JSON instead of a decimal byte array.

Actual logs

{"Message":"Received response from HIS with status code: 403 and tracking id: x","LogType":"ConfigAgentTrace","LogLevel":"Information","Environment":"prod","Role":"ClusterConfigAgent","Location":"northeurope","ArmId":"x","CorrelationId":"","AgentName":"ClusterIdentityRequestController","AgentVersion":"1.35.4","AgentTimestamp":"2026/07/21 13:25:14.055"}
{"Message":"failed retrieving certificate from HIS with a non-successful status code 403, error: [123 34 101 114 114 111 114 34 58 123 34 99 111 100 101 34 58 34 72 67 82 80 52 48 51 34 44 34 109 101 115 115 97 103 101 34 58 34 65 99 99 101 115 115 32 116 111 32 116 104 101 32 114 101 113 117 101 115 116 101 100 32 114 101 115 111 117 114 99 101 32 104 97 115 32 98 101 101 110 32 114 101 106 101 99 116 101 100 46 32 84 104 101 32 99 111 110 110 101 99 116 105 111 110 32 117 115 101 100 32 97 32 80 114 105 118 97 116 101 32 69 110 100 112 111 105 110 116 32 98 117 116 32 110 111 32 80 114 105 118 97 116 101 32 76 105 110 107 32 83 99 111 112 101 32 104 97 115 32 98 101 101 110 32 97 115 115 105 103 110 101 100 32 116 111 32 116 104 101 32 114 101 115 111 117 114 99 101 46 34 44 34 116 97 114 103 101 116 34 58 34 47 116 121 112 101 47 67 111 110 110 101 99 116 101 100 67 108 117 115 116 101 114 115 47 105 100 101 110 116 105 116 121 34 125 125]","LogType":"ConfigAgentTrace","LogLevel":"Error","Environment":"prod","Role":"ClusterConfigAgent","Location":"northeurope","ArmId":"x","CorrelationId":"","AgentName":"ClusterIdentityRequestController","AgentVersion":"1.35.4","AgentTimestamp":"2026/07/21 13:25:14.055"}

The decimal byte array decodes to the following HTTP response body:

Get Cluster Certificate failed: status code 403

{
  "error": {
    "code": "HCRP403",
    "message": "Access to the requested resource has been rejected. The connection used a Private Endpoint but no Private Link Scope has been assigned to the resource.",
    "target": "/type/ConnectedClusters/identity"
  }
}

Screenshots

Not applicable.

Environment

  • OS: Rocky Linux 9
  • Kubernetes Distribution: RKE2
  • Azure Arc Agent Version: 1.35.4
  • Kubernetes Version: v1.35.5+rke2r2

Additional context

This appears to be a logging issue only.

The HTTP response body seems to be logged as a Go []byte rather than as decoded text. As a result, users see decimal byte values instead of the actual JSON error returned by HIS.

This significantly slows down troubleshooting because users must manually decode the response to determine the real error.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions