Skip to content

fix: support to skip Terminate and Describe Instances in the affected az#9031

Open
sumukha-radhakrishna wants to merge 6 commits intoaws:mainfrom
sumukha-radhakrishna:skip-terminates-affected-az
Open

fix: support to skip Terminate and Describe Instances in the affected az#9031
sumukha-radhakrishna wants to merge 6 commits intoaws:mainfrom
sumukha-radhakrishna:skip-terminates-affected-az

Conversation

@sumukha-radhakrishna
Copy link
Copy Markdown
Contributor

AWS has an AZ degradation event in me-south-1b, causing some controllers to run into 401s after some time.
The 401s are due to the way aws-sdk-go setting the clockskew for hung up requests poisioning the system.

This patch supports skipping Terminate and Describe Instances call in the affected AZ.

Fixes #N/A

Description

Set this env variable to skip calling Terminates/Describes in the zone.

 - name: IMPAIRED_ZONE
    value: me-south-1b

Note: This is a hack please use caution and test befire using this in production.

How was this change tested?

Ran a test locally by simulating AZ failure.

Does this change impact docs?

  • Yes, PR includes docs updates
  • Yes, issue opened: #
  • No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@sumukha-radhakrishna sumukha-radhakrishna requested a review from a team as a code owner March 24, 2026 00:16
@sumukha-radhakrishna sumukha-radhakrishna enabled auto-merge (squash) March 24, 2026 00:18
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 24, 2026

Preview deployment ready!

Preview URL: https://pr-9031.d18coufmbnnaag.amplifyapp.com

Built from commit f62faceb52923c22db69bdb6fa71fb95c2cf77e8

@sumukha-radhakrishna sumukha-radhakrishna force-pushed the skip-terminates-affected-az branch from ceba7d7 to d96e1ca Compare March 24, 2026 00:33
Copy link
Copy Markdown
Contributor Author

@sumukha-radhakrishna sumukha-radhakrishna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/snapshot

sumukha-radhakrishna

This comment was marked as resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant