Skip to content

ci: run integration tests in 2 batches #391

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
olebel wants to merge 1 commit into
base: main
Choose a base branch
from
Open

ci: run integration tests in 2 batches #391

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .github/workflows/integration-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ name: Integration tests
# yamllint disable-line rule:truthy
on:
workflow_call:
inputs:
e2e_target:
description: 'Make target for E2E tests'
required: false
default: 'test-e2e-integration'
type: string

env:
KUBE_SSH_NODES: kind
Expand Down Expand Up @@ -144,7 +150,7 @@ jobs:
- name: Run E2E Tests
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
run: |
make test-e2e-integration
make ${{ inputs.e2e_target }}
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Validate and quote e2e_target before shell execution.

Line [153] executes a raw workflow input in bash. That allows command injection if an unexpected value is passed via workflow_call.

🔒 Proposed fix 
-      - name: Run E2E Tests
-        working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
-        run: |
-          make ${{ inputs.e2e_target }}
+      - name: Run E2E Tests
+        working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider
+        run: |
+          target='${{ inputs.e2e_target }}'
+          case "$target" in
+            test-e2e-integration|test-e2e-integration-batch1|test-e2e-integration-batch2) ;;
+            *) echo "invalid e2e_target: $target" >&2; exit 1 ;;
+          esac
+          make "$target"
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/integration-tests.yaml at line 153, The workflow runs user
input unsafely via the line containing make ${{ inputs.e2e_target }}, so
validate and quote the e2e_target input before executing: constrain
inputs.e2e_target to an allowlist or a strict regex (e.g., only
letters/numbers/[-_]) and then invoke make with the validated, quoted value (use
the inputs.e2e_target identifier) so the shell executes a single safe token
instead of allowing command injection.

- name: Print operator inventory logs
if: always()
working-directory: ${{ env.GOPATH }}/src/github.com/akash-network/provider/_run/kube
Expand Down
12 changes: 10 additions & 2 deletions .github/workflows/tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,15 @@ jobs:
uses: ./.github/actions/setup-ubuntu
- run: make shellcheck

integration-tests:
integration-tests-batch1:
uses: ./.github/workflows/integration-tests.yaml
with:
e2e_target: test-e2e-integration-batch1

integration-tests-batch2:
uses: ./.github/workflows/integration-tests.yaml
with:
e2e_target: test-e2e-integration-batch2

Comment on lines +107 to 116
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# Find and inspect the integration-tests.yaml workflow
find .github -name "*.yaml" -o -name "*.yml" | head -20

Repository: akash-network/provider

Length of output: 314

🏁 Script executed:

# Check if integration-tests.yaml exists and read relevant sections
if [ -f ".github/workflows/integration-tests.yaml" ]; then
  echo "=== integration-tests.yaml exists ==="
  wc -l ".github/workflows/integration-tests.yaml"
  echo ""
  echo "=== Lines around 24 (KIND_NAME) ==="
  sed -n '15,35p' ".github/workflows/integration-tests.yaml"
  echo ""
  echo "=== Inputs section ==="
  rg -A 20 "^inputs:" ".github/workflows/integration-tests.yaml" | head -40
fi

Repository: akash-network/provider

Length of output: 678

🏁 Script executed:

# Check the tests.yaml file for how it calls the reusable workflow
if [ -f ".github/workflows/tests.yaml" ]; then
  echo "=== Lines 107-116 from tests.yaml ==="
  sed -n '107,116p' ".github/workflows/tests.yaml"
  echo ""
  echo "=== Full context of integration-tests calls ==="
  sed -n '100,140p' ".github/workflows/tests.yaml"
fi

Repository: akash-network/provider

Length of output: 1395

🏁 Script executed:

# Search for any kind_name or KIND_NAME usage across workflows
echo "=== Searching for kind_name or KIND_NAME in workflows ==="
rg -i "kind.?name" .github/workflows/ -A 2 -B 2

Repository: akash-network/provider

Length of output: 904

🏁 Script executed:

# Get full structure of integration-tests.yaml to understand jobs
echo "=== Full integration-tests.yaml with job names ==="
cat -n ".github/workflows/integration-tests.yaml" | grep -E "^\s*[0-9]+\s+(jobs:|[a-z-]+:)" | head -40

Repository: akash-network/provider

Length of output: 1633

🏁 Script executed:

# Get the inputs section and full job list
echo "=== Integration-tests.yaml - inputs and jobs section ==="
sed -n '1,80p' ".github/workflows/integration-tests.yaml" | cat -n

Repository: akash-network/provider

Length of output: 3486

🏁 Script executed:

# Search for all job definitions and their conditions in integration-tests.yaml
echo "=== All jobs in integration-tests.yaml ==="
rg "^  [a-z-]+:" ".github/workflows/integration-tests.yaml" -A 5 | head -80

Repository: akash-network/provider

Length of output: 158

🏁 Script executed:

# Get complete job names from integration-tests.yaml
echo "=== All job definitions in integration-tests.yaml ==="
rg "^  [a-z0-9-]+:$" ".github/workflows/integration-tests.yaml"

Repository: akash-network/provider

Length of output: 137

🏁 Script executed:

# Check the full file to see structure
echo "=== Line count and structure ==="
wc -l ".github/workflows/integration-tests.yaml"
echo ""
echo "=== Section headers (jobs, needs, etc) ==="
grep -n "^jobs:\|^  [a-z-]*:\|^    if:\|^    needs:" ".github/workflows/integration-tests.yaml" | head -50

Repository: akash-network/provider

Length of output: 206

🏁 Script executed:

# Get the lines where e2e_target is used
echo "=== Where e2e_target is used in integration-tests.yaml ==="
rg -n "e2e_target" ".github/workflows/integration-tests.yaml" -B 2 -A 2

Repository: akash-network/provider

Length of output: 481

🏁 Script executed:

# Check if kind_name input exists in integration-tests.yaml
echo "=== Check inputs section of integration-tests.yaml ==="
sed -n '5,15p' ".github/workflows/integration-tests.yaml"
echo ""
echo "=== Search for any kind_name or cluster isolation in integration-tests.yaml ==="
rg -i "kind.name|cluster.name" ".github/workflows/integration-tests.yaml" -B 3 -A 3

Repository: akash-network/provider

Length of output: 934

🏁 Script executed:

# Check if there's any unique naming strategy or runner isolation config
echo "=== Check for runner labels or self-hosted config ==="
rg "runs-on:|self-hosted" ".github/workflows/" -B 2 -A 2 | head -60

Repository: akash-network/provider

Length of output: 2545

🏁 Script executed:

# Check if github.run_id or other unique vars are used anywhere for isolation
echo "=== Check for unique run identifiers used in Kind setup ==="
rg "github\.(run_id|run_number|job)" ".github/workflows/integration-tests.yaml" -B 2 -A 2

Repository: akash-network/provider

Length of output: 125

🏁 Script executed:

# Final confirmation: kind_name does not exist as input
echo "=== All inputs in integration-tests.yaml ==="
sed -n '5,13p' ".github/workflows/integration-tests.yaml"

Repository: akash-network/provider

Length of output: 296

Parallel batches collide on hardcoded Kind cluster name.

Both batch jobs invoke crd-e2e from the same reusable workflow in parallel, and KIND_NAME: kube is hardcoded (.github/workflows/integration-tests.yaml, line 24). On core-e2e self-hosted runners sharing a Docker daemon, concurrent jobs will conflict on the same cluster name, causing flaky test failures.

Add a kind_name input to the reusable workflow and pass unique values per batch (e.g., kube-batch1, kube-batch2) to isolate cluster namespaces:

Suggested mitigation

In .github/workflows/integration-tests.yaml, add to inputs:

kind_name:
  description: 'Kind cluster name'
  required: false
  default: 'kube'
  type: string

Then update the crd-e2e job to use it instead of the hardcoded value, and in .github/workflows/tests.yaml, pass unique names:

integration-tests-batch1:
  uses: ./.github/workflows/integration-tests.yaml
  with:
    e2e_target: test-e2e-integration-batch1
    kind_name: kube-batch1

integration-tests-batch2:
  uses: ./.github/workflows/integration-tests.yaml
  with:
    e2e_target: test-e2e-integration-batch2
    kind_name: kube-batch2
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tests.yaml around lines 107 - 116, The reusable workflow
hardcodes KIND_NAME causing parallel batch collision; add a new input named
kind_name to the reusable workflow (default "kube") and update the crd-e2e job
to use this input instead of the hardcoded KIND_NAME, then pass unique values
from the callers (integration-tests-batch1 and integration-tests-batch2) via the
with: block (e.g., kind_name: kube-batch1 and kind_name: kube-batch2) so each
batch uses a distinct Kind cluster name.

dispatch-release:
runs-on: ubuntu-latest
Expand All @@ -118,7 +125,8 @@ jobs:
- coverage
- yamlcheck
- shellcheck
- integration-tests
- integration-tests-batch1
- integration-tests-batch2
steps:
- uses: actions/checkout@v4
with:
Expand Down
21 changes: 21 additions & 0 deletions integration/e2e_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -761,6 +761,27 @@ func TestIntegrationTestSuite(t *testing.T) {
suite.Run(t, &E2EIPAddress{IntegrationTestSuite{ipMarketplace: true}})
}

func TestIntegrationBatch1(t *testing.T) {
integrationTestOnly(t)

suite.Run(t, new(E2EContainerToContainer))
suite.Run(t, new(E2EAppNodePort))
suite.Run(t, new(E2EDeploymentUpdate))
suite.Run(t, new(E2EApp))
suite.Run(t, new(E2EMigrateHostname))
}

func TestIntegrationBatch2(t *testing.T) {
integrationTestOnly(t)

suite.Run(t, new(E2EPersistentStorageDefault))
suite.Run(t, new(E2EPersistentStorageBeta2))
suite.Run(t, new(E2EPersistentStorageDeploymentUpdate))
suite.Run(t, new(E2EStorageClassRam))
suite.Run(t, new(E2ECustomCurrency))
suite.Run(t, &E2EIPAddress{IntegrationTestSuite{ipMarketplace: true}})
}

// TestQueryApp enables rapid testing of the querying functionality locally
// Not for CI tests.
func TestQueryApp(t *testing.T) {
Expand Down
8 changes: 8 additions & 0 deletions make/test-integration.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,14 @@ test-e2e-integration:
# ```
$(KIND_VARS) $(INTEGRATION_VARS) $(GO_TEST) -count=1 -p 4 -tags "e2e" -v ./integration/... -run TestIntegrationTestSuite -timeout 3000s

.PHONY: test-e2e-integration-batch1
test-e2e-integration-batch1:
$(KIND_VARS) $(INTEGRATION_VARS) $(GO_TEST) -count=1 -tags "e2e" -v ./integration/... -run TestIntegrationBatch1 -timeout 1500s

.PHONY: test-e2e-integration-batch2
test-e2e-integration-batch2:
$(KIND_VARS) $(INTEGRATION_VARS) $(GO_TEST) -count=1 -tags "e2e" -v ./integration/... -run TestIntegrationBatch2 -timeout 1500s

.PHONY: test-e2e-integration-k8s
test-e2e-integration-k8s:
$(INTEGRATION_VARS) \
Expand Down
Loading