Skip to content

Fix nginx buildpack PR automation to update integration test expectations - #605

Open
ivanovac wants to merge 2 commits into
masterfrom
fix-nginx-test-updates
Open

Fix nginx buildpack PR automation to update integration test expectations#605
ivanovac wants to merge 2 commits into
masterfrom
fix-nginx-test-updates

Conversation

@ivanovac

@ivanovac ivanovac commented Apr 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the automatic nginx version PR generation to also update integration test expectations and override buildpack fixture, preventing test failures in all nginx version update PRs.

Note: This PR previously included a pipeline change to add a passed constraint to the update job. That change has been removed because it doesn't solve the root cause of the multi-stack issue. The proper fix for that is in PR #608 (atomic commits).

Problem

All automatically generated nginx version update PRs (#392, #394, #389, #390, #384, #385, etc.) are failing integration tests with two types of errors:

  1. Override test: "Expected an error, got nil" - deployment succeeds when it should fail
  2. Version tests: Timeout waiting for version strings that don't match
  3. Available versions test: Expects old version list

Root Cause:
The automation in tasks/update-buildpack-dependency/run.rb updates manifest.yml with new nginx versions and adjusts version_lines (mainline/stable), but doesn't update:

  • Integration test file src/nginx/integration/default_test.go (hardcoded version strings)
  • Override buildpack fixture fixtures/util/override_buildpack/override.yml (fake version for testing)

Example: When nginx 1.29.7 is added and mainline changes from 1.28.x to 1.29.x:

Tests still expect:

  • mainline => 1.28.
  • Available versions: mainline, stable, 1.28.x, 1.29.x
  • Override fixture has 1.28.999

But buildpack outputs:

  • mainline => 1.29.
  • Available versions: mainline, stable, 1.26.x, 1.28.x, 1.29.x
  • Override 1.28.999 doesn't match mainline 1.29.x, so override doesn't apply

Changes

Enhanced tasks/update-buildpack-dependency/run.rb with nginx-specific file updates:

1. Update Integration Test Expectations

Updates src/nginx/integration/default_test.go patterns:

  • using mainline => X.Y. → Updated to current mainline version
  • Requested nginx version: mainline => X.Y. → Updated to current mainline version
  • stable => X.Y. → Updated to current stable version
  • Available versions: ... → Recalculated from manifest dependencies

2. Update Override Buildpack Fixture

Updates fixtures/util/override_buildpack/override.yml to match current mainline:

  • Extracts mainline version (e.g., 1.29.x)
  • Generates fake version (e.g., 1.29.999)
  • Updates version_lines.mainline in override.yml
  • Updates nginx dependency version
  • Updates URI to reference fake version
  • Keeps SHA256 intentionally wrong (for testing validation)

The override test uses a fake version with wrong SHA256 to verify the buildpack properly validates checksums. If the fake version doesn't match the mainline version line, the override won't apply and the test passes when it should fail.

3. Follows Existing Patterns

Uses same file-editing pattern as JRuby Gemfile updates - stores files in a hash and commits them together.

Testing Plan

After this PR is merged:

  1. Close all failing nginx version PRs
  2. Trigger regeneration of nginx 1.29.7 PR (or manually create one)
  3. Verify the new PR includes updates to:
    • manifest.yml
    • src/nginx/integration/default_test.go
    • fixtures/util/override_buildpack/override.yml
  4. Verify all integration tests pass (including override test)
  5. Regenerate remaining nginx version PRs

Related Issues

Dependencies

Should be merged before regenerating nginx version PRs to ensure new PRs have correct test expectations.

ivanovac added 2 commits April 3, 2026 17:23
…ions

When nginx version PRs are automatically generated, the manifest.yml is updated
with new nginx versions and the version_lines (mainline/stable) are adjusted.
However, the integration tests in src/nginx/integration/default_test.go contain
hardcoded version strings that were not being updated, causing all nginx version
update PRs to fail integration tests.

This fix adds logic to automatically update the nginx integration test file when
a nginx dependency PR is created. The script now:

1. Detects when updating nginx dependency in nginx buildpack
2. Extracts the current mainline and stable versions from version_lines
3. Calculates available nginx version lines from the manifest
4. Updates all version string patterns in default_test.go:
   - 'using mainline => X.Y.' pattern
   - 'Requested nginx version: mainline => X.Y.' pattern
   - 'stable => X.Y.' pattern
   - 'Available versions: mainline, stable, X.Y.x, ...' list
5. Adds the updated test file to the git commit

This ensures that when new nginx versions are added, the tests automatically
expect the correct version strings and don't fail due to hardcoded expectations.

Fixes the issue where all nginx version update PRs (#392, #394, #389, etc.) were
failing integration tests.
The override buildpack integration test uses a fake nginx version (X.Y.999) with
an intentionally incorrect SHA256 to verify the buildpack properly validates
checksums and fails appropriately.

The fake version needs to match the current mainline version line in the manifest.
When nginx version PRs change mainline from 1.28.x to 1.29.x, the override fixture
must also update from 1.28.999 to 1.29.999, otherwise the override won't apply
and the test will pass when it should fail.

This enhancement adds logic to automatically update the override.yml fixture:
1. Extracts the current mainline version (e.g., '1.29.x')
2. Generates fake version (e.g., '1.29.999')
3. Updates version_lines.mainline in override.yml
4. Updates nginx dependency version in override.yml
5. Updates URI to reference the fake version
6. Adds the file to the commit

This ensures the override test continues to work correctly as nginx versions evolve.
@ivanovac
ivanovac force-pushed the fix-nginx-test-updates branch from c75a393 to f84ebed Compare April 3, 2026 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant