diff --git a/README.md b/README.md index 000bb5fb..349c70b9 100644 --- a/README.md +++ b/README.md @@ -40,9 +40,7 @@ steps: command: .buildkite/deploy.sh ``` -The plugin is a thin wrapper around the hidden `buildkite-gha plugin` entrypoint. It uses mise to install and verify the selected CLI release, and defaults to the latest stable release. During the preview, leaving `version` unset means there is no CLI version to update as new stable releases ship. - -Use `workflow: "*"` to select every tracked `.yml` and `.yaml` file directly under `.github/workflows`. +The plugin is a thin wrapper around the hidden `buildkite-gha plugin` entrypoint. It uses mise to install and verify the selected CLI release, and defaults to the latest stable release. During the preview, leaving `version` unset means there is no CLI version to update as new stable releases ship. Set `workflow` to one explicit path or `workflows` to an explicit path list; plugin configuration does not accept directories or glob patterns. To hold the CLI at a specific release instead, set `version` to an exact stable release from `0.9.0` onward: @@ -72,7 +70,7 @@ Configured Linux profiles use the matching Noble or Jammy hosted-toolchains image. A macOS label selects native Darwin/arm64, not a GitHub image or Xcode inventory. -The imported workflow is a dynamic part of the Buildkite pipeline. Upload can take one tracked workflow glob or a list of explicit workflow paths and creates one aggregate group per directly runnable workflow in a single transaction. Workflows that do not declare the selected event become skipped groups. Each `:github: ` group depends on the importer; its GitHub check is named `Buildkite / ()`. This approach lets you keep existing workflows while moving jobs to native Buildkite steps over time. +The imported workflows are a dynamic part of the Buildkite pipeline. The plugin creates one aggregate group per explicitly listed, directly runnable workflow in a single transaction. Workflows that do not declare the selected event become skipped groups. Each `:github: ` group depends on the importer; its GitHub check is named `Buildkite / ()`. This approach lets you keep existing workflows while moving jobs to native Buildkite steps over time. Buildkite owns build creation and schedule configuration. Within that build, `buildkite-gha` maps push, pull request, manual/API, and scheduled builds to `push`, `pull_request`, `workflow_dispatch`, and `schedule`, then applies the matching `on:` branch, tag, base-branch, and pull request activity filters. Cross-event workflows are excluded before event-dependent compilation and retained as skipped groups. diff --git a/docs/cli.md b/docs/cli.md index 86c10366..1093a4ac 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -101,26 +101,19 @@ buildkite-gha upload .github/workflows/ci.yml The importer must run on Linux/amd64 with `BUILDKITE=true` and `BUILDKITE_STEP_KEY`. -The hidden zero-argument `buildkite-gha plugin` entry point reads `workflows` and -`runners` from `BUILDKITE_PLUGIN_CONFIGURATION`; `workflows` accepts one selector -string or an array of explicit paths. It also accepts the plugin-owned `version`, -`source-ref`, and `minimum-release-age` fields. The legacy singular `workflow` -field remains supported for released plugin compatibility but cannot be combined -with `workflows`. The Linux/amd64 importer fetches the same release's Darwin -runtime only when a workflow requires it. Custom importers can use the public -flags below. +The hidden zero-argument `buildkite-gha plugin` entry point reads `workflow`, +`workflows`, and `runners` from `BUILDKITE_PLUGIN_CONFIGURATION`. Set `workflow` +to one explicit path or `workflows` to a non-empty array of explicit paths; the +fields are mutually exclusive. Every path must identify a regular, tracked +`.yml` or `.yaml` file inside the repository; directories and glob patterns are +rejected. It also accepts the plugin-owned `version`, `source-ref`, and +`minimum-release-age` fields. The Linux/amd64 importer fetches the same release's +Darwin runtime only when a workflow requires it. Custom importers can use the +public flags below. ### Select workflows -Use `*` for every tracked `.yml` and `.yaml` file directly under `.github/workflows`: - -```sh -buildkite-gha upload '*' -``` - -Quote `*` in shells and YAML. A single operand can also be a literal file, directory, or tracked glob. Matches are canonicalized, sorted, and deduplicated before workflow identities and job-key namespaces are assigned. Existing filenames containing `*`, `?`, or `[` remain literal. - -Two or more operands switch to explicit-list mode: +Pass every workflow path explicitly: ```sh buildkite-gha upload -- \ @@ -128,7 +121,7 @@ buildkite-gha upload -- \ .github/workflows/release.yml ``` -Every list entry must resolve to one regular, tracked `.yml` or `.yaml` file inside the repository. Aliases and duplicates are canonicalized, deduplicated, and sorted, so reversed arguments produce the same pipeline. Directories, missing or untracked files, files outside the repository, other extensions, and symlinks are rejected before any workflow is parsed or Buildkite command runs. A tracked filename containing glob metacharacters remains literal, but an unmatched glob mixed into a list—or two glob operands—is rejected rather than expanded independently. +Every operand must name one regular `.yml` or `.yaml` file. When uploading more than one workflow, every path must be tracked inside the repository. Aliases and duplicates are canonicalized, deduplicated, and sorted, so reversed arguments produce the same pipeline. Directories, globs, missing files, other extensions, and symlinks are rejected before any workflow is parsed or Buildkite command runs; aggregate uploads also reject untracked and outside paths. `--` ends option parsing and is required when a path operand begins with `-`; options must appear before it. Without `--`, a leading-dash operand is an unknown option. The CLI does not split shell strings or decode a JSON or YAML list from one argument: custom wrappers should pass each path as a separate argument and use `--` before externally supplied operands. diff --git a/docs/compatibility.md b/docs/compatibility.md index d5e93ba1..3670cf2b 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -59,7 +59,7 @@ Steps remain inside one job because they share a workspace, environment files, a ### Aggregate workflow upload -`*` selects every tracked `.yml` and `.yaml` file directly under `.github/workflows`. Upload also accepts one literal, directory, or tracked glob, or two or more explicit workflow paths. Lists require regular, tracked `.yml` or `.yaml` files inside the repository; directories, missing or untracked files, outside paths, symlinks, and globs fail. Inputs are canonicalized, sorted, and deduplicated before workflow identities and job-key namespaces are assigned. +The plugin accepts one explicit `workflow` path or a non-empty `workflows` path array. Plugin paths and aggregate `upload` operands must identify regular, tracked `.yml` or `.yaml` files inside the repository; directories, missing or untracked files, outside paths, symlinks, and globs fail. A custom importer may upload one explicit regular workflow outside the repository. Inputs are canonicalized, sorted, and deduplicated before workflow identities and job-key namespaces are assigned. All directly runnable workflows are represented in one artifact and pipeline transaction. Each becomes one aggregate group labeled `:github: `, with its canonical path as the fallback for an unnamed workflow. A workflow that declares the effective event compiles into child jobs. A workflow that does not declare it becomes a skipped group with an ignored placeholder and no plan artifacts. The label is static across events. The group-level GitHub check is named `Buildkite / ()`. Each group depends on the importer, while its child jobs omit that redundant dependency and their own check notifications. diff --git a/internal/cli/cli.go b/internal/cli/cli.go index 6ee86a45..55850b84 100644 --- a/internal/cli/cli.go +++ b/internal/cli/cli.go @@ -55,7 +55,7 @@ Run "buildkite-gha help " for command help. var commandUsage = map[string]string{ "validate": "Usage: buildkite-gha validate [--event-path ] [--profile hosted] [--format text|json] \n", "compile": "Usage: buildkite-gha compile --event-path [--format pipeline|ir-json] \n", - "upload": "Usage: buildkite-gha upload [--event-path ] [--runner-queue =]... [--runner-image =]... [--runtime-distribution =]... [--runtime-queue hosted] [--] [...]\n", + "upload": "Usage: buildkite-gha upload [--event-path ] [--runner-queue =]... [--runner-image =]... [--runtime-distribution =]... [--runtime-queue hosted] [--] [...]\n", "run-job": "Usage: buildkite-gha run-job (--plan | --plan-digest --plan-producer ) [--result ] [--hosted-tool-cache]\n", } @@ -67,7 +67,7 @@ func writeCommandHelp(stdout io.Writer, command string) { case "compile": _, _ = fmt.Fprint(stdout, "\nPipeline output references content-addressed plans; compile does not materialize or upload those artifacts.\n") case "upload": - _, _ = fmt.Fprint(stdout, "\nThe * shorthand selects tracked .yml and .yaml files directly under .github/workflows. One workflow operand preserves literal, directory, and tracked glob expansion. Two or more operands are explicit tracked .yml/.yaml paths; use -- before paths that begin with a dash. Inputs are uploaded as one aggregate pipeline with one group per directly runnable workflow; reusable-only workflow_call files are imported through callers but do not become groups. Scheduled groups select only build.source == schedule: Buildkite schedules retain cron ownership, so every scheduled workflow group is eligible on any Buildkite scheduled build. Each repeatable --runner-queue argument maps one supported runs-on label to a Buildkite queue. Configured Linux profiles default to the matching immutable hosted-toolchains image; --runner-image overrides it. Duplicate or unsupported mappings fail, unmapped supported Linux labels retain their default targeting, and every macOS label requires an explicit queue. Each repeatable --runtime-distribution argument binds linux/amd64 or darwin/arm64 to a verified executable. Upload importers must run on linux/amd64; the Linux runtime defaults to the importer executable when omitted, and macOS has no default. The deprecated --runtime-queue hosted option is accepted for plugin compatibility but does not select a queue. Event precedence is an explicit event file, Buildkite's reserved webhook metadata, then reduced-fidelity Buildkite environment compatibility data; every source remains unsigned. Verified checkout jobs automatically use Buildkite repository-provider Git credentials when the job enables them; the deprecated --private-checkout option is accepted as a no-op.\n") + _, _ = fmt.Fprint(stdout, "\nEvery workflow operand must be an explicit .yml or .yaml path; use -- before paths that begin with a dash. Multiple operands must be tracked files inside the checked-out repository. Inputs are uploaded as one aggregate pipeline with one group per directly runnable workflow; reusable-only workflow_call files are imported through callers but do not become groups. Scheduled groups select only build.source == schedule: Buildkite schedules retain cron ownership, so every scheduled workflow group is eligible on any Buildkite scheduled build. Each repeatable --runner-queue argument maps one supported runs-on label to a Buildkite queue. Configured Linux profiles default to the matching immutable hosted-toolchains image; --runner-image overrides it. Duplicate or unsupported mappings fail, unmapped supported Linux labels retain their default targeting, and every macOS label requires an explicit queue. Each repeatable --runtime-distribution argument binds linux/amd64 or darwin/arm64 to a verified executable. Upload importers must run on linux/amd64; the Linux runtime defaults to the importer executable when omitted, and macOS has no default. The deprecated --runtime-queue hosted option is accepted for plugin compatibility but does not select a queue. Event precedence is an explicit event file, Buildkite's reserved webhook metadata, then reduced-fidelity Buildkite environment compatibility data; every source remains unsigned. Verified checkout jobs automatically use Buildkite repository-provider Git credentials when the job enables them; the deprecated --private-checkout option is accepted as a no-op.\n") } } @@ -178,7 +178,7 @@ func plugin(args []string, stdout, stderr io.Writer, version string, runner tran } return uploadParsed(parsedUploadArgs{ workflowOperands: configuration.Workflows, - explicitWorkflowPaths: configuration.explicitWorkflowPaths, + explicitWorkflowPaths: true, runnerTargets: configuration.runnerTargets, pluginAcquisition: &pluginRuntimeAcquisition{version: version}, }, stdout, stderr, version, transport.Agent{Runner: runner}) @@ -192,9 +192,8 @@ func validateImporterPlatform(goos, goarch string) error { } type pluginConfiguration struct { - Workflows []string - explicitWorkflowPaths bool - runnerTargets map[string]compiler.RunnerTarget + Workflows []string + runnerTargets map[string]compiler.RunnerTarget } func parsePluginConfiguration(source string) (pluginConfiguration, error) { @@ -223,40 +222,32 @@ func parsePluginConfiguration(source string) (pluginConfiguration, error) { return pluginConfiguration{}, fmt.Errorf("%s contains unknown field %q", pluginConfigurationEnvironment, key) } } - legacyWorkflow, hasLegacyWorkflow := encoded["workflow"] + workflowValue, hasWorkflow := encoded["workflow"] workflowsValue, hasWorkflows := encoded["workflows"] - if hasLegacyWorkflow && hasWorkflows { + if hasWorkflow && hasWorkflows { return pluginConfiguration{}, fmt.Errorf("%s workflow and workflows are mutually exclusive", pluginConfigurationEnvironment) } var workflows []string - explicitWorkflowPaths := false - if hasLegacyWorkflow { - workflow, ok := legacyWorkflow.(string) + if hasWorkflow { + workflow, ok := workflowValue.(string) if !ok || strings.TrimSpace(workflow) == "" { return pluginConfiguration{}, fmt.Errorf("%s workflow must be a non-empty string", pluginConfigurationEnvironment) } workflows = []string{workflow} } else { - switch value := workflowsValue.(type) { - case string: - if strings.TrimSpace(value) != "" { - workflows = []string{value} - } - case []any: - if len(value) != 0 { - explicitWorkflowPaths = true - workflows = make([]string, len(value)) - for index, entry := range value { - workflow, ok := entry.(string) - if !ok || strings.TrimSpace(workflow) == "" { - return pluginConfiguration{}, fmt.Errorf("%s workflows entry %d must be a non-empty string", pluginConfigurationEnvironment, index) - } - workflows[index] = workflow + values, ok := workflowsValue.([]any) + if ok && len(values) != 0 { + workflows = make([]string, len(values)) + for index, entry := range values { + workflow, ok := entry.(string) + if !ok || strings.TrimSpace(workflow) == "" { + return pluginConfiguration{}, fmt.Errorf("%s workflows entry %d must be a non-empty string", pluginConfigurationEnvironment, index) } + workflows[index] = workflow } } if len(workflows) == 0 { - return pluginConfiguration{}, fmt.Errorf("%s workflows is required and must be a non-empty string or array of non-empty strings", pluginConfigurationEnvironment) + return pluginConfiguration{}, fmt.Errorf("%s workflow or workflows is required; workflows must be a non-empty array of non-empty strings", pluginConfigurationEnvironment) } } targets := make(map[string]compiler.RunnerTarget) @@ -303,7 +294,7 @@ func parsePluginConfiguration(source string) (pluginConfiguration, error) { targets[label] = target } } - return pluginConfiguration{Workflows: workflows, explicitWorkflowPaths: explicitWorkflowPaths, runnerTargets: targets}, nil + return pluginConfiguration{Workflows: workflows, runnerTargets: targets}, nil } func normalizePluginCommit(ctx context.Context, getenv func(string) string, setenv func(string, string) error, runner transport.Runner) error { @@ -1406,7 +1397,7 @@ func uploadParsed(uploadArguments parsedUploadArgs, stdout, stderr io.Writer, ve if uploadArguments.explicitWorkflowPaths { workflows, err = expandExplicitWorkflowPaths(workflowOperands) } else { - workflows, err = expandWorkflowOperands(workflowOperands) + workflows, err = resolveWorkflowOperands(workflowOperands) } if err != nil { _, _ = fmt.Fprintf(stderr, "buildkite-gha: upload: %v\n", err) @@ -1432,11 +1423,7 @@ func uploadParsed(uploadArguments parsedUploadArgs, stdout, stderr io.Writer, ve } } if runnableWorkflowCount == 0 { - if len(workflowOperands) == 1 { - _, _ = fmt.Fprintf(stderr, "buildkite-gha: upload: workflow pattern %q matched only reusable workflow_call workflows; there is nothing to upload\n", workflowOperands[0]) - } else { - _, _ = fmt.Fprintln(stderr, "buildkite-gha: upload: workflow paths matched only reusable workflow_call workflows; there is nothing to upload") - } + _, _ = fmt.Fprintln(stderr, "buildkite-gha: upload: workflow paths matched only reusable workflow_call workflows; there is nothing to upload") return 1 } ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) @@ -1822,17 +1809,35 @@ type workflowInput struct { ReusableOnly, Applicable bool } -func expandWorkflowOperands(operands []string) ([]workflowInput, error) { - if len(operands) == 0 { - return nil, fmt.Errorf("workflow path is required") +func resolveWorkflowOperands(operands []string) ([]workflowInput, error) { + if len(operands) != 1 { + return expandExplicitWorkflowPaths(operands) } - if len(operands) == 1 { - return expandWorkflowPattern(operands[0]) + path, err := filepath.Abs(operands[0]) + if err != nil { + return nil, fmt.Errorf("resolve workflow path %q: %w", operands[0], err) + } + if err := requireRegularWorkflowFile(path, operands[0]); err != nil { + return expandExplicitWorkflowPaths(operands) + } + extension := filepath.Ext(path) + if extension != ".yml" && extension != ".yaml" { + return nil, fmt.Errorf("workflow path %q must end in .yml or .yaml", operands[0]) + } + canonical := filepath.ToSlash(filepath.Clean(operands[0])) + if rootBytes, rootErr := exec.Command("git", "rev-parse", "--show-toplevel").Output(); rootErr == nil { + root := filepath.Clean(strings.TrimSpace(string(rootBytes))) + if relative, relativeErr := filepath.Rel(root, path); relativeErr == nil && relative != ".." && !strings.HasPrefix(relative, ".."+string(filepath.Separator)) { + canonical = filepath.ToSlash(filepath.Clean(relative)) + } } - return expandExplicitWorkflowPaths(operands) + return workflowInputs([]workflowInput{{Path: path, CanonicalPath: canonical}}, false) } func expandExplicitWorkflowPaths(operands []string) ([]workflowInput, error) { + if len(operands) == 0 { + return nil, fmt.Errorf("workflow path is required") + } rootBytes, err := exec.Command("git", "rev-parse", "--show-toplevel").Output() if err != nil { return nil, fmt.Errorf("locate checked-out git repository: %w", err) @@ -1872,77 +1877,6 @@ func expandExplicitWorkflowPaths(operands []string) ([]workflowInput, error) { return workflowInputs(matches, true) } -func expandWorkflowPattern(pattern string) ([]workflowInput, error) { - allWorkflows := pattern == "*" - patternHasMeta := allWorkflows || strings.ContainsAny(pattern, "*?[") - if info, err := os.Stat(pattern); !allWorkflows && err == nil && !info.IsDir() { - // An existing path is always literal, even when its filename contains - // glob metacharacters. This preserves the pre-pattern CLI contract. - patternHasMeta = false - } - rootBytes, rootErr := exec.Command("git", "rev-parse", "--show-toplevel").Output() - if rootErr != nil { - if !patternHasMeta { - if info, err := os.Stat(pattern); err == nil && !info.IsDir() { - return workflowInputs([]workflowInput{{Path: pattern, CanonicalPath: filepath.ToSlash(filepath.Clean(pattern))}}, false) - } - } - return nil, fmt.Errorf("locate checked-out git repository: %w", rootErr) - } - root := strings.TrimSpace(string(rootBytes)) - pathspecs := []string{ - ":(top,glob).github/workflows/*.yml", - ":(top,glob).github/workflows/*.yaml", - } - if !allWorkflows { - absolutePattern, err := filepath.Abs(pattern) - if err != nil { - return nil, fmt.Errorf("resolve workflow pattern %q: %w", pattern, err) - } - relativePattern, err := filepath.Rel(root, absolutePattern) - if err != nil || relativePattern == ".." || strings.HasPrefix(relativePattern, ".."+string(filepath.Separator)) { - if !patternHasMeta { - if info, statErr := os.Stat(pattern); statErr == nil && !info.IsDir() { - return workflowInputs([]workflowInput{{Path: pattern, CanonicalPath: filepath.ToSlash(filepath.Clean(pattern))}}, false) - } - } - return nil, fmt.Errorf("workflow pattern %q is outside the checked-out git repository", pattern) - } - pathspecMagic := ":(literal)" - if patternHasMeta { - pathspecMagic = ":(glob)" - } - pathspecs = []string{pathspecMagic + filepath.ToSlash(relativePattern)} - } - commandArgs := append([]string{"-C", root, "ls-files", "-z", "--"}, pathspecs...) - command := exec.Command("git", commandArgs...) - output, err := command.Output() - if err != nil { - return nil, fmt.Errorf("expand workflow pattern %q against tracked files: %w", pattern, err) - } - var matches []workflowInput - for _, entry := range bytes.Split(output, []byte{0}) { - if len(entry) == 0 { - continue - } - canonical := string(entry) - path := filepath.Join(root, filepath.FromSlash(canonical)) - if err := requireRegularWorkflowFile(path, canonical); err != nil { - return nil, err - } - matches = append(matches, workflowInput{Path: path, CanonicalPath: canonical}) - } - if len(matches) == 0 && !patternHasMeta { - if info, statErr := os.Stat(pattern); statErr == nil && !info.IsDir() { - return workflowInputs([]workflowInput{{Path: pattern, CanonicalPath: filepath.ToSlash(filepath.Clean(pattern))}}, false) - } - } - if len(matches) == 0 { - return nil, fmt.Errorf("workflow pattern %q matched no tracked files", pattern) - } - return workflowInputs(matches, patternHasMeta || len(matches) > 1) -} - func requireRegularWorkflowFile(path, displayPath string) error { info, err := os.Lstat(path) if err != nil || !info.Mode().IsRegular() { diff --git a/internal/cli/cli_test.go b/internal/cli/cli_test.go index 3167a102..4affd359 100644 --- a/internal/cli/cli_test.go +++ b/internal/cli/cli_test.go @@ -105,7 +105,7 @@ func TestUploadHelpFormsMatch(t *testing.T) { } outputs = append(outputs, stdout.String()) } - if outputs[0] != outputs[1] || !strings.Contains(outputs[0], "tracked .yml and .yaml files directly under .github/workflows") || !strings.Contains(outputs[0], "--runner-image") || !strings.Contains(outputs[0], "every scheduled workflow group is eligible") { + if outputs[0] != outputs[1] || !strings.Contains(outputs[0], "explicit .yml or .yaml path") || !strings.Contains(outputs[0], "--runner-image") || !strings.Contains(outputs[0], "every scheduled workflow group is eligible") { t.Fatalf("upload help outputs differ or omit runner profile or aggregate workflow options:\nhelp command: %q\nhelp flag: %q", outputs[0], outputs[1]) } } @@ -163,23 +163,16 @@ func TestParsePluginConfiguration(t *testing.T) { if !slices.Equal(configuration.Workflows, []string{".github/workflows/ci.yml", ".github/workflows/release.yml"}) || len(configuration.runnerTargets) != 2 { t.Fatalf("configuration = %#v", configuration) } - if !configuration.explicitWorkflowPaths { - t.Fatal("workflows array did not retain explicit path semantics") - } if got := configuration.runnerTargets["ubuntu-latest"]; got != (compiler.RunnerTarget{Queue: "hosted", Platform: compiler.PlatformLinuxAMD64, Image: image}) { t.Fatalf("Linux target = %#v", got) } if got := configuration.runnerTargets["macos-14"]; got != (compiler.RunnerTarget{Queue: "macos-sonoma-arm64", Platform: compiler.PlatformDarwinARM64}) { t.Fatalf("Darwin target = %#v", got) } - minimal, err := parsePluginConfiguration(`{"workflows":"workflow.yml"}`) - if err != nil || !slices.Equal(minimal.Workflows, []string{"workflow.yml"}) || minimal.explicitWorkflowPaths || len(minimal.runnerTargets) != 0 { + minimal, err := parsePluginConfiguration(`{"workflow":"workflow.yml"}`) + if err != nil || !slices.Equal(minimal.Workflows, []string{"workflow.yml"}) || len(minimal.runnerTargets) != 0 { t.Fatalf("minimal configuration = %#v, %v", minimal, err) } - legacy, err := parsePluginConfiguration(`{"workflow":"legacy.yml"}`) - if err != nil || !slices.Equal(legacy.Workflows, []string{"legacy.yml"}) { - t.Fatalf("legacy configuration = %#v, %v", legacy, err) - } for _, test := range []struct { name string @@ -187,12 +180,12 @@ func TestParsePluginConfiguration(t *testing.T) { want string }{ {name: "malformed", source: `{`, want: "decode"}, - {name: "missing workflows", source: `{}`, want: "workflows is required"}, + {name: "missing workflow selection", source: `{}`, want: "workflow or workflows is required"}, {name: "duplicate workflow", source: `{"workflow":"one.yml","workflow":"two.yml"}`, want: "duplicate object key"}, {name: "both workflow fields", source: `{"workflow":"one.yml","workflows":"two.yml"}`, want: "mutually exclusive"}, {name: "empty workflow", source: `{"workflow":""}`, want: "workflow must be a non-empty string"}, - {name: "empty workflows string", source: `{"workflows":""}`, want: "non-empty string or array"}, - {name: "empty workflows array", source: `{"workflows":[]}`, want: "non-empty string or array"}, + {name: "workflows string", source: `{"workflows":"one.yml"}`, want: "workflows must be a non-empty array"}, + {name: "empty workflows array", source: `{"workflows":[]}`, want: "workflows must be a non-empty array"}, {name: "non-string workflow entry", source: `{"workflows":["one.yml",null]}`, want: "workflows entry 1 must be a non-empty string"}, {name: "empty workflow entry", source: `{"workflows":["one.yml",""]}`, want: "workflows entry 1 must be a non-empty string"}, {name: "unknown top-level field", source: `{"workflow":"ci.yml","runnerss":[]}`, want: "unknown field"}, @@ -289,8 +282,8 @@ func TestPluginUsesJSONConfigurationAndOnlyRequiredRuntime(t *testing.T) { t.Fatal(err) } configuration, err := json.Marshal(map[string]any{ - "workflows": workflowPath, - "version": "0.8.0", + "workflow": workflowPath, + "version": "0.8.0", "runners": []map[string]string{ {"runs-on": "ubuntu-latest", "queue": "hosted"}, }, @@ -366,35 +359,47 @@ func TestPluginUploadsPluralWorkflowList(t *testing.T) { } } -func TestPluginRejectsGlobInSingleEntryWorkflowList(t *testing.T) { +func TestPluginRejectsNonExplicitWorkflowSelectors(t *testing.T) { requireImporterHost(t) - configuration, err := json.Marshal(map[string]any{ - "workflows": []string{filepath.Join("..", "..", "testdata", "smoke", ".github", "workflows", "*.yml")}, - }) - if err != nil { - t.Fatal(err) - } - t.Setenv(pluginConfigurationEnvironment, string(configuration)) - setCLIPluginBuildkiteEnvironment(t, "plugin-workflows-glob") - runner := &cliCaptureRunner{} - var stdout, stderr bytes.Buffer - if code := run([]string{"plugin"}, &stdout, &stderr, "dev", runner); code != 1 || !strings.Contains(stderr.String(), "explicit paths") { - t.Fatalf("run() code/stderr = %d / %q", code, stderr.String()) - } - if stdout.Len() != 0 || len(runner.commands) != 0 || len(runner.uploaded) != 0 { - t.Fatalf("invalid workflow list reached Buildkite: stdout %q, commands %#v, uploads %#v", stdout.String(), runner.commands, runner.uploaded) + workflowDirectory := filepath.Join("..", "..", "testdata", "smoke", ".github", "workflows") + for _, test := range []struct { + name string + field string + workflows any + want string + }{ + {name: "all shorthand", field: "workflow", workflows: "*", want: "explicit paths"}, + {name: "string glob", field: "workflow", workflows: filepath.Join(workflowDirectory, "*.yml"), want: "explicit paths"}, + {name: "array glob", field: "workflows", workflows: []string{filepath.Join(workflowDirectory, "*.yml")}, want: "explicit paths"}, + {name: "directory", field: "workflow", workflows: workflowDirectory, want: "does not name a regular tracked file"}, + } { + t.Run(test.name, func(t *testing.T) { + configuration, err := json.Marshal(map[string]any{test.field: test.workflows}) + if err != nil { + t.Fatal(err) + } + t.Setenv(pluginConfigurationEnvironment, string(configuration)) + setCLIPluginBuildkiteEnvironment(t, "plugin-workflows-explicit") + runner := &cliCaptureRunner{} + var stdout, stderr bytes.Buffer + if code := run([]string{"plugin"}, &stdout, &stderr, "dev", runner); code != 1 || !strings.Contains(stderr.String(), test.want) { + t.Fatalf("run() code/stderr = %d / %q", code, stderr.String()) + } + if stdout.Len() != 0 || len(runner.commands) != 0 || len(runner.uploaded) != 0 { + t.Fatalf("invalid workflow selection reached Buildkite: stdout %q, commands %#v, uploads %#v", stdout.String(), runner.commands, runner.uploaded) + } + }) } } func TestPluginPublishesMixedRuntimeDistributions(t *testing.T) { requireImporterHost(t) const fullCommit = "0123456789abcdef0123456789abcdef01234567" - root := t.TempDir() - workflowPath := filepath.Join(root, "mixed.yml") - workflow := []byte("on: push\njobs:\n linux:\n runs-on: ubuntu-latest\n steps:\n - run: echo linux\n macos:\n needs: linux\n runs-on: macos-15\n steps:\n - run: echo macos\n") - if err := os.WriteFile(workflowPath, workflow, 0o600); err != nil { - t.Fatal(err) - } + repository := writeUploadWorkflowRepository(t, map[string]string{ + "mixed.yml": "on: push\njobs:\n linux:\n runs-on: ubuntu-latest\n steps:\n - run: echo linux\n macos:\n needs: linux\n runs-on: macos-15\n steps:\n - run: echo macos\n", + }) + t.Chdir(repository) + workflowPath := filepath.Join(".github", "workflows", "mixed.yml") linuxPath, err := os.Executable() if err != nil { t.Fatal(err) @@ -409,7 +414,7 @@ func TestPluginPublishesMixedRuntimeDistributions(t *testing.T) { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, } - darwinPath := filepath.Join(root, "buildkite-gha-darwin") + darwinPath := filepath.Join(t.TempDir(), "buildkite-gha-darwin") if err := os.WriteFile(darwinPath, darwinContents, 0o700); err != nil { t.Fatal(err) } @@ -487,10 +492,23 @@ func TestPluginPublishesMixedRuntimeDistributions(t *testing.T) { Command string }{Image: step.Image, Queue: step.Agents["queue"], Command: step.Command} } - if linux := steps["gha-linux"]; linux.Queue != "linux" || linux.Image != image || !strings.Contains(linux.Command, "--hosted-tool-cache") || !strings.Contains(linux.Command, strings.TrimPrefix(cliTestRuntimeDigest(), "sha256:")) { + var linux, macos struct { + Image string + Queue string + Command string + } + for key, step := range steps { + switch { + case strings.HasSuffix(key, "-linux"): + linux = step + case strings.HasSuffix(key, "-macos"): + macos = step + } + } + if linux.Queue != "linux" || linux.Image != image || !strings.Contains(linux.Command, "--hosted-tool-cache") || !strings.Contains(linux.Command, strings.TrimPrefix(cliTestRuntimeDigest(), "sha256:")) { t.Fatalf("Linux pipeline step = %#v", linux) } - if macos := steps["gha-macos"]; macos.Queue != "macos" || macos.Image != "" || strings.Contains(macos.Command, "--hosted-tool-cache") || !strings.Contains(macos.Command, strings.TrimPrefix(darwinDigest, "sha256:")) { + if macos.Queue != "macos" || macos.Image != "" || strings.Contains(macos.Command, "--hosted-tool-cache") || !strings.Contains(macos.Command, strings.TrimPrefix(darwinDigest, "sha256:")) { t.Fatalf("Darwin pipeline step = %#v", macos) } } @@ -2418,111 +2436,7 @@ func TestRunUploadPublishesMixedRuntimeDistributions(t *testing.T) { } } -func TestExpandWorkflowPatternSortsDeduplicatesAndRejectsNoMatch(t *testing.T) { - pattern := filepath.Join("..", "..", "testdata", "smoke", ".github", "workflows", "*e*.yml") - first, err := expandWorkflowPattern(pattern) - if err != nil { - t.Fatal(err) - } - second, err := expandWorkflowPattern(pattern) - if err != nil { - t.Fatal(err) - } - wantNames := []string{"artifact-multi-prefix.yml", "cache-v5.yml", "cache-v6.yml", "concurrent.yml", "shell.yml"} - gotNames := make([]string, 0, len(first)) - for _, workflow := range first { - gotNames = append(gotNames, filepath.Base(workflow.CanonicalPath)) - } - if !reflect.DeepEqual(first, second) || !reflect.DeepEqual(gotNames, wantNames) { - t.Fatalf("expanded workflows = %#v and %#v", first, second) - } - for _, workflow := range first { - if workflow.Identity == "" || workflow.StepKeyNamespace != workflow.Identity { - t.Fatalf("glob workflow identity = %#v", workflow) - } - } - duplicated, err := workflowInputs([]workflowInput{first[1], first[0], first[1]}, true) - if err != nil || len(duplicated) != 2 || duplicated[0].CanonicalPath != first[0].CanonicalPath || duplicated[1].CanonicalPath != first[1].CanonicalPath { - t.Fatalf("deduplicated workflows/error = %#v / %v", duplicated, err) - } - if _, err := expandWorkflowPattern(filepath.Join("..", "..", "testdata", "smoke", ".github", "workflows", "missing-*.yml")); err == nil || !strings.Contains(err.Error(), "matched no tracked files") { - t.Fatalf("no-match error = %v", err) - } -} - -func TestExpandWorkflowPatternPreservesLiteralMetacharacterPath(t *testing.T) { - path := filepath.Join(t.TempDir(), "workflow[1].yml") - if err := os.WriteFile(path, []byte("on: push\n"), 0o600); err != nil { - t.Fatal(err) - } - inputs, err := expandWorkflowPattern(path) - if err != nil { - t.Fatal(err) - } - if len(inputs) != 1 || inputs[0].Path != path || inputs[0].StepKeyNamespace != "" { - t.Fatalf("literal metacharacter input = %#v", inputs) - } -} - -func TestExpandWorkflowPatternNamespacesLiteralDirectoryMatches(t *testing.T) { - directory := filepath.Join("..", "..", "testdata", "smoke", ".github", "workflows") - inputs, err := expandWorkflowPattern(directory) - if err != nil { - t.Fatal(err) - } - if len(inputs) != 10 { - t.Fatalf("literal directory matched %d workflows, want 10", len(inputs)) - } - for _, input := range inputs { - if input.StepKeyNamespace == "" || input.StepKeyNamespace != input.Identity { - t.Fatalf("literal directory workflow is not namespaced: %#v", input) - } - } -} - -func TestExpandWorkflowPatternStarSelectsTopLevelYAMLWorkflows(t *testing.T) { - workflowSource := "on: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps: [{run: true}]\n" - repository := writeUploadWorkflowRepository(t, map[string]string{ - "first.yml": workflowSource, - "second.yaml": workflowSource, - }) - repository, err := filepath.EvalSymlinks(repository) - if err != nil { - t.Fatal(err) - } - workflowDirectory := filepath.Join(repository, ".github", "workflows") - if err := os.WriteFile(filepath.Join(workflowDirectory, "notes.txt"), []byte("not a workflow\n"), 0o600); err != nil { - t.Fatal(err) - } - nestedDirectory := filepath.Join(workflowDirectory, "nested") - if err := os.MkdirAll(nestedDirectory, 0o755); err != nil { - t.Fatal(err) - } - if err := os.WriteFile(filepath.Join(nestedDirectory, "nested.yml"), []byte(workflowSource), 0o600); err != nil { - t.Fatal(err) - } - if err := os.WriteFile(filepath.Join(repository, "root.yml"), []byte(workflowSource), 0o600); err != nil { - t.Fatal(err) - } - if output, err := exec.Command("git", "-C", repository, "add", ".").CombinedOutput(); err != nil { - t.Fatalf("git add shorthand fixtures: %v: %s", err, output) - } - t.Chdir(repository) - inputs, err := expandWorkflowOperands([]string{"*"}) - if err != nil { - t.Fatal(err) - } - got := make([]string, 0, len(inputs)) - for _, input := range inputs { - got = append(got, input.CanonicalPath) - } - want := []string{".github/workflows/first.yml", ".github/workflows/second.yaml"} - if !reflect.DeepEqual(got, want) { - t.Fatalf("* workflows = %q, want %q", got, want) - } -} - -func TestRunUploadRejectsTrackedSymlinksMatchedByGlob(t *testing.T) { +func TestRunUploadRejectsExplicitTrackedSymlinks(t *testing.T) { requireImporterHost(t) workflowSource := "on: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps: [{run: true}]\n" for _, test := range []struct { @@ -2561,7 +2475,7 @@ func TestRunUploadRejectsTrackedSymlinksMatchedByGlob(t *testing.T) { t.Setenv("BUILDKITE_STEP_KEY", "symlink-importer") runner := &cliCaptureRunner{} var stdout, stderr bytes.Buffer - if code := run([]string{"upload", "--event-path", eventPath, ".github/workflows/*.yml"}, &stdout, &stderr, "dev", runner); code != 1 || !strings.Contains(stderr.String(), "does not name a regular tracked file") { + if code := run([]string{"upload", "--event-path", eventPath, ".github/workflows/linked.yml"}, &stdout, &stderr, "dev", runner); code != 1 || !strings.Contains(stderr.String(), "does not name a regular tracked file") { t.Fatalf("run() code/stderr = %d / %q", code, stderr.String()) } if stdout.Len() != 0 || len(runner.commands) != 0 || len(runner.uploaded) != 0 { @@ -2571,7 +2485,7 @@ func TestRunUploadRejectsTrackedSymlinksMatchedByGlob(t *testing.T) { } } -func TestExpandWorkflowOperandsCanonicalizesExplicitTrackedPaths(t *testing.T) { +func TestExpandExplicitWorkflowPathsCanonicalizesTrackedPaths(t *testing.T) { workflowSource := "on: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps: [{run: true}]\n" repository := writeUploadWorkflowRepository(t, map[string]string{ "a.yml": workflowSource, @@ -2609,11 +2523,11 @@ func TestExpandWorkflowOperandsCanonicalizesExplicitTrackedPaths(t *testing.T) { aPath := filepath.Join(".github", "workflows", "a.yml") bPath := filepath.Join(".github", "workflows", "b.yaml") - first, err := expandWorkflowOperands([]string{filepath.Join(repository, bPath), "./" + aPath, aPath}) + first, err := expandExplicitWorkflowPaths([]string{filepath.Join(repository, bPath), "./" + aPath, aPath}) if err != nil { t.Fatal(err) } - second, err := expandWorkflowOperands([]string{aPath, filepath.Join(repository, bPath)}) + second, err := expandExplicitWorkflowPaths([]string{aPath, filepath.Join(repository, bPath)}) if err != nil { t.Fatal(err) } @@ -2625,7 +2539,7 @@ func TestExpandWorkflowOperandsCanonicalizesExplicitTrackedPaths(t *testing.T) { t.Fatalf("explicit workflow identity = %#v", input) } } - metacharacter, err := expandWorkflowOperands([]string{filepath.Join(".github", "workflows", "workflow[1].yml"), aPath}) + metacharacter, err := expandExplicitWorkflowPaths([]string{filepath.Join(".github", "workflows", "workflow[1].yml"), aPath}) if err != nil || len(metacharacter) != 2 || metacharacter[1].CanonicalPath != ".github/workflows/workflow[1].yml" { t.Fatalf("literal metacharacter list = %#v, %v", metacharacter, err) } @@ -2633,7 +2547,7 @@ func TestExpandWorkflowOperandsCanonicalizesExplicitTrackedPaths(t *testing.T) { if err != nil { t.Fatal(err) } - leadingDash, err := expandWorkflowOperands(operands) + leadingDash, err := expandExplicitWorkflowPaths(operands) if err != nil || len(leadingDash) != 2 || leadingDash[0].CanonicalPath != "-leading.yml" { t.Fatalf("leading-dash explicit path = %#v, %v", leadingDash, err) } @@ -2653,8 +2567,8 @@ func TestExpandWorkflowOperandsCanonicalizesExplicitTrackedPaths(t *testing.T) { {name: "symlink", operands: []string{symlinkPath, aPath}, want: "regular tracked file"}, } { t.Run(test.name, func(t *testing.T) { - if _, err := expandWorkflowOperands(test.operands); err == nil || !strings.Contains(err.Error(), test.want) { - t.Fatalf("expandWorkflowOperands(%q) error = %v, want %q", test.operands, err, test.want) + if _, err := expandExplicitWorkflowPaths(test.operands); err == nil || !strings.Contains(err.Error(), test.want) { + t.Fatalf("expandExplicitWorkflowPaths(%q) error = %v, want %q", test.operands, err, test.want) } }) } @@ -2710,28 +2624,42 @@ func TestRunUploadExplicitPathsAreAtomicAndOrderIndependent(t *testing.T) { } } -func TestRunUploadRejectsMultipleOperandGlobBeforeBuildkite(t *testing.T) { +func TestRunUploadRejectsWorkflowSelectorsBeforeBuildkite(t *testing.T) { requireImporterHost(t) workflowSource := "on: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps: [{run: true}]\n" repository := writeUploadWorkflowRepository(t, map[string]string{"a.yml": workflowSource}) t.Chdir(repository) t.Setenv("BUILDKITE", "true") t.Setenv("BUILDKITE_STEP_KEY", "invalid-list-importer") - runner := &cliCaptureRunner{} - var stdout, stderr bytes.Buffer - if code := run([]string{"upload", filepath.Join(".github", "workflows", "*.yml"), filepath.Join(".github", "workflows", "a.yml")}, &stdout, &stderr, "dev", runner); code != 1 || !strings.Contains(stderr.String(), "glob pattern") { - t.Fatalf("run() code/stderr = %d / %q", code, stderr.String()) - } - if stdout.Len() != 0 || len(runner.commands) != 0 || len(runner.uploaded) != 0 { - t.Fatalf("invalid explicit list reached Buildkite: stdout %q, commands %#v, uploads %#v", stdout.String(), runner.commands, runner.uploaded) + for _, operands := range [][]string{ + {"*"}, + {filepath.Join(".github", "workflows", "*.yml")}, + {filepath.Join(".github", "workflows", "*.yml"), filepath.Join(".github", "workflows", "a.yml")}, + } { + runner := &cliCaptureRunner{} + var stdout, stderr bytes.Buffer + args := append([]string{"upload"}, operands...) + if code := run(args, &stdout, &stderr, "dev", runner); code != 1 || !strings.Contains(stderr.String(), "glob pattern") { + t.Fatalf("run(%q) code/stderr = %d / %q", operands, code, stderr.String()) + } + if stdout.Len() != 0 || len(runner.commands) != 0 || len(runner.uploaded) != 0 { + t.Fatalf("invalid workflow selector reached Buildkite: stdout %q, commands %#v, uploads %#v", stdout.String(), runner.commands, runner.uploaded) + } } } -func TestRunUploadAggregatesGlobAtomicallyWithNamespacedJobs(t *testing.T) { +func TestRunUploadAggregatesExplicitPathsAtomicallyWithNamespacedJobs(t *testing.T) { requireImporterHost(t) - pattern := filepath.Join("..", "..", "testdata", "smoke", ".github", "workflows", "*e*.yml") + workflowDirectory := filepath.Join("..", "..", "testdata", "smoke", ".github", "workflows") + workflowPaths := []string{ + filepath.Join(workflowDirectory, "artifact-multi-prefix.yml"), + filepath.Join(workflowDirectory, "cache-v5.yml"), + filepath.Join(workflowDirectory, "cache-v6.yml"), + filepath.Join(workflowDirectory, "concurrent.yml"), + filepath.Join(workflowDirectory, "shell.yml"), + } eventPath := filepath.Join("..", "..", "testdata", "smoke", "events", "push.json") - inputs, err := expandWorkflowPattern(pattern) + inputs, err := expandExplicitWorkflowPaths(workflowPaths) if err != nil { t.Fatal(err) } @@ -2739,7 +2667,8 @@ func TestRunUploadAggregatesGlobAtomicallyWithNamespacedJobs(t *testing.T) { t.Setenv("BUILDKITE_STEP_KEY", "aggregate-importer") runner := &cliCaptureRunner{} var stdout, stderr bytes.Buffer - if code := run([]string{"upload", "--event-path", eventPath, pattern}, &stdout, &stderr, "dev", runner); code != 0 { + args := append([]string{"upload", "--event-path", eventPath}, workflowPaths...) + if code := run(args, &stdout, &stderr, "dev", runner); code != 0 { t.Fatalf("run() code = %d, stderr = %q", code, stderr.String()) } if !strings.Contains(stdout.String(), "Uploaded 12 jobs from 5 workflows") || stderr.Len() != 0 || len(runner.commands) != 14 { @@ -2852,7 +2781,14 @@ func TestRunUploadNamesAggregateGitHubChecksFromWorkflowLabels(t *testing.T) { t.Setenv("BUILDKITE_STEP_KEY", "checks-importer") runner := &cliCaptureRunner{} var stdout, stderr bytes.Buffer - if code := run([]string{"upload", "--event-path", eventPath, ".github/workflows/*.yml"}, &stdout, &stderr, "dev", runner); code != 0 { + workflowPaths := []string{ + ".github/workflows/a.yml", + ".github/workflows/b.yml", + ".github/workflows/unnamed.yml", + ".github/workflows/reusable.yml", + } + args := append([]string{"upload", "--event-path", eventPath}, workflowPaths...) + if code := run(args, &stdout, &stderr, "dev", runner); code != 0 { t.Fatalf("run() code = %d, stderr = %q", code, stderr.String()) } var pipeline struct { @@ -2994,6 +2930,12 @@ func TestRunUploadIsolatesExplicitEffectiveEventsBeforeCompilation(t *testing.T) "workflow_dispatch": writeUploadEvent(t, repository, "workflow_dispatch", "refs/heads/main", map[string]any{}), "schedule": writeUploadEvent(t, repository, "schedule", "refs/heads/main", map[string]any{}), } + workflowPaths := []string{ + ".github/workflows/dispatch.yml", + ".github/workflows/pull-request.yml", + ".github/workflows/push.yml", + ".github/workflows/schedule.yml", + } for _, test := range []struct { name, event, source, workflow string }{ @@ -3009,7 +2951,8 @@ func TestRunUploadIsolatesExplicitEffectiveEventsBeforeCompilation(t *testing.T) t.Setenv("BUILDKITE_SOURCE", test.source) runner := &cliCaptureRunner{webhookErr: errors.New("metadata must not be read with --event-path")} var stdout, stderr bytes.Buffer - if code := run([]string{"upload", "--event-path", events[test.event], ".github/workflows/*.yml"}, &stdout, &stderr, "dev", runner); code != 0 { + args := append([]string{"upload", "--event-path", events[test.event]}, workflowPaths...) + if code := run(args, &stdout, &stderr, "dev", runner); code != 0 { t.Fatalf("run() code = %d, stderr = %q", code, stderr.String()) } var pipeline struct { @@ -3075,6 +3018,12 @@ func TestRunUploadAlignsBuildkiteFallbackWithEffectiveEvent(t *testing.T) { "push.yml": "name: Push\non: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps: [{run: true}]\n", "schedule.yml": "name: Schedule\non:\n schedule:\n - cron: '0 0 * * *'\njobs:\n test:\n runs-on: ubuntu-latest\n steps: [{run: true}]\n", }) + workflowPaths := []string{ + ".github/workflows/dispatch.yml", + ".github/workflows/pull-request.yml", + ".github/workflows/push.yml", + ".github/workflows/schedule.yml", + } for _, test := range []struct { name, source, event, workflow string pullRequest bool @@ -3102,7 +3051,8 @@ func TestRunUploadAlignsBuildkiteFallbackWithEffectiveEvent(t *testing.T) { t.Setenv("BUILDKITE_SOURCE", test.source) runner := &cliCaptureRunner{} var stdout, stderr bytes.Buffer - if code := run([]string{"upload", ".github/workflows/*.yml"}, &stdout, &stderr, "dev", runner); code != 0 { + args := append([]string{"upload"}, workflowPaths...) + if code := run(args, &stdout, &stderr, "dev", runner); code != 0 { t.Fatalf("run() code = %d, stderr = %q", code, stderr.String()) } var pipeline struct {