Skip to content

Batch randomly cancels in-progress jobs if re-submitting #62

Description

@ppinchuk

Sometimes kicking off a batch job while it is already running cancels the set that was kicked off first (but does not cancel all sets that are running).

You submit the batch, then change the configs and kick off the batch again - when you do that, only one of the two sets that were running get canceled. However, you wouldn't expect anything to be cancelled because it was "submitted"/running.

No config paths were changed.

Use case for this would be to kick batch off again for some failed jobs while others are still running.

This is all using --monitor-background.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

p-mediumPriority: mediumtopic-batchIssues/pull requests related to batched pipeline execution

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions