Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ type CompareFunc func(actual, expected interface{}) (equal bool, differences str
| `.Comparer(fn)` | Set custom comparison function |
| `.Timeout(d)` | Set max duration per case |
| `.Parallel()` | Enable parallel execution for subtests (use with `SubTest()`) |
| `.KnownIssue(reason)` | Add message for known flaky tests |

## Case Fields

Expand Down
20 changes: 20 additions & 0 deletions docs/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -364,6 +364,26 @@ trial.New(fn, cases).

---

## Known Issue Visibility

Use `KnownIssue()` to annotate failures with a warning reason when a test is known to be flaky. This is visibility only: failures still fail.

```go
trial.New(fn, cases).
KnownIssue("tracked flaky behavior - TODO fix #123").
SubTest(t)
```

When a case fails, output includes a suffix like:

```text
FAIL: "case name" ... (known issue: tracked flaky behavior - TODO fix #123)
```

Passing cases are unchanged.

---

## Complete Example

Copy-paste-ready test file:
Expand Down
30 changes: 25 additions & 5 deletions trial.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,14 @@ func colorGreen(s string) string {
return s
}

// colorYellow wraps text in yellow ANSI color codes if colors are enabled.
func colorYellow(s string) string {
if colorEnabled {
return "\033[33m" + s + "\033[0m"
}
return s
}

type (
// TestFunc a wrapper function used to setup the method being tested.
TestFunc func(in Input) (result interface{}, err error)
Expand All @@ -70,11 +78,12 @@ type Comparer interface {

// Trial framework used to test different logical states
type Trial[In any, Out any] struct {
cases map[string]Case[In, Out]
testFn testFunc[In, Out]
equalFn CompareFunc
timeout time.Duration
parallel bool
cases map[string]Case[In, Out]
testFn testFunc[In, Out]
equalFn CompareFunc
timeout time.Duration
parallel bool
knownIssueReason string
}

// Cases made during the trial
Expand Down Expand Up @@ -126,6 +135,14 @@ func (t *Trial[In, Out]) Parallel() *Trial[In, Out] {
return t
}

// KnownIssue marks this trial run with a known-issue reason.
// If a case fails, the reason is appended to improve visibility.
// This does not change pass/fail behavior.
func (t *Trial[In, Out]) KnownIssue(reason string) *Trial[In, Out] {
t.knownIssueReason = reason
return t
}

// SubTest runs all cases as individual subtests
func (t *Trial[In, Out]) SubTest(tst testing.TB) {
if h, ok := tst.(tHelper); ok {
Expand Down Expand Up @@ -220,6 +237,9 @@ func (t *Trial[In, Out]) testCase(msg string, test Case[In, Out]) result {
result.pass("PASS: %q", msg)
}
}
if !result.Success && t.knownIssueReason != "" {
result.Message += colorYellow(fmt.Sprintf(" (known issue: %s)", t.knownIssueReason))
}
Comment on lines +240 to +242

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

1. Knownissue skipped on timeout 🐞 Bug ≡ Correctness

Trial.testCase appends the known-issue suffix only at the end of the function, but timeout and
panic-related code paths return early, so those failing cases never include the KnownIssue
annotation.
This contradicts the documented behavior (“When a case fails, output includes a suffix…”) and
reduces visibility for some of the most important failure modes (timeouts/panics).
Agent Prompt
## Issue description
`KnownIssue()` is intended to append a suffix to *any failing case*, but `testCase()` currently appends the suffix only at the end of the normal path. Early returns (timeout, panic-handling) bypass this, so those failing results lack the known-issue annotation.

## Issue Context
- Timeout failures (`ctx.Done()`) and panic-related failures (`panicCheck`) return before reaching the KnownIssue append block.
- Docs explicitly state that when a case fails, output includes the known-issue suffix.

## Fix Focus Areas
- trial.go[191-244]

### Implementation direction
- Centralize the KnownIssue decoration in a helper (e.g., `t.decorateResult(r *result)` that appends when `!r.Success && t.knownIssueReason != ""`) and call it immediately before *every* return in `testCase()`.
- Alternatively, refactor `testCase()` to a single exit path where the decoration always runs before returning.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

return *result
}

Expand Down
23 changes: 23 additions & 0 deletions trial_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,29 @@ func TestParallel(t *testing.T) {
New(fn, cases).Parallel().SubTest(t)
}

func TestKnownIssue(t *testing.T) {

fn := func(in int) (int, error) {
return in * 2, nil
}
cases := map[string]Case[int, int]{
"mismatch": {Input: 2, Expected: 999},
"match": {Input: 2, Expected: 4},
}
testRunner := New(fn, cases).KnownIssue("Issue #1234")

for testName, test := range cases {
result := testRunner.testCase(testName, test)
if result.Success && strings.Contains(result.Message, "Issue #1234") {
t.Logf("FAIL: Unexpecte Known issue %v", result.Message)
t.Fail()
} else if !result.Success && !strings.Contains(result.Message, "Issue #1234") {
t.Logf("FAIL: Expected Known Issue in %v", result.Message)
t.Fail()
}
}
}

func TestColorDiagnostics(t *testing.T) {
if colorEnabled {
t.Log("Color Enabled " + colorGreen("GREEN") + " " + colorRed("RED"))
Expand Down
Loading