Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .config/dotnet-tools.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"version": 1,
"isRoot": true,
"tools": {
"dotnet-stryker": {
"version": "4.14.0",
"commands": [
"dotnet-stryker"
],
"rollForward": false
}
}
}
40 changes: 40 additions & 0 deletions .github/workflows/mutation-testing.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: Mutation Testing

on:
schedule:
- cron: '0 3 * * 1' # Every Monday at 03:00 UTC
workflow_dispatch:

permissions:
contents: read

jobs:
mutation-testing:
runs-on: ubuntu-latest
timeout-minutes: 60

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Setup .NET
uses: actions/setup-dotnet@v3
with:
dotnet-version: '8.x'

- name: Restore tools
run: dotnet tool restore

- name: Install dependencies
run: dotnet restore

- name: Run Stryker mutation testing
run: dotnet stryker

- name: Upload mutation report
if: always()
uses: actions/upload-artifact@v4
with:
name: mutation-report
path: '**/StrykerOutput/**/reports/**'
retention-days: 14
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -401,6 +401,9 @@ FodyWeavers.xsd
**/.teapie/cache/
**/.teapie/reports/

# Stryker mutation testing
**/StrykerOutput/

# User-defined
*launchSettings*.json
docs/_site
Expand Down
70 changes: 70 additions & 0 deletions docs/mutation-testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Mutation Testing

TeaPie uses [Stryker.NET](https://stryker-mutator.io/docs/stryker-net/introduction/) for mutation testing to verify the effectiveness of the test suite.

## Overview

Mutation testing systematically introduces small changes (*mutants*) into the source code and checks whether the existing tests detect them. A killed mutant means the tests caught the change; a surviving mutant may indicate a gap in test assertions.

## Quick Start

```bash
# Restore the local tool
dotnet tool restore

# Run mutation testing
dotnet stryker
```

The HTML report is generated in the `StrykerOutput/` directory.

## Configuration

Stryker is configured via [`stryker-config.json`](../stryker-config.json) in the repository root. Key settings:

| Setting | Value | Description |
|---------|-------|-------------|
| `project` | `TeaPie.csproj` | Source project under test |
| `test-projects` | `TeaPie.Tests.csproj` | Test project |
| `reporters` | html, progress, cleartext | Output formats |
| `thresholds.high` | 80 | Score above this is green |
| `thresholds.low` | 50 | Score below this is red |
| `thresholds.break` | 37 | Score below this fails the build |

## CI Integration

Mutation testing runs as a **scheduled weekly workflow** (`.github/workflows/mutation-testing.yml`) every Monday at 03:00 UTC. It can also be triggered manually via `workflow_dispatch`. The HTML report is uploaded as a build artifact.

## Baseline Score

The initial baseline mutation score is **37.83%**, established with Stryker.NET 4.14.0.

| Metric | Count |
|--------|-------|
| Killed | 787 |
| Survived | 707 |
| Timeout | 6 |
| No Coverage | 596 |
| Compile Error | 145 |
| Ignored | 677 |

A score of ~38% is **low**. In industry practice, mutation scores typically fall into these ranges:

| Range | Assessment |
|-------|------------|
| **80%+** | Good — strong test suite that catches most regressions |
| **60–79%** | Acceptable — reasonable safety net, room to improve |
| **40–59%** | Below average — significant gaps in test assertions |
| **Below 40%** | Low — most mutations survive undetected |

The main contributors to the low score are the high **No Coverage** count (596 mutants in untested code) and the large number of **Survived** mutants (707) in areas like console rendering, pipeline orchestration, and file I/O that lack assertions. Improving coverage of core logic and adding targeted assertions will have the biggest impact.

## Thresholds

The thresholds are set to prevent regression and drive improvement:

- **`break` (37%)** — At the baseline. The score is already low, so any regression fails the CI build immediately. This ensures the mutation score can only go up.
- **`low` (50%)** — Mid-term improvement target. Reaching this means the test suite catches at least half of all mutations, a meaningful milestone.
- **`high` (80%)** — Long-term goal aligned with industry standards for a good mutation score. Scores above this are reported as green.

These thresholds should be raised as the test suite improves and the mutation score increases.
13 changes: 13 additions & 0 deletions stryker-config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"stryker-config": {
"project": "TeaPie.csproj",
"test-projects": ["./tests/TeaPie.Tests/TeaPie.Tests.csproj"],
"solution": "./TeaPie.sln",
"reporters": ["html", "progress", "cleartext"],
"thresholds": {
"high": 80,
"low": 50,
"break": 37
}
}
}
67 changes: 67 additions & 0 deletions tests/TeaPie.Tests/ApplicationContextOptionsBuilderShould.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
using FluentAssertions;

namespace TeaPie.Tests;

public class ApplicationContextOptionsBuilderShould
{
[Fact]
public void Build_ReturnsDefaultOptions_WhenNothingSet()
{
var options = new ApplicationContextOptionsBuilder().Build();

options.TempFolderPath.Should().Be(Constants.SystemTemporaryFolderPath);
options.Environment.Should().Be(string.Empty);
options.EnvironmentFilePath.Should().Be(string.Empty);
options.ReportFilePath.Should().Be(string.Empty);
options.InitializationScriptPath.Should().Be(string.Empty);
options.CacheVariables.Should().BeTrue();
}

[Fact]
public void SetEnvironment_SetsEnvironment() =>
new ApplicationContextOptionsBuilder()
.SetEnvironment("dev")
.Build()
.Environment.Should().Be("dev");

[Fact]
public void SetReportFilePath_SetsReportFilePath() =>
new ApplicationContextOptionsBuilder()
.SetReportFilePath("report.xml")
.Build()
.ReportFilePath.Should().Be("report.xml");

[Fact]
public void SetVariablesCaching_SetsCachingFlag() =>
new ApplicationContextOptionsBuilder()
.SetVariablesCaching(false)
.Build()
.CacheVariables.Should().BeFalse();

[Fact]
public void EachSetter_ReturnsBuilder_ForFluentChaining()
{
var builder = new ApplicationContextOptionsBuilder();
builder.SetEnvironment("dev").Should().BeSameAs(builder);
}

[Fact]
public void SetTempFolderPath_WithNull_UsesEmptyString() =>
new ApplicationContextOptionsBuilder()
.SetTempFolderPath(null)
.Build()
.TempFolderPath.Should().Be(string.Empty);

[Fact]
public void SetEnvironment_WithNull_UsesEmptyString() =>
new ApplicationContextOptionsBuilder()
.SetEnvironment(null)
.Build()
.Environment.Should().Be(string.Empty);

[Fact]
public void Build_WithNoTempPath_UsesConstantsSystemTemporaryFolderPath() =>
new ApplicationContextOptionsBuilder()
.Build()
.TempFolderPath.Should().Be(Constants.SystemTemporaryFolderPath);
}
57 changes: 57 additions & 0 deletions tests/TeaPie.Tests/ApplicationContextOptionsShould.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
using FluentAssertions;

namespace TeaPie.Tests;

public class ApplicationContextOptionsShould
{
[Fact]
public void DefaultConstructor_SetsEmptyStringsAndCacheVariablesTrue()
{
var options = new ApplicationContextOptions();

options.TempFolderPath.Should().Be(string.Empty);
options.Environment.Should().Be(string.Empty);
options.EnvironmentFilePath.Should().Be(string.Empty);
options.ReportFilePath.Should().Be(string.Empty);
options.InitializationScriptPath.Should().Be(string.Empty);
options.CacheVariables.Should().BeTrue();
}

[Fact]
public void Constructor_WithValues_SetsPropertiesCorrectly()
{
var options = new ApplicationContextOptions(
"/temp", "prod", "env.json", "report.xml", "init.csx", false);

options.TempFolderPath.Should().Be("/temp");
options.Environment.Should().Be("prod");
options.EnvironmentFilePath.Should().Be("env.json");
options.ReportFilePath.Should().Be("report.xml");
options.InitializationScriptPath.Should().Be("init.csx");
options.CacheVariables.Should().BeFalse();
}

[Fact]
public void NullParameters_DefaultToEmptyStrings()
{
var options = new ApplicationContextOptions(null, null, null, null, null);

options.TempFolderPath.Should().Be(string.Empty);
options.Environment.Should().Be(string.Empty);
options.EnvironmentFilePath.Should().Be(string.Empty);
options.ReportFilePath.Should().Be(string.Empty);
options.InitializationScriptPath.Should().Be(string.Empty);
}

[Fact]
public void CacheVariables_DefaultsToTrue() =>
new ApplicationContextOptions().CacheVariables.Should().BeTrue();

[Fact]
public void Properties_AreSettable()
{
var options = new ApplicationContextOptions();
options.TempFolderPath = "new";
options.TempFolderPath.Should().Be("new");
}
}
70 changes: 70 additions & 0 deletions tests/TeaPie.Tests/ConstantsShould.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
using FluentAssertions;

namespace TeaPie.Tests;

public class ConstantsShould
{
[Fact]
public void HaveCorrectRequestFileExtension() =>
Constants.RequestFileExtension.Should().Be(".http");

[Fact]
public void HaveCorrectScriptFileExtension() =>
Constants.ScriptFileExtension.Should().Be(".csx");

[Fact]
public void HaveCorrectPreRequestSuffix() =>
Constants.PreRequestSuffix.Should().Be("-init");

[Fact]
public void HaveCorrectRequestSuffix() =>
Constants.RequestSuffix.Should().Be("-req");

[Fact]
public void HaveCorrectPostResponseSuffix() =>
Constants.PostResponseSuffix.Should().Be("-test");

[Fact]
public void HaveCorrectDefaultEnvironmentFileName() =>
Constants.DefaultEnvironmentFileName.Should().Be("env");

[Fact]
public void HaveCorrectEnvironmentFileExtension() =>
Constants.EnvironmentFileExtension.Should().Be(".json");

[Fact]
public void HaveCorrectDefaultEnvironmentName() =>
Constants.DefaultEnvironmentName.Should().Be("$shared");

[Fact]
public void HaveCorrectPascalCasePattern() =>
Constants.PascalCasePattern.Should().Be("([A-Z][a-z]*|[a-z]+)");

[Fact]
public void HaveCorrectApplicationName() =>
Constants.ApplicationName.Should().Be("TeaPie");

[Fact]
public void HaveCorrectDefaultInitializationScriptName() =>
Constants.DefaultInitializationScriptName.Should().Be("init");

[Fact]
public void HaveCorrectTeaPieFolderName() =>
Constants.TeaPieFolderName.Should().Be(".teapie");

[Fact]
public void HaveCorrectUnixEndOfLine() =>
Constants.UnixEndOfLine.Should().Be("\n");

[Fact]
public void HaveCorrectWindowsEndOfLine() =>
Constants.WindowsEndOfLine.Should().Be("\r\n");

[Fact]
public void HaveCorrectSecretVariableTag() =>
Constants.SecretVariableTag.Should().Be("secret");

[Fact]
public void HaveCorrectNoCacheVariableTag() =>
Constants.NoCacheVariableTag.Should().Be("no-cache");
}
42 changes: 42 additions & 0 deletions tests/TeaPie.Tests/Environments/EnvironmentShould.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
using FluentAssertions;
using TeaPie.Variables;
using Environment = TeaPie.Environments.Environment;

namespace TeaPie.Tests.Environments;

public class EnvironmentShould
{
[Fact]
public void SetNameFromConstructor()
{
var env = new Environment("dev", new Dictionary<string, object?>());

env.Name.Should().Be("dev");
}

[Fact]
public void SetVariablesFromConstructor()
{
var vars = new Dictionary<string, object?> { { "key", "value" } };
var env = new Environment("dev", vars);

env.Variables.Should().ContainKey("key");
}

[Fact]
public void ApplySetsAllVariablesOnTargetCollection()
{
var vars = new Dictionary<string, object?>
{
{ "baseUrl", "http://localhost" },
{ "port", 8080 }
};
var env = new Environment("dev", vars);
var collection = new VariablesCollection();

env.Apply(collection);

collection.Get<string>("baseUrl").Should().Be("http://localhost");
collection.Get<int>("port").Should().Be(8080);
}
}
Loading
Loading