Merge pull request #1 from agammann/fix/pytest-final-outcomes

Respect final pytest xfail outcomes

Author: agammann

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 0.4.0a2 — pytest outcome correction
+
+- Record final pytest outcomes after expected-failure processing; mixed suites containing xfail or xpass remain inconclusive.
+- Preserve ordinary assertion failures, runtime errors, and existing unittest behavior.
+- Align the package, CLI, and report version.
+
 ## 0.4.0a1 — public alpha
 
 - Compare selected unittest or pytest cases on a Git baseline and a committed or working-tree fix.

README.md

@@ -9,7 +9,7 @@ It runs the same selected Python tests against a Git baseline and your current
 code, including uncommitted edits and new, non-ignored files. It records individual
 test outcomes, source hashes, snapshots, and raw logs.
 
-Version 0.4.0a1 is a public alpha for Python projects with a known pre-fix Git
+Version 0.4.0a2 is a public alpha for Python projects with a known pre-fix Git
 commit. It has no runtime package dependencies, account, or model API. It has
 been tested locally on Windows with Python 3.12; see [VERIFICATION.md](VERIFICATION.md)
 for the exact results and current limits.
@@ -19,12 +19,12 @@ for the exact results and current limits.
 Requires Python 3.11+ and Git. Install the alpha from its tagged source:
 
 ```sh
-python -m pip install "git+https://github.com/agammann/bugbix.git@v0.4.0-alpha.1"
+python -m pip install "git+https://github.com/agammann/bugbix.git@v0.4.0-alpha.2"
 bugbix --version
 ```
 
-Alternatively, download the wheel from the [alpha release](https://github.com/agammann/bugbix/releases/tag/v0.4.0-alpha.1)
-and install that file with `python -m pip install /path/to/bugbix-0.4.0a1-py3-none-any.whl`.
+Alternatively, download the wheel from the [alpha release](https://github.com/agammann/bugbix/releases/tag/v0.4.0-alpha.2)
+and install that file with `python -m pip install /path/to/bugbix-0.4.0a2-py3-none-any.whl`.
 Bugbix is not published to PyPI.
 
 ## Quick start
@@ -91,9 +91,10 @@ Exit `0` means `regression_observed`; exit `1` means another completed verdict;
 exit `2` means configuration, snapshot, or orchestration failure. CLI output is JSON.
 For `inconclusive`, the CLI and report name the blocking run, test, and exception
 type when available. Runner failures include a short cause; the full trace remains
-in the evidence folder. An assertion failure in one test does not override an error
-or skip in another selected test. Select the relevant test and rerun, or repair
-the test error. The report lists Git-visible changed files and links both stdout
+in the evidence folder. An assertion failure in one test does not override an error,
+skip, expected failure (`xfail`), or unexpected pass (`xpass`) in another selected
+test. Remove an obsolete expected-failure marker when checking a fixed test. Select the relevant test and rerun, or
+repair the test error. The report lists Git-visible changed files and links both stdout
 and stderr. A failure-to-pass transition cannot be reported as a regression when
 the only Git-visible changes are test files. Test-file recognition uses common
 Python names and `test/` or `tests/` folders; review the changed-file list.

VERIFICATION.md

@@ -5,7 +5,29 @@ known pre-fix Git commit from a fix. This record separates local observations
 from broader product claims. The project does not yet have independent user
 adoption evidence.
 
-## Local product checks
+## 0.4.0a2 checks — 2026-10-02
+
+Windows 11, CPython 3.12.14, pytest 9.1.1:
+
+- All 22 product tests passed without skips, including mixed ordinary/expected
+  failures and strict/non-strict unexpected passes.
+- Before the fix, a mixed pytest suite incorrectly returned `regression_observed`
+  and recorded an xfail-to-xpass case as an ordinary failure-to-pass transition.
+  The corrected runner reads pytest's final report and returns `inconclusive`.
+- A wheel was installed in a separate virtual environment and run outside the
+  source checkout. Its six Python module files matched the source. The package,
+  CLI, and report version agreed on `0.4.0a2`.
+- The installed CLI passed five disposable example checks: a real assertion
+  transition, a weak test, an import error, a skipped test, and a timeout.
+- The installed CLI checked Bugbix's new mixed-xfail regression test against
+  baseline `3176427c3744009772c5be2bc1a276b8464c3fa7`: two assertion failures
+  before the fix, two passes after it, yielding `regression_observed`.
+
+These are local controlled checks. The earlier third-party repository trials
+below were not rerun for this correction. Arbitrary pytest plugins and detached
+background processes remain outside the verified scope.
+
+## Earlier alpha checks — 2026-09-27
 
 Test host: Windows 11 build 26200, CPython 3.12.14, 2026-09-27. The source
 suite exercises positive and negative verdicts, pytest and unittest, skipped
@@ -15,7 +37,7 @@ in a separate virtual environment and invoked outside the source tree.
 
 | Check | Observed result |
 | --- | --- |
-| [Source suite on the alpha source](evidence/alpha-source-tests.txt) | 20 tests passed |
+| [Source suite on the first alpha source](evidence/alpha-source-tests.txt) | 20 tests passed |
 | Installed wheel smoke check | `bugbix --version` returned `Bugbix 0.4.0a1` |
 | Package consistency | Wheel Python source bytes matched the files in this repository |
 
@@ -55,7 +77,7 @@ python -m pip wheel --no-deps --wheel-dir dist .
 ```
 
 The GitHub Actions workflow runs the suite on Windows and Ubuntu with Python
-3.12. A passing workflow is separate evidence from the local results above;
+3.11 and 3.12. A passing workflow is separate evidence from the local results above;
 check the latest run before assuming either platform passed remotely.
 
 ## Boundaries

bugbix/__init__.py

@@ -1,3 +1,3 @@
 """Bugbix: regression evidence for the change in front of you."""
 
-__version__ = "0.1.0"
+__version__ = "0.4.0a2"

bugbix/cli.py

@@ -3,12 +3,13 @@
 import subprocess
 import sys
 
+from . import __version__
 from .core import check
 
 
 def main(argv=None):
     parser = argparse.ArgumentParser(description="Does your regression test catch the old bug?")
-    parser.add_argument("--version", action="version", version="Bugbix 0.4.0a1")
+    parser.add_argument("--version", action="version", version=f"Bugbix {__version__}")
     commands = parser.add_subparsers(dest="command", required=True)
     run = commands.add_parser("check", help="Compare selected tests on old and current code")
     run.add_argument("--repo", default=".")

bugbix/core.py

@@ -13,6 +13,8 @@
 import sys
 import time
 
+from . import __version__
+
 MAX_FILE = 10 * 1024 * 1024
 MAX_TOTAL = 100 * 1024 * 1024
 
@@ -310,7 +312,7 @@ def check(repo, base, head, tests, output, python=sys.executable, timeout=30.0,
         blockers = [{"run": "all", "issue": "no_non_test_changes"}]
     else:
         blockers = blockers_for(runs) if verdict == "inconclusive" else []
-    report = {"schema_version": 2, "product": "Bugbix", "version": "0.4.0a1",
+    report = {"schema_version": 2, "product": "Bugbix", "version": __version__,
               "verdict": verdict, "reason": reason, "regression_tests": transitions,
               "blockers": blockers, "changed_files": changed_files,
               "repo": str(repo), "base": base_oid, "head": head_oid, "tests": tests,

bugbix/pytest_runner.py

@@ -28,12 +28,9 @@ def pytest_collectreport(self, report):
         if report.failed:
             self.collection_errors.append(report.nodeid)
 
-    def pytest_runtest_makereport(self, item, call):
-        # This hook sees the original exception, allowing an assertion failure
-        # to be distinguished from a setup/runtime error.
-        import pytest
-
-        report = pytest.TestReport.from_item_and_call(item, call)
+    def record_report(self, report, call):
+        # Inspect the final report after pytest has applied xfail and other hooks,
+        # while retaining the original exception for assertion/error distinction.
         if report.failed:
             if hasattr(report, "wasxfail"):
                 status = "unexpected_success"
@@ -46,8 +43,8 @@ def pytest_runtest_makereport(self, item, call):
         elif report.skipped:
             self.record(report.nodeid, "expected_failure" if hasattr(report, "wasxfail") else "skip")
         elif call.when == "call":
-            self.record(report.nodeid, "pass")
-        return report
+            self.record(report.nodeid,
+                        "unexpected_success" if hasattr(report, "wasxfail") else "pass")
 
 
 def main():
@@ -58,7 +55,13 @@ def main():
     try:
         import pytest
 
-        recorder = Recorder()
+        class Plugin(Recorder):
+            @pytest.hookimpl(hookwrapper=True, tryfirst=True)
+            def pytest_runtest_makereport(self, item, call):
+                outcome = yield
+                self.record_report(outcome.get_result(), call)
+
+        recorder = Plugin()
         exit_code = int(pytest.main(["-q", "-p", "no:cacheprovider", *paths], plugins=[recorder]))
         payload = {"events": [{"id": nodeid, "status": status,
                                **({"error_type": recorder.error_types[nodeid]}

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "bugbix"
-version = "0.4.0a1"
+version = "0.4.0a2"
 description = "Check whether a regression test catches the bug before your fix."
 requires-python = ">=3.11"
 readme = "README.md"

tests/test_product.py

@@ -177,6 +177,35 @@ def test_pytest_function_catches_dirty_tree_fix(self):
                         self.root / "focused", repeat=1, runner="pytest")
         self.assertEqual(focused["verdict"], "regression_observed")
 
+    def test_pytest_xfail_outcomes_block_mixed_regression(self):
+        try:
+            import pytest  # noqa: F401
+        except ImportError:
+            self.skipTest("pytest is not installed in this test environment")
+        self.change_test("import pytest\nfrom pricing import total\n"
+                         "def test_discount():\n    assert total(100, 20, 10) == 88\n"
+                         "@pytest.mark.xfail(reason='pending', strict=False)\n"
+                         "def test_marked():\n    assert total(100, 20, 10) == 88\n")
+        report = self.run_check(runner="pytest")
+        self.assertEqual(report["verdict"], "inconclusive")
+        outcomes = {(item["run"], item.get("status")) for item in report["blockers"]}
+        self.assertIn(("base-1", "expected_failure"), outcomes)
+        self.assertIn(("head-1", "unexpected_success"), outcomes)
+
+    def test_pytest_xpass_does_not_count_as_an_ordinary_pass(self):
+        try:
+            import pytest  # noqa: F401
+        except ImportError:
+            self.skipTest("pytest is not installed in this test environment")
+        for strict in (False, True):
+            with self.subTest(strict=strict):
+                self.change_test("import pytest\n"
+                                 f"@pytest.mark.xfail(reason='pending', strict={strict})\n"
+                                 "def test_marked():\n    assert True\n")
+                report = check(self.repo, "HEAD", "working", ["test_pricing.py"],
+                               self.root / f"xpass-{strict}", repeat=1, runner="pytest")
+                self.assertEqual(report["verdict"], "inconclusive")
+
     def test_pytest_runtime_error_is_inconclusive(self):
         try:
             import pytest  # noqa: F401