CI Failure Doctor🏥 CI Failure Investigation — Run #27414475006 · Flaky Embedding Tests (Network-Dependent)

[github-actions[bot]]

Summary

A transient Test (ubuntu-latest) failure was detected in CI run #27414475006 (commit 11f96c646288, PR #232). The same tests passed on macOS and Windows, and the next CI run passed on all platforms. Root cause: embedding integration tests download the BGE-M3 model from HuggingFace Hub at test runtime with no CI-level cache for model files, making them vulnerable to transient network failures.

---

Failure Details

Field	Value
Failed Run	27414475006
Commit	11f96c646288 — chore(ci): upgrade gh-aw workflows to v0.79.6 (#232)
Trigger	push to main
Failed Job	Test (ubuntu-latest)
Failed Step	Run tests
Exit Code	101 (test assertion failures)
Test Duration	20 seconds (failed) vs 61 seconds (macOS, passed)
Resolved By	Next commit 3e99ec8f CI run passed all platforms

---

Root Cause Analysis

What Failed

Commit 11f96c646288 changed only .github/ workflow files — no Rust source or Cargo.lock changes. Despite this, Test (ubuntu-latest) failed with exit code 101 (test runtime failures).

Why It Failed

The integration test suite (tests/integration_test.rs) includes several embedding tests that are not marked #[ignore] and therefore run unconditionally as part of cargo test --all-features:

// tests/integration_test.rs — search_tests module
#[test]
fn test_embedding_single_text() {
    let embedder = create_embedder().expect("embedder creation failed");
    let result = embedder.embed("Hello, world!");  // ← triggers BGE-M3 model download
    ...
}

#[test]
fn test_embedding_batch() {
    let embedder = create_embedder().expect("embedder creation failed");
    let result = embedder.embed_batch(&texts);  // ← triggers BGE-M3 model download
    ...
}

Calling embedder.embed() triggers lazy model initialization in src/embedding/fastembed_impl.rs:

fn get_model() -> Result<...> {
    let options = fastembed::InitOptions::new(fastembed::EmbeddingModel::BGEM3)
        .with_show_download_progress(false);
    let model = fastembed::TextEmbedding::try_new(options)  // ← downloads model if not cached
        .map_err(|e| StorageError::Embedding(...))?;
    ...
}

The BGE-M3 model (~1 GB) is downloaded from HuggingFace Hub at test runtime. The CI cache configuration does not include the model directory:

# .github/workflows/ci.yml — missing model cache
- uses: actions/cache@...
  with:
    path: |
      ~/.cargo/registry
      ~/.cargo/git
      target
    # ↑ No ~/.cache/fastembed or ~/.cache/huggingface

On every CI run, the model must be freshly downloaded. If HuggingFace Hub is transiently unreachable from the ubuntu runner (rate limiting, network hiccup, DNS failure), the test fails.

Evidence

Platform	Duration	Result
ubuntu-latest	20 seconds	❌ FAILURE
macos-latest	61 seconds	✅ SUCCESS
windows-latest	(different test command, --features fastembed-embeddings only)	✅ SUCCESS

The ubuntu tests completed in only 20 seconds — far too fast for a successful BGE-M3 model download. macOS took 61 seconds (consistent with either a cached model or a successful download). The disparity strongly indicates a network failure on the ubuntu runner.

---

Failed Jobs and Errors

Job: Test (ubuntu-latest)  — FAILURE
  Steps:
    ✅ Set up job
    ✅ Checkout repository
    ✅ Install Rust toolchain
    ✅ Cache cargo registry  (cache HIT)
    ❌ Run tests             (20s, exit code 101)

Job: All Checks Pass — FAILURE
  (cascade from ubuntu test failure)

Annotation: Process completed with exit code 101 at step 613 of the CI run.

---

Investigation Findings

1. Not a code regression — the failing commit changed only .github/ workflow files. The Rust source and Cargo.lock were unchanged.
2. Platform-specific failure — only ubuntu was affected. This rules out code logic bugs.
3. Transient failure — the next CI run (different commit, same test suite) passed on all platforms including ubuntu.
4. Fast failure timing — 20 seconds is consistent with a quick network timeout, not a slow download or test logic failure.
5. No model file cache — the CI actions/cache configuration does not include the fastembed/HuggingFace model cache directory.
6. Unit tests are correctly guarded — src/embedding/fastembed_impl.rs marks download-requiring tests #[ignore], but the integration tests in tests/integration_test.rs do not use #[ignore].

---

Recommended Actions

• Add model cache to CI — include the fastembed model directory in actions/cache:

  path: |
    ~/.cargo/registry
    ~/.cargo/git
    target
    ~/.cache/fastembed     # add this
  key: ${{ runner.os }}-cargo-test-models-${{ hashFiles('**/Cargo.lock') }}

• Mark download-dependent integration tests — add #[ignore = "requires fastembed model download"] to test_embedding_single_text, test_embedding_batch, and similar tests in tests/integration_test.rs; add a separate CI job that runs them with -- --include-ignored on a schedule or with retry logic.
• Add retry logic — wrap the fastembed try_new() call in get_model() with a retry loop (2–3 attempts with exponential back-off) to gracefully handle transient network errors.

---

Prevention Strategies

1. Cache model files in CI — the single most impactful change. Models rarely change; cache key on Cargo.lock hash is sufficient.
2. Separate network-dependent tests — run them in a dedicated CI job with continue-on-error: true or retry middleware, so they don't block the main test matrix.
3. Fail-fast vs. flaky detection — consider adding a --retry-flaky-tests flag or wrapping cargo test with a retry script for the embedding job.

---

AI Team Self-Improvement

Paste into instructions.md / agent prompt for future AI coding agents:

# Flaky Test Prevention — Embedding/Network Tests

When adding tests that use `create_embedder()`, `embed()`, or any function that
downloads ML models from HuggingFace Hub or similar external services at runtime:

1. ALWAYS mark these tests with `#[ignore = "requires model download"]` in integration tests.
2. DO NOT add network-dependent tests to the default (non-ignored) test suite.
3. When modifying `.github/workflows/ci.yml` caches, ensure `~/.cache/fastembed`
   (or relevant model cache dir) is included in `actions/cache` paths.
4. If a CI test fails in <30 seconds on one platform but succeeds on others for a
   commit with no Rust source changes, assume transient network failure first.

---

Historical Context

• No prior issues found for this specific failure pattern (fastembed model download flakiness).
• Previous CI runs on the same codebase succeeded; this is the first observed instance of this failure mode.
• The failure self-resolved on the next commit's CI run, consistent with a transient infrastructure issue.

Generated by CI Failure Doctor · 488.9 AIC · ⌖ 25 AIC · ⊞ 32K · ◷

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/ci-doctor.md@adea00ee5830135f24022f1000d706c7f5fa24c4

• expires on Jun 19, 2026, 12:32 PM UTC