Skip to content

Retry each code sample once on failure to absorb transient backend errors#5182

Merged
btshrewsbury-viam merged 1 commit into
mainfrom
ci/retry-transient-sample-failures
Jul 14, 2026
Merged

Retry each code sample once on failure to absorb transient backend errors#5182
btshrewsbury-viam merged 1 commit into
mainfrom
ci/retry-transient-sample-failures

Conversation

@btshrewsbury-viam

@btshrewsbury-viam btshrewsbury-viam commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Problem

Every Test Code Samples failure since the sample-bug fixes in #5120 (7/1) has been a transient backend error, not a code bug — triage across the failing runs in #5119:

  • gRPC INTERNAL on DeleteDataPipeline (pipeline-delete.py/.go, pipeline-list.py teardown, pipeline-create.go cleanup) — hit 6 of 8 runs on 7/13
  • DEADLINE_EXCEEDED on the shared data-query SQL query (once taking out all three language variants in the same run)
  • brief test-machine connection drops

These clear within seconds: in run 29241295626, the pipeline orphaned by a failed delete was successfully deleted by the next sample's setup cleanup 3 seconds later.

Fix

Run each failed sample a second time after a 30-second delay before counting it as failed, in all three language steps. Only persistent failures fail the job. Samples are already re-runnable by design (the suite runs weekly and on every push, and each sample cleans up prior state on entry), so a retry is equivalent to the next scheduled run.

Also drops the redundant inline VAR="$VAR" prefixes on the run commands: each step's env: block already exports a superset of those variables to every process in the step.

Validation (8 dispatched runs on 7/13)

  • The retry works: across three dispatches of this branch it absorbed three would-be job failures (pipeline-delete.py, pipeline-list.py, pipeline-create.go — each failed once, passed 30s later), while two plain-main control dispatches failed outright on the same INTERNAL error.
  • The TS connect failures seen during validation are unrelated to this change: they initially looked branch-correlated, but bisection (TS step byte-identical to main still failed) and finally a dispatch of unmodified main (run 29266526730) reproduced them. They are a flapping backend incident in the TS cloud-WebRTC dial path, tracked on CI failure: Test Code Samples #5119; Go/Python samples are immune because they reach the runner-local server via mDNS.

What this does not fix

Failures that persist longer than the 30s retry window — the flapping TS WebRTC incident, extended test-machine outages, and the late-afternoon persistent DeleteDataPipeline errors — still fail the job. Those are backend/infra issues tracked on #5119.

Tracking issue: #5119

🤖 Generated with Claude Code

Every Test Code Samples failure since the sample-bug fixes in #5120 has
been a transient error that cleared within seconds: gRPC INTERNAL on
DeleteDataPipeline, DEADLINE_EXCEEDED on data-query, or brief test
machine connection drops. Run each sample a second time after a 30s
delay before counting it as failed, so only persistent failures fail
the job. Sample credentials already flow through each step's env block,
so the redundant inline variable prefixes are dropped.

Tracking issue: #5119
@netlify

netlify Bot commented Jul 13, 2026

Copy link
Copy Markdown

Deploy Preview for viam-docs ready!

Name Link
🔨 Latest commit 08b190c
🔍 Latest deploy log https://app.netlify.com/projects/viam-docs/deploys/6a54feb9263a4c000817211b
😎 Deploy Preview https://deploy-preview-5182--viam-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
Lighthouse
Lighthouse
1 paths audited
Performance: 47 (🟢 up 10 from production)
Accessibility: 100 (no change from production)
Best Practices: 100 (no change from production)
SEO: 92 (no change from production)
PWA: 60 (no change from production)
View the detailed breakdown and full score reports

To edit notification comments on pull requests, go to your Netlify project configuration.

@viambot viambot added the safe to build This pull request is marked safe to build from a trusted zone label Jul 13, 2026
@btshrewsbury-viam

Copy link
Copy Markdown
Collaborator Author

Validation summary (8 dispatched runs on 7/13, details on #5119):

The retry works and is needed. Across three dispatches of this branch, the retry absorbed three would-be job failures — pipeline-delete.py, pipeline-list.py, and pipeline-create.go, each failing once with the backend's transient gRPC INTERNAL on DeleteDataPipeline and passing 30s later. Two control dispatches of main without the retry failed outright on that same error.

The TypeScript connect failures seen during validation are unrelated to this change. All three branch dispatches failed the 7 machine-dialing TS samples with Failed to connect to robot, which initially looked branch-correlated. Bisection ruled that out: a branch with the TS step byte-identical to main still failed them, and finally a plain dispatch of unmodified main (run 29266526730) failed all 7 the same way. It is a flapping backend/infra incident in the TS cloud-WebRTC dial path (signaling completes and remote ICE candidates arrive, then the exchange terminates within ~50 ms), tracked on #5119. Go and Python samples are immune because they discover the runner-local server via mDNS and bypass the cloud dial path.


Generated by Claude Code

@btshrewsbury-viam btshrewsbury-viam changed the title Retry code samples once on failure to absorb transient backend errors Retry each code sample once on failure to absorb transient backend errors Jul 13, 2026
@btshrewsbury-viam btshrewsbury-viam merged commit 0681581 into main Jul 14, 2026
14 of 17 checks passed
@btshrewsbury-viam btshrewsbury-viam deleted the ci/retry-transient-sample-failures branch July 14, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to build This pull request is marked safe to build from a trusted zone

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants