Skip to content

Fix various DDM stuck remove/pending issues#43382

Open
MagnusHJensen wants to merge 10 commits intomainfrom
40332-fix-ddm-pending-issues
Open

Fix various DDM stuck remove/pending issues#43382
MagnusHJensen wants to merge 10 commits intomainfrom
40332-fix-ddm-pending-issues

Conversation

@MagnusHJensen
Copy link
Copy Markdown
Member

@MagnusHJensen MagnusHJensen commented Apr 9, 2026

Related issue: Resolves #40322 (Second part)

Checklist for submitter

If some of the following don't apply, delete the relevant line.

  • Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.
    See Changes files for more information.

  • Input data is properly validated, SELECT * is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.

  • Timeouts are implemented and retries are limited to avoid infinite loops

  • If paths of existing endpoints are modified without backwards compatibility, checked the frontend/CLI for any necessary changes

Testing

  • Added/updated automated tests
  • QA'd all new/changed functionality manually

Summary by CodeRabbit

  • Bug Fixes

    • Cleans up orphaned pending Apple MDM profile removals so pending remove rows no longer get stuck during batch processing.
    • Fixes DDM status handling so remove operations are skipped/handled based on the incoming update, preventing incorrect flips.
    • Reconciler now self-heals stuck pending removal profiles across batches.
  • Tests

    • Added unit tests covering cross-batch cleanup, orphaned pending removes, and DDM status/reporting edge cases.
  • Documentation

    • Updated changelog entry for DDM pending/remove fixes.

@MagnusHJensen MagnusHJensen requested a review from a team as a code owner April 9, 2026 22:37
Copilot AI review requested due to automatic review settings April 9, 2026 22:37
Copy link
Copy Markdown

@claude claude bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@MagnusHJensen
Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 9, 2026

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes several Apple DDM “stuck remove/pending” scenarios by preventing incorrect status updates, making batch processing deterministic for cleanup, and adding a safety-net cleanup for already-stuck rows.

Changes:

  • Add tests covering token-collision status reports, cross-batch duplicate remove/install cleanup, and orphaned pending removes.
  • Ensure MDMAppleBatchSetHostDeclarationState always runs an orphaned remove/pending cleanup.
  • Reorder “changed declarations” query to process removals first; adjust status report logic to skip updating existing remove rows.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 4 comments.

File Description
server/datastore/mysql/apple_mdm_ddm_test.go Adds regression tests for stuck remove/pending rows and batch-order edge cases.
server/datastore/mysql/apple_mdm.go Adds an orphaned-remove cleanup safety net; processes removals first; fixes status report update condition.
changes/40322-fix-ddm-pending-issues Adds user-facing release notes for the DDM stuck remove/pending fixes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 9, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ce96247b-afa5-4751-89ad-c163cc2f3f7e

📥 Commits

Reviewing files that changed from the base of the PR and between 57382d3 and 3aa45c9.

📒 Files selected for processing (1)
  • server/datastore/mysql/apple_mdm_ddm_test.go
✅ Files skipped from review due to trivial changes (1)
  • server/datastore/mysql/apple_mdm_ddm_test.go

Walkthrough

Adds cleanUpOrphanedPendingRemoves to delete host-level remove/pending declaration rows when a matching install declaration exists with status in (verified,verifying). MDMAppleBatchSetHostDeclarationState now invokes this cleanup after successful DB batch updates regardless of whether declarations changed. The declarations query ordering was changed to return remove rows before install rows. MDMAppleStoreDDMStatusReport skip/update logic now uses the incoming update's OperationType. Three tests were added covering skip/remove behavior, cross-batch duplicate cleanup, and orphaned pending remove cleanup; change notes were updated.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The PR title clearly and concisely summarizes the main focus: fixing DDM stuck remove/pending issues, which directly relates to all code changes addressing stale declaration states.
Description check ✅ Passed The PR description mostly follows the template with required sections completed: related issue noted, checklist items marked for validation/SQL security, testing coverage, and manual QA. However, the 'Where appropriate, automated tests simulate multiple hosts' checkbox is not explicitly confirmed as done.
Linked Issues check ✅ Passed The code changes directly address the #40322 objectives: cleanup functions handle stale remove/pending entries, batch processing improvements prevent stuck operations, and fixes ensure self-healing within the expected timeframe via reconciler runs.
Out of Scope Changes check ✅ Passed All changes are directly scoped to fixing DDM stuck remove/pending issues: database cleanup logic, batch processing fixes, test coverage, and changelog updates align with #40322 requirements.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 40332-fix-ddm-pending-issues

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
server/datastore/mysql/apple_mdm_ddm_test.go (1)

393-399: Clarify misleading comment about identifier.

The comment states "same content/identifier" but the Identifier field differs between d1 ("com.example.cleanup") and d2 ("com.example.cleanup.new"). The token matches because raw_json is identical (which contains the embedded identifier "com.example.cleanup"), not because the struct's Identifier field is the same.

📝 Suggested clarification
-	// D2 has different name but same content/identifier — same token
+	// D2 has different name and different Identifier field, but same raw_json content — same token
+	// (token is derived from raw_json, not from the struct's Identifier field)
 	d2, err := ds.NewMDMAppleDeclaration(ctx, &fleet.MDMAppleDeclaration{
 		DeclarationUUID: "decl-new",
 		Name:            "New Declaration",
 		Identifier:      "com.example.cleanup.new",
 		RawJSON:         declJSON,
 	})
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@server/datastore/mysql/apple_mdm_ddm_test.go` around lines 393 - 399, The
comment above the NewMDMAppleDeclaration call for variable d2 is misleading:
update it to say the token matches because RawJSON (declJSON) is identical and
contains the embedded identifier, not because the struct's Identifier field
equals d1.Identifier; reference the variables d1 and d2 and mention the
Identifier and RawJSON fields so the comment clarifies that identical raw_json
(declJSON) drives token equality.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@server/datastore/mysql/apple_mdm_ddm_test.go`:
- Around line 393-399: The comment above the NewMDMAppleDeclaration call for
variable d2 is misleading: update it to say the token matches because RawJSON
(declJSON) is identical and contains the embedded identifier, not because the
struct's Identifier field equals d1.Identifier; reference the variables d1 and
d2 and mention the Identifier and RawJSON fields so the comment clarifies that
identical raw_json (declJSON) drives token equality.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: bcdd423e-9839-4767-909e-fe81a2a4680b

📥 Commits

Reviewing files that changed from the base of the PR and between 8509b18 and 09ebc4f.

📒 Files selected for processing (3)
  • changes/40332-fix-ddm-pending-issues
  • server/datastore/mysql/apple_mdm.go
  • server/datastore/mysql/apple_mdm_ddm_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 9, 2026

Codecov Report

❌ Patch coverage is 91.48936% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.89%. Comparing base (0cc037f) to head (3aa45c9).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
server/datastore/mysql/apple_mdm.go 91.48% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #43382      +/-   ##
==========================================
- Coverage   66.89%   66.89%   -0.01%     
==========================================
  Files        2588     2589       +1     
  Lines      207563   207680     +117     
  Branches     9284     9284              
==========================================
+ Hits       138840   138918      +78     
- Misses      56097    56123      +26     
- Partials    12626    12639      +13     
Flag Coverage Δ
backend 68.67% <91.48%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Changing the name of a DDM configuration profile caused a stuck remove operation

2 participants