Bootstrap repairs for Ubuntu 26.04: sudo, PHP packages, PHP-FPM, DNS and CA trust

CHANGELOG.md

@@ -9,11 +9,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **Bootstrap Never Repaired a Broken Sudoers File** - 2.1.1 generates valid rules, but `magebox bootstrap` skipped the whole step whenever `/etc/sudoers.d/magebox` already existed, so every machine upgrading from an older release kept its rejected wildcard rules and nothing improved. The step now always runs; the installer decides whether a rewrite is needed and leaves a correct file alone.
+- **Bootstrap Aborted Before It Could Fix the PHP Repository** - `InstallPrerequisites` ran `apt update` first and returned on any error. A PPA with no packages for the running release makes apt exit 100, so bootstrap gave up before reaching the code that repairs exactly that repository. Both `apt update` calls are now warnings rather than failures, and `add-apt-repository`'s own exit code is ignored for the same reason.
+- **One Missing Certificate Took Every Site Offline** - nginx refuses to start when a vhost references a certificate that is not on disk, so a single stale project stopped every other one. Bootstrap now checks the certificates every vhost references, regenerates the missing ones it manages, and names the file to remove for any it does not.
+- **"Docker Is Not Running" When Docker Was Running** - A permission error on the Docker socket was reported as a stopped daemon, sending users to restart something that was already up. Bootstrap now distinguishes the two and, for the permission case, prints the command that adds the user to the docker group.
+- **One Half-Configured Package Blocked Every Install** - A package left half-configured by an earlier failure makes `apt install` exit 100 whatever it is asked for. Bootstrap treated that as fatal while installing its base tools, so it never reached the repository configuration that would have made PHP installable, and every PHP version was then reported as unavailable. Bootstrap now finishes configuring broken packages with `dpkg --configure -a` before installing, and a failure to install base tools is a warning rather than the end of the run.
+- **Every PHP Version Skipped Because apt Could Not Read a Source File** - MageBox filters out packages `apt-cache` does not know, but apt returns the same empty answer whether a package is missing or a sources file could not be read. A repository file MageBox itself wrote root-only therefore made every `php8.1` … `php8.4` package look unavailable, so bootstrap silently installed none and reported success. The check now treats a permission warning as "cannot tell" and lets apt decide, so an install either works or fails with a message worth reading.
+- **"Local CA Already Installed" While Browsers Trusted a Different One** - The check only looked for `rootCA.pem` on disk, so on a machine restored from another install MageBox reported the certificate authority as installed while the browser still trusted the old one, and every local HTTPS site warned. Bootstrap now compares the authority's serial against the browser trust store and reinstalls it when they differ.
+- **systemd-resolved Silently Ignored MageBox's Configuration** - MageBox writes config through a temp file, which `os.CreateTemp` creates as root-readable only, and `cp` keeps that mode. systemd-resolved reads its configuration after dropping privileges, so it refused the drop-in with "Permission denied" at every restart and `.test` names never resolved, while MageBox reported DNS as configured. Config files are now written world-readable, and callers that need something stricter, such as sudoers, still set their own mode.
+- **Bootstrap Reported Working DNS When Only dnsmasq Answered** - The check queried dnsmasq directly, which proves dnsmasq works and nothing else. Everything on the machine goes through the system resolver, so browsers and curl kept failing after a "resolves" line. Bootstrap now checks the system resolver too, and says which of the two answered.
+- **`.test` Names Failed Outright After a dnsmasq Fallback** - The systemd-resolved drop-in that sends `.test` lookups to MageBox's dnsmasq is written before dnsmasq is known to work. When dnsmasq could not be started, bootstrap fell back to `/etc/hosts` but left the drop-in in place, pointing every `.test` lookup at a resolver that was not running, so names failed instead of falling through to the hosts file. The fallback now removes the drop-in and restarts systemd-resolved.
+- **Bootstrap Ran as Root Broke the Machine** - Run with `sudo`, bootstrap built the whole environment under `/root`: the config, the certificate authority, PHP-FPM pools and an nginx include pointing into a directory nginx cannot read, which stopped nginx from starting at all. Bootstrap now refuses to run as root and says to run it as the normal user, which asks for a password where it needs one.
+- **php-fpm.conf Accumulated Includes Until PHP-FPM Would Not Start** - Every bootstrap run appended another MageBox pools include. Duplicates define the same pool twice, and an include left by a run under another user matches nothing; either stops PHP-FPM from starting, and a PHP-FPM that cannot start makes every later `dpkg` operation on the package fail, which cascaded into failed Blackfire and dnsmasq installs. Bootstrap now rewrites the file to hold exactly one include, dropping duplicates and includes naming another user's home.
+- **dnsmasq Reported as Installed When Only the Library Was** - Ubuntu's `dnsmasq-base` provides the binary without a systemd unit, so MageBox skipped the install and then failed to start the service with a bare exit code. Installation now also requires the service unit on systemd machines.
 - **Passwordless Sudo Broken on Ubuntu 26.04** - Ubuntu 26.04 ships sudo-rs, which refuses to parse wildcards in command arguments. MageBox wrote rules such as `systemctl start php*-fpm`, `cp /tmp/magebox-* /etc/nginx/nginx.conf` and `apt install -y blackfire*`, so `/etc/sudoers.d/magebox` failed to parse entirely and every MageBox operation, including `sudo -s`, printed parse errors and asked for a password. Rules are now generated as complete commands, one line per PHP version and action, and are identical whether they come from the Go installers or the YAML installer definitions. Bootstrap validates the file with `visudo` before installing it and replaces an existing file that the local sudo rejects, so upgrading and re-running `magebox bootstrap` repairs a broken system.
-- **PHP 8.1 to 8.4 Could Not Be Installed on Ubuntu 26.04** - Ondrej Sury's PPA publishes nothing for Ubuntu releases it has not caught up with, so on 26.04 apt had no `php8.1` … `php8.4` packages and only Ubuntu's own PHP 8.5 could be installed. Bootstrap now checks which suites the PPA publishes, pins the newest one available, and says so. Packages built for the previous release normally install cleanly; a version that fails is reported and bootstrap continues.
-
-### Changed
-
+- **PHP 8.1 to 8.4 Could Not Be Installed on Ubuntu 26.04** - Ondrej Sury's PPA publishes nothing past Ubuntu 24.04 and is being folded into packages.sury.org, which does cover 26.04. Bootstrap now picks the repository that covers the running release: the PPA where it still does, packages.sury.org where it does not, and neither with a clear warning when no repository covers the release at all. Pinning the PPA's older suite was tried first and is worse than useless, because those packages depend on library versions (libxml2, libicu74, libzip4t64) the newer release no longer ships, so every install fails on unsatisfiable dependencies.
 - **Ubuntu 26.04 is recognised as a supported release** - It no longer triggers the "not officially tested" warning.
 - **Passwordless sudo is limited to service control** - Only starting, stopping, reloading and restarting nginx, PHP-FPM and the Blackfire agent, plus `nginx -t` and `nginx -s reload`, run without a password. Commands that run during bootstrap or an explicit install ask for one, as do Xdebug and Blackfire configuration edits. This also removes the previous `sed -i *` and `ln -s *` rules, which allowed arbitrary arguments and amounted to unrestricted root for the MageBox user.
 

cmd/magebox/bootstrap.go

@@ -69,6 +69,18 @@ func init() {
 func runBootstrap(cmd *cobra.Command, args []string) error {
 	verbose.Section("Bootstrap Starting")
 
+	// Run as root, bootstrap would build the whole environment under /root:
+	// the config, certificates, PHP-FPM pools and an nginx include pointing at
+	// a directory nginx cannot read, which stops nginx from starting at all.
+	if installer.ShouldRefuseRootBootstrap(os.Geteuid()) {
+		cli.PrintError("Do not run bootstrap as root.")
+		fmt.Println()
+		fmt.Println("  Run it as your own user; it asks for a sudo password where it needs one:")
+		fmt.Println("    magebox bootstrap")
+		fmt.Println()
+		return fmt.Errorf("bootstrap must not run as root")
+	}
+
 	p, err := getPlatform()
 	if err != nil {
 		verbose.Debug("Failed to detect platform: %v", err)
@@ -165,12 +177,22 @@ func runBootstrap(cmd *cobra.Command, args []string) error {
 		errors = append(errors, "Docker is not installed. Install: "+bootstrapper.DockerInstallInstructions())
 	} else {
 		verbose.Debug("Docker binary found, checking daemon...")
-		// Check if Docker daemon is running
-		if !bootstrapper.CheckDockerRunning() {
+		// Check whether Docker answers, and why it does not
+		switch issue := bootstrapper.CheckDocker(); issue {
+		case bootstrap.DockerPermissionDenied:
+			verbose.Debug("Docker socket is not accessible for this user")
+			cli.PrintWarning("Docker is running but this user cannot reach its socket.")
+			cli.PrintInfo("Fix: %s", issue.Advice())
+			errors = append(errors, "Docker socket is not accessible for this user")
+		case bootstrap.DockerNotRunning:
 			verbose.Debug("Docker daemon is not running")
 			cli.PrintWarning("Docker is installed but not running. Please start Docker.")
 			errors = append(errors, "Docker daemon is not running")
-		} else {
+		case bootstrap.DockerUnknownFailure:
+			verbose.Debug("Docker did not answer")
+			cli.PrintWarning("Docker did not answer. %s", issue.Advice())
+			errors = append(errors, "Docker is not answering")
+		default:
 			verbose.Debug("Docker daemon is running")
 		}
 	}
@@ -674,6 +696,38 @@ func runBootstrap(cmd *cobra.Command, args []string) error {
 		}
 	}
 
+	// Repair certificates referenced by vhosts but missing on disk. nginx
+	// refuses to start over a single one, taking every project offline.
+	fmt.Print("  Checking vhost certificates... ")
+	if missing, err := nginx.MissingCertificates(vhostsDir); err != nil {
+		fmt.Println(cli.Warning("skipped"))
+	} else if len(missing) == 0 {
+		fmt.Println(cli.Success("ok"))
+	} else {
+		fmt.Println(cli.Warning(fmt.Sprintf("%d vhost(s) reference missing certificates", len(missing))))
+		for vhost, certs := range missing {
+			domain := ""
+			for _, cert := range certs {
+				if d := nginx.DomainFromCertPath(cert); d != "" {
+					domain = d
+					break
+				}
+			}
+			if domain == "" {
+				cli.PrintWarning("%s references a certificate MageBox does not manage: %s", filepath.Base(vhost), certs[0])
+				continue
+			}
+			fmt.Printf("    Regenerating certificate for %s... ", domain)
+			if _, err := sslMgr.EnsureCert(domain, domain); err != nil {
+				fmt.Println(cli.Error("failed"))
+				cli.PrintWarning("Could not regenerate %s: %v", domain, err)
+				cli.PrintInfo("Remove %s or run 'magebox start' in that project to fix it.", vhost)
+			} else {
+				fmt.Println(cli.Success("done"))
+			}
+		}
+	}
+
 	// Test and reload nginx
 	fmt.Print("  Testing nginx config... ")
 	if err := nginxCtrl.Test(); err != nil {
@@ -800,12 +854,37 @@ func runBootstrap(cmd *cobra.Command, args []string) error {
 		fmt.Println("  Falling back to /etc/hosts mode")
 		globalCfg.DNSMode = "hosts"
 		_ = config.SaveGlobalConfig(homeDir, globalCfg)
+
+		// The systemd-resolved drop-in is written before dnsmasq is known to
+		// work. Left behind, it sends every .test lookup to a resolver that is
+		// not running, so names fail instead of falling through to /etc/hosts.
+		if dns.NeedsResolvedCleanup(false, dns.ResolvedDropInPresent()) {
+			fmt.Print("  Removing the systemd-resolved override... ")
+			if err := dnsManager.RemoveSystemdResolvedConfig(); err != nil {
+				fmt.Println(cli.Error("failed"))
+				cli.PrintWarning("Remove %s by hand, or .test names will not resolve: %v", dns.ResolvedDropInPath, err)
+			} else {
+				fmt.Println(cli.Success("done"))
+			}
+		}
+
 		cli.PrintInfo("Domains will be added to /etc/hosts when you run %s", cli.Command("magebox start"))
 	}
 
 	// Test DNS resolution if dnsmasq was configured
 	if dnsmasqConfigured {
 		testDomain := fmt.Sprintf("test.%s", tld)
+
+		// dnsmasq answering says nothing about the rest of the machine: the
+		// systemd-resolved drop-in may not have been read, in which case
+		// browsers and curl still fail.
+		if dns.SystemResolves(testDomain) {
+			fmt.Printf("  System resolver answers for %s %s\n", testDomain, cli.Success("✓"))
+		} else if dnsManager.TestResolution(testDomain) {
+			cli.PrintWarning("dnsmasq answers for %s but the system resolver does not.", testDomain)
+			cli.PrintInfo("Check that systemd-resolved can read %s, then restart it: sudo systemctl restart systemd-resolved", dns.ResolvedDropInPath)
+		}
+
 		fmt.Printf("  Testing DNS resolution for %s... ", testDomain)
 		if dnsManager.TestResolution(testDomain) {
 			fmt.Println(cli.Success("✓ resolves to 127.0.0.1"))
@@ -916,17 +995,15 @@ func runBootstrap(cmd *cobra.Command, args []string) error {
 	if p.Type == platform.Linux {
 		fmt.Println(cli.Header("Step 10: Sudoers Configuration"))
 
-		sudoersFile := "/etc/sudoers.d/magebox"
-		if _, err := os.Stat(sudoersFile); err == nil {
-			fmt.Println("  Sudoers already configured " + cli.Success("✓"))
+		// Never skip because the file exists: a file written by an older
+		// MageBox contains wildcards that today's sudo rejects, which
+		// disables every rule in it.
+		fmt.Print("  Setting up passwordless nginx/php-fpm control... ")
+		if err := bootstrapper.ConfigureSudoers(); err != nil {
+			fmt.Println(cli.Error("failed"))
+			cli.PrintWarning("Failed to setup sudoers: %v", err)
 		} else {
-			fmt.Print("  Setting up passwordless nginx/php-fpm control... ")
-			if err := bootstrapper.ConfigureSudoers(); err != nil {
-				fmt.Println(cli.Error("failed"))
-				cli.PrintWarning("Failed to setup sudoers: %v", err)
-			} else {
-				fmt.Println(cli.Success("done"))
-			}
+			fmt.Println(cli.Success("done"))
 		}
 		fmt.Println()
 	}

internal/bootstrap/docker.go

@@ -0,0 +1,61 @@
+package bootstrap
+
+import (
+	"os/exec"
+	"strings"
+)
+
+// DockerIssue explains why talking to Docker failed.
+type DockerIssue int
+
+const (
+	// DockerOK means the daemon answered.
+	DockerOK DockerIssue = iota
+	// DockerNotRunning means the daemon is not up.
+	DockerNotRunning
+	// DockerPermissionDenied means the daemon is up but this user may not
+	// reach its socket — usually a missing membership of the docker group.
+	DockerPermissionDenied
+	// DockerUnknownFailure means the command failed for another reason.
+	DockerUnknownFailure
+)
+
+// Advice returns what the user should do about the issue.
+func (d DockerIssue) Advice() string {
+	switch d {
+	case DockerNotRunning:
+		return "start Docker and run bootstrap again"
+	case DockerPermissionDenied:
+		return "add yourself to the docker group, then log out and back in: sudo usermod -aG docker $USER"
+	case DockerUnknownFailure:
+		return "run 'docker info' to see what Docker reports"
+	default:
+		return ""
+	}
+}
+
+// ClassifyDockerFailure reads the output of a failed "docker info".
+//
+// A permission error means the daemon is running and reachable by others;
+// telling the user to start Docker would send them the wrong way.
+func ClassifyDockerFailure(output string) DockerIssue {
+	lower := strings.ToLower(output)
+	switch {
+	case strings.Contains(lower, "permission denied"):
+		return DockerPermissionDenied
+	case strings.Contains(lower, "cannot connect to the docker daemon"),
+		strings.Contains(lower, "is the docker daemon running"):
+		return DockerNotRunning
+	default:
+		return DockerUnknownFailure
+	}
+}
+
+// CheckDocker reports whether Docker answers, and why it does not.
+func (b *Bootstrapper) CheckDocker() DockerIssue {
+	output, err := exec.Command("docker", "info").CombinedOutput()
+	if err == nil {
+		return DockerOK
+	}
+	return ClassifyDockerFailure(string(output))
+}

internal/bootstrap/docker_test.go

@@ -0,0 +1,72 @@
+package bootstrap
+
+import "testing"
+
+// "docker info" fails for two very different reasons, and telling the user to
+// start a daemon that is already running sends them the wrong way.
+func TestClassifyDockerFailure(t *testing.T) {
+	tests := []struct {
+		name       string
+		output     string
+		want       DockerIssue
+		wantAdvice string
+	}{
+		{
+			name:       "user is not in the docker group",
+			output:     "permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock",
+			want:       DockerPermissionDenied,
+			wantAdvice: "docker group",
+		},
+		{
+			name:       "daemon is down",
+			output:     "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?",
+			want:       DockerNotRunning,
+			wantAdvice: "start Docker",
+		},
+		{
+			name:   "anything else",
+			output: "something unexpected",
+			want:   DockerUnknownFailure,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got := ClassifyDockerFailure(tt.output)
+			if got != tt.want {
+				t.Errorf("ClassifyDockerFailure() = %v, want %v", got, tt.want)
+			}
+			if tt.wantAdvice != "" && !containsFold(got.Advice(), tt.wantAdvice) {
+				t.Errorf("advice %q does not mention %q", got.Advice(), tt.wantAdvice)
+			}
+		})
+	}
+}
+
+func containsFold(haystack, needle string) bool {
+	for i := 0; i+len(needle) <= len(haystack); i++ {
+		if equalFold(haystack[i:i+len(needle)], needle) {
+			return true
+		}
+	}
+	return false
+}
+
+func equalFold(a, b string) bool {
+	if len(a) != len(b) {
+		return false
+	}
+	for i := range a {
+		x, y := a[i], b[i]
+		if 'A' <= x && x <= 'Z' {
+			x += 'a' - 'A'
+		}
+		if 'A' <= y && y <= 'Z' {
+			y += 'a' - 'A'
+		}
+		if x != y {
+			return false
+		}
+	}
+	return true
+}

internal/bootstrap/installer/aptcache_test.go

@@ -0,0 +1,40 @@
+package installer
+
+import "testing"
+
+// apt-cache answers "unknown package" the same way whether the package really
+// is missing or apt could not read a sources file. Treating an unreadable
+// source as "package unavailable" made bootstrap skip every PHP package and
+// report success, leaving the machine with only the distribution's PHP.
+func TestAptCacheAnswerReliable(t *testing.T) {
+	tests := []struct {
+		name   string
+		stderr string
+		want   bool
+	}{
+		{name: "clean run", stderr: "", want: true},
+		{
+			name:   "sources file not readable",
+			stderr: "W: Unable to read /etc/apt/sources.list.d/magebox-php.list - open (13: Permission denied)",
+			want:   false,
+		},
+		{
+			name:   "lists directory not readable",
+			stderr: "E: Could not open file /var/lib/apt/lists/… - open (13: Permission denied)",
+			want:   false,
+		},
+		{
+			name:   "ordinary warning about a missing package",
+			stderr: "N: Unable to locate package php9.9-fpm",
+			want:   true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := aptCacheAnswerReliable(tt.stderr); got != tt.want {
+				t.Errorf("aptCacheAnswerReliable(%q) = %v, want %v", tt.stderr, got, tt.want)
+			}
+		})
+	}
+}

internal/bootstrap/installer/base.go

@@ -71,7 +71,15 @@ func (b *BaseInstaller) WriteFile(path, content string) error {
 	tmpFile.Close()
 
 	// Copy to destination with sudo (use RunSudo to allow password prompt)
-	return b.RunSudo("cp", tmpPath, path)
+	if err := b.RunSudo("cp", tmpPath, path); err != nil {
+		return err
+	}
+
+	// os.CreateTemp makes the file 0600, and cp keeps that mode. Daemons that
+	// read their configuration after dropping privileges (systemd-resolved,
+	// for one) then cannot open it and silently ignore the file. Callers that
+	// need something stricter, such as sudoers, set it afterwards.
+	return b.RunSudo("chmod", "0644", path)
 }
 
 // CommandExists checks if a command is available