From e0f074cd8c28229b7081f3b3e4bd1837a709abad Mon Sep 17 00:00:00 2001
From: anti <samuel@securejump.cl>
Date: Fri, 24 Apr 2026 21:58:33 -0400
Subject: [PATCH] docs(troubleshooting): branch buildx recovery on leaked-mount
 count

prune+restart alone doesn't evict already-held mounts; full recipe
is stop-pkill-umount-start. Add a separate recipe for the count==0
case (driver state corruption, no mounts to clean) and call out the
strict signature DECNET uses to classify the wedge.
---
 Troubleshooting.md | 44 ++++++++++++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 12 deletions(-)

diff --git a/Troubleshooting.md b/Troubleshooting.md
index 7abc9ff..43e0cfa 100644
--- a/Troubleshooting.md
+++ b/Troubleshooting.md
@@ -78,23 +78,43 @@ mount | grep -c '/var/lib/docker/tmp/buildkit-mount'
 
 Anything past single digits is pathological. We've seen hosts sitting on hundreds after a few botched mass-scale topologies.
 
-**Fix.**
-
-```bash
-docker buildx prune -af
-sudo systemctl restart docker
-```
-
-Restart is the operative step — it drops every leaked mount. `prune -af` also discards the build cache so the next deploy rebuilds from scratch; skip it if you want the cache preserved.
-
-If the activity dir itself is corrupted (rare):
+**Fix — leaked mounts present (count > 0).**
+
+`prune -af && systemctl restart docker` is **not enough** — leaked mounts often outlive the daemon because zombie `buildkitd` / `containerd-shim` processes still hold them. Full recipe:
 
 ```bash
+sudo systemctl stop docker.socket docker.service
+sudo pkill -9 -f buildkitd
+sudo pkill -9 -f containerd-shim
+for m in $(mount | awk '$3 ~ /buildkit-mount/ {print $3}'); do
+  sudo umount -l "$m"
+done
 rm -rf ~/.docker/buildx/activity
-docker buildx create --use
+sudo systemctl start docker
+docker buildx create --use --name default
+docker buildx inspect --bootstrap
 ```
 
-**How DECNET handles it.** The engine's `_compose_with_retry` counts leaked buildkit mounts before every build and refuses to start if the count crosses 10 — you get the recovery recipe in the error payload instead of a cryptic EROFS surfaced three retries deep. Mid-build failures that match the known wedge signature also short-circuit the retry loop with the same hint.
+The `umount -l` step is the one most recipes online miss.
+
+**Fix — driver corruption (count == 0).**
+
+If `mount | grep -c buildkit-mount` already prints 0 and you still hit the wedge, the buildx driver state itself is inconsistent. Rebuild it:
+
+```bash
+docker buildx rm default 2>/dev/null
+rm -rf ~/.docker/buildx/activity ~/.docker/buildx/instances/default
+docker buildx create --use --name default
+docker buildx inspect --bootstrap
+```
+
+**How DECNET handles it.** The engine's `_compose_with_retry`:
+
+* Pre-flights leaked mounts before every build; if the count crosses 10, refuses to start and emits the leaked-mount recipe.
+* Catches the wedge signature mid-build (`failed to update builder last activity time` + `read-only file system`) and short-circuits the retry loop, branching the recipe on whether mounts are 0 or >0.
+* Preserves the original compose stderr in the error so you can see what actually broke alongside the recipe.
+
+Unrelated `read-only file system` errors (e.g. a config file mount) are NOT classified as a wedge — both sentinel phrases must match.
 
 ---