Add pipeline durability fix and stuck build cleanup script

Mount jenkins/init.groovy.d/ into the container to auto-clean corrupted
flow executions on startup (JENKINS-50407). Document both the prevention
(Performance-optimized durability setting) and recovery (groovy cleanup
script) in the setup guide.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

4 files changed, 99 insertions, 0 deletions

diff --git a/CLAUDE.md b/CLAUDE.md
index 27e3799..cb9cbbc 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -58,6 +58,8 @@ Bastion is a self-hosted server system on Debian that runs multiple services in
 - Accessible at `https://jenkins.swave.lol` and `https://swave.lol/jenkins`
 - Ports: 8080 (web UI), 50000 (agent communication)
 - Volume: `/var/jenkins_home`
+- Pipeline durability: set to "Performance-optimized" to prevent flow execution corruption
+- Startup script: `jenkins/init.groovy.d/clear-stuck-builds.groovy` auto-cleans stuck builds on restart
 
 ### Nginx (reverse proxy)
 - Dir: `nginx/`
@@ -89,6 +91,7 @@ Each stack has its own Docker network. Nginx joins all of them to reverse proxy:
 - `doc/Jenkinsfile.kernel-example` — example kernel build pipeline using kernel-builder image
 - `doc/jenkins-cpp-build.md` — guide for building C++ apps in Jenkins with Docker agent
 - `jenkins/kernel-builder/Dockerfile` — custom Docker image for building the Linux kernel
+- `jenkins/init.groovy.d/clear-stuck-builds.groovy` — startup script to clean stuck pipeline executions
 - `ghost/.env.example` — template for Ghost env vars
 - `.gitignore` — excludes `.env` files (secrets)
 
diff --git a/doc/setup-guide.md b/doc/setup-guide.md
index 6dc8d44..43c2b20 100644
--- a/doc/setup-guide.md
+++ b/doc/setup-guide.md
@@ -229,6 +229,64 @@ pipeline {
 
 See `doc/Jenkinsfile.example` for a full multi-stage example.
 
+### 6.5 Pipeline durability and stuck builds
+
+Jenkins has a known bug ([JENKINS-50407](https://issues.jenkins.io/browse/JENKINS-50407))
+where pipeline flow executions can become corrupted. This happens when a build
+is interrupted uncleanly — for example, by a timeout, manual abort, Jenkins
+restart mid-build, or a disk issue. When it occurs, new pipeline builds will
+silently fail: they show "Started by user ..." in the console output and
+nothing else. The Jenkins log will contain warnings like:
+
+```
+WARNING o.j.p.w.f.FlowExecutionList$DefaultStorage#unregister:
+<job>#<build> was not in the list to begin with: []
+```
+
+Two measures are in place to prevent and recover from this:
+
+#### Prevention: Pipeline durability setting
+
+By default, Jenkins aggressively writes pipeline state to disk at every step
+so that running builds can be resumed after a crash. This persistence layer
+is what gets corrupted. Since our builds run in ephemeral Docker containers
+(DooD), there is nothing useful to resume — a fresh build is always needed.
+
+Switching to "Performance-optimized" durability disables most of the state
+persistence, which eliminates the main source of corruption and also makes
+pipelines run faster.
+
+After initial Jenkins setup:
+
+1. Go to **Manage Jenkins** > **Configure System**
+2. Find **Pipeline Speed/Durability Setting**
+3. Change it to **Performance-optimized: much less durability**
+4. Click **Save**
+
+The tradeoff: if Jenkins crashes mid-build, the running build is lost and
+cannot be resumed. This is acceptable because the Docker build container
+is also lost on crash, so there is nothing to resume anyway.
+
+#### Recovery: Startup cleanup script
+
+The file `jenkins/init.groovy.d/clear-stuck-builds.groovy` is mounted into
+the Jenkins container at `/var/jenkins_home/init.groovy.d/`. Jenkins
+automatically executes all `.groovy` files in this directory on every startup.
+
+The script iterates through all registered pipeline flow executions and
+removes any that are marked as complete but are still tracked in the
+execution list. This prevents stuck executions from blocking future builds.
+
+If builds ever get stuck again (showing only "Started by user ..." with no
+further output), simply restart the Jenkins container:
+
+```bash
+docker restart jenkins
+```
+
+The cleanup script will run automatically and clear the stuck state. No
+manual deletion of build directories is needed.
+
 
 ## 7. Gerrit (Code Review)
 
diff --git a/jenkins/docker-compose.yaml b/jenkins/docker-compose.yaml
index 843980b..fbbe3c7 100644
--- a/jenkins/docker-compose.yaml
+++ b/jenkins/docker-compose.yaml
@@ -12,6 +12,7 @@ services:
     volumes:
       - /var/jenkins_home:/var/jenkins_home
       - /var/run/docker.sock:/var/run/docker.sock
+      - ./init.groovy.d:/var/jenkins_home/init.groovy.d:ro
     networks:
       jenkins_network:
         ipv4_address: 172.23.0.2
diff --git a/jenkins/init.groovy.d/clear-stuck-builds.groovy b/jenkins/init.groovy.d/clear-stuck-builds.groovy
new file mode 100644
index 0000000..0b3a6b7
--- /dev/null
+++ b/jenkins/init.groovy.d/clear-stuck-builds.groovy
@@ -0,0 +1,37 @@
+// clear-stuck-builds.groovy
+//
+// This script runs automatically every time Jenkins starts.
+// It cleans up corrupted/stuck pipeline flow executions that can occur
+// when a build is interrupted uncleanly (timeout, abort, crash, etc.).
+//
+// Without this, Jenkins can enter a state where new pipeline builds
+// silently fail — they show "Started by user ..." and nothing else.
+// The Jenkins log will contain warnings like:
+//
+//   WARNING o.j.p.w.f.FlowExecutionList$DefaultStorage#unregister:
+//   <job>#<build> was not in the list to begin with: []
+//
+// This is a known Jenkins bug: https://issues.jenkins.io/browse/JENKINS-50407
+//
+// The fix: on startup, iterate through all registered flow executions
+// and remove any that are already complete but still tracked, preventing
+// them from blocking future builds.
+
+import org.jenkinsci.plugins.workflow.flow.FlowExecutionList
+
+println "[clear-stuck-builds] Checking for stuck flow executions..."
+
+int cleaned = 0
+FlowExecutionList.get().each { exec ->
+    if (exec.isComplete()) {
+        println "[clear-stuck-builds] Cleaning up completed execution: ${exec}"
+        FlowExecutionList.get().unregister(exec.owner)
+        cleaned++
+    }
+}
+
+if (cleaned > 0) {
+    println "[clear-stuck-builds] Cleaned up ${cleaned} stuck execution(s)"
+} else {
+    println "[clear-stuck-builds] No stuck executions found"
+}