summaryrefslogtreecommitdiff
path: root/doc/setup-guide.md
diff options
context:
space:
mode:
Diffstat (limited to 'doc/setup-guide.md')
-rw-r--r--doc/setup-guide.md58
1 files changed, 58 insertions, 0 deletions
diff --git a/doc/setup-guide.md b/doc/setup-guide.md
index 6dc8d44..43c2b20 100644
--- a/doc/setup-guide.md
+++ b/doc/setup-guide.md
@@ -229,6 +229,64 @@ pipeline {
See `doc/Jenkinsfile.example` for a full multi-stage example.
+### 6.5 Pipeline durability and stuck builds
+
+Jenkins has a known bug ([JENKINS-50407](https://issues.jenkins.io/browse/JENKINS-50407))
+where pipeline flow executions can become corrupted. This happens when a build
+is interrupted uncleanly — for example, by a timeout, manual abort, Jenkins
+restart mid-build, or a disk issue. When it occurs, new pipeline builds will
+silently fail: they show "Started by user ..." in the console output and
+nothing else. The Jenkins log will contain warnings like:
+
+```
+WARNING o.j.p.w.f.FlowExecutionList$DefaultStorage#unregister:
+<job>#<build> was not in the list to begin with: []
+```
+
+Two measures are in place to prevent and recover from this:
+
+#### Prevention: Pipeline durability setting
+
+By default, Jenkins aggressively writes pipeline state to disk at every step
+so that running builds can be resumed after a crash. This persistence layer
+is what gets corrupted. Since our builds run in ephemeral Docker containers
+(DooD), there is nothing useful to resume — a fresh build is always needed.
+
+Switching to "Performance-optimized" durability disables most of the state
+persistence, which eliminates the main source of corruption and also makes
+pipelines run faster.
+
+After initial Jenkins setup:
+
+1. Go to **Manage Jenkins** > **Configure System**
+2. Find **Pipeline Speed/Durability Setting**
+3. Change it to **Performance-optimized: much less durability**
+4. Click **Save**
+
+The tradeoff: if Jenkins crashes mid-build, the running build is lost and
+cannot be resumed. This is acceptable because the Docker build container
+is also lost on crash, so there is nothing to resume anyway.
+
+#### Recovery: Startup cleanup script
+
+The file `jenkins/init.groovy.d/clear-stuck-builds.groovy` is mounted into
+the Jenkins container at `/var/jenkins_home/init.groovy.d/`. Jenkins
+automatically executes all `.groovy` files in this directory on every startup.
+
+The script iterates through all registered pipeline flow executions and
+removes any that are marked as complete but are still tracked in the
+execution list. This prevents stuck executions from blocking future builds.
+
+If builds ever get stuck again (showing only "Started by user ..." with no
+further output), simply restart the Jenkins container:
+
+```bash
+docker restart jenkins
+```
+
+The cleanup script will run automatically and clear the stuck state. No
+manual deletion of build directories is needed.
+
## 7. Gerrit (Code Review)