diff options
Diffstat (limited to 'doc/setup-guide.md')
| -rw-r--r-- | doc/setup-guide.md | 58 |
1 files changed, 58 insertions, 0 deletions
diff --git a/doc/setup-guide.md b/doc/setup-guide.md index 6dc8d44..43c2b20 100644 --- a/doc/setup-guide.md +++ b/doc/setup-guide.md @@ -229,6 +229,64 @@ pipeline { See `doc/Jenkinsfile.example` for a full multi-stage example. +### 6.5 Pipeline durability and stuck builds + +Jenkins has a known bug ([JENKINS-50407](https://issues.jenkins.io/browse/JENKINS-50407)) +where pipeline flow executions can become corrupted. This happens when a build +is interrupted uncleanly — for example, by a timeout, manual abort, Jenkins +restart mid-build, or a disk issue. When it occurs, new pipeline builds will +silently fail: they show "Started by user ..." in the console output and +nothing else. The Jenkins log will contain warnings like: + +``` +WARNING o.j.p.w.f.FlowExecutionList$DefaultStorage#unregister: +<job>#<build> was not in the list to begin with: [] +``` + +Two measures are in place to prevent and recover from this: + +#### Prevention: Pipeline durability setting + +By default, Jenkins aggressively writes pipeline state to disk at every step +so that running builds can be resumed after a crash. This persistence layer +is what gets corrupted. Since our builds run in ephemeral Docker containers +(DooD), there is nothing useful to resume — a fresh build is always needed. + +Switching to "Performance-optimized" durability disables most of the state +persistence, which eliminates the main source of corruption and also makes +pipelines run faster. + +After initial Jenkins setup: + +1. Go to **Manage Jenkins** > **Configure System** +2. Find **Pipeline Speed/Durability Setting** +3. Change it to **Performance-optimized: much less durability** +4. Click **Save** + +The tradeoff: if Jenkins crashes mid-build, the running build is lost and +cannot be resumed. This is acceptable because the Docker build container +is also lost on crash, so there is nothing to resume anyway. + +#### Recovery: Startup cleanup script + +The file `jenkins/init.groovy.d/clear-stuck-builds.groovy` is mounted into +the Jenkins container at `/var/jenkins_home/init.groovy.d/`. Jenkins +automatically executes all `.groovy` files in this directory on every startup. + +The script iterates through all registered pipeline flow executions and +removes any that are marked as complete but are still tracked in the +execution list. This prevents stuck executions from blocking future builds. + +If builds ever get stuck again (showing only "Started by user ..." with no +further output), simply restart the Jenkins container: + +```bash +docker restart jenkins +``` + +The cleanup script will run automatically and clear the stuck state. No +manual deletion of build directories is needed. + ## 7. Gerrit (Code Review) |
