diff options
| -rw-r--r-- | CLAUDE.md | 3 | ||||
| -rw-r--r-- | doc/setup-guide.md | 58 | ||||
| -rw-r--r-- | jenkins/docker-compose.yaml | 1 | ||||
| -rw-r--r-- | jenkins/init.groovy.d/clear-stuck-builds.groovy | 37 |
4 files changed, 99 insertions, 0 deletions
@@ -58,6 +58,8 @@ Bastion is a self-hosted server system on Debian that runs multiple services in - Accessible at `https://jenkins.swave.lol` and `https://swave.lol/jenkins` - Ports: 8080 (web UI), 50000 (agent communication) - Volume: `/var/jenkins_home` +- Pipeline durability: set to "Performance-optimized" to prevent flow execution corruption +- Startup script: `jenkins/init.groovy.d/clear-stuck-builds.groovy` auto-cleans stuck builds on restart ### Nginx (reverse proxy) - Dir: `nginx/` @@ -89,6 +91,7 @@ Each stack has its own Docker network. Nginx joins all of them to reverse proxy: - `doc/Jenkinsfile.kernel-example` — example kernel build pipeline using kernel-builder image - `doc/jenkins-cpp-build.md` — guide for building C++ apps in Jenkins with Docker agent - `jenkins/kernel-builder/Dockerfile` — custom Docker image for building the Linux kernel +- `jenkins/init.groovy.d/clear-stuck-builds.groovy` — startup script to clean stuck pipeline executions - `ghost/.env.example` — template for Ghost env vars - `.gitignore` — excludes `.env` files (secrets) diff --git a/doc/setup-guide.md b/doc/setup-guide.md index 6dc8d44..43c2b20 100644 --- a/doc/setup-guide.md +++ b/doc/setup-guide.md @@ -229,6 +229,64 @@ pipeline { See `doc/Jenkinsfile.example` for a full multi-stage example. +### 6.5 Pipeline durability and stuck builds + +Jenkins has a known bug ([JENKINS-50407](https://issues.jenkins.io/browse/JENKINS-50407)) +where pipeline flow executions can become corrupted. This happens when a build +is interrupted uncleanly — for example, by a timeout, manual abort, Jenkins +restart mid-build, or a disk issue. When it occurs, new pipeline builds will +silently fail: they show "Started by user ..." in the console output and +nothing else. The Jenkins log will contain warnings like: + +``` +WARNING o.j.p.w.f.FlowExecutionList$DefaultStorage#unregister: +<job>#<build> was not in the list to begin with: [] +``` + +Two measures are in place to prevent and recover from this: + +#### Prevention: Pipeline durability setting + +By default, Jenkins aggressively writes pipeline state to disk at every step +so that running builds can be resumed after a crash. This persistence layer +is what gets corrupted. Since our builds run in ephemeral Docker containers +(DooD), there is nothing useful to resume — a fresh build is always needed. + +Switching to "Performance-optimized" durability disables most of the state +persistence, which eliminates the main source of corruption and also makes +pipelines run faster. + +After initial Jenkins setup: + +1. Go to **Manage Jenkins** > **Configure System** +2. Find **Pipeline Speed/Durability Setting** +3. Change it to **Performance-optimized: much less durability** +4. Click **Save** + +The tradeoff: if Jenkins crashes mid-build, the running build is lost and +cannot be resumed. This is acceptable because the Docker build container +is also lost on crash, so there is nothing to resume anyway. + +#### Recovery: Startup cleanup script + +The file `jenkins/init.groovy.d/clear-stuck-builds.groovy` is mounted into +the Jenkins container at `/var/jenkins_home/init.groovy.d/`. Jenkins +automatically executes all `.groovy` files in this directory on every startup. + +The script iterates through all registered pipeline flow executions and +removes any that are marked as complete but are still tracked in the +execution list. This prevents stuck executions from blocking future builds. + +If builds ever get stuck again (showing only "Started by user ..." with no +further output), simply restart the Jenkins container: + +```bash +docker restart jenkins +``` + +The cleanup script will run automatically and clear the stuck state. No +manual deletion of build directories is needed. + ## 7. Gerrit (Code Review) diff --git a/jenkins/docker-compose.yaml b/jenkins/docker-compose.yaml index 843980b..fbbe3c7 100644 --- a/jenkins/docker-compose.yaml +++ b/jenkins/docker-compose.yaml @@ -12,6 +12,7 @@ services: volumes: - /var/jenkins_home:/var/jenkins_home - /var/run/docker.sock:/var/run/docker.sock + - ./init.groovy.d:/var/jenkins_home/init.groovy.d:ro networks: jenkins_network: ipv4_address: 172.23.0.2 diff --git a/jenkins/init.groovy.d/clear-stuck-builds.groovy b/jenkins/init.groovy.d/clear-stuck-builds.groovy new file mode 100644 index 0000000..0b3a6b7 --- /dev/null +++ b/jenkins/init.groovy.d/clear-stuck-builds.groovy @@ -0,0 +1,37 @@ +// clear-stuck-builds.groovy +// +// This script runs automatically every time Jenkins starts. +// It cleans up corrupted/stuck pipeline flow executions that can occur +// when a build is interrupted uncleanly (timeout, abort, crash, etc.). +// +// Without this, Jenkins can enter a state where new pipeline builds +// silently fail — they show "Started by user ..." and nothing else. +// The Jenkins log will contain warnings like: +// +// WARNING o.j.p.w.f.FlowExecutionList$DefaultStorage#unregister: +// <job>#<build> was not in the list to begin with: [] +// +// This is a known Jenkins bug: https://issues.jenkins.io/browse/JENKINS-50407 +// +// The fix: on startup, iterate through all registered flow executions +// and remove any that are already complete but still tracked, preventing +// them from blocking future builds. + +import org.jenkinsci.plugins.workflow.flow.FlowExecutionList + +println "[clear-stuck-builds] Checking for stuck flow executions..." + +int cleaned = 0 +FlowExecutionList.get().each { exec -> + if (exec.isComplete()) { + println "[clear-stuck-builds] Cleaning up completed execution: ${exec}" + FlowExecutionList.get().unregister(exec.owner) + cleaned++ + } +} + +if (cleaned > 0) { + println "[clear-stuck-builds] Cleaned up ${cleaned} stuck execution(s)" +} else { + println "[clear-stuck-builds] No stuck executions found" +} |