path: root/doc/setup-guide.md

# Bastion Server — Setup Guide

Full instruction to build the server from a fresh Debian installation.

## Prerequisites

- Fresh Debian server
- Root access
- Domain `swave.lol` with DNS A records pointing to the server:
  - `swave.lol`
  - `blog.swave.lol`
  - `jenkins.swave.lol`
  - `cgit.swave.lol`
  - `gerrit.swave.lol`


## 1. Install Docker

Based on https://docs.docker.com/engine/install/debian/#install-using-the-repository

```bash
# Install prerequisites
apt-get update
apt-get install -y ca-certificates curl gnupg

# Add Docker GPG key
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

# Add Docker repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Verify
docker run hello-world
```


## 2. Clone the bastion repository

```bash
cd /root/Projects
git clone <your-repo-url> bastion
cd bastion
```


## 3. Start Portainer

Portainer manages all other containers and stacks via web UI.

```bash
docker run -d \
  --name="portainer" \
  --restart on-failure \
  -p 9000:9000 \
  -p 8000:8000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data \
  portainer/portainer-ce:latest
```

Access Portainer at `http://<server-ip>:9000` and create admin account.


## 4. Git Server

### 4.1 Create host directories

```bash
# Directory for SSH public keys (for git access)
mkdir -p /var/git_ssh_keys

# Directory for git repositories
mkdir -p /var/git/repos

# Copy your public SSH keys for git access (use ed25519, RSA is not supported)
# cp ~/.ssh/id_ed25519.pub /var/git_ssh_keys/mykey.pub
```

### 4.2 Build and start

```bash
cd /root/Projects/bastion/git-server

# Build the git-server image
docker build -t git-server .

# Start (this also creates the git-network)
docker compose -f server.yaml up -d
```

### 4.3 Create a test repository

```bash
# On the server
cd /var/git/repos
git init --bare test.git
chown -R 1000:1000 test.git  # git user inside container

# From a client machine
git clone ssh://git@<server-ip>/repos/test.git
or
git clone git://<server-ip>/test.git
```

### 4.4 Import an existing repository

To make a one-time copy of an existing repository (e.g., from GitHub):

```bash
cd /var/git/repos
git clone --bare https://github.com/user/repo.git
chown -R 1000:1000 repo.git
```

This copies all branches, tags, and history. The remote reference is removed,
so it becomes a standalone copy with no link back to the original.

To keep syncing from the original source, use `--mirror` instead:

```bash
cd /var/git/repos
git clone --mirror https://github.com/user/repo.git
chown -R 1000:1000 repo.git
```

This keeps the remote configured so you can periodically pull updates:

```bash
cd /var/git/repos/repo.git
git remote update
```

`--mirror` syncs all refs exactly, including deleted branches.


## 5. Ghost (Blog)

### 5.1 Prepare environment

```bash
cd /root/Projects/bastion/ghost

# Copy the mysql init script to the expected location
mkdir -p /var/lib/docker/mysql-init-script
cp mysql-init-script/create-multiple-databases.sh /var/lib/docker/mysql-init-script/

# Edit stack.env (or .env for non-Portainer use) with your credentials
# IMPORTANT: Change default passwords before first run!
```

Key settings in `stack.env`:
- `DOMAIN` — your domain (e.g. `swave.lol`)
- `DATABASE_ROOT_PASSWORD` — MySQL root password
- `DATABASE_USER` / `DATABASE_PASSWORD` — Ghost database credentials
- `UPLOAD_LOCATION` — where Ghost stores content

### 5.2 Start Ghost

```bash
cd /root/Projects/bastion/ghost
docker compose up -d
```

Ghost will be available at `http://<server-ip>:2368` for testing before Nginx is configured.

To start with ActivityPub support (optional):
```bash
docker compose --profile activitypub up -d
```


## 6. Jenkins

### 6.1 Create host directories

```bash
mkdir -p /var/jenkins_home
chown -R 1000:1000 /var/jenkins_home  # jenkins user inside container runs as UID 1000
```

### 6.2 Build and start

```bash
cd /root/Projects/bastion/jenkins

# Build Jenkins image (includes Docker CLI and docker-workflow plugin)
docker compose build

# Start Jenkins
docker compose up -d
```

### 6.3 Get initial admin password

```bash
docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
```

Jenkins will be available at `http://<server-ip>:8080/jenkins` for initial setup.

### 6.4 Jenkins build setup (DooD)

Jenkins uses Docker-out-of-Docker: it spawns sibling containers on the host for builds.
Each project defines its own build image in the Jenkinsfile:

```groovy
pipeline {
    agent {
        docker { image 'node:20' }
    }
    stages {
        stage('Build') {
            steps {
                sh 'npm install && npm run build'
            }
        }
    }
}
```

See `doc/Jenkinsfile.example` for a full multi-stage example.

### 6.5 Pipeline durability and stuck builds

Jenkins has a known bug ([JENKINS-50407](https://issues.jenkins.io/browse/JENKINS-50407))
where pipeline flow executions can become corrupted. This happens when a build
is interrupted uncleanly — for example, by a timeout, manual abort, Jenkins
restart mid-build, or a disk issue. When it occurs, new pipeline builds will
silently fail: they show "Started by user ..." in the console output and
nothing else. The Jenkins log will contain warnings like:

```
WARNING o.j.p.w.f.FlowExecutionList$DefaultStorage#unregister:
<job>#<build> was not in the list to begin with: []
```

Two measures are in place to prevent and recover from this:

#### Prevention: Pipeline durability setting

By default, Jenkins aggressively writes pipeline state to disk at every step
so that running builds can be resumed after a crash. This persistence layer
is what gets corrupted. Since our builds run in ephemeral Docker containers
(DooD), there is nothing useful to resume — a fresh build is always needed.

Switching to "Performance-optimized" durability disables most of the state
persistence, which eliminates the main source of corruption and also makes
pipelines run faster.

After initial Jenkins setup:

1. Go to **Manage Jenkins** > **Configure System**
2. Find **Pipeline Speed/Durability Setting**
3. Change it to **Performance-optimized: much less durability**
4. Click **Save**

The tradeoff: if Jenkins crashes mid-build, the running build is lost and
cannot be resumed. This is acceptable because the Docker build container
is also lost on crash, so there is nothing to resume anyway.

#### Recovery: Startup cleanup script

The file `jenkins/init.groovy.d/clear-stuck-builds.groovy` is mounted into
the Jenkins container at `/var/jenkins_home/init.groovy.d/`. Jenkins
automatically executes all `.groovy` files in this directory on every startup.

The script iterates through all registered pipeline flow executions and
removes any that are marked as complete but are still tracked in the
execution list. This prevents stuck executions from blocking future builds.

If builds ever get stuck again (showing only "Started by user ..." with no
further output), simply restart the Jenkins container:

```bash
docker restart jenkins
```

The cleanup script will run automatically and clear the stuck state. No
manual deletion of build directories is needed.


## 7. Gerrit (Code Review)

### 7.1 Create host directories

```bash
mkdir -p /var/gerrit/{etc,git,db,index,cache}
chown -R 1000:1000 /var/gerrit  # gerrit user inside container runs as UID 1000
```

### 7.2 Start Gerrit

Gerrit is part of the git-server stack. It requires `jenkins_network` to exist,
so Jenkins must be started first.

```bash
cd /root/Projects/bastion/git-server
docker compose -f server.yaml up -d gerrit
```

Gerrit shares `/var/git/repos` with git-server for repository access.


## 8. Nginx (Reverse Proxy + SSL)

Nginx must be started AFTER Git Server, Ghost, Jenkins, and Gerrit, because it joins
their networks as external.

### 8.1 Create host directories

```bash
# Nginx config directory
mkdir -p /var/nginx/conf

# Copy nginx.conf
cp /root/Projects/bastion/nginx/nginx.conf /var/nginx/conf/

# DH parameters (this takes a few minutes)
mkdir -p /var/dh_param
openssl dhparam -out /var/dh_param/dhparam-2048.pem 2048

# Let's Encrypt directories
mkdir -p /var/letsencrypt/etc
mkdir -p /var/letsencrypt/lts_site

# Copy the ACME challenge placeholder page
cp /root/Projects/bastion/nginx/letsencrypt/index.html /var/letsencrypt/lts_site/
```

### 8.2 First run — HTTP only (no SSL yet)

Before we have SSL certificates, we need to temporarily disable the SSL server
blocks so Nginx can start and serve the ACME challenge for certbot.

```bash
# Edit the nginx.conf copy on the host to comment out SSL server blocks
# Keep only the port 80 server block
nano /var/nginx/conf/nginx.conf
```

Comment out everything from `# Ghost — swave.lol` to the end of file.
Keep only the `listen 80` server block.

```bash
cd /root/Projects/bastion/nginx
docker compose up -d
```

### 8.3 Obtain SSL certificates

```bash
cd /root/Projects/bastion/nginx
bash run_certbot.sh
```

This requests certificates for:
- `swave.lol`
- `blog.swave.lol`
- `jenkins.swave.lol`
- `cgit.swave.lol`
- `gerrit.swave.lol`

IMPORTANT: All DNS records must be pointing to the server before running certbot.

### 8.4 Enable SSL

```bash
# Restore full nginx.conf with SSL blocks
cp /root/Projects/bastion/nginx/nginx.conf /var/nginx/conf/

# Restart Nginx
cd /root/Projects/bastion/nginx
docker compose restart
```

### 8.5 Verify

All services should now be accessible:

| URL | Service |
|-----|---------|
| `https://swave.lol` | Ghost (blog) |
| `https://blog.swave.lol` | Ghost (blog, alias) |
| `https://swave.lol/jenkins` | Jenkins (path-based) |
| `https://jenkins.swave.lol` | Jenkins (subdomain) |
| `https://swave.lol/cgit` | Cgit (path-based) |
| `https://cgit.swave.lol` | Cgit (subdomain) |
| `https://swave.lol/gerrit` | Gerrit (path-based) |
| `https://gerrit.swave.lol` | Gerrit (subdomain) |
| `http://<server-ip>:9000` | Portainer |
| `ssh://git@<server-ip>/repos/<repo>.git` | Git (SSH) |
| `git://<server-ip>/<repo>.git` | Git (daemon, read-only) |


## Network Architecture

```
git-network (172.22.0.0/16)
├── git-server    (172.22.0.2)
├── cgit           (172.22.0.3)
├── gerrit         (172.22.0.4)
└── nginx          (172.22.0.254)

ghost_network
├── ghost
├── ghost-db
├── activitypub (optional)
└── nginx

jenkins_network (172.23.0.0/16)
├── jenkins        (172.23.0.2)
├── git-server     (172.23.0.3)
├── gerrit         (172.23.0.4)
└── nginx
```

Nginx sits on all networks so it can reverse proxy to every service.


## Startup Order

The correct order to start all services:

```bash
# 1. Portainer (standalone, no dependencies)
docker run -d --name="portainer" --restart on-failure \
  -p 9000:9000 -p 8000:8000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data portainer/portainer-ce:latest

# 2. Git Server (creates git-network)
docker compose -f /root/Projects/bastion/git-server/server.yaml up -d

# 3. Ghost (creates ghost_network)
docker compose -f /root/Projects/bastion/ghost/compose.yml up -d

# 4. Jenkins (creates jenkins_network)
docker compose -f /root/Projects/bastion/jenkins/docker-compose.yaml up -d

# 5. Gerrit (joins git-network + jenkins_network, both must exist)
docker compose -f /root/Projects/bastion/git-server/server.yaml up -d gerrit

# 6. Nginx (joins all networks — must be last)
docker compose -f /root/Projects/bastion/nginx/docker-compose.yaml up -d
```


## Ports Summary

| Port | Service | Protocol |
|------|---------|----------|
| 22 | Git Server (SSH) | TCP |
| 80 | Nginx (HTTP, redirects to 443) | TCP |
| 443 | Nginx (HTTPS) | TCP |
| 2368 | Ghost (direct, for testing) | TCP |
| 8080 | Jenkins (direct, for testing) | TCP |
| 9000 | Portainer (web UI) | TCP |
| 9418 | Git Server (git daemon) | TCP |
| 50000 | Jenkins (agent communication) | TCP |