# Bastion Server — Setup Guide

Full instruction to build the server from a fresh Debian installation.

## Prerequisites

- Fresh Debian server
- Root access
- Domain `swave.lol` with DNS A records pointing to the server:
  - `swave.lol`
  - `blog.swave.lol`
  - `jenkins.swave.lol`
  - `cgit.swave.lol`
  - `gerrit.swave.lol`
  - `nexus.swave.lol`
  - `registry.swave.lol`
  - `auth.swave.lol`


## 1. Install Docker

Based on https://docs.docker.com/engine/install/debian/#install-using-the-repository

```bash
# Install prerequisites
apt-get update
apt-get install -y ca-certificates curl gnupg

# Add Docker GPG key
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

# Add Docker repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Verify
docker run hello-world
```


## 2. Clone the bastion repository

```bash
cd /root/Projects
git clone <your-repo-url> bastion
cd bastion
```


## 3. Start Portainer

Portainer manages all other containers and stacks via web UI.

```bash
docker run -d \
  --name="portainer" \
  --restart on-failure \
  -p 9000:9000 \
  -p 8000:8000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data \
  portainer/portainer-ce:latest
```

Access Portainer at `http://<server-ip>:9000` and create admin account.


## 4. Git Server

### 4.1 Create host directories

```bash
# Directory for SSH public keys (for git access)
mkdir -p /var/git_ssh_keys

# Directory for git repositories
mkdir -p /var/git/repos

# Copy your public SSH keys for git access (use ed25519, RSA is not supported)
# cp ~/.ssh/id_ed25519.pub /var/git_ssh_keys/mykey.pub
```

### 4.2 Build and start

```bash
cd /root/Projects/bastion/git-server

# Build the git-server image
docker build -t git-server .

# Start (this also creates the git-network)
docker compose -f server.yaml up -d
```

### 4.3 Create a test repository

```bash
# On the server
cd /var/git/repos
git init --bare test.git
chown -R 1000:1000 test.git  # git user inside container

# From a client machine
git clone ssh://git@<server-ip>/repos/test.git
or
git clone git://<server-ip>/test.git
```

### 4.4 Import an existing repository

To make a one-time copy of an existing repository (e.g., from GitHub):

```bash
cd /var/git/repos
git clone --bare https://github.com/user/repo.git
chown -R 1000:1000 repo.git
```

This copies all branches, tags, and history. The remote reference is removed,
so it becomes a standalone copy with no link back to the original.

To keep syncing from the original source, use `--mirror` instead:

```bash
cd /var/git/repos
git clone --mirror https://github.com/user/repo.git
chown -R 1000:1000 repo.git
```

This keeps the remote configured so you can periodically pull updates:

```bash
cd /var/git/repos/repo.git
git remote update
```

`--mirror` syncs all refs exactly, including deleted branches.


## 5. Ghost (Blog)

### 5.1 Prepare environment

```bash
cd /root/Projects/bastion/ghost

# Copy the mysql init script to the expected location
mkdir -p /var/lib/docker/mysql-init-script
cp mysql-init-script/create-multiple-databases.sh /var/lib/docker/mysql-init-script/

# Edit stack.env (or .env for non-Portainer use) with your credentials
# IMPORTANT: Change default passwords before first run!
```

Key settings in `stack.env`:
- `DOMAIN` — your domain (e.g. `swave.lol`)
- `DATABASE_ROOT_PASSWORD` — MySQL root password
- `DATABASE_USER` / `DATABASE_PASSWORD` — Ghost database credentials
- `UPLOAD_LOCATION` — where Ghost stores content

### 5.2 Start Ghost

```bash
cd /root/Projects/bastion/ghost
docker compose up -d
```

Ghost will be available at `http://<server-ip>:2368` for testing before Nginx is configured.

To start with ActivityPub support (optional):
```bash
docker compose --profile activitypub up -d
```


## 6. Jenkins

### 6.1 Create host directories

```bash
mkdir -p /var/jenkins_home
chown -R 1000:1000 /var/jenkins_home  # jenkins user inside container runs as UID 1000
```

### 6.2 Build and start

```bash
cd /root/Projects/bastion/jenkins

# Build Jenkins image (includes Docker CLI and docker-workflow plugin)
docker compose build

# Start Jenkins
docker compose up -d
```

### 6.3 Get initial admin password

```bash
docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
```

Jenkins will be available at `http://<server-ip>:8080/jenkins` for initial setup.

### 6.4 Jenkins build setup (DooD)

Jenkins uses Docker-out-of-Docker: it spawns sibling containers on the host for builds.
Each project defines its own build image in the Jenkinsfile:

```groovy
pipeline {
    agent {
        docker { image 'node:20' }
    }
    stages {
        stage('Build') {
            steps {
                sh 'npm install && npm run build'
            }
        }
    }
}
```

See `doc/Jenkinsfile.example` for a full multi-stage example.

### 6.5 Pipeline durability and stuck builds

Jenkins has a known bug ([JENKINS-50407](https://issues.jenkins.io/browse/JENKINS-50407))
where pipeline flow executions can become corrupted. This happens when a build
is interrupted uncleanly — for example, by a timeout, manual abort, Jenkins
restart mid-build, or a disk issue. When it occurs, new pipeline builds will
silently fail: they show "Started by user ..." in the console output and
nothing else. The Jenkins log will contain warnings like:

```
WARNING o.j.p.w.f.FlowExecutionList$DefaultStorage#unregister:
<job>#<build> was not in the list to begin with: []
```

Two measures are in place to prevent and recover from this:

#### Prevention: Pipeline durability setting

By default, Jenkins aggressively writes pipeline state to disk at every step
so that running builds can be resumed after a crash. This persistence layer
is what gets corrupted. Since our builds run in ephemeral Docker containers
(DooD), there is nothing useful to resume — a fresh build is always needed.

Switching to "Performance-optimized" durability disables most of the state
persistence, which eliminates the main source of corruption and also makes
pipelines run faster.

After initial Jenkins setup:

1. Go to **Manage Jenkins** > **Configure System**
2. Find **Pipeline Speed/Durability Setting**
3. Change it to **Performance-optimized: much less durability**
4. Click **Save**

The tradeoff: if Jenkins crashes mid-build, the running build is lost and
cannot be resumed. This is acceptable because the Docker build container
is also lost on crash, so there is nothing to resume anyway.

#### Recovery: Startup cleanup script

The file `jenkins/init.groovy.d/clear-stuck-builds.groovy` is mounted into
the Jenkins container at `/var/jenkins_home/init.groovy.d/`. Jenkins
automatically executes all `.groovy` files in this directory on every startup.

The script iterates through all registered pipeline flow executions and
removes any that are marked as complete but are still tracked in the
execution list. This prevents stuck executions from blocking future builds.

If builds ever get stuck again (showing only "Started by user ..." with no
further output), simply restart the Jenkins container:

```bash
docker restart jenkins
```

The cleanup script will run automatically and clear the stuck state. No
manual deletion of build directories is needed.

### 6.6 Adding a permanent build agent (optional)

By default, Jenkins runs all builds on the controller node using Docker
containers (DooD). This works well for a single-user server. If you need
to scale (heavy builds making the UI unresponsive, builds on different
OS/architectures, or running many builds in parallel), you can add
dedicated build agents.

A **build agent** is a separate machine that Jenkins connects to and
delegates builds to. The controller handles scheduling and the UI, while
agents do the actual work.

#### Prerequisites on the agent machine

The agent machine needs:
- Java (same major version as the controller)
- SSH access from Jenkins
- A dedicated directory for Jenkins workspace (e.g., `/home/jenkins`)
- Docker (if you want to use Docker agents on the remote machine)

```bash
# On the agent machine
apt-get update && apt-get install -y default-jdk
useradd -m -d /home/jenkins jenkins
```

#### Configure the agent in Jenkins

1. Go to **Manage Jenkins** > **Nodes** > **New Node**
2. Enter a name (e.g., `build-agent-1`), select **Permanent Agent**, click **Create**
3. Configure:
   - **Remote root directory**: `/home/jenkins`
   - **Labels**: space-separated tags to route jobs (e.g., `linux x86_64 docker`)
   - **Usage**: "Use this node as much as possible" (or "Only build jobs
     with label expressions matching this node" if you want explicit routing)
   - **Launch method**: "Launch agents via SSH"
   - **Host**: the agent machine's IP or hostname
   - **Credentials**: add SSH credentials for the `jenkins` user on the agent
   - **Host Key Verification Strategy**: "Non verifying" for initial setup
     (switch to "Known hosts" for production)
4. Click **Save** — Jenkins will connect to the agent via SSH

#### Routing builds to agents

Use the `label` directive in your Jenkinsfile to target specific agents:

```groovy
pipeline {
    agent { label 'linux' }
    stages {
        stage('Build') {
            steps {
                sh 'make'
            }
        }
    }
}
```

You can combine labels with logical operators:

```groovy
// Run on a node that has BOTH labels
agent { label 'linux && docker' }

// Run on a node that has EITHER label
agent { label 'linux || macos' }
```

#### Disabling builds on the controller

Once agents are set up, you can stop running builds on the controller:

1. Go to **Manage Jenkins** > **Nodes** > **Built-In Node** > **Configure**
2. Set **Number of executors** to `0`
3. Click **Save**

Now all builds will be routed to agents only.


## 7. Nexus (Artifact Repository & Docker Registry)

### 7.1 Create host directories

```bash
mkdir -p /var/nexus-data
chown 200:200 /var/nexus-data  # Nexus runs as UID 200 inside container
```

### 7.2 Start Nexus

```bash
cd /root/Projects/bastion/nexus
docker compose up -d
```

Nexus takes ~2 minutes to start. Check logs with:

```bash
docker logs -f nexus
```

### 7.3 Get initial admin password

```bash
docker exec nexus cat /nexus-data/admin.password
```

Access Nexus at `http://<server-ip>:8081` for initial setup (before Nginx is configured),
or at `https://nexus.swave.lol` / `https://swave.lol/nexus` after Nginx is running.

Complete the setup wizard: set a new admin password and configure anonymous access.

### 7.4 Configure Docker hosted repository

After completing initial setup, create a Docker registry in Nexus:

1. Log into Nexus UI
2. Go to **Settings** (gear icon) > **Repositories** > **Create Repository**
3. Choose **docker (hosted)**
4. Configure:
   - **Name**: `docker-hosted`
   - **HTTP**: check the box, set port to **5000**
   - **Enable Docker V1 API**: leave unchecked
5. Click **Create Repository**

Docker clients can now use `registry.swave.lol` as the registry address:

```bash
# Log in
docker login registry.swave.lol

# Tag and push an image
docker tag my-image:latest registry.swave.lol/my-image:latest
docker push registry.swave.lol/my-image:latest

# Pull an image
docker pull registry.swave.lol/my-image:latest
```


## 8. Gerrit (Code Review)

### 8.1 Create host directories

```bash
mkdir -p /var/gerrit/{etc,git,db,index,cache}
chown -R 1000:1000 /var/gerrit  # gerrit user inside container runs as UID 1000
```

### 8.2 Start Gerrit

Gerrit is part of the git-server stack. It requires `jenkins_network` to exist,
so Jenkins must be started first.

```bash
cd /root/Projects/bastion/git-server
docker compose -f server.yaml up -d gerrit
```

Gerrit shares `/var/git/repos` with git-server for repository access.


## 9. Cockpit & Netdata (Monitoring)

See `cockpit/setup.md` for full Cockpit installation and configuration details.

### 9.1 Install Cockpit (native, on the host)

```bash
apt install cockpit cockpit-storaged cockpit-networkmanager
systemctl enable --now cockpit.socket
```

Configure Cockpit for subpath access. Edit `/etc/cockpit/cockpit.conf`:

```ini
[WebService]
Origins = https://swave.lol wss://swave.lol
ProtocolHeader = X-Forwarded-Proto
UrlRoot=/cockpit
```

```bash
systemctl restart cockpit
```

See `cockpit/setup.md` for full details.

### 9.2 Start Netdata (Docker)

```bash
cd /root/Projects/bastion/netdata
docker compose up -d
```

This creates `monitoring_network` (172.24.0.0/16) with Netdata at 172.24.0.2.

### 9.3 Add location blocks to nginx.conf

Add the contents of `cockpit/nginx-cockpit.conf` and `netdata/nginx-netdata.conf`
to the `swave.lol` HTTPS server block in `/var/nginx/conf/nginx.conf`.

After updating, reload nginx:

```bash
docker exec nginx nginx -s reload
```

Cockpit and Netdata will be available at:
- `https://swave.lol/cockpit/`
- `https://swave.lol/netdata/`


## 10. Nginx (Reverse Proxy + SSL)

Nginx must be started AFTER Git Server, Ghost, Jenkins, and Gerrit, because it joins
their networks as external.

### 10.1 Create host directories

```bash
# Nginx config directory
mkdir -p /var/nginx/conf

# Copy nginx.conf
cp /root/Projects/bastion/nginx/nginx.conf /var/nginx/conf/

# DH parameters (this takes a few minutes)
mkdir -p /var/dh_param
openssl dhparam -out /var/dh_param/dhparam-2048.pem 2048

# Let's Encrypt directories
mkdir -p /var/letsencrypt/etc
mkdir -p /var/letsencrypt/lts_site

# Copy the ACME challenge placeholder page
cp /root/Projects/bastion/nginx/letsencrypt/index.html /var/letsencrypt/lts_site/
```

### 10.2 First run — HTTP only (no SSL yet)

Before we have SSL certificates, we need to temporarily disable the SSL server
blocks so Nginx can start and serve the ACME challenge for certbot.

```bash
# Edit the nginx.conf copy on the host to comment out SSL server blocks
# Keep only the port 80 server block
nano /var/nginx/conf/nginx.conf
```

Comment out everything from `# Ghost — swave.lol` to the end of file.
Keep only the `listen 80` server block.

```bash
cd /root/Projects/bastion/nginx
docker compose up -d
```

### 10.3 Obtain SSL certificates

```bash
cd /root/Projects/bastion/nginx
bash run_certbot.sh
```

This requests certificates for:
- `swave.lol`
- `blog.swave.lol`
- `jenkins.swave.lol`
- `cgit.swave.lol`
- `gerrit.swave.lol`
- `nexus.swave.lol`
- `registry.swave.lol`

IMPORTANT: All DNS records must be pointing to the server before running certbot.

### 10.4 Enable SSL

```bash
# Restore full nginx.conf with SSL blocks
cp /root/Projects/bastion/nginx/nginx.conf /var/nginx/conf/

# Restart Nginx
cd /root/Projects/bastion/nginx
docker compose restart
```

### 10.5 Verify

All services should now be accessible:

| URL | Service | Auth |
|-----|---------|------|
| `https://swave.lol` | Ghost (blog) | Public |
| `https://blog.swave.lol` | Ghost (blog, alias) | Public |
| `https://swave.lol/jenkins` | Jenkins (path-based) | Authelia (one_factor) |
| `https://jenkins.swave.lol` | Jenkins (subdomain) | Authelia OIDC |
| `https://swave.lol/cgit` | Cgit (path-based) | Public |
| `https://cgit.swave.lol` | Cgit (subdomain) | Public |
| `https://swave.lol/gerrit` | Gerrit (path-based) | Authelia (one_factor) |
| `https://gerrit.swave.lol` | Gerrit (subdomain) | Authelia (one_factor) |
| `https://swave.lol/nexus` | Nexus (path-based) | Authelia (one_factor) |
| `https://nexus.swave.lol` | Nexus (subdomain) | Authelia (one_factor) |
| `https://registry.swave.lol` | Docker Registry (via Nexus) | Authelia (one_factor) |
| `https://swave.lol/cockpit/` | Cockpit (server admin) | Authelia (one_factor) |
| `https://swave.lol/netdata/` | Netdata (metrics) | Authelia (one_factor) |
| `https://auth.swave.lol` | Authelia (SSO portal) | — |
| `http://<server-ip>:9000` | Portainer | — |
| `ssh://git@<server-ip>/repos/<repo>.git` | Git (SSH) | — |
| `git://<server-ip>/<repo>.git` | Git (daemon, read-only) | — |


## 11. Authelia (SSO)

Authelia provides a unified authentication layer for protected services.
See `authelia/setup.md` for the full step-by-step guide.

Authelia must start **after** Netdata (which creates `monitoring_network`) and
**before** Nginx.

### 11.1 Quick start

```bash
# Create host directories
mkdir -p /var/authelia/{config,data,redis,secrets}

# Generate secrets (see authelia/setup.md for details)
openssl rand -hex 64 > /var/authelia/secrets/jwt_secret
openssl rand -hex 64 > /var/authelia/secrets/session_secret
openssl rand -hex 64 > /var/authelia/secrets/storage_encryption_key
openssl rand -hex 64 > /var/authelia/secrets/oidc_hmac_secret
openssl genrsa -out /var/authelia/secrets/oidc_rsa_key.pem 4096
chmod 600 /var/authelia/secrets/*

# Copy config and create users database
cp /root/Projects/bastion/authelia/config/configuration.yml /var/authelia/config/
cp /root/Projects/bastion/authelia/config/users_database.yml.example \
   /var/authelia/config/users_database.yml
# Edit /var/authelia/config/users_database.yml — fill in real password hashes

# Start Authelia
cd /root/Projects/bastion/authelia
docker compose up -d
```

### 11.2 Configured auth methods

| Service | Auth method |
|---------|-------------|
| Ghost | Public (no auth) |
| cgit | Public (no auth) |
| Netdata | Forward-auth (Authelia) |
| Cockpit | Forward-auth (Authelia) |
| Gerrit | Forward-auth + HTTP header (`X-Forwarded-User`) |
| Jenkins | OIDC (Authelia as provider) |
| Nexus | Forward-auth (Authelia) |
| Docker Registry | Forward-auth (Authelia) |

For Gerrit HTTP header auth and Jenkins OIDC plugin setup, see `authelia/setup.md`.


## Network Architecture

```
git-network (172.22.0.0/16)
├── git-server    (172.22.0.2)
├── cgit           (172.22.0.3)
├── gerrit         (172.22.0.4)
└── nginx          (172.22.0.254)

ghost_network
├── ghost
├── ghost-db
├── activitypub (optional)
└── nginx

jenkins_network (172.23.0.0/16)
├── jenkins        (172.23.0.2)
├── git-server     (172.23.0.3)
├── gerrit         (172.23.0.4)
└── nginx

nexus_network (172.25.0.0/16)
├── nexus          (172.25.0.2)
└── nginx

monitoring_network (172.24.0.0/16)
├── netdata        (172.24.0.2)
└── nginx

authelia_network (172.26.0.0/16)
├── authelia       (172.26.0.2, port 9091)
├── authelia-redis (172.26.0.3, port 6379)
└── nginx

Cockpit runs natively on the host (port 9090).
Nginx reaches it via host.docker.internal (host-gateway).
```

Nginx sits on all networks so it can reverse proxy to every service.


## Startup Order

The correct order to start all services:

```bash
# 1. Portainer (standalone, no dependencies)
docker run -d --name="portainer" --restart on-failure \
  -p 9000:9000 -p 8000:8000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data portainer/portainer-ce:latest

# 2. Git Server (creates git-network)
docker compose -f /root/Projects/bastion/git-server/server.yaml up -d

# 3. Ghost (creates ghost_network)
docker compose -f /root/Projects/bastion/ghost/compose.yml up -d

# 4. Jenkins (creates jenkins_network)
docker compose -f /root/Projects/bastion/jenkins/docker-compose.yaml up -d

# 5. Nexus (creates nexus_network)
docker compose -f /root/Projects/bastion/nexus/docker-compose.yaml up -d

# 6. Gerrit (joins git-network + jenkins_network, both must exist)
docker compose -f /root/Projects/bastion/git-server/server.yaml up -d gerrit

# 7. Netdata (creates monitoring_network)
docker compose -f /root/Projects/bastion/netdata/docker-compose.yaml up -d

# 8. Authelia (creates authelia_network)
docker compose -f /root/Projects/bastion/authelia/docker-compose.yaml up -d

# 9. Nginx (joins all networks — must be last)
docker compose -f /root/Projects/bastion/nginx/docker-compose.yaml up -d
```


## Ports Summary

| Port | Service | Protocol |
|------|---------|----------|
| 22 | Git Server (SSH) | TCP |
| 80 | Nginx (HTTP, redirects to 443) | TCP |
| 443 | Nginx (HTTPS) | TCP |
| 2368 | Ghost (direct, for testing) | TCP |
| 8080 | Jenkins (direct, for testing) | TCP |
| 9000 | Portainer (web UI) | TCP |
| 8081 | Nexus (direct, for testing) | TCP |
| 9090 | Cockpit (native, host only — proxied via nginx) | TCP |
| 9091 | Authelia (internal only — proxied via nginx) | TCP |
| 9418 | Git Server (git daemon) | TCP |
| 50000 | Jenkins (agent communication) | TCP |