Homelab: Forgejo Registry & Patroni Troubleshooting

Quick reference for three issues hit on the Docker Swarm homelab (docker-nix worker @ 192.168.69.120, NixOS flake at /home/ed/flake).

1. Forgejo returns `401 Not authenticated (no forwarded identity)`

Symptom: http://192.168.69.120:3000 returns JSON 401, but Forgejo's own logs look healthy.

Cause: Not a Forgejo error. The Docker Swarm routing mesh publishes ports 3000/3001 (Pangolin) across every swarm node. The DOCKER-INGRESS iptables chain runs before the normal DOCKER chain in PREROUTING, so IPv4 traffic to :3000 is grabbed by the swarm service and never reaches the standalone Forgejo container. (IPv6/loopback weren't covered, so localhost→200 but the LAN IP→401.)

Diagnosis:

docker ps                                   # confirm forgejo container is Up
curl -i http://127.0.0.1:3000/              # via container path
sudo iptables -t nat -S | grep -iE "3000|DNAT"   # look for DOCKER-INGRESS -> :3000

Fix: Move Forgejo off the swarm-published port. Edit docker-compose.yml ("3300:3000") and app.ini ROOT_URL = http://192.168.69.120:3300/, then docker compose up -d. Forgejo now lives at :3300.

2. `docker pull` → `http: server gave HTTP response to HTTPS client`

Cause: Forgejo's registry serves plain HTTP; Docker defaults to HTTPS.

Fix: Mark the registry insecure on every Docker daemon that pulls from it, then restart Docker.

NixOS (/home/ed/flake/configuration.nix):

virtualisation.docker.daemon.settings.insecure-registries = [ "192.168.69.120:3300" ];

then sudo nixos-rebuild switch --flake .

Non-NixOS (/etc/docker/daemon.json):
```
{ "insecure-registries": ["192.168.69.120:3300"] }
```
then sudo systemctl restart docker

Also remember:

Open the port in the firewall (networking.firewall.allowedTCPPorts must include 3300). Tailscale nodes are exempt — tailscale0 is a trusted interface.
Re-tag any image still referencing :3000 → :3300.

3. Patroni replica stuck on `database starting` forever

Symptom: A patroni member's container starts but PostgreSQL never finishes coming up.

Cause: The replica's data dir is wedged (timeline divergence / half-applied WAL). Don't fight it — re-clone from the leader.

Before wiping: confirm a healthy leader exists elsewhere.

docker exec <patroni-container> patronictl list   # one node must show Leader / running

Recovery (on the broken node):

# 1. Old swarm task containers pin the volumes ("volume is in use" even when drained).
docker ps -a --filter volume=pg_patroni-2-data --format '{{.ID}} {{.Status}} {{.Image}}'
docker rm -f <those container IDs>

# 2. Delete the data + raft volumes
docker volume rm pg_patroni-2-data pg_patroni-2-raft
docker volume prune -f          # sweep orphaned anonymous volumes

# 3. From a MANAGER node, un-drain so Patroni re-clones from the leader
docker node update --availability active pg2

Prevention: Run docker container prune -f periodically on swarm nodes so dead task containers don't pin volumes.

Handy facts

docker-nix is a swarm worker — docker service/stack/node ls only work on a manager (pg* / traefik* nodes).
Swarm published ports are visible in iptables -t nat -S under DOCKER-INGRESS.
NixOS config is the flake at /home/ed/flake, applied with sudo nixos-rebuild switch --flake . (/etc/nixos is stale).

New Page