Proxmox Cluster Behind Traefik — Reverse Proxy & Load Balancing

Overview

This guide covers exposing a Proxmox VE cluster through Traefik reverse proxy with a single domain (proxmox.fullmetaled.com). It addresses the common VNC/console WebSocket issues and provides three load balancing strategies.

Environment

Component	Detail
Proxmox Node 1	192.168.1.101:8006
Proxmox Node 2	192.168.1.102:8006
Proxmox Node 3	192.168.1.103:8006
Traefik	v3.6.6 (Docker)
Domain	prox.fullmetaled.com
TLS	Cloudflare Origin Certs

Architecture

┌──────────────────────────────────────────────────────────┐
│  Browser                                                 │
│  https://prox.fullmetaled.com                         │
└────────────────────────┬─────────────────────────────────┘
                         │
                    Cloudflare
                         │
                         ▼
┌──────────────────────────────────────────────────────────┐
│  Traefik                                                 │
│                                                          │
│  • TLS termination (Cloudflare origin certs)             │
│  • WebSocket passthrough (VNC/xterm.js consoles)         │
│  • Health checks against each Proxmox node               │
│  • Load balancing / failover                             │
│  • Sticky sessions (auth token stays on one node)        │
└──────┬───────────────────┬───────────────────┬───────────┘
       │                   │                   │
  ┌────▼─────┐       ┌────▼─────┐       ┌────▼─────┐
  │  Node 1  │       │  Node 2  │       │  Node 3  │
  │  :8006   │       │  :8006   │       │  :8006   │
  └──────────┘       └──────────┘       └──────────┘
        ◄── Cluster Comms (Corosync/SSH) ──►

---

The WebSocket Problem

Symptoms

• VNC/xterm.js console freezes after ~30 seconds
• Console requires constant page refreshes
• Console works fine when accessing Proxmox directly via IP
• TrueNAS shell resets when switching browser tabs

Root Cause

Traefik's default readTimeout and writeTimeout are finite values. WebSocket connections (used by VNC and xterm.js consoles) are long-lived and don't send HTTP-level data after the initial upgrade. Traefik interprets this as an idle connection and kills it.

Default behaviour (broken):
  0s   → WebSocket opens (101 Switching Protocols)
  30s  → No HTTP-level data (it's a WebSocket now)
  30s  → Traefik: "idle connection" → kills it
  30s  → Console freezes, user refreshes

With timeout fix:
  0s   → WebSocket opens (101 Switching Protocols)
  ∞    → Connection stays alive as long as both ends want it
  ✓    → Console works indefinitely

Fix

These timeout settings must be applied to the Traefik static configuration (docker-compose command args). They apply globally to the websecure entrypoint.

# Entrypoint timeouts — disables the WebSocket killing behaviour
- "--entrypoints.websecure.transport.respondingTimeouts.readTimeout=0"
- "--entrypoints.websecure.transport.respondingTimeouts.writeTimeout=0"
- "--entrypoints.websecure.transport.respondingTimeouts.idleTimeout=600s"

# Backend forwarding timeouts — same fix for the Traefik→Proxmox direction
- "--serverstransport.forwardingtimeouts.responseHeaderTimeout=0"
- "--serverstransport.forwardingtimeouts.idleConnTimeout=600s"

Note: Setting a timeout to 0 means disabled/infinite. This is safe — the underlying TCP connection and WebSocket protocol handle their own keepalives.

---

Traefik Static Configuration

docker-compose.yml

This is the full Traefik docker-compose with all WebSocket fixes included.

services:
  traefik:
    image: traefik:v3.6.6
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true

    networks:
      - traefik

    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./certs:/certs:ro
      - ./dynamic:/dynamic:ro
      - ./logs:/logs

    command:
      # ─── Entrypoints ───
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.web.http.redirections.entrypoint.permanent=true"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.websecure.http.tls=true"

      # ─── WebSocket / VNC Console Fix ───
      # Without these, VNC and xterm.js consoles freeze after ~30s
      - "--entrypoints.websecure.transport.respondingTimeouts.readTimeout=0"
      - "--entrypoints.websecure.transport.respondingTimeouts.writeTimeout=0"
      - "--entrypoints.websecure.transport.respondingTimeouts.idleTimeout=600s"
      - "--serverstransport.forwardingtimeouts.responseHeaderTimeout=0"
      - "--serverstransport.forwardingtimeouts.idleConnTimeout=600s"

      # ─── Backend TLS ───
      # Required for services with self-signed certs (Proxmox, TrueNAS, etc)
      - "--serverstransport.insecureskipverify=true"

      # ─── Providers ───
      - "--providers.file.directory=/dynamic/"
      - "--providers.file.watch=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=traefik"

      # ─── API & Dashboard ───
      - "--api.dashboard=true"
      - "--api.insecure=false"

      # ─── Logging ───
      - "--log.level=INFO"
      - "--accesslog=true"
      - "--accesslog.format=json"
      - "--accesslog.filepath=/logs/access.log"
      - "--metrics.prometheus=false"

      # ─── Cloudflare Trusted IPs ───
      # Required so real client IPs are visible (for CrowdSec, etc)
      - "--entrypoints.websecure.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22"

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik-dash.fullmetaled.com`)"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.middlewares=secure-access@file"

networks:
  traefik:
    name: traefik

---

Health Checks

Proxmox's API requires authentication. The health check path must be an unauthenticated endpoint.

Path	Status	Use for health check?
/	200 — login page HTML	✅ Yes
/api2/json	401 — requires auth	❌ No

healthCheck:
  path: "/"
  interval: "10s"
  timeout: "5s"
  scheme: "https"

When a health check fails, Traefik removes that node from the pool. When it recovers, Traefik adds it back automatically.

---

Sticky Sessions

Proxmox authentication generates a ticket (cookie) that is only valid on the node that issued it. Without sticky sessions:

Request 1 → Node1 → Login → Get auth ticket for Node1
Request 2 → Node3 → Auth ticket invalid → 401 → Broken

With sticky sessions, Traefik sets its own cookie that pins the browser to one backend:

Request 1 → Node1 → Login → Get auth ticket + sticky cookie
Request 2 → Node1 → Sticky cookie routes here → Works
Request 3 → Node1 → Same → Works
Node1 dies → Next request → Node2 → Re-login → New sticky cookie

sticky:
  cookie:
    name: "proxmox_node"
    secure: true
    httpOnly: true
    sameSite: "lax"

---

Shared Transport Configuration

All three methods below use the same serversTransport block. This handles the Traefik→Proxmox connection and must allow self-signed certs and long-lived WebSockets.

  serversTransports:
    proxmox-transport:
      insecureSkipVerify: true
      forwardingTimeouts:
        dialTimeout: "30s"
        responseHeaderTimeout: "0"
        idleConnTimeout: "600s"

---

Load Balancing Methods

Method 1: Failover (Primary + Fallback)

Best for: "Always use Node1, only fail over if it dies."

Node1 receives 100% of traffic. If it goes down, traffic moves to Node2/Node3. When Node1 recovers, traffic returns to it automatically.

Normal:     Browser → Node1 (always)
Node1 down: Browser → Node2 or Node3 (round-robin with sticky)
Node1 back: Browser → Node1 (automatic)

dynamic/proxmox.yaml

http:
  routers:
    proxmox-router:
      rule: "Host(`proxmox.fullmetaled.com`)"
      entryPoints:
        - websecure
      middlewares:
        - secure-access@file
      service: proxmox-failover
      tls: {}

  services:
    proxmox-failover:
      failover:
        service: proxmox-primary
        fallback: proxmox-secondary
        healthCheck: {}

    proxmox-primary:
      loadBalancer:
        servers:
          - url: "https://192.168.1.101:8006"
        healthCheck:
          path: "/"
          interval: "10s"
          timeout: "5s"
          scheme: "https"
        passHostHeader: true
        serversTransport: "proxmox-transport"

    proxmox-secondary:
      loadBalancer:
        sticky:
          cookie:
            name: "proxmox_node"
            secure: true
            httpOnly: true
            sameSite: "lax"
        servers:
          - url: "https://192.168.1.102:8006"
          - url: "https://192.168.1.103:8006"
        healthCheck:
          path: "/"
          interval: "10s"
          timeout: "5s"
          scheme: "https"
        passHostHeader: true
        serversTransport: "proxmox-transport"

  serversTransports:
    proxmox-transport:
      insecureSkipVerify: true
      forwardingTimeouts:
        dialTimeout: "30s"
        responseHeaderTimeout: "0"
        idleConnTimeout: "600s"

---

Method 2: Weighted (Priority + Spillover)

Best for: "Strongly prefer Node1 but allow occasional distribution."

Weights control the probability of initial placement. Once a sticky cookie is set, the user stays on that node regardless of weight. With 100:1:1, ~98% of new sessions go to Node1.

New session: 98% → Node1, 1% → Node2, 1% → Node3
After cookie: Always same node (sticky)
Node1 down:  Rebalance 1:1 between Node2 and Node3
Node1 back:  New sessions resume 100:1:1 distribution

dynamic/proxmox.yaml

http:
  routers:
    proxmox-router:
      rule: "Host(`proxmox.fullmetaled.com`)"
      entryPoints:
        - websecure
      middlewares:
        - secure-access@file
      service: proxmox-weighted
      tls: {}

  services:
    proxmox-weighted:
      weighted:
        services:
          - name: proxmox-node1
            weight: 100
          - name: proxmox-node2
            weight: 1
          - name: proxmox-node3
            weight: 1
        sticky:
          cookie:
            name: "proxmox_node"
            secure: true
            httpOnly: true
            sameSite: "lax"

    proxmox-node1:
      loadBalancer:
        servers:
          - url: "https://192.168.1.101:8006"
        healthCheck:
          path: "/"
          interval: "10s"
          timeout: "5s"
          scheme: "https"
        passHostHeader: true
        serversTransport: "proxmox-transport"

    proxmox-node2:
      loadBalancer:
        servers:
          - url: "https://192.168.1.102:8006"
        healthCheck:
          path: "/"
          interval: "10s"
          timeout: "5s"
          scheme: "https"
        passHostHeader: true
        serversTransport: "proxmox-transport"

    proxmox-node3:
      loadBalancer:
        servers:
          - url: "https://192.168.1.103:8006"
        healthCheck:
          path: "/"
          interval: "10s"
          timeout: "5s"
          scheme: "https"
        passHostHeader: true
        serversTransport: "proxmox-transport"

  serversTransports:
    proxmox-transport:
      insecureSkipVerify: true
      forwardingTimeouts:
        dialTimeout: "30s"
        responseHeaderTimeout: "0"
        idleConnTimeout: "600s"

---

Method 3: Round-Robin with Sticky Sessions

Best for: "Distribute evenly across all nodes."

Standard round-robin distributes new sessions equally. Sticky sessions keep each user on their assigned node. All nodes are treated as equal peers.

New sessions: Node1 → Node2 → Node3 → Node1 → ...
After cookie: Always same node (sticky)
Node dies:    Removed from pool, remaining nodes share load
Node back:    Re-enters rotation for new sessions

dynamic/proxmox.yaml

http:
  routers:
    proxmox-router:
      rule: "Host(`proxmox.fullmetaled.com`)"
      entryPoints:
        - websecure
      middlewares:
        - secure-access@file
      service: proxmox-service
      tls: {}

  services:
    proxmox-service:
      loadBalancer:
        sticky:
          cookie:
            name: "proxmox_node"
            secure: true
            httpOnly: true
            sameSite: "lax"
        servers:
          - url: "https://192.168.1.101:8006"
          - url: "https://192.168.1.102:8006"
          - url: "https://192.168.1.103:8006"
        healthCheck:
          path: "/"
          interval: "10s"
          timeout: "5s"
          scheme: "https"
        passHostHeader: true
        serversTransport: "proxmox-transport"

  serversTransports:
    proxmox-transport:
      insecureSkipVerify: true
      forwardingTimeouts:
        dialTimeout: "30s"
        responseHeaderTimeout: "0"
        idleConnTimeout: "600s"

---

Method Comparison

	Failover	Weighted	Round-Robin
Normal traffic	100% Node1	~98% Node1	33% each
Node1 down	Node2/3 round-robin	Node2/3 50/50	Node2/3 50/50
Node1 recovers	Immediate return	New sessions return	Re-enters rotation
Sticky sessions	On fallback only	All nodes	All nodes
Complexity	Medium	Higher	Simplest
Best use case	Dedicated primary	Soft preference	True HA

---

Verifying It Works

Check Health Status

# Traefik dashboard
# Navigate to traefik-dash.fullmetaled.com → HTTP Services → proxmox
# All nodes should show green

# Or via API
curl -s http://localhost:8080/api/http/services | python3 -m json.tool

Check WebSocket Connections

Open browser DevTools → Network tab → Filter: WS

Open a VM console and look for:
  ✅ Status: 101 Switching Protocols → WebSocket opened
  ✅ Connection stays open           → Timeouts fixed
  ❌ Connection closes after ~30s    → Timeouts not applied
  ❌ Status: 502                     → Backend unreachable

Check Traefik Logs

# Watch for health check warnings
docker logs -f traefik 2>&1 | grep -i health

# Healthy = no output (warnings stop)
# Unhealthy = repeated WRN lines for specific node URLs

Test Failover

# On node1 — simulate outage
systemctl stop pveproxy

# Wait 10s (health check interval)
# Refresh proxmox.fullmetaled.com
# Should land on node2 or node3

# Restore
systemctl start pveproxy

# New sessions should return to node1 (failover/weighted methods)

Test Direct Connectivity

# Verify each node responds (run from Traefik host)
for ip in 101 102 103; do
  echo -n "Node 192.168.1.$ip: "
  curl -sk -o /dev/null -w "%{http_code}" https://192.168.1.$ip:8006/
  echo
done

# Expected output:
# Node 192.168.1.101: 200
# Node 192.168.1.102: 200
# Node 192.168.1.103: 200

---

Troubleshooting

Symptom	Cause	Fix
Console freezes after ~30s	Traefik timeout killing WebSocket	Add readTimeout=0 and writeTimeout=0 to static config
Health check 401 errors	Health check path requires auth	Change path to /
502 Bad Gateway	Traefik can't reach backend	Check insecureSkipVerify: true and node firewall
Auth loops / random logouts	No sticky sessions	Add sticky.cookie to loadBalancer
Console works on IP, not domain	Timeout issue in proxy layer	Apply all timeout settings in this guide
Health check passes but console fails	Only entrypoint timeouts set, not forwarding	Add responseHeaderTimeout=0 to serversTransport