SeaweedFS — Technical Reference

Stack File

All services deployed via single Docker Swarm stack. Config and secrets managed by Swarm — no manual file distribution to nodes.

project/
├── seaweedfs.yml    ← stack file
├── filer.toml       ← filer config (Swarm config)
└── s3.json          ← S3 credentials (Swarm secret)

filer.toml

[leveldb2]
enabled = true
dir     = "/data/filer"

s3.json

{
  "identities": [
    {
      "name": "admin",
      "credentials": [
        {
          "accessKey": "YOUR_ACCESS_KEY",
          "secretKey": "YOUR_SECRET_KEY"
        }
      ],
      "actions": [
        "Admin",
        "Read",
        "Write",
        "List",
        "ReadAcp",
        "WriteAcp"
      ]
    },
    {
      "name": "some_service",
      "credentials": [
        {
          "accessKey": "SERVICE_ACCESS_KEY",
          "secretKey": "SERVICE_SECRET_KEY"
        }
      ],
      "actions": ["Read", "Write", "List"],
      "buckets": ["service-bucket-name"]
    }
  ]
}

seaweedfs.yml

Key design decisions:

• network_mode: host does not work in Docker Swarm (silently ignored)
• All services use overlay network + mode: host port publishing
• Every service needs explicit -ip=<host-ip> and -ip.bind=0.0.0.0
• Volume servers must be pinned per-node (not global) to set -ip correctly
• S3 credential file must be mode: 0444 (container may not run as root)

version: "3.8"

configs:
  seaweed_filer_config:
    file: ./filer.toml

secrets:
  seaweed_s3_config:
    file: ./s3.json

networks:
  seaweed:
    driver: overlay
    attachable: true

services:

  # ── Masters ─────────────────────────────────────────────────
  # Repeat for each node with correct -ip and placement constraint

  master-node1:
    image: chrislusf/seaweedfs:latest
    command: >
      master
      -port=9333
      -ip=NODE_1_IP
      -ip.bind=0.0.0.0
      -mdir=/data/master
      -peers=NODE_1_IP:9333,NODE_2_IP:9333,NODE_3_IP:9333
      -defaultReplication=001
    volumes:
      - /mnt/seaweed-data/master:/data/master
    networks:
      - seaweed
    ports:
      - target: 9333
        published: 9333
        protocol: tcp
        mode: host
      - target: 19333
        published: 19333
        protocol: tcp
        mode: host
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == NODE_1_HOSTNAME
      restart_policy:
        condition: on-failure
        delay: 5s
      resources:
        limits:
          memory: 256M
        reservations:
          memory: 64M

  # master-node2, master-node3: identical except -ip and hostname

  # ── Volume Servers ───────────────────────────────────────────
  # One per node, pinned, explicit -ip

  volume-node1:
    image: chrislusf/seaweedfs:latest
    command: >
      volume
      -port=9080
      -ip=NODE_1_IP
      -ip.bind=0.0.0.0
      -mserver=NODE_1_IP:9333,NODE_2_IP:9333,NODE_3_IP:9333
      -dir=/data/volume
      -max=0
      -minFreeSpacePercent=5
      -index=leveldb
    volumes:
      - /mnt/seaweed-data/volume:/data/volume
    networks:
      - seaweed
    ports:
      - target: 9080
        published: 9080
        protocol: tcp
        mode: host
      - target: 19080
        published: 19080
        protocol: tcp
        mode: host
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == NODE_1_HOSTNAME
      restart_policy:
        condition: on-failure
        delay: 10s
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 256M

  # volume-node2, volume-node3: identical except -ip and hostname

  # ── Filers ───────────────────────────────────────────────────

  filer-node1:
    image: chrislusf/seaweedfs:latest
    command: >
      filer
      -port=8888
      -ip=NODE_1_IP
      -ip.bind=0.0.0.0
      -master=NODE_1_IP:9333,NODE_2_IP:9333,NODE_3_IP:9333
      -filerGroup=homelab
    volumes:
      - /mnt/seaweed-data/filer:/data/filer
    configs:
      - source: seaweed_filer_config
        target: /etc/seaweedfs/filer.toml
        mode: 0444
    networks:
      - seaweed
    ports:
      - target: 8888
        published: 8888
        protocol: tcp
        mode: host
      - target: 18888
        published: 18888
        protocol: tcp
        mode: host
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == NODE_1_HOSTNAME
      restart_policy:
        condition: on-failure
        delay: 10s
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 128M

  # filer-node2, filer-node3: identical except -ip and hostname

  # ── S3 Gateways ──────────────────────────────────────────────

  s3-node1:
    image: chrislusf/seaweedfs:latest
    command: >
      s3
      -port=8333
      -filer=NODE_1_IP:8888
      -config=/run/secrets/s3_config
    secrets:
      - source: seaweed_s3_config
        target: s3_config
        mode: 0444
    networks:
      - seaweed
    ports:
      - target: 8333
        published: 8333
        protocol: tcp
        mode: host
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == NODE_1_HOSTNAME
      restart_policy:
        condition: on-failure
        delay: 10s
      resources:
        limits:
          memory: 256M
        reservations:
          memory: 64M

  # s3-node2, s3-node3: identical except -filer IP and hostname

Node Preparation

# On each node — create data directories
sudo mkdir -p /mnt/seaweed-data/{master,volume,filer}
sudo chmod -R 777 /mnt/seaweed-data/

NixOS FUSE Mount

{ config, pkgs, lib, ... }:

{
  environment.systemPackages = with pkgs; [ seaweedfs fuse3 ];
  programs.fuse.userAllowOther = true;

  systemd.tmpfiles.rules = [
    "d /mnt/seaweed 0777 root root -"
  ];

  systemd.services.seaweedfs-mount = {
    description = "SeaweedFS FUSE Mount";
    wantedBy    = [ "multi-user.target" ];
    after       = [ "network-online.target" ];
    wants       = [ "network-online.target" ];
    before      = [ "docker.service" ];

    serviceConfig = {
      Type      = "simple";
      ExecStart = lib.concatStringsSep " " [
        "${pkgs.seaweedfs}/bin/weed"
        "mount"
        "-filer=NODE_1_IP:8888,NODE_2_IP:8888,NODE_3_IP:8888"
        "-filer.path=/"
        "-dir=/mnt/seaweed"
        "-allowOthers=true"
        "-dirAutoCreate=true"
        "-cacheMetaTtlSec=2"
        "-cacheCapacityMB=256"
        "-umask=000"
      ];
      ExecStop    = "${pkgs.fuse3}/bin/fusermount3 -u /mnt/seaweed";
      Restart     = "on-failure";
      RestartSec  = "10s";
      KillMode    = "process";
      TimeoutStartSec = "60s";
    };
  };

  systemd.services.docker = {
    after    = [ "seaweedfs-mount.service" ];
    requires = [ "seaweedfs-mount.service" ];
  };
}

Deployment Commands

# Initial deploy
docker stack deploy -c seaweedfs.yml seaweed

# Update after config change
docker stack deploy -c seaweedfs.yml seaweed

# Rotating S3 credentials
docker stack rm seaweed
docker secret rm seaweed_s3_config
docker config rm seaweed_filer_config
# edit s3.json
docker stack deploy -c seaweedfs.yml seaweed

# Force restart a specific service
docker service update --force seaweed_filer-node1

Swarm Config & Secrets

Docker configs = non-sensitive, stored in plaintext in Raft log
  Used for: filer.toml

Docker secrets = sensitive, stored encrypted, decrypted only inside container
  Used for: s3.json (contains access keys)

Both are IMMUTABLE — to update, remove and recreate.
Both are read from local files at deploy time, then stored in Swarm.

Storage & Volumes

Terminology

Volume server = the process (one per node, runs as a container)
Volume        = a ~30GB file on disk managed by the volume server

These are NOT Docker volumes. These are SeaweedFS internal storage units.
One volume server manages many volumes automatically.

How Disk Space Is Used

/mnt/seaweed-data/volume/
├── 1.dat    ← volume 1 data (grows up to 30GB)
├── 1.idx    ← volume 1 needle index
├── 2.dat    ← volume 2 (auto-created when 1 filled up)
├── 2.idx
├── 3.dat
├── 3.idx
└── ...      ← keeps going until disk is full

Lifecycle

Volume 1 reaches 30GB
  → master marks it read-only
  → master auto-creates volume 2
  → new writes go to volume 2
  → no intervention needed

Node disk reaches 95% (minFreeSpacePercent=5)
  → volume server reports "full" to master
  → master stops assigning new volumes to that node
  → writes go to other nodes
  → existing data still readable

All nodes full
  → "No writable volumes" error
  → writes fail
  → add more disk space or a new node

Capacity Planning

# Check current usage
curl -s http://NODE_IP:9333/dir/status

# Output includes per-node:
#   "Max": 31      ← how many 30GB volumes fit on this disk
#   "Free": 29     ← unused volume slots
#   "Volumes": 2   ← in-use volumes

# Quick math:
#   1TB disk ÷ 30GB = ~31 volumes = ~930GB usable
#   3 nodes × 1TB = 3TB raw
#   replication 001 (2 copies) = ~1.5TB usable data

Volume Size

Default: 30GB per volume (set by master -volumeSizeLimitMB=30000)

Smaller volumes = faster vacuum/compaction, more files on disk
Bigger volumes  = fewer files, slower maintenance operations
30GB default is fine for homelab.

Running Out of Space

Option 1: Add bigger disk to existing node
  Stop volume server → migrate data → start server → master auto-detects

Option 2: Add a 4th node to the cluster
  Add volume-node4 to stack → master starts using it automatically

Option 3: Reclaim space from deleted files
  weed shell -master=NODE_IP:9333
  > volume.vacuum -garbageThreshold=0.3

  Deleted files leave gaps in .dat files.
  Vacuum compacts them and frees disk space.

You Never Need More Volume Containers

One volume server per node is all you ever need.
It manages all disk space on its mount point automatically.
More space needed → bigger disk or more nodes, not more containers.

Health Checks

# Master cluster status
curl -s http://NODE_IP:9333/cluster/status

# Volume servers registered
curl -s http://NODE_IP:9333/dir/status

# Filer responding
curl -s http://NODE_IP:8888/

# Volume assignment working
curl -s http://NODE_IP:9333/dir/assign

# Write test
curl -F "file=@/etc/hostname" "http://NODE_IP:8888/test/hostname.txt"

# Cross-node read test
curl http://OTHER_NODE_IP:8888/test/hostname.txt

# FUSE mount status
systemctl status seaweedfs-mount
mountpoint /mnt/seaweed

# Port check
sudo ss -tlnp | grep -E '9333|9080|8888|8333'

weed shell — Admin CLI

# Enter interactive shell
nix-shell -p seaweedfs
weed shell -master=NODE_IP:9333

# Useful commands
> volume.list
> volume.balance -force
> volume.fix.replication
> volume.vacuum -garbageThreshold=0.3
> s3.bucket.list
> fs.ls /
> fs.ls /buckets/

Troubleshooting

Symptom	Cause	Fix
curl returns empty	Master not elected / unreachable	Check IPs. Check firewall. Check Raft state.
exit (255) on master	Permission denied on data dir, OR bind address conflict, OR corrupted Raft	Check docker service logs. Clean /mnt/seaweed-data/master/* on all nodes.
exit (2) on any service	Invalid command flag	Check docker service logs. Run docker run --rm chrislusf/seaweedfs:latest <command> -help to see valid flags.
No writable volumes	Replication can't be satisfied (e.g. 010 but all nodes in same rack)	Change to 001 or set rack labels on volume servers.
DeadlineExceeded on filer write	Filer can't reach volume gRPC, OR volumes advertising overlay IPs	Ensure every service has explicit -ip=<host-ip>. Check dir/status — URLs should show real IPs not 10.0.x.x.
S3 help text printed then exit	Invalid flag in S3 command	Remove unsupported flags. Check weed s3 -help for valid flags.
S3 permission denied on secret	Secret mode too restrictive	Use mode: 0444 instead of 0400.
Volume host-mode port already in use	Another service (e.g. Traefik) using that port	Change volume port (e.g. 9080 instead of 8080).
network_mode: host silently ignored	Docker Swarm does not support network_mode: host	Use overlay network + mode: host port publishing.
FUSE mount stale data (15-60s delay)	Metadata cache TTL default is 60s	Set -cacheMetaTtlSec=2 on weed mount.
FUSE mount permission denied	Mount running as root, no umask set	Add -umask=000 to mount flags.
Corrupted Raft / single-node cluster	Masters started at different times or with wrong IPs	rm -rf /mnt/seaweed-data/master/* on all nodes, then redeploy.

Using SeaweedFS in Application Stacks

# FUSE mount approach — bind mount from host
services:
  myapp:
    image: myapp:latest
    volumes:
      - /mnt/seaweed/myapp/data:/app/data
    deploy:
      replicas: 2    # runs on any node

# S3 API approach — for apps with native S3 support
services:
  myapp:
    image: myapp:latest
    environment:
      S3_ENDPOINT: http://NODE_IP:8333
      S3_ACCESS_KEY: key_from_s3_json
      S3_SECRET_KEY: secret_from_s3_json
      S3_BUCKET: myapp-bucket

Important Caveats

PostgreSQL / databases:
  Do NOT put database data directories on FUSE mount.
  FUSE doesn't guarantee full POSIX semantics.
  Risk of data corruption.
  Pin databases to one node with local volumes.

SQLite:
  Same warning — don't use SQLite on FUSE mount.
  Single-writer apps only, and even then risky.

Large files:
  SeaweedFS chunks large files automatically.
  Performance is good but not as fast as local disk.

Backup:
  Replication is NOT backup.
  If you delete a file, the replica is also deleted.
  Implement separate backup strategy
  (S3 sync to offsite, ZFS snapshots, etc).