Wiki Page 1: SeaweedFS — Overview & Topology

What Is SeaweedFS

A distributed file system that provides shared storage across Docker Swarm nodes. It eliminates the need for NFS, GlusterFS, or manual file syncing. Any container on any node reads and writes the same files instantly.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     3-Node Docker Swarm                      │
│                                                              │
│  Node A                  Node B                Node C        │
│                                                              │
│  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐ │
│  │ master         │  │ master         │  │ master         │ │
│  │ volume         │  │ volume         │  │ volume         │ │
│  │ filer          │  │ filer          │  │ filer          │ │
│  │ s3 gateway     │  │ s3 gateway     │  │ s3 gateway     │ │
│  └───────┬────────┘  └───────┬────────┘  └───────┬────────┘ │
│          │                   │                   │           │
│  ┌───────▼───────────────────▼───────────────────▼────────┐  │
│  │              FUSE mount: /mnt/seaweed                  │  │
│  │           (systemd service on each NixOS host)         │  │
│  │                                                        │  │
│  │  All 3 nodes see identical filesystem contents          │  │
│  │  Write on A → readable on B and C within ~2 seconds    │  │
│  └────────────────────────────────────────────────────────┘  │
│                                                              │
│  Docker Swarm services bind-mount /mnt/seaweed/appname/      │
│  → works on any node, Swarm places services freely           │
└─────────────────────────────────────────────────────────────┘

Components — What Each Thing Does

Component	Role	Count	What Happens If It Dies
Master	Traffic coordinator. Decides where new files go. Holds no file data. Uses Raft consensus for leader election.	3 (one per node)	1 dies → new leader elected in ~3s. 2 die → writes stop, reads may work.
Volume Server	Actual disk storage. Stores files as packed blobs.	3 (one per node)	1 dies → replicated files still readable from other nodes.
Filer	Metadata server. Maps paths/filenames to volume locations. Backed by local LevelDB. Syncs metadata across filers via -filerGroup.	3 (one per node)	1 dies → FUSE clients failover to another filer in ~2-30s.
S3 Gateway	Translates S3 API calls into filer operations. Completely stateless.	3 (one per node)	1 dies → use another node's S3 endpoint.
FUSE Mount	Runs on the NixOS host (not a container). Makes the filer appear as /mnt/seaweed.	3 (one per node)	Restart systemd service. Docker services using the mount will hang until restored.

How Data Flows

WRITE: Container writes /mnt/seaweed/app/file.txt
  → FUSE mount catches write
  → sends to local filer
  → filer asks master "where should I store this?"
  → master says "volume server on node B"
  → filer sends data to node B's volume server
  → volume server replicates to node C (replication=001 = 2 copies)
  → filer stores metadata: "/app/file.txt → volume X, needle Y"
  → done (~milliseconds)

READ: Container reads /mnt/seaweed/app/file.txt
  → FUSE mount asks filer "where is this file?"
  → filer checks metadata: "volume X, needle Y, on node B"
  → FUSE fetches data from node B's volume server
  → returns to container

Two Ways to Access Storage

Method 1: FUSE mount (filesystem)
  /mnt/seaweed/myapp/data → bind mount into container
  Best for: apps that need a regular filesystem
  Used by: most services

Method 2: S3 API (HTTP)
  endpoint: http://<any-node>:8333
  Best for: apps with native S3 support (better performance)
  Used by: Nextcloud, Gitea LFS, Loki, etc.

Replication

Setting: 001 = 2 copies across the cluster (same rack)

Every file exists on 2 of 3 nodes.
Any single node can die without data loss.
Master decides which 2 nodes get each file (load balanced).

S3 Buckets and Filer Namespace

Everything lives in one unified namespace.
S3 and FUSE are two views of the same data.

S3 bucket "photos" with file "cat.jpg"
  → stored at filer path: /buckets/photos/cat.jpg
  → visible via FUSE at: /mnt/seaweed/buckets/photos/cat.jpg

Files written via FUSE at /mnt/seaweed/myapp/
  → visible via filer HTTP at http://<node>:8888/myapp/
  → NOT visible via S3 (unless written under /buckets/)

Ports

Port	Service	Protocol
9333	Master HTTP	HTTP
19333	Master gRPC	gRPC
9080	Volume HTTP	HTTP
19080	Volume gRPC	gRPC
8888	Filer HTTP + Web UI	HTTP
18888	Filer gRPC	gRPC
8333	S3 Gateway	HTTP

Web UIs

Master UI:  http://<any-node>:9333  — topology, volume status, leader info
Filer UI:   http://<any-node>:8888  — file browser, upload, create folders

Failure Scenarios

What Dies	Impact	Recovery
1 node	Full operation. Raft re-elects if leader died. FUSE fails over to another filer. Replicated data still available.	Automatic, ~2-30s
2 nodes	Writes stop (no Raft quorum). Last node has some data. Reads may work for files replicated to surviving node.	Manual — bring a node back
All 3 nodes	Cluster down. Data safe on disk.	Restart nodes, cluster reforms