Skip to content
IT related

Rclone

Updated 4 min read

On this page 13 sections
  1. Rclone Manual
  2. Overview
  3. When and Why to Use Rclone
  4. Basic Setup
  5. Quick Start: Common Commands
  6. Resume-Friendly Large File Downloads (example: Backblaze B2)
  7. Basic Uses
  8. Advanced Uses
  9. Tips and Tricks
  10. Troubleshooting
  11. Alternatives
  12. Practical B2 Example (End-to-End)
  13. References

Rclone Manual

Overview

Rclone is a command-line tool for managing files across cloud storage services and local filesystems. It supports Backblaze B2, S3-compatible storage, Google Drive, OneDrive, Dropbox, SFTP/FTP/WebDAV, and many more. Rclone excels at reliable transfers, resume, verification, encryption, mounting cloud storage, and automating syncs.

Key features:

  • Copy, sync, move, and check with retries and resume
  • Hash verification (uses cloud-native checksums where available)
  • Multi-threaded transfers for large files
  • Encryption (client-side) with the Crypt remote
  • Mount cloud storage as a filesystem (FUSE/VFS)
  • Serve files over HTTP, WebDAV, SFTP, FTP
  • Powerful filters and include/exclude rules
  • Cross-platform: Linux, macOS, Windows, BSD

When and Why to Use Rclone

Use rclone when you need:

  • Reliable, resumable transfers of very large files (e.g., multi-hundred-GB backups)
  • Cross-cloud or cloud-to-local sync with verification
  • Client-side encryption without vendor lock-in
  • Automation/scripting-friendly tooling
  • A single tool that works across multiple providers consistently
  • Ad-hoc downloads/uploads with filtering and bandwidth control
  • To mount cloud storage as a virtual drive

Rclone is not a full backup system by itself (no built-in deduplication or point-in-time snapshot set management). Pair with tooling like restic/duplicacy/borg if you need snapshot-based backups.

Basic Setup

You’ll need credentials for your cloud (e.g., Backblaze B2 Key ID and Application Key). It will show a list of the clouds you can add details for.

Two ways to configure:

  1. Quick, non-interactive
rclone config create myb2 b2 account YOUR_KEY_ID key YOUR_APP_KEY
  1. Guided
rclone config
# n) New remote
# name: myb2
# storage: b2
# account: YOUR_KEY_ID
# key: YOUR_APP_KEY
# accept defaults unless you have special needs
# y) Yes to save

Security tips:

  • Obscure a single secret:
    rclone obscure YOUR_APP_KEY
  • Encrypt your entire config (recommended on shared systems):
    rclone config
# Choose "s) Set configuration password"
    Non-interactive:
    export RCLONE_CONFIG_PASS='your-master-passphrase'

Quick Start: Common Commands

List buckets/remotes:

rclone lsd myb2:

List files:

rclone ls myb2:my-bucket | head

Copy a single file (local -> cloud or cloud -> local):

rclone copyto /path/file.zip myb2:my-bucket/file.zip --progress
rclone copyto myb2:my-bucket/file.zip /path/file.zip --progress

Copy a folder (no deletes on destination):

rclone copy myb2:my-bucket/snapshots /mnt/kar-data/snapshots --progress

Sync a folder (make destination match source: add/update/delete):

rclone sync myb2:my-bucket/snapshots /mnt/kar-data/snapshots --progress

Show sizes:

rclone size myb2:my-bucket/snapshots

Dry run (preview actions):

rclone sync ... --dry-run -v

Logs:

rclone copy ... -vv --log-file /var/log/rclone.log

Resume-Friendly Large File Downloads (example: Backblaze B2)

Rclone supports resuming partial downloads when the backend supports range requests (B2 does). You can safely rerun the same command and it will continue where it left off.

Single large file:

rclone copyto myb2:b2-snapshots-223239e41e5d/bzsnapshot_2025-09-24-15-15-06.zip \
  /mnt/kar-data/bzsnapshot_2025-09-24-15-15-06.zip \
  --transfers 1 \
  --multi-thread-streams 4 \
  --retries 999 --retries-sleep 30s \
  --progress

Notes:

  • Increase or decrease --multi-thread-streams depending on your disk/network (try 2–8).
  • You can rerun the command any time; rclone will resume and verify.
  • Verify integrity afterward:
    rclone check myb2:b2-snapshots-223239e41e5d/bzsnapshot_2025-09-24-15-15-06.zip \
  /mnt/kar-data/bzsnapshot_2025-09-24-15-15-06.zip -vv

B2-specific helpful flags:

  • Use a specific download host:
    --b2-download-url https://f000.backblazeb2.com
  • If you’ve set file versions and want “hard delete” on sync:
    --b2-hard-delete

Basic Uses

  • Copy vs Sync
    • copy: only adds/updates files at destination
    • sync: makes destination exactly match source (deletes extras)

Examples:

# Copy new/changed files to B2
rclone copy /data myb2:my-bucket/data --progress

# Sync local with B2 (warning: deletes at destination if not present locally)
rclone sync /data myb2:my-bucket/data --progress
  • Filters
# Include only *.zip and *.log, exclude others
rclone copy /data myb2:my-bucket/data --include "*.zip" --include "*.log" --exclude "*"

# Use a filter file
rclone copy /data myb2:my-bucket/data --filter-from filters.txt
  • Bandwidth limits and progress
rclone copy ... --bwlimit 80M --progress
  • Safe backups with versioned destination
# Copy, sending deleted/overwritten files to a backup dir with timestamp suffix
rclone sync /data myb2:my-bucket/data --backup-dir myb2:my-bucket/_deleted/$(date +%F) --suffix .del

Advanced Uses

Performance Tuning

  • Concurrency:
    • --transfers N: parallel files transferred
    • --checkers N: parallel metadata checks
  • Multi-threaded parts of a single large file:
    • --multi-thread-streams N
    • --multi-thread-cutoff 250M (default; below this size, use single-thread)

Example:

rclone copy /big myb2:bucket/big --transfers 8 --checkers 16 --multi-thread-streams 8 --progress
  • Retries and timeouts:
--retries 999 --low-level-retries 100 --retries-sleep 30s --timeout 5m --contimeout 30s

Verification and Checksums

  • Prefer checksum-based comparison when supported:
rclone copy src: dst: --checksum
  • Validate equality:
rclone check src: dst: -vv
  • Print checksums:
rclone sha1sum myb2:my-bucket/file.zip
rclone md5sum /path/file.zip

Encryption (Crypt Remote)

Encrypt data client-side before storing in the cloud.

Create a crypt remote on top of an existing remote path:

rclone config
# n) New remote
# name: secret
# storage: crypt
# remote: myb2:my-bucket/secure
# Choose passphrase(s)

Use it like any other remote:

rclone copy /secure-data secret: --progress

Mount Cloud Storage (VFS/FUSE)

Mount a remote as a filesystem (Linux/macOS require FUSE; Windows uses WinFsp).

rclone mount myb2:my-bucket /mnt/b2 \
  --daemon \
  --vfs-cache-mode full \
  --dir-cache-time 72h --vfs-cache-max-age 72h \
  --vfs-cache-max-size 50G \
  --transfers 8 --multi-thread-streams 8 \
  -vv --log-file /var/log/rclone-mount.log

Unmount: fusermount -u /mnt/b2 (Linux), umount /mnt/b2 (macOS), or close the process on Windows.

Bi-directional Sync (bisync)

Keep two folders in sync both ways (use with care; test first).

rclone bisync local/dir myb2:my-bucket/dir --resync --check-access
# Subsequent runs:
rclone bisync local/dir myb2:my-bucket/dir --filters-file filters.txt

Serving Files

Expose a remote via standard protocols.

  • HTTP:
    rclone serve http myb2:my-bucket --addr :8080 --read-only
  • WebDAV:
    rclone serve webdav myb2:my-bucket --addr :8081 --user USER --pass PASS
  • SFTP/FTP similarly available via rclone serve sftp/ftp.

Server-Side Transfers

When both source and destination are on the same cloud provider (or supported backends), rclone uses server-side copy/move to save bandwidth:

rclone copy myb2:bucketA/path myb2:bucketB/path

Backend-Specific Options (B2 examples)

  • Choose region endpoint:
    --b2-endpoint https://s3.us-west-004.backblazeb2.com
  • Preferred download URL:
    --b2-download-url https://f000.backblazeb2.com
  • Hard delete on sync (remove previous versions):
    --b2-hard-delete

Tips and Tricks

  • Always start with --dry-run when testing new filters or sync directions.
  • For single-file operations, use copyto/moveto to explicitly name the destination file.
  • Preserve timestamps/metadata by default; use --metadata on some remotes that support it.
  • Reduce memory usage on constrained systems by lowering concurrency and streams.
  • Long paths/odd characters: quote your paths or use --files-from with a file list.
  • Log rotate: run with --log-file and use your OS log rotation.
  • Use restricted/lifecycle-managed keys on your cloud for safety.
  • Schedule tasks with systemd timers, cron, or Windows Task Scheduler.

Example systemd unit:

# /etc/systemd/system/rclone-backup.service
[Unit]
Description=Nightly rclone backup

[Service]
Type=oneshot
ExecStart=/usr/bin/rclone sync /data myb2:my-bucket/data --log-file /var/log/rclone-backup.log -vv

Timer:

# /etc/systemd/system/rclone-backup.timer
[Unit]
Description=Run rclone backup nightly

[Timer]
OnCalendar=03:00
Persistent=true

[Install]
WantedBy=timers.target

Troubleshooting

  • Quota or rate limit errors: increase --retries, back off with --retries-sleep, and reduce concurrency.
  • Too many open files: lower --transfers/--checkers or raise OS limits (ulimit -n).
  • Slow mounts or “file busy”: tune VFS flags, especially --vfs-cache-mode full and cache sizes.
  • Permission denied on mount: ensure user has rights to mountpoint; on Linux add user to fuse group; on Windows install WinFsp.
  • Unexpected deletes with sync: always run --dry-run and consider --backup-dir to preserve removed files.

Alternatives

Depending on your use case:

  • Backblaze B2 CLI (b2/backblaze-b2)
    • Native B2 operations; simple and supported by Backblaze.
    • Limited resume/robustness vs rclone for very large single-file downloads.
  • MinIO Client (mc)
    • S3-compatible CLI; good for S3 providers (B2 S3 API, AWS, Wasabi).
  • AWS CLI / s5cmd
    • For S3; s5cmd is extremely fast for massive object counts.
  • AzCopy (Azure), gsutil (Google Cloud Storage)
    • Optimized for their ecosystems.
  • rsync
    • Excellent for local/SSH, but not cloud object stores.
  • aria2, wget, curl
    • Great for HTTP(S) downloads with resume; limited cloud metadata/checksums.
  • GUI clients
    • Cyberduck, Mountain Duck, Transmit, WinSCP: user-friendly, less scriptable.
  • Snapshot backup tools
    • restic, duplicacy, borgbackup, duplicity: versioned backups with deduplication; can use B2/S3 as backends.

Practical B2 Example (End-to-End)

  1. Configure:
rclone config create myb2 b2 account YOUR_KEY_ID key YOUR_APP_KEY
  1. Resume-friendly large file download:
rclone copyto myb2:b2-snapshots-223239e41e5d/bzsnapshot_2025-09-24-15-15-06.zip \
  /mnt/kar-data/bzsnapshot_2025-09-24-15-15-06.zip \
  --transfers 1 --multi-thread-streams 4 \
  --retries 999 --retries-sleep 30s --progress
  1. Verify:
rclone check myb2:b2-snapshots-223239e41e5d/bzsnapshot_2025-09-24-15-15-06.zip \
  /mnt/kar-data/bzsnapshot_2025-09-24-15-15-06.zip -vv

References
IT related RSS feed Published with Oblish

Keyboard shortcuts

/
Search
[
Previous page
]
Next page
?
Show this list
Esc
Close