Why ZFS Changes Everything About Linux Storage
Traditional Linux storage is a stack of independent layers. You have physical disks, partitions, maybe RAID, then LVM for volume management, then a filesystem like ext4 or XFS on top. Each layer has its own tools, its own failure modes, and its own limitations. When something goes wrong — silent data corruption, a failed disk, running out of space — you're juggling multiple abstractions that weren't designed to work together.
ZFS eliminates this complexity by integrating everything into a single coherent system. Volume management, RAID, filesystem, and integrity checking are all part of one codebase with one set of tools. The result is storage that's dramatically easier to administer while being more reliable than anything you can build with the traditional Linux stack.
On a Swiss dedicated server with multiple NVMe drives or a large HDD array, ZFS unlocks capabilities that are simply impossible with ext4 or XFS: instant snapshots that consume no space until data changes, built-in compression that's often faster than uncompressed I/O, automatic data integrity verification that catches silent corruption before it propagates to backups, and native replication that makes disaster recovery straightforward.
This guide covers ZFS deployment on Linux from first principles. We'll build pools, create datasets, configure snapshots and replication, tune performance, and cover the operational practices that keep ZFS healthy in production.
Installing ZFS on Linux
ZFS on Linux (OpenZFS) is available for all major distributions. The installation method depends on your distro, but the ZFS commands and concepts are identical everywhere.
Ubuntu/Debian:
# Ubuntu 20.04+ includes ZFS in the default repos
apt update
apt install zfsutils-linux
# Verify installation
zfs version
# zfs-2.1.5-1ubuntu6~22.04.2
# zfs-kmod-2.1.5-1ubuntu6~22.04.2
# Check kernel module is loaded
lsmod | grep zfs
# zfs 4161536 0
# zunicode 335872 1 zfs
# ... (several more modules)RHEL/Rocky/AlmaLinux:
# Add ZFS repository
dnf install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm
# Install ZFS (kABI-tracking kmod by default)
dnf install zfs
# Load the kernel module
modprobe zfs
# Enable auto-loading on boot
echo zfs > /etc/modules-load.d/zfs.conf
# Verify
zfs versionArch Linux:
# ZFS is in the AUR - use archzfs repo for binary packages
# Add to /etc/pacman.conf:
# [archzfs]
# Server = https://archzfs.com/$repo/$arch
pacman -S zfs-linux zfs-utils
# Enable services
systemctl enable zfs-import-cache.service
systemctl enable zfs-mount.service
systemctl enable zfs.targetAfter installation, verify that the ZFS kernel module version matches the userspace tools. Mismatches cause subtle problems:
# Check for version alignment
zfs version
# Both lines should show the same version number
# If mismatched after kernel update, rebuild DKMS
dkms autoinstallZFS Architecture: Pools, Vdevs, and Datasets
Understanding ZFS terminology is essential before creating storage. The hierarchy is straightforward once you see it:
Pool (zpool): The top-level storage container. A pool aggregates one or more vdevs into a single namespace. All storage capacity, redundancy, and I/O comes from the pool. Think of it as the entire storage system.
Vdev (virtual device): A building block within a pool. A vdev can be a single disk, a mirror of disks, a RAIDZ group (ZFS's software RAID), or a special device like a cache or log. The pool's redundancy comes from vdev configuration — lose a vdev and you lose the pool.
Dataset: A logical subdivision within a pool. Datasets are like directories but with independent properties (compression, quotas, snapshots). You create datasets for different purposes — one for databases, one for logs, one for user files — and configure each appropriately.
Zvol: A block device exported from a pool. Use zvols when you need a raw block device (for VMs, iSCSI targets, or applications that bypass the filesystem).
Here's the relationship visualized:
Pool: tank
├── Vdev: mirror-0 (disk1 + disk2)
├── Vdev: mirror-1 (disk3 + disk4)
├── Dataset: tank/databases (compression=lz4, recordsize=8K)
├── Dataset: tank/logs (compression=zstd, sync=disabled)
├── Dataset: tank/backups (compression=zstd-9, quota=500G)
└── Zvol: tank/vm-disk (volblocksize=64K, 100GB)Creating Your First Pool
Let's build a production-ready pool. We'll assume a bare metal server with four identical SSDs for the main pool and optionally an NVMe for cache acceleration.
Identify your disks:
# List all block devices with details
lsblk -d -o NAME,SIZE,MODEL,SERIAL,TRAN
# For production, always use disk-by-id paths (stable across reboots)
ls -la /dev/disk/by-id/ | grep -v part
# Example output:
# ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M123456 -> ../../sda
# ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M234567 -> ../../sdb
# ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M345678 -> ../../sdc
# ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M456789 -> ../../sdd
# nvme-Samsung_SSD_970_PRO_512GB_S463NF0M123456 -> ../../nvme0n1Choose a redundancy level:
• mirror: Two or more disks with identical data. Can lose n-1 disks per vdev. Best random I/O performance. 50% capacity efficiency (with 2 disks).
• raidz1: Single-parity RAID. Can lose one disk per vdev. Good sequential performance. ~75% capacity efficiency (with 4 disks).
• raidz2: Double-parity RAID. Can lose two disks per vdev. Better protection for large arrays. ~67% efficiency (with 6 disks).
• raidz3: Triple-parity. For very large arrays where multiple simultaneous failures are likely during rebuild.
For most dedicated servers, mirrored vdevs offer the best balance of performance, flexibility, and reliability. You can add capacity by adding mirror pairs, and the random I/O performance is excellent.
Create a mirrored pool:
# Create pool 'tank' with two mirror vdevs (4 disks total)
zpool create tank \
mirror \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M123456 \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M234567 \
mirror \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M345678 \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M456789
# Verify pool status
zpool status tank
# pool: tank
# state: ONLINE
# config:
# NAME STATE
# tank ONLINE
# mirror-0 ONLINE
# ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M123456 ONLINE
# ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M234567 ONLINE
# mirror-1 ONLINE
# ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M345678 ONLINE
# ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M456789 ONLINE
# Check available space
zpool list tank
# NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH
# tank 928G 624K 928G - - 0% 0% 1.00x ONLINECreate a raidz1 pool (alternative):
# Single raidz1 vdev with 4 disks (1 disk parity)
zpool create tank raidz1 \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M123456 \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M234567 \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M345678 \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_S3Z2NB0M456789
# This gives ~1.4TB usable from 4x 500GB drives (vs ~1TB with mirrors)Pool-Level Configuration
After creating the pool, configure essential properties:
# Set ashift appropriately for your drives
# 12 = 4K sectors (most modern SSDs/HDDs)
# 13 = 8K sectors (some enterprise NVMe)
# Check first: cat /sys/block/sda/queue/physical_block_size
# If 4096, use ashift=12; if 8192, use ashift=13
# NOTE: ashift must be set at pool creation time
# For a new pool: zpool create -o ashift=12 tank ...
# Enable autotrim for SSDs (discard support)
zpool set autotrim=on tank
# Verify pool properties
zpool get all tank | grep -E 'ashift|autotrim|version|feature'Add a fast cache (L2ARC) if you have spare NVMe:
# L2ARC caches frequently-read data on fast storage
# Only useful if your working set exceeds RAM
zpool add tank cache /dev/disk/by-id/nvme-Samsung_SSD_970_PRO_512GB_S463NF0M123456
# Verify cache device
zpool status tank
# ...
# cache
# nvme-Samsung_SSD_970_PRO_512GB_S463NF0M123456 ONLINE
# Check cache hit rate (after some usage)
zpool iostat -v tankAdd a separate log device (SLOG) for synchronous writes:
# SLOG accelerates sync writes (databases, NFS)
# Use a high-endurance, low-latency device
# Mirror it for safety — losing SLOG loses uncommitted writes
zpool add tank log mirror \
/dev/disk/by-id/nvme-Intel_Optane_P1600X_001 \
/dev/disk/by-id/nvme-Intel_Optane_P1600X_002
# Verify log device
zpool status tank
# ...
# logs
# mirror-2 ONLINE
# nvme-Intel_Optane_P1600X_001 ONLINE
# nvme-Intel_Optane_P1600X_002 ONLINECreating and Configuring Datasets
Datasets are where you organize your storage and apply specific settings. Create datasets for different workloads rather than using the pool root directly:
# Create datasets for different purposes
zfs create tank/databases
zfs create tank/logs
zfs create tank/backups
zfs create tank/vms
zfs create tank/home
# List all datasets
zfs list
# NAME USED AVAIL REFER MOUNTPOINT
# tank 1.2M 900G 96K /tank
# tank/backups 96K 900G 96K /tank/backups
# tank/databases 96K 900G 96K /tank/databases
# tank/home 96K 900G 96K /tank/home
# tank/logs 96K 900G 96K /tank/logs
# tank/vms 96K 900G 96K /tank/vmsConfigure compression:
ZFS compression is a no-brainer for almost all workloads. LZ4 is so fast that compressed I/O is often faster than uncompressed because you're moving less data to/from storage:
# Enable LZ4 compression (fast, moderate ratio)
zfs set compression=lz4 tank/databases
zfs set compression=lz4 tank/home
# Use ZSTD for archival data (slower, better ratio)
zfs set compression=zstd tank/backups
zfs set compression=zstd-9 tank/logs # Higher level for logs
# Check compression ratio after writing data
zfs get compressratio tank/backups
# NAME PROPERTY VALUE SOURCE
# tank/backups compressratio 2.84x -
# See actual vs logical space
zfs list -o name,used,logicalused,compressratioTune recordsize for your workload:
ZFS writes data in records (blocks). The default 128K works well for general-purpose workloads, but some applications benefit from tuning:
# Databases with small random I/O (PostgreSQL, MySQL InnoDB)
zfs set recordsize=8K tank/databases
# Sequential large files (video, backups, images)
zfs set recordsize=1M tank/backups
# VMs (match guest block size, typically 64K for Windows)
zfs set recordsize=64K tank/vms
# General files (default is fine)
zfs get recordsize tank/home
# NAME PROPERTY VALUE SOURCE
# tank/home recordsize 128K defaultSet quotas and reservations:
# Limit how much space a dataset can use
zfs set quota=500G tank/backups
zfs set quota=100G tank/logs
# Guarantee minimum space for critical datasets
zfs set reservation=50G tank/databases
# Check limits
zfs get quota,reservation,available tank/databasesControl sync behavior for performance-sensitive workloads:
# For logs where losing recent writes is acceptable
# (application handles durability elsewhere)
zfs set sync=disabled tank/logs
# For databases (keep default)
zfs get sync tank/databases
# NAME PROPERTY VALUE SOURCE
# tank/databases sync standard default
# With SLOG device, sync writes are fast anywaySnapshots: Point-in-Time Recovery
ZFS snapshots are one of its killer features. A snapshot captures the exact state of a dataset at a moment in time. Creating a snapshot is instantaneous and consumes no space initially — space is only used as the original data changes (copy-on-write).
# Create a snapshot
zfs snapshot tank/databases@before-migration
# List snapshots
zfs list -t snapshot
# NAME USED AVAIL REFER MOUNTPOINT
# tank/databases@before-migration 0B - 245M -
# Create recursive snapshot of all datasets
zfs snapshot -r tank@daily-2026-11-12
# List all snapshots recursively
zfs list -t snapshot -r tankAccess snapshot data:
# Snapshots are accessible via hidden .zfs directory
ls /tank/databases/.zfs/snapshot/
# before-migration
# Browse files as they existed at snapshot time
ls /tank/databases/.zfs/snapshot/before-migration/
# Copy a file from snapshot
cp /tank/databases/.zfs/snapshot/before-migration/important.sql /tmp/Rollback to a snapshot:
# Rollback discards all changes since the snapshot
# WARNING: This destroys data written after the snapshot
# First, stop any applications using the dataset
systemctl stop postgresql
# Rollback
zfs rollback tank/databases@before-migration
# For recursive rollback (all child datasets)
zfs rollback -r tank@daily-2026-11-12Clone a snapshot:
# Clone creates a writable copy sharing blocks with the original
# Perfect for development/test environments
zfs clone tank/databases@before-migration tank/databases-test
# The clone is independent and can be modified
# Check space usage
zfs list -o name,used,refer tank/databases tank/databases-test
# NAME USED REFER
# tank/databases 245M 245M
# tank/databases-test 0B 245M # Shares blocks until modified
# When done, destroy the clone
zfs destroy tank/databases-testAutomated snapshot management:
# Install sanoid for automated snapshot policies
apt install sanoid # or from GitHub
# Configure /etc/sanoid/sanoid.conf
cat > /etc/sanoid/sanoid.conf << 'EOF'
[tank/databases]
use_template = production
recursive = no
[tank/logs]
use_template = logs
recursive = no
[tank/home]
use_template = production
recursive = yes
[template_production]
hourly = 24
daily = 30
monthly = 12
yearly = 2
autosnap = yes
autoprune = yes
[template_logs]
hourly = 6
daily = 7
monthly = 0
yearly = 0
autosnap = yes
autoprune = yes
EOF
# Enable sanoid timer
systemctl enable --now sanoid.timer
# Check snapshot creation
zfs list -t snapshot | headZFS Replication for Disaster Recovery
ZFS can send snapshots incrementally to a remote system, creating an efficient backup and disaster recovery mechanism. This is fundamentally different from rsync — you're replicating the entire dataset state, including filesystem metadata, at the block level.
# Send a full snapshot to a remote server
zfs send tank/databases@before-migration | \
ssh remote-server zfs recv backup-tank/databases
# Send incremental changes (much faster for regular backups)
zfs send -i tank/databases@monday tank/databases@tuesday | \
ssh remote-server zfs recv backup-tank/databases
# Use compression during transfer
zfs send tank/databases@tuesday | \
zstd -c | \
ssh remote-server 'zstd -d | zfs recv backup-tank/databases'
# Resume interrupted transfers (ZFS 0.8+)
zfs send -t | ssh remote-server zfs recv backup-tank/databases Automated replication with syncoid (from sanoid):
# syncoid handles incremental send/recv automatically
syncoid tank/databases root@remote-server:backup-tank/databases
# With compression
syncoid --compress=zstd tank/databases root@remote-server:backup-tank/databases
# Recursive replication
syncoid -r tank root@remote-server:backup-tank
# Add to cron for regular replication
# /etc/cron.d/zfs-replication
0 * * * * root syncoid -q tank/databases root@dr-site:backup/databases
0 2 * * * root syncoid -qr tank root@dr-site:backupFor disaster recovery to a remote Swiss dedicated server, ZFS replication provides RPO (Recovery Point Objective) as low as your replication frequency — hourly snapshots mean at most one hour of data loss. RTO (Recovery Time Objective) is however long it takes to boot the DR server and import the pool.
Data Integrity and Self-Healing
ZFS checksums every block of data and metadata. When you read data, ZFS verifies the checksum and can automatically repair corruption if redundancy exists (mirrors, RAIDZ):
# Check pool health and error counts
zpool status tank
# Look for CKSUM errors (checksum failures = data corruption detected)
# Run a scrub to verify all data integrity
zpool scrub tank
# Monitor scrub progress
zpool status tank
# scan: scrub in progress since Wed Nov 12 10:00:00 2026
# 500G scanned at 2.1G/s, 200G issued at 800M/s, 928G total
# 0B repaired, 21.55% done, 0 days 00:15:30 to go
# Check scrub history
zpool history tank | grep scrub
# Schedule regular scrubs (weekly recommended)
# Already handled by zfs-scrub timer on most distros
systemctl status zfs-scrub@tank.timerHandling errors:
# If scrub reports errors, investigate
zpool status -v tank
# Shows which files have errors
# For repairable errors (mirror/RAIDZ)
# ZFS automatically repairs using redundant copy
# No action needed — just verify after scrub
# For unrepairable errors (no redundancy)
# Data is corrupt. Restore from snapshot or backup.
# Clear error counters after resolving issues
zpool clear tankPerformance Monitoring
ZFS provides detailed I/O statistics to help identify bottlenecks:
# Basic pool I/O statistics
zpool iostat tank 5
# capacity operations bandwidth
# pool alloc free read write read write
# ---------- ----- ----- ----- ----- ----- -----
# tank 200G 728G 152 234 12.3M 18.7M
# Per-vdev statistics
zpool iostat -v tank 5
# Shows I/O for each vdev, cache, log device
# Latency histograms (detailed performance analysis)
zpool iostat -l tank 5
# Shows read/write latency distribution
# ARC (memory cache) statistics
arc_summary
# Or manually:
cat /proc/spl/kstat/zfs/arcstats
# Key ARC metrics
grep -E 'hits|misses|size' /proc/spl/kstat/zfs/arcstats
# High hit ratio = good, working set fits in RAM
# Low hit ratio = consider more RAM or L2ARCTuning ARC (memory cache):
# Check current ARC size limits
cat /sys/module/zfs/parameters/zfs_arc_max
cat /sys/module/zfs/parameters/zfs_arc_min
# Set maximum ARC size (default is ~50% of RAM)
# For dedicated ZFS server, allow more
# For server running other services, reduce
echo 34359738368 > /sys/module/zfs/parameters/zfs_arc_max # 32GB
# Make persistent in /etc/modprobe.d/zfs.conf
echo 'options zfs zfs_arc_max=34359738368' >> /etc/modprobe.d/zfs.confExpanding and Modifying Pools
ZFS pools can grow over time as you add capacity:
# Add a new mirror vdev to expand capacity
zpool add tank mirror \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_new1 \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_new2
# Verify new capacity
zpool list tank
# Replace a failed drive in a mirror
zpool replace tank \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_failed \
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_new
# Monitor resilver (rebuild) progress
zpool status tank
# scan: resilver in progress since Wed Nov 12 14:00:00 2026
# 245G scanned at 1.8G/s, 180G issued at 1.5G/s
# 184G resilvered, 73.47% done, 0 days 00:01:45 to go
# Expand pool after replacing drives with larger ones
# (enable autoexpand or run online expansion)
zpool set autoexpand=on tank
# Or manually: zpool online -e tank /dev/disk/by-id/larger-diskImportant limitations:
• You cannot remove vdevs from a pool (with some recent exceptions for special vdevs)
• You cannot change RAIDZ to mirror or vice versa without recreating
• You can add vdevs but each vdev type (mirror vs raidz) remains fixed
• Plan your initial pool layout carefully
Importing and Exporting Pools
ZFS pools are portable. You can export from one system and import on another — useful for moving storage between servers or disaster recovery:
# Export pool (required before physically moving disks)
zpool export tank
# List importable pools (after moving disks to new server)
zpool import
# pool: tank
# id: 12345678901234567890
# state: ONLINE
# action: The pool can be imported using its name or numeric identifier
# config:
# tank ONLINE
# mirror-0 ONLINE
# mirror-1 ONLINE
# Import pool
zpool import tank
# Import with a different name
zpool import tank new-tank-name
# Import to alternate mount point
zpool import -R /mnt/recovery tank
# Force import if pool was not cleanly exported
zpool import -f tankZFS for Databases
Databases have specific requirements that ZFS can accommodate:
# PostgreSQL-optimized dataset
zfs create -o recordsize=8K \
-o compression=lz4 \
-o atime=off \
-o logbias=throughput \
-o primarycache=metadata \
tank/postgresql
# Move PostgreSQL data directory
systemctl stop postgresql
mv /var/lib/postgresql /tank/postgresql/
ln -s /tank/postgresql /var/lib/postgresql
chown -R postgres:postgres /tank/postgresql
systemctl start postgresql
# MySQL/MariaDB InnoDB-optimized dataset
zfs create -o recordsize=16K \
-o compression=lz4 \
-o atime=off \
-o primarycache=metadata \
tank/mysql
# For databases, consider disabling ZFS checksums
# if the database has its own checksumming
# (controversial — most keep ZFS checksums on)
# zfs set checksum=off tank/mysql # Only if you trust InnoDB checksumsDatabase backup with ZFS:
# Consistent backup via snapshot (PostgreSQL)
# 1. Start backup mode
psql -c "SELECT pg_backup_start('zfs-snapshot');"
# 2. Create snapshot
zfs snapshot tank/postgresql@backup-$(date +%Y%m%d)
# 3. End backup mode
psql -c "SELECT pg_backup_stop();"
# 4. Snapshot is now a consistent point-in-time backup
# Can replicate to remote: syncoid tank/postgresql@backup-... remote:...ZFS for Virtual Machines
ZFS works excellently as VM storage, either with zvols (block devices) or datasets for disk images:
# Create a zvol for a VM disk
zfs create -V 100G \
-o volblocksize=64K \
-o compression=lz4 \
tank/vm/windows-server
# The zvol appears as /dev/zvol/tank/vm/windows-server
# Use this in your hypervisor (KVM, bhyve, etc.)
# For qcow2 images on a dataset
zfs create -o recordsize=64K \
-o compression=lz4 \
tank/vm-images
# Snapshot a VM for backup
zfs snapshot tank/vm/windows-server@before-update
# Clone for testing
zfs clone tank/vm/windows-server@before-update tank/vm/windows-testProduction Checklist
Before going live with ZFS in production on your Swiss dedicated server, verify:
Hardware:
• ECC RAM strongly recommended (ZFS checksums detect corruption, but ECC prevents it)
• Use quality power supplies and consider UPS
• Use disk-by-id paths, not /dev/sdX names
• Match ashift to physical sector size
Configuration:
• Compression enabled on all datasets (lz4 minimum)
• atime=off on datasets that don't need access time tracking
• recordsize tuned for major workloads
• ARC size appropriately limited if sharing server with other services
• autotrim enabled for SSD pools
Monitoring:
• Weekly scrubs scheduled
• Pool health monitored (DEGRADED/FAULTED alerts)
• Disk space monitored (keep pools below 80% full)
• ARC hit ratio tracked
Backup:
• Automated snapshots via sanoid
• Off-site replication via syncoid
• Tested restore procedure documented
Conclusion
ZFS fundamentally changes what's possible with Linux storage. Features that require complex setups with traditional tools — snapshots, replication, data integrity verification, compression — are built in and work reliably. The learning curve is real, but once you understand pools, vdevs, and datasets, the operational simplicity is remarkable.
For production workloads on Swiss VPS or dedicated infrastructure, ZFS provides enterprise-grade storage without enterprise-grade complexity. The combination of data integrity, efficient snapshots, and native replication makes it ideal for databases, virtual machines, file servers, and any workload where data matters.
Start with a test pool on spare disks. Learn the commands. Practice snapshots and rollbacks. Once you're comfortable, migrate production data with confidence that ZFS will protect it better than anything else available on Linux.