No space left on device although df shows free space

What the message looks like

cp: cannot create regular file '/var/www/nextcloud/data/upload.tmp': No space left on device

Common causes

• All inodes used up by millions of small files (PHP sessions, cache, mail queue, Maildir).
• Deleted but still open files (e.g. logs) keep using space until the process closes them.
• Blocks reserved for root (5% by default on ext4): regular users already see the disk as full.
• A different filesystem is full, e.g. /tmp on tmpfs or a Docker volume.
• A user or group quota is reached.

Check

1. Space per filesystem
   df -hCopy
2. Inode usage per filesystem
   df -iCopy
3. Find deleted files that are still open
   sudo lsof +L1Copy
4. Find directories with the most files
   sudo du --inodes -x /var 2>/dev/null | sort -n | tail -n 15Copy
5. Show reserved blocks (ext4)
   sudo tune2fs -l /dev/sda1 | grep -i 'reserved block count'Copy

Fix

• Delete masses of small files in a targeted way, e.g. old sessions: find /var/lib/php/sessions -type f -mmin +1440 -delete.
• Restart the process holding a deleted file, or truncate it via : > /proc/<PID>/fd/<FD>.
• Configure logrotate properly (postrotate with reload or copytruncate) so open logs are not deleted.
• On pure data disks, lower the reservation: tune2fs -m 1 /dev/sdb1.
• Automate cleanup via cron and monitor inode usage.

FAQ

Can I increase the inode count afterwards?

Not on ext4 without reformatting. XFS and Btrfs allocate inodes dynamically. Long term, remove the cause of all the small files.

Why is no space freed after deleting the big log file?

The process still has the file open. The kernel only frees the space after the service is restarted or reloaded – lsof +L1 shows these cases.