Incident note: No space left on device despite free space

Read the signal

Start with the exact wording of the error and its timestamp in the log. Then narrow down the cause before changing anything.

cp: cannot create regular file '/var/www/nextcloud/data/upload.tmp': No space left on device

Narrow down possible causes

• All inodes used up by millions of small files (PHP sessions, cache, mail queue, Maildir).
• Deleted but still open files (e.g. logs) keep using space until the process closes them.
• Blocks reserved for root (5% by default on ext4): regular users already see the disk as full.
• A different filesystem is full, e.g. /tmp on tmpfs or a Docker volume.

Check before changing anything

1. Space per filesystem

   df -h

2. Inode usage per filesystem

   df -i

3. Find deleted files that are still open

   sudo lsof +L1

4. Find directories with the most files

   sudo du --inodes -x /var 2>/dev/null | sort -n | tail -n 15

Fix and test again

1. Delete masses of small files in a targeted way, e.g. old sessions: find /var/lib/php/sessions -type f -mmin +1440 -delete.
2. Restart the process holding a deleted file, or truncate it via : > /proc/<PID>/fd/<FD>.
3. Configure logrotate properly (postrotate with reload or copytruncate) so open logs are not deleted.
4. On pure data disks, lower the reservation: tune2fs -m 1 /dev/sdb1.

Then repeat the original action and compare the result with the initial error.

Keep a note for the next ticket

Record the exact error, the check result, the change and the retest. That gives the team a useful starting point next time.

Can I increase the inode count afterwards?

Not on ext4 without reformatting. XFS and Btrfs allocate inodes dynamically. Long term, remove the cause of all the small files.

Full error guide: No space left on device although df shows free space