[lug] du vs df

[Zan Lynx]

On 03/11/2013 01:47 PM, Michael J. Hammel wrote:
> I ran into an interesting problem today.  We have hg configured on
> a /home partition and a user was reporting out of disk space when using
> it.  I checked with df -m and sure enough the partition was full.
>
> So I figured someone screwed up and filled up their home directory with
> junk.  I did a du -sm /home and found only about 40% of the partition
> size of data.  Du reported plenty of disk space.  df said the partition
> was full.  I don't expect them to be identical, just close.  du was 60%
> less than df (roughly) on a partition with about 220GB of total space.
>
> I'm assuming this was some kind of sparse file, though I'm not clear why
> it showed itself on /home.  /home is just for user data, including our
> mercurial and internal web repositories.
>
> While researching this a bit, I watched as df showed the partition being
> cleared of the extra usage.  Lots of space being freed up very fast.  du
> still showed the same thing.
>
> Any thoughts on what could make df think the partition was full but du
> not think that?  I'm trying to figure out what got us into that
> situation to begin with.  I suppose I could move the scm and web stuff
> to another partition but I'd like to know *how* this all came about.
>
> One possible event during this was a junior engineer was running a find
> on the entire filesystem looking for files with IP addresses in them
> (the boss asked if we logged web accesses, and that was his first idea
> to find out.  He has been properly flogged.).  I suppose if he was
> saving the output to a file, then deleted the file while the process
> ran, something bad might have happened.  Even that seems a bit of a
> stretch.
>
> Other than that, I'm not clear what might have been running to cause
> this.
>

If you redirect output to a file and then delete that file while the 
original program is still running, that file is not actually deleted. 
Not until the program writing output has finished.

Files have two link counts. A count of filename links on disk and a 
count of file handles in kernel. Only when those are both zero does the 
file actually disappear.

Another thing that can cause this is some filesystems delete files in 
the background. I believe EXT4 is one of these. The filename can be 
unlinked but the actual file blocks are not reclaimed immediately.

Another thing that can cause a disk to be full is the new btrfs 
filesystem. It works on B+ trees and if those trees get badly 
fragmented, the disk can be full while only a portion of it is actually 
used for files. The btrfs filesystem defrag command can fix that problem.