[lug] du vs df

[Michael J. Hammel]

On Mon, 2013-03-11 at 16:10 -0600, Zan Lynx wrote:
> If you redirect output to a file and then delete that file while the 
> original program is still running, that file is not actually deleted. 
...
> Another thing that can cause a disk to be full is the new btrfs 

Both good comments (as were the responses from Jose, Jeffrey and Bear).
The filesystem type is an easy check - /home is ext3.

The idea that a file isn't gone until both the fs inode and kernel
handles are cleared probably points to why du and df did not match.
Unfortunately, it doesn't say anything about how that missing file
filled up 119GB of disk space.  And then, when the file went away, that
space cleared out, in increments, very fast (but not immediately, as you
might suspect with clearing an inode).

A quick test:
  time dd if=/dev/zero of=testfile bs=1M count=1000
shows it takes ~20s to create the file, so a 119GB file would take about
40m. 

To delete it:
   time rm testfile
shows it takes less than 1/4s to remove the file, mostly because the
file isn't opened - it's inode is just removed.

So what I saw was what appeared to be a single file (perhaps multiple
files) that was still open but with no inode that was actually drained
of content while the file was still opened.  Which means there was some
process filling up a file with no inode and then stuck around to drain
it.  To my knowledge we have no such processes running on that machine.

Weird.  Ghosts in the machine?  Or just a jr. eng. with a knack for
confusing old farts like me.  I prefer the former explanation, of
course.

-- 
Michael J. Hammel <mjhammel at graphics-muse.org>