[lug] zfs-fuse failure/hanging

[Rob Nagler]

I was playing around with pushing a zfs file system to offline storage
with a send/receive. That was taking forever (this is a known problem)
so I put mbuffer in the middle.  It till was taking forever so I
killed (-TERM, not -KILL) the send/receive/mbuffer processes.  This
left around a temporary snapshot.  zfs-fuse was busy so I went to bed,
and waited till the morning.  I was able to delete the snapshot on the
receiving file system, but the next level up file system.  I got this:

# zfs destroy archive/20120715
connect: Connection refused
Please make sure that the zfs-fuse daemon is running.
internal error: failed to initialize ZFS library

At this point, the zfs-fuse daemon was dead.  It's important to
realize what this means: all files systems are unavailable.
Fortunately, nothing important was on any of them, just backups.

So then I tried tried to restart the server:

# service zfs-fuse start
Cleaning up stale zfs-fuse PID file in /var/run/zfs-fuse.pid
Starting zfs-fuse:                                         [  OK  ]
Immunizing zfs-fuse against OOM kills                      [  OK  ]
Mounting zfs partitions:

It's hung.  zfs list hangs.  The daemon is running consuming about 5%
of CPU.  Not sure what it's doing.  The disks are functioning (can dd
from them).   I tried to kill zfs-fuse with a TERM, but I don't want
to do a KILL.  I'll leave it running.  It seems to be reading from
disk.

Any ideas about what I should do?

Rob