[lug] Monitoring file descriptors

[Bear Giles]

You can set max number of files via rlimit.  I've only done this in my 
own app, as part of security hardening, but I thought there was a way 
for root to set the values for any running process.

If you can find that, you can have a process that periodically sets the 
max file descriptor limit for every running process.  The rogue one will 
eventually get an EMFILE signal and you can see what dies.  Or at worst, 
just keep it from taking out the rest of the system.

Ben Whaley wrote:
> Folks,
>
> I had a CentOS box that was continually hanging every 2-3 days due to 
> some out of control process consuming all available file descriptors. 
> I've done a bunch of stuff now (patched, fixed config problems with 
> various services, etc) that may have helped, but I also wrote a bash 
> one-liner to monitor available file descriptors going forward. Thought 
> somebody might find this useful.
>
> if [ `cat /proc/sys/fs/file-nr | awk '{ diff=$3 - $1; print diff }'` 
> -lt 20000 ]; then echo "Low on file descriptors - `cat 
> /proc/sys/fs/file-nr | awk '{ diff=$3 - $1; print diff }'`" | mail -s 
> "ftp1 file descriptors" ben at atrust.com <mailto:ben at atrust.com>; fi
>
> To break it down:
>
> cat /proc/sys/fs/file-nr
> - file-nr contains the number of file descriptors in use and the total 
> number available (the limit is adjustable through 
> /etc/security/limits.conf on a Red Hat-like system)
>
> awk '{ diff=$3 - $1; print diff }'
> - This prints the difference between the total number of available 
> descriptors and the number currently in use
>
> - If the number available is less than 2,000 it alerts via email. I 
> run it out of cron every 5 minutes.
>
> Just trying to get us back on topic ;) Can't we all just be friends?
>
> - Ben
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: lug.boulder.co.us port=6667 channel=#colug