[lug] High load with 'w'

[John Starkey]

On Tue, 2004-05-18 at 06:03, rm at fabula.de wrote: 
> Have you ever investigated what really causes the high load? Since you mention
> the DB server this might well be disk IO.

While there is a lot of overlap in our jobs, I'm actually not the sys
admin on these two servers, just the lowly webmaster ;) But, I'm sure
some of it's disk IO on the DB server (a 15k rpm drive). We're fine
until the spiders make it to the forum part of the site, which has 2.5
million posts, .5 million threads and .5 million members. Every page
request is at least 60 hits to the DB. Since our users are already
requesting approx 5 of those pages per second, add another 5 or so with
the spiders and you have 600 queries per second. Most of them use sorts
and joins. Most updates are low priority so they shouldn't be too bad.

There is also the problem of mass downloaders, but they're generally
stopped in a few pages.

> While high IO rates per se aren't causing
> high load (see a previous poster's comments on this) it most likely increases the
> average time a request takes to be served. This will result in a higher number of
> apache processes and that _will_ push up the load.

Yah, at the time of this writing (I always wanted to say that :D), there
are 1277 httpd processes running. We run two httpd binaries: 1 tweaked
for images and 1 for php pages. The load is at normal levels now, I'll
have to count the processes next time the harvestors hit us.

> Even worse, on older kernels
> every request will "wake up" all listening (read: idle and waiting for requests)
> apache children, so, unless you tweak you high/low marks for the number of
> servers, even a short time high number of parallel processes will result in a lot
> of (unneccessary) context switches (which again are bad for load and responsiveness).

We're using the latest 2.4s.

> BTW, i assume you have enough upstream bandwidth? Those 7Mbit look suspiciously 
> like an almost saturated 10Mbit cable ;-)

I was incorrect in my statement. We average at 16 Mbit per second and
peak at 37 Mbit. We have a 100Mbit dedicated going out and a shared with
about 160 other sites going in. So outbound saturation doesn't appear to
be an issue.

John