[lug] Linux boxes drop off the net? Router problem? Answers t o your questions.

[Gary Frerking (TurboPower)]

Thanks much to the people who have pitched in with questions/suggestions so
far. I'll address the questions inline in one lump here.

*** From: Kevin Fenzi <kevin at scrye.com> ***

>> This happens to all your linux boxes? <<

Yes.

>> at the same time? or diffrent times? for how long? <<

Not positive. I'll do some more checking to see if I can quantify the
problem further.

>> do they come back themselves? or have to be rebooted or network
restarted? <<

They come back by themselves. I'm not sure what prompts them to come back.

>> what happens when you are on one of the affected boxes when it's not
talking to the outside? can you ping out? is the network interface up? are
the routes correct? <<

Yes, yes, yes.

>> I think the first thing you should do is gather more information and see
if that leads you to a cause. <<

Agreed. Part of my problem is I don't know the best info to gather. Ntop was
suggested elsewhere, so I'll get that going. I'm also getting a shell
account set up on one of our ISP's machines so I can do some testing from
there.

>>Perhaps you can install something like netsaint or big brother and monitor
the connectivity of all the linux machines? This would tell you how often it
happens, how long the outages last, and if they happen at the same time. <<

I'll look into those, thanks for the tips.

>> You said that the linux boxes don't have identical hardware. Could they
be similar enough to where they are all using the same ethernet driver for
instance? (ie, the tulip driver works on a vast array of ethernet cards). <<

I don't think so. Most of the boxes are running Intel EtherExpress NICs, but
a couple of them are running ancient generic NE2000s.

*** From: "Sebastian Sobolewski" <spsobole at mindless.com> ***

>> 1. If you go to a Linux box that has "disappeared" and ping a remote site
like yahoo. Can you now ping that Linux box externally? This would indicate
the switch or router loosing track of the machine. In this case you may want
to set up your linux boxes to do a ping once a minute and watch and see if
they still drop of the network. <<

I'll give that a go, thanks.

>> 2.  From my understanding the win2k boxes and Linux boxes are
intermingled on the same hubs, But are all the Linux boxes on the same
subnet, which is 
different from the one the win2k boxes are on? A different subnet would
point to a routing problem. <<

Same subnet.

>> 3.  Are you using the VLAN features of the LinkSwitch 1000? Your VLAN's
may be miss configured. (If you have VLAN support of this should not be an
issue)

>> 4. Does your network run a DHCP Server? <<

Yes. But all dynamic IPs come from a different Class C block than the static
IPs on the network.

>> 5. Are you running any "strange" services on your linux boxes.  Check
/var/log/messages for any entries specifying that your Ethernet is in
promiscuous mode.  If you find it most likely means your boxes have been
compromised. <<

No. All boxes are checked periodically for things like that. I have a script
that I run from an internal machine looking for promiscuous mode, and also
NMAPs my boxes looking for unexpected open ports (there are only 2 or 3
ports open on each machine). I also run Tripwire on the boxes and monitor
CPU usage for anything out of the ordinary. I'm pretty confident the
machines are not compromised.

>> 6. I'm not familiar with the 2516 but you may want to check if it's been
getting hit by any denial of service attacks.  The 2516 may be blocking port
access after sensing a DOS.  (Router filter rules)  However that should ALSO
cause your win2k boxes to disappear. <<

We checked this -- my SysAdmin says this isn't happening.

*** From: "D. Stimits" <stimits at idcomm.com> ***

>> I haven't seen this problem myself, but you might be interested in
installing "ntop" <<

Yep, good idea. I'm familiar with ntop -- I'll get it going and see if it
gives us any useful info.


-- Gary