[lug] Sort-of-dead server

[Jeff Schroeder]

Hey all,

I have a mysterious problem with a web+database server.  The network 
connection is active, and using 'nmap' to see the running services 
returns exactly what I'd expect:

# nmap 192.168.1.42

Starting nmap 3.30 ( http://www.insecure.org/nmap/ )
Interesting ports on 192.168.1.42:
(The 1638 ports scanned but not shown below are in state: closed)
Port       State       Service
22/tcp     open        ssh
80/tcp     open        http
443/tcp    open        https
3306/tcp   open        mysql
Nmap run completed -- 1 IP address (1 host up) scanned in 0.616 seconds

So my SSH, Apache, and MySQL servers appear to be running just fine.  
But when I try to connect to the web site, or access the database, or 
even SSH to the server, the connection hangs.

For example, here's an SSH request (with debugging output):

# ssh -v -v 192.168.1.42

OpenSSH_3.7.1p2, SSH protocols 1.5/2.0, OpenSSL 0.9.7c 30 Sep 2003
debug1: Reading configuration data /etc/conf/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.1.42 [192.168.1.42] port 22.
debug1: Connection established.
.
.
debug1: Server accepts key: pkalg ssh-dss blen 433
debug2: input_userauth_pk_ok: fp 
a6:ae:b5:50:ec:d3:53:0f:48:03:6e:c1:f3:11:42:e2
debug1: read PEM private key done: type DSA
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Entering interactive session.

At this point it simply hangs, and nothing (including control-C) has any 
affect.  Yet it appears the SSH channel has been opened, and the 
session is ready.

Has anyone seen this sort of behavior?  I'm mystified because the 
services respond to nmap's requests, but they don't do anything else.  
I can't login to reboot, and sending "init 6" via SSH fails at the same 
point as above.  The real problem is that failovers aren't working 
because (1) the backup server thinks the primary is still running fine, 
and (2) even if it did failover, I'd now have two machines with the 
same virtual IP.  Bleah.

Any help is appreciated!

TIA,
Jeff