[lug] Sort-of-dead server

[Dan Ferris]

Did bash get nuked on the server?

Can you log in via console?

Dan

Jeff Schroeder wrote:

> Hey all,
> 
> I have a mysterious problem with a web+database server.  The network 
> connection is active, and using 'nmap' to see the running services 
> returns exactly what I'd expect:
> 
> # nmap 192.168.1.42
> 
> Starting nmap 3.30 ( http://www.insecure.org/nmap/ )
> Interesting ports on 192.168.1.42:
> (The 1638 ports scanned but not shown below are in state: closed)
> Port       State       Service
> 22/tcp     open        ssh
> 80/tcp     open        http
> 443/tcp    open        https
> 3306/tcp   open        mysql
> Nmap run completed -- 1 IP address (1 host up) scanned in 0.616 seconds
> 
> So my SSH, Apache, and MySQL servers appear to be running just fine.  
> But when I try to connect to the web site, or access the database, or 
> even SSH to the server, the connection hangs.
> 
> For example, here's an SSH request (with debugging output):
> 
> # ssh -v -v 192.168.1.42
> 
> OpenSSH_3.7.1p2, SSH protocols 1.5/2.0, OpenSSL 0.9.7c 30 Sep 2003
> debug1: Reading configuration data /etc/conf/ssh/ssh_config
> debug2: ssh_connect: needpriv 0
> debug1: Connecting to 192.168.1.42 [192.168.1.42] port 22.
> debug1: Connection established.
> .
> .
> debug1: Server accepts key: pkalg ssh-dss blen 433
> debug2: input_userauth_pk_ok: fp 
> a6:ae:b5:50:ec:d3:53:0f:48:03:6e:c1:f3:11:42:e2
> debug1: read PEM private key done: type DSA
> debug1: Authentication succeeded (publickey).
> debug1: channel 0: new [client-session]
> debug2: channel 0: send open
> debug1: Entering interactive session.
> 
> At this point it simply hangs, and nothing (including control-C) has any 
> affect.  Yet it appears the SSH channel has been opened, and the 
> session is ready.
> 
> Has anyone seen this sort of behavior?  I'm mystified because the 
> services respond to nmap's requests, but they don't do anything else.  
> I can't login to reboot, and sending "init 6" via SSH fails at the same 
> point as above.  The real problem is that failovers aren't working 
> because (1) the backup server thinks the primary is still running fine, 
> and (2) even if it did failover, I'd now have two machines with the 
> same virtual IP.  Bleah.
> 
> Any help is appreciated!
> 
> TIA,
> Jeff
> 
> _______________________________________________
> Web Page:  http://lug.boulder.co.us
> Mailing List: http://lists.lug.boulder.co.us/mailman/listinfo/lug
> Join us on IRC: lug.boulder.co.us port=6667 channel=#colug
>