[lug] Hacked SSH Daemon

[George Sexton]

George Sexton wrote:
> dio2002 at indra.com wrote:
>>
>> I'm curious about the trackback procedure you used to discern this. 
>> Obviously the size tipped you off and maybe you stopped right there.  But
>> if you used any other methods to trace the actions, including the logs
>> mentioned above, i'd like to know what steps you took and any logs you
>> found clues in (if you have the time).
> 

A little more information. It looks like the vulnerability was a 
directory traversal error that would allow the bad guy to read any 
arbitrary file. I found lots of entries where different machines 
retrieved the /etc/shadow file, and the ~/.bash_history file for all users.

The retrieval of the .bash_history files was kind of a shocker to me. I 
had a genuine "Oh, CRAP!" moment. Fortunately, mine didn't inadvertently 
reveal any passwords, but it did expose a few host names. It's clear 
that .bash_history is a double-edged sword. I think I'm going to add a 
cron-job to my systems to kill them after backup has run. If I need it 
for forensics, I can go to the backups.

Another real bummer to note is that the retrieval of /etc/shadow, 
.bash_history, and other files started some 4-5 months before the 
machine was ultimately gotten into.

It then appears that they found the root password in a lookup 
dictionary. The password was a dictionary word with one digit 
substituted for a letter.

I believe this because they appear to have logged into WebMin, and then 
using WebMin's shell command started the simple perl shell that gave 
them remote access to then install the trojaned ssh daemon.

I guess there are a few things to learn from this story.

1) If you're going to use packages that aren't supplied by your 
distribution you need to subscribe to the security mailing list so you 
don't get hurt.

2) Real, complex passwords are REALLY important, especially for ROOT 
accounts. My rule from now on is that passwords for root accounts will 
come from a random password generator.

3) Using unique passwords on each system is REALLY important. 
Fortunately, in this case I did. I'll admit. I haven't been real 
faithful about this rule in the past. I will be now.

4) Anytime ANYTHING hinky happens with SSH assume that you've been 
compromised.

5) It's possible that Tripwire would have caught it. Unfortunately, for 
a large number of systems, Tripwire is a total PITA to keep going. I 
might write a cron script to do a RPM -Va and have it check for modified 
binaries in the /sbin /bin /usr/bin and /usr/sbin directory. It wouldn't 
take as much maintenance as Tripwire, but it also wouldn't be as secure. 
Still, in this case, these guys were not even trying to cover their 
tracks and it would have found the issue right away.

6) Daemons and systems that have remote access components should have 
their logs monitored. I use logwatch, but it wasn't watching the webmin 
logs.




-- 
George Sexton
MH Software, Inc.
Voice: +1 303 438 9585
URL:   http://www.mhsoftware.com/