-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
 
Hi,
 
*NB* bear in mind that:
  a) the information on the break-in in comes from compromised machines
     and thus has to be taken with appropriate skepticism.
  b) the investigation is still ongoing - as I was writing this draft
     further information came to light which may invalidate a lot of
     it.  [Or not - as it turns out].
 
                              Detection
                              ---------
 
On November 20 it was noticed that master was kernel oops-ing
lots. While investigating this it was discovered that murphy was
showing the exact same oops, which was an overly suspicious
coincidence.  Also klecker, murphy and gluck have aide installed to
monitor filesystem changes and at around the same time it started
warning that /sbin/init had been replaced and that the mtime and ctime
timestamps for /usr/lib/locale/en_US had changed.
 
Investigation revealed the cause for both these things to be the
suckit root kit (see the "Suckit" appendix for more info).
 
 
                            What happened?
                            --------------
 
On Wednesday 19th November (2003), at approximately 5pm GMT, a sniffed
password was used to access an (unprivileged) account on
klecker.debian.org.  Somehow they got root on klecker and installed
suckit.  The same account was then used to log into master and gain
root (and install suckit) there too.  They then tried to get to murphy
with the same account.  This failed because murphy is a restricted box
that only a small subset of developers can log into.  They then used
their root access on master to access to an administrative account
used for backup purposes and used that to gain access to Murphy.  They
got root on murphy and installed Suckit there too.  The next day they
used a password sniffed on master to login into gluck, got root there
and installed suckit.
 
See the "Time-line" appendix for more details on times.
 
                               Response
                               --------
 
Gluck was powered down and an image has been made of it's disks for
forensic analysis.
 
Since we didn't have direct physical access to klecker it's Internet
connection was shut down and disk images were made via serial console
to a local machine on a firewalled net connection.
 
master and murphy were kept running for a short while in order to make
an announcement of the compromise, after which they were also taken
off-line and imaged.
 
                               Cleanup
                               -------
 
After a thorough cleanup and reinstall of modified files the non-US and
security archives were verified by looking at mirror logs for changes and
comparing MD5 checksums of the files on Klecker and those on three
different trusted mirrors.
 
Gluck, Master and Murphy were wiped and reinstalled from CD.  data and
services are in the process of being restored.
 
All machines and data were checked for devices outside of /dev, suid
executables, writable files, etc. and all suspicious files were
removed.  Services (and their scripts/programs) are being compared to
known-good sources and sanity checked before being re-enabled.
 
Since we now knew we had compromised accounts and sniffers on our
hands we had to assume that that an unknown number of accounts were
now compromised, so all accounts were locked, passwords invalidated
and ssh authorised keys removed.
 
                        How could this happen?
                        ----------------------
 
All the compromised machines were running recent kernels[1] and were
up-to-date with almost all security updates[2].
 
However there was two problems.
 
(1) The kernels running on the machines in question didn't all get a
    ptrace fixed kernel as fast one might have liked.  Master, Klecker
    and Murphy got new kernels in May but Gluck for various reasons
    didn't get upgraded till August (although I believe it had
    /proc/sys/kernel/modprobe fixed to at least block the most common
    exploit before that).
 
(2) Master had a copy it's old harddrive still lying around by
    accident.  Unfortunately it had a lot of old, unpatched suid
    binaries on it.
 
Although these could have been the attack vector, I don't believe they
were.  (2) seems unlikely simply because master wasn't, AFAWK, the
first host compromised.  Although it's possible an attacker with local
access to gluck got root through (1), it seems unlikely they'd sit on
that for <n> months and then use it on several machines only to
comeback and rootkit several debian.org machines and at least one
(that we know of) other unrelated system at the same time (and which
didn't have an extended ptrace vulnerability exposure.)
 
Based on that and the forensics on the unrelated system mentioned
above, I believe that there was an as of yet unknown local root
exploit used to go from having local unprivileged access to having
root.
 
                        Where do we go from here?
                        -------------------------
 
Unfortunately due to the fact there is (I believe) an unknown local
root exploit in the wild, we can't yet unlock the Debian accounts.
Obviously we can't continue without LDAP accounts for very long
either.  At the moment I'd ask for a little more patience both a)
while the painful and painstaking task of restoring machines one by
one is completed and b) while we try and exhaust all reasonable
avenues of investigation to determine how the attacker went from
unprivileged to root.
 
Obviously we're looking at hardening our boxes and tightening up our
procedures to try and stop this from happening again.  I'll send more
details on that later.
 
          Finally
          -------
 
Developers worried about their own machines might like to have a look
at:
 
  http://www.wiggy.net/debian/developer-securing/
 
================================================================================
 
                              Appendices
                              ~~~~~~~~~~
 
Thanks
- ------
 
 o Adam Heath and Brian Wolfe for their work on master & murphy.
 o Wichert Akkerman for his work on klecker.
 o Dann Frazier and Matt Taggart for their work on gluck.
 o Michael Stone and Robert van der Meulen for their forensics work.
 o Jaakko Niemi for his work on checking and re-enabling lists.debian.org.
 o Colin Watson for his work on checking and re-enabling bugs.debian.org.
 o Josip Rodin for his work on checking and re-enabling the lists web archives.
 
[This text is based on a draft by Wichert Akkerman.]
 
========================================
 
Time-line
- ---------
 
All times in GMT.
 
o Klecker init timestamp: Nov 19 17:08
o Master sk timestamp: Nov 19 17:47
o Murphy sk timestamp: Nov 19 18:35
o Oopses on Murphy start: Nov 19 19:25
o Oopses on Master start: Nov 20 05:38
o Gluck init timestamp: Nov 20 20:54
 
========================================
 
Suckit
- ------
 
Suckit is a rootkit which installs a sniffer, a process hider, a file
hider and a backdoor login in a running kernel. Apparently there was a
flaw in its kernel code which caused the kernel to oops on master and
murphy. This also explained why /sbin/init was replaced: the new init
loads suckit into the kernel and then proceeds to start the real init,
making sure that it is still active after a reboot.
 
========================================
 
Footnotes:
 
[1] Klecker: 2.4.22, Master & Murphy: 2.4.21-rc2, Gluck: 2.4.22rc2
 
[2] klecker was missing the latest postgresql updated.  ssh on all
    machines was a DSA-customized version which was missing only the
    3rd and final round (i.e. Solar Designer's patches) of ssh
    updates.
 
========================================
 
P.S. As always, I speak only for myself.
 
- --  
James
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8 <http://mailcrypt.sourceforge.net/>
 
iEYEARECAAYFAj/GnvoACgkQgD/uEicUG7BG8wCgriGB843o5TwmZZszRGlVPosL
b/0An2cpuoQEAaq4iMls0KlWrLcBlkEo
=i8am
-----END PGP SIGNATURE-----