[mrtg] MRTG and the viruse

[John McKay]

We originally received the virus this morning at about 5:30 pdt (8:30 edt).  We have two aging 7000's, each with 22 lan interfaces, one runs ipx for novell connectivity, the other, only ip, for a total user community of over 2000.  The one running ipx has always been an indicator of overall network health, just by looking at the cpu utilization. This morning, I got a call from Net Ops about 7:00 pdt, so I checked the cpu utilization, and I could see that a problem was in the making, as it was over 75%.  I then looked at the bytes in/out on the lan interfaces, they looked ok, so I looked at the packets/second in/out, and they were extremely high, over 2000/second in/out at some points on some interfaces.  I grabbed a person from Networking Computing as I was heading downstairs to Net Ops.  We plugged in the sniffer into the span port of one of the segments , and we began to sort of see what was happening, but clueless as to exactly what. At this time, we called for more people to help, as we could see, this was going to get bloody, as we were still an hour and a half away from the time that the majority of our users login into our systems.  We ran down a couple of mac the addresses, found the arp entry in the ip router, did a quick lookup in the dns to get the user alias, then to the email directory to get the user name and location.  Our hardware analyst then sprinted over to the users to investigate.  Our worst fears were confirmed....  the users had opened it and got the message.  Of course, many other users did also during the next few hours, adding to the problem.  We finally learned in more detail what the virus was doing to our lan.... destroying data on all the connected drives... our file servers. Once we learned the identity of the virus, the proper filter was applied to the mail server, but the damage was already done.  Even after this, we still had users opening it up.  By the end of the day, we had all the workstations reboot with the virus scan fix, and identified all the files that needed to be restored on our servers.  All in all, it was a busy day, involving all of our net ops, lan ops, network computing, and network services staffs to remove this virus.  Obviously, we have some cleanup to do, and some education to the user community about opening email of unknown origin and suspicious subject matter. 

Thanks to have been running MRTG for over 2 years, we have an extremely good feel of what is going on, and know when things are about to go south. Without MRTG, we would have had no clue to where to begin to look for this problem, or any other problem.  THANKS Tobie!

p.s.  the 7000's have about one month to live.  they are in the process of being replaced with new cisco switches. the next items we will be replacing are the 5200's and older pix boxes, which will bring us up to current technology.

--
Unsubscribe mailto:mrtg-request at list.ee.ethz.ch?subject=unsubscribe
Archive     http://www.ee.ethz.ch/~slist/mrtg
FAQ         http://faq.mrtg.org    Homepage     http://www.mrtg.org