The plan was to get Christmas out of the way then get back to writing and being generally productive in a form that didn’t involve trips to IKEA and hours assembling office furniture.
It wasn’t to be.
In mid-December both the Win2k3 servers were hit by a zero-day Trojan. This was despite protection that included a hardware firewall and live-updated anti-virus and anti-spyware software on both servers, plus similar protection down to packet level within the mail server software. The first sign I had that anything was wrong was slow Internet connections. When I checked, the activity indicators on the router and modem were solidly lit – not the normal case round here. Just as I was pondering the cause an email arrived from my ISP advising me that my network had been reported as a source of spam … Aaaaarrrrrggghhhhh!!!!
Sure enough, both servers had been compromised and were spraying out junk mail at a rate of several hundred messages per minute. In the few gaps this left in my outbound bandwidth, they were also sending out port-probes methodically working their way through several IP sub-nets at the command of their new masters looking for other systems that the Trojan could compromise.
Now, despite all the protection in place here (protection I’d still say is among the best available) a successful attack like this is a risk run by anyone with a computer connected to the ‘net. Especially servers as they generally sit on fast pipes connected directly into the ‘net so can do the most work for the crooks behind the Trojan. My own servers sit on a domestic ADSL line with upstream bandwidth of only around 500-800kbps (my main web servers are hosted elsewhere) so they lucked out with me but even so, they managed to get several tens of thousands of spam messages out before I discovered and could solve the problem.
And the cost …!!
Like many people these days, I rely on email – as do the family and friends who also have email addresses here. So just “pulling the plug” was not an option unless I had a way of getting at least the email server back on line without too much delay.
More importantly, the other server also provided a raft of network services to all the other machines so taking it down causes everything else to collapse with, if delayed, still inevitable certainty. Not good. Having identified the nature of the beast, my first approach to fixing the problem was to download the removal tools and patches from the AV vendors and MS. So … both machines taken off the network (internal and external) removal software run – check OK – then patches installed.
Reconnect to Internet and ….
… BOOM! The blasted Trojan was back again!
Several hours later and it was obvious that the Trojan removal software had not done its work. Despite its efforts and mine (painstakingly trawling through system files all over a large server is NOT fun) it had managed to conceal some part of its payload somewhere that would take more time than I had or wanted to spend playing detective to find.
At this point, I want to restate that tired old advice to everyone who has or uses a computer anywhere for any purpose. Back it up. In fact, do more than that. Back it up in a way that allows you to go back and restore it to any point in time. If you’re sensible, you’ll back up the backup too.
Thankfully, that’s pretty much what happens here. I’ve used Acronis to back up both servers and PCs to a large NAS server at least nightly with other specialised software archiving the email repository and other fast-updating files in almost real time.
Plan B swung into effect. Machines taken off the Internet and restored to the point a few hours before the Trojan struck. Next the patches were installed to prevent the crooks getting back in the hot seat. Finally, the email and other databases were rolled forward to their last current position and, with a deep intake of breath, the servers were put back on-line …
Plan B worked. The servers stayed clean – though monitoring the firewall saw several thousand attempts by other distributed servers to talk to the now exterminated Trojans.
No comments:
Post a Comment