Member Article

Hacked server? What should you do?

It’s a server administrator’s worst nightmare, and it’s one that almost every sysadmin will have to face at some point. Your server’s been hacked. Perhaps you’ve been notified by your hosting provider that it’s being used in a DDoS attack. Or you have noticed a spike in resource use that can’t be tied to legitimate activity. Or your malware scanners have located what looks like a backdoor. The sinking feeling on realising that your server has been exploited is horrible, especially when you consider the potential damage to your business, but it’s important that you deal with it rationally.

You next steps will depend on unique factors, but there is some general advice that I would give to everyone facing this scenario.

Keep Calm

Your reaction might be to start trying desperately to fix the problem, essentially applying band aids to your server in the hope that you’ll be able to mitigate the data loss or the impact immediately. It’s a natural reaction but it can cause more problems than it solves. To properly handle a hacked server, you need information, and acting without a full understanding of the situation can make things worse.

According to Eric Cole director of the SANS Cyber Defense Program at the SANS Institute, “It is easy to panic and start trying to control the damage. But very often, without a proper plan, you could actually be destroying evidence and making things worse.”

So, remain calm and try to develop an understanding of what exactly happened. Consult logs, find out what is running on your server, run malware scans, and, if you don’t have the expertise to assess the situation yourself, call in outside help.

First Steps

First, you want to deprive the attacker of their means of access to your server. Hopefully, you’ve been able to figure out how they got in, but fixing the immediate problem isn’t sufficient. If a hacker has been able to access your server, you can no longer trust any of its passwords or those of any other servers on your network.

You should implement a complete security overhaul, changing SSH and FTP passwords, email passwords, account passwords, and everything else that could conceivably been accessed by the attacker.

In many cases, you aren’t going to be able to avoid doing a complete reinstall of the server. If a hacker has remote root access and has managed to install a rootkit, nothing on the server can be trusted.

Implement The Fix

If you’re certain that your operating system hasn’t been exploited, and it’s just specific applications or scripts running on the server that are causing the problem, you may be able to get away with blowing away just that app and reinstalling.

As noted by Rimhosting:

The first thing most webapp exploits do is to install a backdoor. e.g. a URL they can check that re-enables their access, or a cron task that runs to re-add their access. So it may be best to do a reinstall of your server. And be very, very careful what content you bring over from your backups.

That last point is of particular importance. You should only reinstall from your backups if you are certain that the exploit hasn’t been backed up too. Use backups from before the hack. This is why it’s important to keep at least several months of backups.

Notify Users

This is probably the most difficult decision for many businesses, but if there’s a chance user data was leaked, you have an ethical duty to inform your users so that they can take steps to protect themselves.

Don’t Let It Happen Again

Being hacked is a risk we take whenever we put servers on the open Internet. However diligent you are as a server administrator, it’s possible for vulnerabilities in the software you rely on to sink your ship. You shouldn’t be too hard on yourself, but you should learn from the incident and implement processes so the same thing doesn’t happen again. And of course, if you weren’t sticking to basic security best practices like using secure passwords and ensuring that your server’s software was updated, hopefully you’ll learn that lesson too.

A hacked server is not something any of us want to deal with, but if you tackle it calmly, rationally, and transparently, there’s no reason it should hurt your business to badly.

About Graeme Caldwell—Graeme works as an inbound marketer for InterWorx, a revolutionary web hosting control panel for hosts who need scalability and reliability. Follow InterWorx on Twitter at @interworx, Like them on Facebook and check out their blog, http://www.interworx.com/community.

This was posted in Bdaily's Members' News section by Graeme Caldwell .