Security incidents are a complete disruption of my normal day-to-day activities. I love them. I especially like it when they uncover systemic problems we might not otherwise have found out about. We had one of those this week.
At issue: A server that contains Sarbanes-Oxley information is found to be infected by malware.
Action plan: Clean the server until it squeaks, and then get to the bottom of how that infection could have happened.
Our security information and event monitoring (SIEM) tool tipped us off, notifying us that malware had been detected, first on a server, and shortly thereafter on a file server. Those are not resources that should ever have occasion to be exposed to malware, and the fact that the server contained highly sensitive Sarbanes-Oxley information made the malware especially unwelcome -- and surprising, since we have special networks for segmenting our SOX data.
The Romanian Connection
The SIEM notification told us that some of our network traffic had attempted to communicate with a known command-and-control server in Romania. This too was odd, since Internet access from our production servers is supposed to be tightly restricted to little more than security and software update sites.
My first step, of course, was to disable the server switch port. Then I ordered an analysis of the system using EnCase, a very powerful forensics tool and a must for a security practitioner's toolbox. The forensic analysis concluded that the financial server was the point where the malware originated within our network. But how could that have happened, given the restrictions I've described? It turns out that an administrator who had logged on to the server to modify some scripts decided to launch Internet Explorer and log in to his personal webmail account. He then clicked on a phishing link, and you can imagine the rest.
We have retained a third party to analyze malware for us. Its report stated that the malware was programmed to communicate with a website in Romania over SSL (Port 443). The communication attempt that our SIEM observed was an initial check-in -- a heads-up to the command-and-control server that an installation had been successful. Here we got a break, because the Romanian server seemed to be offline. In fact, a lot of the cyberworld's bad guys activate their command-and-control servers for only a very short window.
The report also said that the malware had copied itself to open network file shares. Because the server was mapped to the finance department's file share, the malware was sitting in the root directory of that share, just as our SIEM had found. The server was checked from top to bottom, and we ran additional rootkit detection software to ensure that it was clean. Not wanting to take any chances with this server, I was ready to do a complete wipe and rebuild it from backups.
In the end, that wasn't necessary. Confident that the server was clean, I needed to find out how this incident could have happened in the first place. The answer came with a review of the firewall rules for the network: There were no outbound restrictions in place. They had been in place at one time, but when we moved from an on-premises data center to a hosted facility, some of the firewall rules didn't get migrated correctly. I had the network team close off the firewall hole by restricting outbound access from this network segment to the Internet.
Just as importantly, it seems to me that any systems administrator with privileged access to a critical resource should know that personal use of that system is not a good idea. But quite often, even people in positions of responsibility need training to understand things that just seem like common sense to a security manager. I will have to send an official reminder to the IT department not to use servers for activity other than official business needs.
This week's journal is written by a real security manager, "Mathias Thurman," whose name and employer have been disguised for obvious reasons. Contact him at firstname.lastname@example.org.