This is the 191st article in the Spotlight on IT series. If you'd be interested in writing an article on the subject of backup, security, storage, virtualization, mobile, networking, wireless, DNS, or MSPs for the series PM Eric to get started.
I’m 60 years old, and I love fairy tales. Yes, you read that one right: Some of my favorite stories are ones where some witch or a wizard seeks revenge for some slight, real or imagined. Then, a hero defeats the bad guy and everyone lives happily ever after.
Funny thing is, if you stop and think about it, we are living a bit of a fairy tale. We have people who run companies and wield power, and then there are the wizards behind the scenes — those who run complex networks and computer systems, keep communications systems going and are responsible for tens of thousands of bits of data every day. We call these wizards "system administrators," and much like the wizards of old, some will seek revenge if they’re slighted— and not everyone lives happily ever after.
The issue is system administrators are not normal employees! We’re the guys who make things go. Many are the tales of admins who refuse to give up passwords or use their position to create chaos. Let’s look at the kind of vengeance a sys admin can wreak (and what precautions we can take to limit or stop it).
Three strikes: A true tale of sys admin vengeance
Unlike once-upon-a-time stories, this tale is true and happened just a couple years ago.
Jason Cornish, 37, was an employee of Shionogi Pharmaceuticals in Smrya, Georgia. He’d been with the company for some years, knew the network and how it was put together, and worked for his good friend — a person court documents refer to as “B.N.” (Being a former undercover detective, I know when I see something like that, whoever this initialed individual B.N. is, he probably turned on his friend like a rat in a cheap mafia movie)
As the story goes, one day Cornish got into it with a senior manager. It must have got real ugly because he wasn’t employed there much longer after that. Well, enter B.N. He convinced his bosses that Cornish was too good an asset to just let go, and he suggested they keep him on board on a consulting basis. This B.N. guy must have been a really smooth talker because the company did just that. In baseball terms, we call this strike one.
A few months go by, and the company begins having issues. A massive layoff begins, and B.N. finds himself being among those being let go. In an effort to forestall, sweeten the severance deal, or whatever, he decides holding the network passwords hostage would be a good idea. Instead of being laid off, he was simply fired. Strike two
Here comes strike three: A few more months pass, and the FBI documentation at this point shows that Cornish downloaded VMware’s vSphere Client and attempted to gain access to the Shionogi network several times. Then, on a nice winter day in February 2011, he went to a McDonald’s and used their Wi-Fi to launch his final attack. Records show he got into several ESXi systems and deleted several virtual machines. Essentially, he destroyed their infrastructure with a few clicks of his mouse.
Cornish had planned his attack so that if they traced it (which they did), they’d come back to the McDonald’s. With so many customers every hour, he’d be lost in the crowd. However, Parker’s Law states, "Criminals get caught because they don’t plan on getting caught.” While he did plan on being hidden in plain sight, what he didn’t count on was that once a place comes up on the radar of investigation, the FBI doesn’t just walk away — not until there’s nothing more to learn. They went through everything, including food sales, and wouldn’t you know it, there, minutes before the attack, was a credit card purchase made by Cornish for under $5.
Now they had someone with the knowhow, the motivation and the ability to put the pieces in place and conduct the attack. Cornish confessed, and he's currently taking some time off from IT work in federal prison. B.N., who apparently turned in evidence, has slipped off the radar, and for all we know, he’s reading this post right now.
What’s the point of this story? The entire Cornish incident showcases things not to do.
Sys admins aren’t normal
First and foremost on the list of what went wrong: They treated Cornish like a normal, everyday employee. The company seemed to have given little thought to what a sys admin handles and what might be in his head. But even if a sys admin is leaving under good conditions, there are things we should do beyond simply disabling the account.
Probably the single biggest mistake they made was they brought someone with a grudge back inside. In short, they gave Cornish and B.N. the opportunity to set the stage for the attack. Secondly, they put too much power into the hands on one individual, someone who apparently hadn’t shared or made known to the bosses the passwords that ran everything.
But the biggest issue was something they didn’t do. Dealing with a potential rogue sys admin starts a long time before the decision is ever made to let them go. I’d argue that watching within the perimeter is even more important than watching what happens outside the wire.
This sounds overly paranoid, but, hey, it’s kept me alive for years: “Just because you're paranoid doesn’t mean they aren’t out to get you.” A little paranoia when applied to security is a good thing. Often times the enemy isn’t outside, he’s on the inside — he's someone we trust. So how do we keep an eye on things?
First, we have to be aggressive in how we handle Active Directory and Change Management. Any system is only as good as the accountability built into it, so let’s look at these three factors.
1. Active Directory — Many people get burned in audits because they have AD accounts that are disabled but still hanging around. While we can find all kinds of justification to keep an account, we need to look at them through the lens of security. Ideally, no more than 30 days after a person leaves their account should be deleted, their email dealt with, and so forth.
2. Killing zombie accounts — Next up, we need to watch for “zombie accounts,” accounts that no one has logged into in X amount of days. Now, there are a number of commercial products that will go through and generate a report for you, but you can also do this with built-in tools such as DSQuery and DSGet for free. (This link will give you chapter and verse on the subject.)
But unless you’ve got it scripted, it can turn into a hassle, so I recommend a little automation. Find tools such as Netwrix Active Directory Change Reporter that will generate the information for you. If you don’t have a lot of money, you might take a look at Quest Software’s PowerGUI Administrator. There are add-ins that work great for generating the reports for you.
3. Closing the back door — We also have service accounts; if you stop and think about it, these are heaven-built for an attacker. After all, all they need to do is take one, remember the password, mess with permissions, and you have a back door into the network. One thing we need to consider is keeping these to a minimum, and have every one of them identified and labeled.
Keep an eye on Active Directory. You should be noticing anytime a new account is generated, modified, deleted — whatever. Again, there are a lot of tools that will do this for you, such as the aforementioned Active Directory Change Reporter. It, like so many others, will send you an email every day telling you what’s happened. A good practical exercise to do once in a while is to match the report against tickets. Then, anything out of the norm needs to be questioned.
Preparing to fire a sys admin
Now a lot of people look at this as a waste of time, but then the firing of a system admin is nothing more than an exercise in risk management. Most people are professional enough to just leave, but you just never know about people, so there are some things we need to do if we’re getting ready to give a sys admin the axe.
First, we should make sure they haven’t left anything behind, like a dead man script. This is where a white-hat hacker can be a big asset. These guys know what to look for, and the best time to do bring one in is when the administrator is off on vacation (worst case scenario, arrange a vacation for them). Also, it’s very doubtful that they’ll set it up so that things will fall apart; they expect to come back to work. Any dead man scripts stand a really good chance of being identified by a white-hat. If nothing is found, it’s a good bet they don’t exist.
Another thing a white-hat can help you out with is finding backdoor accounts or tripwires that might be set up. The white-hat should also look through logs — since often would-be rogue sys admins will have started setting things up. And, while you should have the passwords to your network, there are ways to crack passwords if you don’t. Another thing the white-hat can help you with is to find those services or functions that might be running under the sys admin's account.
If you’re not already doing it, you should be auditing your network for changes. We’re looking for things like accounts in AD started or changed. Another thing to keep an eye on is your event logs — if they’re cleared for no reason, someone may be trying to cover their tracks on something. Event IDs to watch for are Event ID 517 and 1102. (Randy Smith’s site Ultimate Windows Security has tons of information on IDs to watch for.)
Before your possible rogue sys admin comes back and before the magic words “you’re fired” are ever uttered, make sure you’ve got a good backup of everything in sight. It also might not be a bad idea to do at least a dry run at disaster recovery just to make sure restores work like they should.
So, at this point we’ve checked for potential issues and (hopefully) found there aren’t any. Now we’re ready to take the sys admin into the conference room and say, “Hasta la vista.”
Between the time the employee is taken in and the time they come out, their account needs to be disabled. During the process, they should be reminded of any confidentiality agreements or the like — and it doesn’t hurt to tell them that a white-hat hacker has come through and checked everything out. (If they haven’t done anything, it serves as a warning. If they have, it can shake them up a little. However, do not discuss findings, if any.)
When they come out, we'll want to make sure we get everything: laptops, cell phones, USBs, keycards, etc. Watch as they clean out their desk, and then make sure they’re escorted from the building.
Some employees may act violently to the news they’re being let go. If you even think there might be an issue either have security standing by or ask for a police officer in the area. I wouldn’t have them as part of the process, since their presence might be seen as provocative, but having them close by isn’t a bad idea.
Another thing to remember: System accounts need to be changed — and there’s no time better to start doing this than the same day they’re let go. (This is one reason the fewer service accounts the better.)
When our now former sys admin is out the door, we need to inform all employees the individual is no longer with the company. While we’re at it, we’ll want to notify any vendors as well.
Lastly comes the siege. At least for a while, we need to be watching, and we need to make sure there are no attempted intrusions. If there are attempts, what you do about them is entirely up to you and your company policy. My gut instinct says to take these very seriously. In the U.S., outfits like InfraGard, the FBI and the Secret Service (both mandated by the government to investigate cyber crime) may be resources for advice and assistance.
With any luck, we'll all live happily ever after when going our separate ways — but I feel it's better to be prepared (and even a little paranoid) when a sys admin rides off into the sunset.
Have you ever had a former sys admin try to retaliate after being let go? What steps do you take when letting someone with the keys to the kingdom go? Share your tips and stories in the comments below!