This is the 38th article in the Spotlight on IT series. If you'd be interested in writing an article on the subject of backup, security, storage, virtualization or MSPs for the series PM Eric to get started.
“Have we been hacked?” Those were the words I heard during a conversation with my CEO, his fellow executives, and my boss.
The day started something like this: A ticket comes across for deactivation of an employee. Let’s call him Ezekiel. Nothing new. I do all my checks to make sure Ezekiel was disabled in AD, blah, blah, blah — you know the routine. An hour later, my boss calls me down to the CEO’s office. He heads me off at the door.
Boss: “You disabled Ezekiel’s account, RIGHT?”
Me: “Of course. Why? What’s going on?”
He moves aside so I can enter the office where I see a room full of top executives, all displaying angry red faces, clutching several printed out documents.
Me: “Hey, everyone. What’s up?” (My voice comes out sounding like Alfalfa’s from “The Little Rascals.”)
Exec 1: “THIS is what’s up,” he says, throwing a small stack of papers into my chest.
A quick glance and I can tell the papers are printouts of emails directed to several inside and outside partners describing, in detail, certain actions of the CEO’s wife, along with proprietary information about the company. I check the sender. To my surprise, its Ezekiel, the very same user I disabled not more than two hours ago.
Boss: “Check the timestamp! Have we been hacked?” he asks, pointing to the top of the paper.
The timestamp showed it was sent out 15 minutes ago, which by everything I was ever taught or knew about being a Systems/Network Administrator was impossible… Or was it? From here, it’s just a bunch of raised voices yelling and talking over one another for the next 10 minutes with all eyes on me while I slowly back myself into a corner cowering like a beaten dog. I don’t have any answers for them but assure them I will get to the bottom of this as I walk out with my tail between my legs.
So, how was it that Ezekiel could still send emails after I disabled his account? Did he hack into the company and enable his account? If he did, why would he only target his email account?
The experiment
Combing through logs on firewalls, servers, and wireless access points, I couldn’t find any sign of an intrusion or anomalies. So I decided to recreate the situation from scratch over the next five days and was alarmed at my findings.
I created five users with the same access as Ezekiel. Two accounts were disabled through Active Directory, just as Ezekiel was. Two had their passwords changed but the accounts remained active — a typical scenario when a supervisor needs time to obtain files, emails and other data from a terminated employee’s account before it’s disabled completely. And for the last one, I disabled just the email account. I documented every step along the way and sent the instructions off to five other Exchange/domain admins at other companies to test along with me.
Each time, the results were the same. Internally, they were immediately disabled and full access was denied to everything. Externally they weren’t able to log into anything either… except Outlook Web Access and ActiveSync.
Microsoft has documented and known about this since Exchange 5.5, introduced in 1997. In fact, they did this on purpose! (There’s more info in KB articles here and here.)
Disabling a users account in Active Directory or changing the user’s password will disable that account immediately on the LAN as expected. However, per Microsoft, the user will still have access to email from any external Internet connection through Outlook Web Access for a default minimum of 15 minutes in ideal conditions.
In our testing, this turned out to be much longer than 15 minutes under non-ideal conditions, such as using Firefox (as opposed to the latest updated version of IE) or an iOS or Android phone (instead of a Windows mobile device). In fact, in my testing, I found I was still able to connect to a disabled account using Firefox for around three hours, and using an Android phone with ActiveSync, up to six hours. The time can also be extended should the user already have his/her OWA account opened outside the organization at the time of disabling the account. What’s more, changing the user’s password does nothing while this token is still cached. You can change the user’s password 100 times and not only will they still be able to use their original password during the 15-minute interval, but they can now successfully use ANY of the 100 passwords you just changed it to, increasing the odds of a successful brute force attack with each password change.
The findings
At this point, anyone even remotely intelligent should be freaking out, and for good reason. So what’s going on here? Why is this even allowed?
In short, it’s to improve performance in IIS. When a user logs into Outlook Web Access, the credentials for the request are used to create a token on the server. This token is then used during the duration of the session to access files (Exchange 2007 allowed browsing of shared server resources, which is no longer allowed in Exchange 2010 — and now we know why) or other system resources such as email. The token is kept in cache by default for 15 minutes. There’s obviously more going on than just this, but that’s the short and sweet. This means that at any given time, changing a user’s password or disabling their account in Active Directory will not take effect on the Exchange server regarding WAN access for up to 15 minutes (under ideal conditions) depending on when IIS last cleared the token cache.
So, why 15 minutes — why don’t we just lower it? There’s a reason to the madness. Each time a user logs into OWA, it creates a storm of ASP packets to the server for authentication along with other things. This isn’t so bad if it’s only happens every 15 minutes, but lower this setting in a larger organization, and your phone starts ringing from users complaining about performance on Exchange or frequent disconnect notices. Microsoft chose 15 minutes as the happy medium between performance and security.
I hope this article has provided the info necessary to approach HR and upper management on the risks associated with using Outlook Web Access in an organization and to develop a policy going forward for terminated employees that may be considered “high risk” as we learned Ezekiel was.
If you’d like to know more, I’ve created a how-to article in the Spiceworks Community to help System/Network Administrator’s mitigate the risk of a potential breach after terminating a high-risk employee.
What security lessons have you learned the hard way? Share your stories and keep the discussion going!