This is the 208th article in the Spotlight on IT series. If you'd be interested in writing an article on the subject of backup, security, storage, virtualization, mobile, networking, wireless, cloud and SaaS, or MSPs for the series PM Eric to get started.
Many jobs ago I was called in as an external contractor for a production company. They ran 24/7 and had just recently lost their resident IT tech due to some light-fingered antics I won't get into. I got the tour, did the usual meet and greet — promptly forgetting most peoples’ names while walking about and trying to get my head around what it was I had to actually look after.
The previous IT tech had some documentation: a small pocketbook with a DYMO label stuck on the front called it “The IT Bible.” With this in hand, I thought I was the luckiest IT pro alive. But that wasn't to last...
Inside were pages and pages of handwritten notes, including passwords. However, the handwriting rivalled my doctor’s in un-readability. There was also a hand-drawn network map of sorts and a couple of network cabinet layouts of the server room.
One of my first jobs was to re-secure the entire network should the disgruntled employee come back for vengeance. Twelve APs with no two identical was a bit of a hassle but my “Bible” was serving me well so far as I found my way around the site with my personal laptop and a couple of Ethernet leads plus a console cable.
With the wireless secured, it was off to the server room to go through 15 (count them 15) physical servers (VMs weren't around back then) running all sorts of things, including Windows 2000 + 2003, SQL, Exchange, Novell, and some in-house bespoke software I thankfully didn't need to touch.
This is where the trouble started. The passwords I had for the servers weren't right. I started rifling through the book looking for an updated page but no luck. I had a few tools up my sleeve for hacking forgotten/lost passwords but most of these required a reboot, which equalled downtime — which of course costs money.
I went back to my boss and passed on the message. He said they could arrange downtime but needed 48 hours notice. I left it with him.
With that taken care of, I started putting out the IT fires that had cropped up while the site was tech-less. Some of the more menial jobs (password resets) were challenging with little to no access to AD, but for the most part I was successful, sorting out quite a few problems. Everything I had done so far was on my laptop in a Word 97 document — any and all changes I had made and everything I had learned. It was also copied onto a floppy disk, just in case.
A loss of power
The next day we had a power outage at around 10 a.m. It was city wide and lasted for 9 hours. The UPS lasted for 18 minutes of it. (Nice to see my predecessor was keeping tabs on battery life!)
I stayed on site waiting for the power to be restored and got a checklist ready for a start-up sequence. The lights came back on at about 7:30 p.m., and I was told by my boss to get everything up pronto — time is money and nine hours of downtime had cost him a pretty penny. I booted all the servers (after hacking the password for the DC and a couple of standalones), figuring the extra 20 minutes was cheaper now than having to down the servers once again just to do later.
Half an hour later all the servers had booted — except one.
It had a boot-up password set in the BIOS, which was also password protected. Back to the pages of the now not-worth-the paper-it-was-written on “Bible” to see what I could find. Nothing, nada, zip.
The boss was back in the server room: “What’s taking so long?”
I explained the problems in layman's terms and told him I was working on it. All the office and admin staff could get back to work, but only 3/4 of the production could restart with the last section resting on this password and my abilities.
I tried a few of the more standard passwords (ones which were listed more than once in the book), unplugged the power, popped the cover, took out the battery, left it five minutes and tried again. I dug through the dust bunnies and found the reset CMOS jumpers. Nothing changed.
I called my predecessor, and he told me to shove it, which I expected but I had to try — I was really grasping at straws by this point.
I dug through his desk, emptied drawers, checked the Post-it notes scattered around his workstation, picked up his selection of floppies and ran back to the server room. Anything that looked remotely like a password, I put into this machine. (Later I realized I had put in some of his shopping list as passwords.) I scoured the floppies for any information. Still a dead duck.
I reset the password on his AD account, logged into his machine as him, hunted around and eventually found something: a Word file saved in the depths of his temporary files called “p-words.” It had approximately 30 passwords listed, but I only needed one to work to unlock the BIOS and I'd be home free.
Thankfully one did work. I was able enter the BIOS and clear the boot password. It had taken me two hours since the power was restored to get everything back up and running at full speed.
Five things to leave behind
When I left the site after two months, my replacement had five documents on a floppy, and I made a copy and gave it to the boss:
- A full password list broken down into sections, with dates when the change was made.
- An Excel workbook with IP allocations, diagrams of each cabinet including port / patch number tracing.
- A full asset register with location, make, model, serial, MAC, asset number and regular user if there was one.
- My document listing any changes I had made, all jobs I had done while I was there (essentially my helpdesk log)
- A half-completed software audit, including versions and licensing. (I had to give him something to do.)
The moral of the story is keep all your documentation updated with changes, in a safe place (preferably backed up and a copy elsewhere) and make sure someone else knows where it is if you were to be hit by a bus tomorrow. Don't leave a world of hurt for the new guy.
---
What do you keep in your IT Bible? Have you ever had to pick up the pieces from a previous tech who didn't keep up with documentation? Share your tales and tips in the comments below!