For several years running, we've run our fingers along the underside of IT in an effort to surface the dirtiest jobs in tech.
Carcasses in computer hardware, endless streams of anatomical close-ups, the occasional encounter with fecal matter both real and metaphoric -- each installment of our Dirty IT Jobs series has revealed an industry of consummate professionals willing to go to any length (or depth) in the name of IT.
[ Also on InfoWorld: Learn the truth about IT job postings -- and what those want ads really mean. | Cut to key news in IT management and development with our summary of top tech happenings. Subscribe to the InfoWorld Daily newsletter. | Get a $50 American Express gift cheque if we publish your tech tale from the trenches. Send it to firstname.lastname@example.org. ]
If there's anything we've learned along the way, it's that when it comes to IT, there will never be a shortage of nasty work to get done.
Compiled here you will find the worst of the worst -- a Hall of Shame, if you will, highlighting the unsung heroes of the unseemly side of IT.
Dirty IT job No. 1: Sludge systems architectThese days, technology goes everywhere: oil rigs, pulp mills, sewage plants, you name it. Somebody's gotta clean up the mess and keep the lights on. But few IT gigs get earthier than Dan King's job as a process control engineer for a Texas sewage treatment facility in the mid-1990s.
"Among other things," King says, "I was responsible for crawling around the sludge dryer -- that's where the poo goes after it's extracted from the water -- trying to figure out how to program the computers to run the conveyors at speeds that would get the sludge dry enough so that it's not a sloppy muddy mess, yet not so dry and dusty that it would catch on fire." A particularly smelly fire was the reason King was assigned to the project in the first place, he adds pungently.
To keep the "sludge" at the right consistency, King used an '80s-era programming language called CL, made by Honeywell Industrial Control Systems, to move the conveyor belts at precisely the right speed and send the right amount of electricity to the dryers. That was the easy part.
"Then I had to crawl around the belt and reach in with my glove to check the consistency of this muddy, slushy mess while watching the temperature."
After that formative experience, King went to grad school. Although he no longer works as a sludge systems architect, he's still encounters the dirty work of a career in IT on a daily basis. "Some days, I'm still up to my hips in poo, but it's bull poo," King says.
Dirty IT job No. 2: Espionage engineerWork in IT long enough, and one day you may be asked to monitor your fellow employees' email, scan their browser histories, or rifle their hard drives looking for evidence they've broken the rules. It's just a fact of doing business, says Roger A. Grimes, a senior security consultant and proprietor of InfoWorld's Security Adviser blog.
The biggest single issue Grimes is asked to investigate? Sex between two employees. "That accounts for 50 to 75 percent of the requests," he says. Second on the list is corporate espionage, usually in the form of soon-to-be-former employees absconding with proprietary company data.
At one company, Grimes discovered that nearly half of the network Web traffic was porn-related. When he informed the CEO, he was gently dissuaded: "'We don't want to be the Internet police,' he told me."
Grimes immediately looked at the CEO's hard drive, where he found a generously endowed cache of gay porn, as well as evidence the executive had booked a session with a male prostitute on a business trip to Miami. At the time, the CEO was days away from getting married.
Two weeks later, the CEO called him into his office. "He said a couple of teenage boys had broken into his home and surfed gay porn on his computer, and now he wanted to know how to get rid of what they left behind," Grimes said.
Shouldn't the chief executive call the police? Grimes asked. No, he just wanted to know how to clear his cache. A few weeks later, the marriage was officially over.
But being an IT spy is not all fun and games. Over the years he's also investigated dozens of employees charged with viewing child pornography at work.
"I try hard to not find images on people's computers," he adds. "There are some things you simply can't unsee. It's an emotionally difficult thing to be involved with."
Sometimes, however, it's hard to avoid. "One time I was asked to clean off the computer of an executive who was leaving the company," says Grimes. "She was in her sixties, with gray hair. Going through her hard drive I found pictures of her in leather bondage with another executive at the same company. I just deleted them. But I never could look at her the same way after that."
Dirty IT job No. 3: Fearless malware hunterHunting malware means crawling the deepest, darkest, nastiest corners of the Web, because that's where the bad stuff usually congregates -- such as drive-by installs on porn and warez sites, says Patrick Morganelli.
"Due to the nature of the sites we need to monitor, one of our first questions in any job interview here is, 'Would you mind viewing the most offensive pornography you've ever seen in your life?' Because that's what a lot of malware research entails."
Even employees not actively involved in malware research can encounter deep nastiness, he says. One time an employee merely passed by a support technician's display while the tech was remotely logged in to a customer's PC. What the employee saw on the tech's screen was so disturbing that he quit shortly thereafter.
"It can definitely wear on people," Morganelli says. "The amount of filth you need to go through on a daily basis just to do your job can be pretty trying, and much of it is extremely disturbing -- bestiality and worse. But there's no way to fight this stuff unless you go out and actively collect it."
Andrew Brandt -- a malware researcher and InfoWorld chronicler of IT admin gaffes, stupid hacker tricks, and colossal QA oversights -- says he was warned before he took the job that he'd see porn that would turn his stomach. But he says he sees less malware distributed via porn sites and more via fake BitTorrents and game cheats sites.
"I would describe my job as rubbing a white glove on the filthy underbelly of the Net and seeing what comes off," says Brandt. "Every day I work with malware that does everything you don't want it to do -- like steal your bank account information, break your computer, or barrage you with ads -- and I do it 20, 30, 40 times a day."
Dirty IT job No. 4: Data crisis counselorWhen disaster strikes and critical data goes down the memory hole, it can generate a gamut of unpleasant emotions: tears, depression, guilt, hopelessness, and rage.
Kelly Chessen has had to listen to it all. As a crisis counselor for a data recovery firm, Chessen received calls from sobbing adults who've lost images or videos of their recently deceased parents. She talked to dentists who were frantic because their systems went down and they had no idea what services their patients needed. She logged hours with IT managers who lost entire Microsoft Exchange servers because they thought they knew how to implement RAID 5 but really didn't.
"I would talk to one IT guy one day and another IT guy from the same company the next day because the first guy had been fired," adds Chessen.
Though she has an undergraduate degree in psychology, it was Chessen's five years on a suicide prevention line that best prepared her for her current position.
"Not everybody can do what I do for living," she says. "You need the skills, the background, and the patience. It's a dirty job, but it's also very rewarding."
And on those rare instances when her firm wasn't able to recover someone's data? "I do grief counseling," she says.
Dirty IT job No. 5: Network sherpaSomebody's got to crawl through the muck dragging Cat 5 cable behind them or get two incompatible wireless technologies on speaking terms. Meet the network sherpa, whose job is to haul his clients up LAN Mountain and deposit them kicking and screaming at the summit.
But the dirtiest part of the job isn't squeezing into tight, dusty, rodent-filled spaces, says Horne, who's spent years as an independent networking consultant. It's dealing with penny-pinching customers unwilling to upgrade their crumbling infrastructures.
"Hell hath no fury like a customer who hears he must pay for a wireless bridge in order to retire several hundred feet of RG-58A/U coaxial cable that's been serving as the Ethernet backbone between two buildings for 20 years," says Horne. "Even though the cable will be buried under the parking lot, damaged by rodents, and hanging from the ground wire the electric company has ordered him to vacate immediately, he will insist his network is still capable of '10 gigs at least.'"
Worse, if you do a good job for your clients, they'll want you to come to their homes and do the same thing there -- like the time an exec at Horne's largest client asked him to fix the Internet feed at his remote New Hampshire vacation house.
"One assumes when there's a problem with an Internet connection, the customer in question actually has Internet access," says Horne. "No such luck."
After a four-hour drive from Boston to the lip of the Canadian border, Horne arrived to find a Linksys wireless access point wrapped in a plastic food bag, duct-taped to a three-meter TVRO satellite dish, which was pointed at a distant hilltop where at one time there had been an unsecured Wi-Fi hotspot. Horne calmly explained to his client it's not a good idea to poach Wi-Fi, and having dependable Net access would require paying a company to provide it, even if you live in the middle of nowhere.
Dirty IT job No. 6: The whistle-blowerNearly every organization has dirty little secrets. More often than not, IT knows where they're kept. Sometimes, it takes a geek to step forward and bring them into the light. But it's not easy.
Just ask Roger Smith (not his real name). As a computer science teacher at an East Coast high school, Smith became concerned when the district bought single-user licenses of Adobe Creative Suite and Microsoft Office, then installed them on network servers where 5,000 users could access them.
Smith says he approached his superiors and the district's IT department and explained why that was wrong, but to no avail. So one day he called the Business Software Alliance and reported them.
"With some software, we were on day 120 of a 30-day free trial," he says. "Part of what we're trying to do is to teach kids ethics. That's hard to do when the software you're using isn't licensed and the kids know that."
The BSA relies on tips from people like Smith, says Jennifer Blank, the BSA's senior director of legal affairs. Often it's disgruntled employees spilling the beans on their boss. Sometimes, though, it's just IT pros who want to do the right thing.
The BSA has its share of critics, who claim the trade group exaggerates the amount of software being pirated, targets smaller businesses that lack the resources to defend themselves, or acts as unofficial enforcers for well-heeled software makers like Microsoft.
"The people who call us names forget that the people they are defending are stealing software," says Blank. "You wouldn't go into Best Buy, put a copy of Windows in your pocket and walk out the door. Only in this case they're taking that copy of Windows and installing it on 100 computers, so they're 100 times worse."
But this dirty IT job is not for the faint of heart. On her blog, The Whistler's Ear, database administrator Nell Walton details her three-year legal battle with her former employer, credit card processor Nova Information Systems (now Elavon), after reporting rampant security breaches. She ultimately lost. The breaches were quite real, but the court decided that, as a database administrator, she could not have had a "reasonable belief" her employer was breaking the law, as required by Sarbanes-Oxley.
"This is not a path I would recommend to anyone unless you have a completely ethical reason for doing so, have a backbone of steel, and a very thick skin (don't think you will make a million dollars, in other words)," she writes. "Something all IT people in the USA need to be aware of [is that] we don't have a lot of protections when it comes to whistle blowing."
Dirty IT job No. 7: Sexy games producerYou spend all day staring at pictures of naked women and talking dirty in three languages. Sounds like dream job, right?
Not exactly, says Patryk Bukowiecki, a game producer and manager who worked for a maker of "sexy" J2ME mobile games in the late 2000s. There he produced games like Lessons of Passion Blackjack, in which game players engaged in simulated sex talk with scantily clad models. Bukowiecki's job was to comb through 300 explicit photos of each model, pick eight of them to use in the game, write "sexy" chat for each girl, then translate it from English into Polish and German.
"I had to make sure the girls talked dirty enough so that guys would play as long as possible, then buy other editions of the same game," he says. "The texting part was actually pretty fun."
But those hot models? More sad than sexy, says Bukowiecki, especially after you've seen their passport photos, which each model had to provide to prove she wasn't underage and was doing this of her own free will.
"I had to approach my job kind of like I was an ob-gyn," he says. "You have to get some distance on these things or you will go mad with all the ideas running through your head. You might be tempted to try and find these girls and save them. Don't do it."
The dirtiest part of this job? Telling people what you do for a living.
"You sit in an open space in the office looking at these pictures on your computer surrounded by 20 people," he says. "They're all passing by your desk, sneaking peeks at what you're doing. Then you get home and your wife or girlfriend asks, 'What did you do at work today, dear?'"
Dirty IT job No. 8: Professional scapegoatWhen big projects go bad, they can go really bad -- and it's somebody's job to make the project happen or die trying.
In the late 1990s, Michael S. Meyers-Jouan was CIO of a small, family-run apparel maker tasked with the job of easing a DOS-era company into what was then the cutting edge.
"When they recruited me, I found myself supporting an ancient order processing system running on an AIX box, a MAS 90 accounting system, and a collection of PCs," says Meyers-Jouan. "I was supporting an HR department that never quite understood Windows, relational databases, or network security. I was supporting process managers who would enter columns of numbers into an Excel worksheet, then add the numbers up on a calculator and enter the sum below the column, and CAD system users whose idea of 'backup' was to make copies on floppies."
Nonetheless, Meyers-Jouan was handed the task of bringing his new employer into the late 20th century. He produced a detailed RFP, and over several months, he winnowed the list of potential vendors from 30 to 3, then finally to a single winner.
"That's when the fun began," he says. "Installing the ERP software was relatively simple. Interviewing the users to develop business process road maps and job descriptions was tedious, but no worse than expected. Configuring the software to match the business needs was challenging and time consuming, but within our capabilities. But getting the users to adopt the new system was simply not going to happen."
The factory workers refused to participate in the process, despite promises it would cut their workload in half. The back-office employees couldn't understand how a factory worker covered in oily dirt had anything useful to tell them; they continued to rely on email, spreadsheets, and 10-key machines to do their jobs. And the last thing top management wanted to hear about was the need for "culture change."
"For the only time in my life, I was fired," says Meyers-Jouan. "The ERP system never did get put into use, and not too long thereafter, the company stopped answering their phones."
Dirty IT job No. 9: Systems sanitation engineerBeer cans. Food wrappers. Cigarette butts. Moldy bread. You'd typically find them in the bottom of your average dumpster -- only in this case, the dumpster is the shell of a discarded computer.
It's all part of the job at Redemtech, an IT asset disposition firm that processes the aging hardware Fortune 500 companies no longer want. Somebody has to go through each piece and muck it out, decide what can be saved and what must be discarded, says Chomroeun "C-Ron" Sith, technical supervisor for Redemtech's Grove City, Ohio, facility (and no relation to the Dark Lord).
Though it varies widely, Sith says approximately half of the systems he sees can be refurbished and resold. The other half gets recycled in an environmentally responsible way. Before that happens, they have to be inspected and cleaned -- and that's where things can get nasty.
"Some of these things look like they've been sitting the back of a warehouse for years," he says. "They come in covered in dust, with cobwebs, rat droppings, and roaches inside. Sometimes they're so rusted that when you pick them up your hands turn orange. One of the systems we got in was covered in makeup. Every time my guys touched it, they got all glittery."
Then there was the time they opened up a desktop CPU and found a dead animal. "It may have been a rodent or a bird," he says. "We couldn't tell for sure. But it was definitely dead."
Sith estimates less than 5 percent of the 6,000 to 10,000 items his facility processes each week arrive in such bad shape they have to be wrapped in plastic to avoid infecting his staff or causing allergic reactions. Still, that's plenty.
Dirty IT job No. 10: The shadowYou probably don't want to know where your coworkers are going on the Web. But sometimes you have no choice. Nancy Hand knows this, well, firsthand.
Several years ago, Hand was a network engineer for a large public utility in the Southwest. When any of that site's 3,000 employees got a malware infection, Hand received an alert. She was then called in to investigate by remotely combing through the employee's browser cache, looking for the source of the attack.
Along the way, Hand got to see where these employees had been surfing while they were allegedly working. Most of the time what she found was benign -- a lot of sites devoted to cooking, fashion, cars, and day trading. Inevitably, though, she'd encounter the darker side.
Like the employee who swore it was a spam email that caused his browser to visit that members-only bondage site, though how that email also managed to create a member profile for him was less clear. Or the company vice president whom Hand discovered had been spending work time visiting TeenageVirginSluts.com. Naturally, he was the VP of IT.
"He was my boss's boss's boss," she says. "After I found this I contacted my manager and said, 'We have a zero-tolerance policy for this kind of thing, so I am officially notifying you of what I found.' To my knowledge nothing was ever done. About a year later that VP got fired for another sexually related infraction."
Some employees ended up being escorted from the facility by armed guards, though Hand never knew whether it was due to something she had found.
"Sometimes it was a little unsettling to be on the machine of someone I'd met in another part of my job who seemed like a very toe-the-line type of person, only to discover it wasn't true," she said.
Dirty IT job No. 11: Black opsSocial engineer, con artist, penetration tester, or white-hat hacker -- whatever you call it, Jim Stickley has a dirty job that actually sounds like fun. As VP of engineering and CTO of TraceSecurity, Stickley gets to talk his way into a client's offices, sneak into their data centers, make off with the company's vitals, then come back later to show them where their internal security broke down.
The best part? He gets to wear disguises. Pest control specialist, AC repairman, OSHA inspector -- Stickley and his crew have a closet full of uniforms. But fireman is a particular favorite. "At one place you're the fire inspector, and girls fall all over you," Stickley says. "The next place you're wearing the pest inspector suit and you're the scum of the earth."
First, Stickley and his team take over the company's email system and schedule an appointment. Then they show up in the appropriate fake attire. Whoever has been assigned to watch them usually leaves after about five minutes, Stickley says. If not, they send her out to get them coffee or offer to show her a (fake) dead mouse they found in the corner. That usually does the trick.
Once she's gone, they sneak into the security room and take all the backup tapes, load Trojans onto the servers, or plug wireless devices into the network and hack it from the parking lot.
"If we can get the backup tapes, we're done," Stickley says. "Every piece of data you'd want -- mothers' maiden names, Social Security numbers -- is on those tapes. We've also walked out with computers, boxes filled with loan documents, and applications for patents that have been drawn up but not submitted. It's amazing."
Stickley says he's penetrated more than 1,000 locations and has yet to be thwarted. The dirty part: Coming back the next day to face the people you just pwned.
"You feel dirty, if nothing else," Stickley says. "People come up to you and they're mad. 'I can't believe I got you a cup of coffee.' But ultimately you're just trying to help them out. Nobody gets fired for screwing up. The whole point is to learn from the experience."
Dirty IT job No. 12: The human server rackThe panicked call at 3 a.m. is a sad fact of life for many sys admins. But not as many are woken in the dead of night and asked to part the floodwaters, perform acts of impromptu structural engineering, or serve as a piece of inanimate equipment.
Brian Saunier got such a call several years ago when he was a sys admin for a small Internet service provider in Georgia. An unusually large summer storm had clogged the drain outside the ISP's building, causing a foot of rainwater to flood the first floor, where the server closet was housed.
Fortunately, the servers were protected by an airtight glass door, says Saunier. Unfortunately, the storm also knocked out the power, causing the cooling system to shut down and putting the servers in danger of overheating.
The door had to be opened. To complicate matters, the machine containing the ISP's customer database was sitting on the floor of the server room, directly in the flood path.
First, Saunier and two fellow sys admins constructed a dam out of cardboard, towels, and anything else they could get their hands on to keep the water out. Then Saunier was elected to run in and grab the server before the waters reached it.
"Our plan was to open the door and run in and pick up the server, which I managed to do without incident," he recalls. "But on the way in my foot clipped the dam and the water started pouring in. I was standing in a flooded server room in two feet of water holding a powered-on server and power cords. That was disconcerting."
After about 10 minutes, Saunier's colleagues located a table that fit inside the closet, so he could put the machine down and commence with mop-up operations, which lasted well into the following evening.
Dirty IT job No. 13: B&E artist Dressed in black camo, hiding in the woods in the dead of night on the edge of a Pennsylvania mountain; it's not your typical IT job.
But that's where Matt Neely found himself a few years back. As vice president of consulting for SecureState at the time, Neely's job was to test the physical security of his firm's clients, which include large federal agencies, major retailers, energy plants, and even entire countries. Trained in the art of picking locks by his previous employer (a bank), Neely uses his breaking-and-entering skills so that organizations can find holes in their perimeter and fill them.
On this cold December night, Neely and a colleague were asked to break into a mining facility just past midnight and steal "trophy data," while two other SecureState penetration testers social-engineered their way in via the front gate. The coal mine was concerned about environmental activists breaking in and tampering with its SCADA systems, causing the mine to shut down. They had good reason to worry.
According to Neely, the mine's external security was so porous that he and his partner were in and out in 10 minutes, or about two hours and 20 minutes less than he'd bargained for. The area around the mine was so remote there was no cellphone coverage, so he had no way to reach the other SecureState team. He and his partner had to hunker down for two hours in a freezing rain before they got picked up.
Roughly 75 percent of the time, Neely says he's able to break in to a facility without getting caught. On the other hand, he says his social-engineering comrades succeed about 90 percent of the time -- and when they fail it's usually because somebody got tipped off a test was coming.
Neely always carries a "get out of jail free" card on his jobs, listing the names and numbers of company personnel who've authorized the security test. So far, he's never had to use it -- even when caught red-handed breaking into a power plant by the cops. Fortunately for Neely, SecureState's policies prohibit performing tests on facilities where the guards carry weapons.
Dirty IT job No. 14: Zombie console monkeyThis job title combines two of the most onerous yet often necessary tasks ever assigned to an IT grunt: analyzing system logs and monitoring network operations, says Lawrence Imeish.
"Doing log file analysis and correlation has to be the most tedious, mundane, perpetually boring job in of all IT," he says. But because logs maintain detailed records of all activity that takes place on a system, they're vital tools for debugging and error detection, he adds.
"Meanwhile, network operations centers usually have a person whose job is to stare at screens waiting for green lights to turn red, signifying a problem with some system," he says. "There are useful messages in all those blips and flashing lights, though, and many of them can go a long way toward preventing problems before they occur."
As companies trim body counts, they often combine these positions into what Imeish calls the zombie console monkey. The utter lack of human interaction combined with little to no exposure to the sun means zombies have been known to become almost transparent over time, he adds.
These days, mature IT organizations use event correlation software and network monitoring apps that can identify anomalies and notify the necessary parties if the network fails. Even then, says Imeish, some companies feel more comfortable with a human being sitting there and watching the dials, just in case.
"It's an entry-level job with not a lot of thought involved. Creative thinking? Forget about it. Your job is to follow a script, written down in a manual, for anything that might happen. That's why we call them 'zombies' -- no brains are required."
Related articles at InfoWorld.com:
- Read the Off the Record blog for stories from IT pros -- and share your own tech tale
- 12 most dreaded help desk requests
- Truth in IT job postings: What those want ads really mean
- IT's biggest enemy co-workers
- 20 IT gotchas: How to avoid these common big blunders
- 12 "best practices" IT should avoid at all costs
- 8 biggest myths about managing geeks
- IT inferno: The nine circles of IT hell
- 10 hard truths IT must learn to accept
- 12 effective habits of indispensible IT pros
- The 9 most endangered species of IT
- Stupid user tricks 6: IT idiocy loves company
This story, "The best of the worst: The dirty IT jobs hall of shame," was originally published at InfoWorld.com. Follow the latest developments in IT careers at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.