Improving the Environment

You probably thought this was going to be about old-growth forests, endangered species, global warming or water rights. However, in the context of troubleshooting, when I say “improve the environment,” I mean your environment!

File under “obvious,” but upgrading the conditions under which you troubleshoot can significantly improve your chances of finding a timely solution. Vice versa, an inhospitable or distracting environment can work against you and significantly delay finding a fix. First, let’s observe that many machines are in locations not among the most comfortable on earth. For example, during my time in the IT world, I spent my fair share of time in colocation facilities (aka, “colos”). These are places where companies buy or rent space to house their computers. Colos are spartan and austerely beautiful places that are designed for one purpose: to keep computers running smoothly. Along that front, they have abundant resources: backup power generators, redundant Internet connections and enough air conditioning to recreate Arctic conditions. I guess you could say that everything has its ideal setting: a rattlesnake is probably happiest while sunning itself on a rock in the desert, a polar bear prefers an iceberg, and a frog on a lily pad in a swampy pond feels quite content. Likewise, I always imagined my computers smiling from ear-to-ear when I placed them in a well-equipped colo. Uninterruptible power, ample cooling, and a fast network: what else could a server want?

What a server wants is probably different from your own desires. What are colos and other industrial environments typically not designed for? That’s right, humans. When you’re in a giant room filled wall-to-wall with servers, your ears will be violated by the deafening whoosh of thousands of cooling fans. The low-frequency electrical hum from all that power being transformed into computing cycles will start to lull you to sleep. If you’re working anywhere near the A/C vents, cold air will blast you and remind you to put your jacket back on. Alternatively, if you’re standing behind a rack of servers expelling the heat from hundreds of CPUs and disk drives, you will feel like you’re on a tropical island and wish you had worn shorts. Oh, and there’s no comfortable place to sit either. Personally, in a setting like this, I feel my problem-solving skills are degraded because my mind is distracted by these environmental factors. For these reasons, I strongly disliked troubleshooting at the colo and my subconscious agenda was always to get in and get out as soon as possible.

Here are some environmental conditions that will impact your troubleshooting abilities:

• Light: of course, you’re better off if you can clearly see what you’re working on. If you can’t use artificial light and it’s night, sometimes it’s best to wait for day.
• Sound: noisy environments are fatiguing and will make communication difficult if you’re troubleshooting with a team (or taking instructions over the phone).
• Space: the area around a broken system must be large enough to accommodate your team and equipment.
• Temperature: extremes of hot or cold will negatively affect your physiology, making repairs more challenging. These type of conditions can also be hard on your tools.
• The Elements: fixing things while exposed to the wind, rain, or sun will require Nature’s cooperation.
• Safety: troubleshooting amidst hazards, man-made or natural, can be very dangerous. Examples; working on a car while it’s stuck in the middle lane of a busy highway, or perched near the edge of a cliff.
• Business Requirements: the need of a firm to continue operating normally may constrain when or how you work. When repairs will be a hindrance, because of the noise, mess, or interruption, be prepared to work after the close of business.
• Scrutiny: the need to maintain a professional demeanor may clash with your ability to efficiently get the job done. This tension is most often felt when you’re forced to work in close contact with a customer. You might encounter a personality conflict, perhaps with a client who likes to micromanage. Likewise, a repair might require something experimental or messy that, to the untrained eye, may cause alarm. People may not understand that troubleshooting requires improvisation, thoughtful pauses to reflect, and frequent course corrections. Put another way, there’s a good reason that sausage is made behind closed doors.
• Time: an important dimension that will affect all of the conditions listed above. Daylight, temperature, tides, weather patterns, sleepiness, fatigue, crowds, etc. will all ebb and flow with the passage of time. You will need to deal with environmental factors the right way and at the right time.

Given that the environment will either help or hinder troubleshooting, you’ll always want to be on the lookout for opportunities to:

• Make the current environment more hospitable.
• Change venues and continue working elsewhere.

Please Turn On The Lights

If you’re troubleshooting at the site of a failure, your ability to modify the environment to suit your needs may be limited. Even so, you can typically make some improvements. I can’t tell you the number of times I have found a group of engineers squinting at a problem in a darkened room. Oh, the look of wonder and awe in their eyes when I would triumphantly flip the light switch on, exclaiming “Let there be light!” Sometimes, I would say it in Latin (“Fiat Lux!”) and the look they’d give me was, “Who’s this crazy guy turning on lights and shouting things in Latin?”

In Latin or English, turning on the lights so you can actually see what you’re doing might have the highest improvement-to-effort ratio when it comes to upgrading your conditions. Others will typically involve a tradeoff (usually time or money). Sometimes, it might be very difficult to make even trivial changes to the place where you’re troubleshooting. What if turning on the lights involves something complicated like finding someone who can operate the automated building management system (like in a stadium)? For those cases, you’d probably wish you had brought along your flashlight!

Where The Grass Is Greener

When circumstances favor relocating to a place where repairs will be easier, faster, cheaper, etc., the factors will typically involve:

• The need for specialized or cumbersome equipment that can’t be brought on-site.
• Repairing at the scene would either be unsafe, uncomfortable, or disrupt business with your presence.
• Downtime is so costly that repairs can’t be done in situ. The faulty piece of equipment must be removed immediately so that a spare can be deployed.
• Better scheduling: repairing on-site may constrain you to the operating hours of the business.
• Stepping out of the spotlight: it’s always better to be able to do your work (and make those inevitable mistakes) without having a customer breathe down your neck.

Again, the decision to relocate will involve tradeoffs and have an economic component. Clearly, if you knew in advance that a fix could be completed in just a few minutes, then bringing a machine back to your workshop an hour away would be a waste of time. On the other hand, if it took a 3 weeks to find the answer, you’d probably be wishing on the third day that you had changed venues sooner. For this reason, you can’t say “Never relocate!” or “Always relocate!” I am only here to remind you of the possible benefits of changing locations to your workshop.

For me, the benefit of a venue change was both physiological and psychological. It was cold, loud, and lonely in the colo. I couldn’t throw Toto on the stereo, make some tea to help me think, or easily bounce an idea off a fellow co-worker. However, I never let my desire for creature comforts get the best of me. Remember, when you retreat with a broken machine to your deluxe, climate-controlled workshop you must eventually return to the same place with a fix. Don’t use a venue change as a stalling tactic. Furthermore, while moving a machine may make you more comfortable, it also might be a colossal waste. Time should be among your most precious resources, and conserving it will sometimes mean standing and fighting. On-site.

References:

• Header image: “Construction helmets”. Pop & Zebra, photographer. Retrieved from Unsplash, https://unsplash.com/photos/wp81DxKUd1E.