Border Lines

In the garage of my apartment building there’s a green tag hanging from the phone wiring box. It’s one of those things that I see every day while driving in and out of my parking space, part of the myriad stimuli that enters my awareness and immediately exits again. When taking the trash out the other day, it caught my attention again, and so I decided to finally see what it was. I flipped it over and saw big letters that read:

“MPOE”

For a moment, I wistfully stared off into the distance and channeled Obi-Wan Kenobi: “MPOE. Now, that’s a name I’ve not heard in a long time… A long time.” Walking back upstairs, I had flashbacks to some of the epic telecom battles I’ve fought over the years. The more I thought about it, the more I realized that the MPOE represents a critical concept for the Troubleshooter.

The Handoff

If you aren’t familiar, MPOE stands for “minimum point of entry.” For a telecom company, it’s the end of the line, the official point where their network terminates and a customer’s begins. This “line in the sand” is critical because it creates a clean and predictable way to assign responsibility when a problem arises.

If you’ve ever had a telecom technician visit your home or business, they likely want to start their investigation outside, at the MPOE. Testing to see if the network works up to this demarcation point is a play straight out of “follow the chain.” Inserting a probe where a line enters a customer’s premises attempts to answer the question: “Is the problem upstream or downstream from this point?” Answering this question leads to some very useful information.

If the network is not working at the MPOE, the implication is obvious: the problem lies somewhere within the telecom company’s domain. This is the tactic of isolation in action! Note too that testing at the demarcation point also allows proper responsibility to be assigned: the customer isn’t expected to either find or pay for problems that originate within a provider’s system.

Conversely, if everything up to and including the MPOE is in working order, then the conclusion is also straightforward: the problem must be somewhere within the client’s infrastructure. If you’re a customer, it would be bizarre to be given a hard hat and asked to get in a cherry picker to help troubleshoot AT&T’s network. Of course, the reverse is also true: unless you pay them, AT&T shouldn’t have to chase down problems with the wiring inside your home or business. That’s your responsibility.

The MPOE is an interesting entity because this junction is much more than technical: that box hanging on the outside of your house is a boundary marker with social, legal, and business-related dimensions. The decision to troubleshoot is good for your personal development precisely because it forces you to understand and engage whatever and whomever lies on the other side of these dividing lines.

Border Crossings

Borders have always fascinated me. When I was young, we did a fair amount of driving around the good ol’ USA on family road trips. I distinctly remember being confused when crossing between states. A sign saying “Welcome To The State Of X…” would pass by, but the view would remain largely the same. I guess I was expecting the landscape to change drastically, so much that the dividing line would be visually apparent. Sometimes, I’d miss the sign and only realize I was in a different state hours later.

Maybe my disappointment also stemmed from those cartoon maps that they make for kids, the ones where every state has an icon representing what it’s known for: a potato on Idaho, a car on Michigan, etc. Is it any wonder I was disappointed after crossing these borders, only to find that Texas wasn’t awash in a sea of barbecue sauce or that Wisconsin wasn’t made of cheese?

I also remember visiting Nogales, Arizona and Nogales, Sonora when I was growing up. These conjoined cities straddle the USA/Mexico border. To my naive mind, this transition was mind-blowing: we had walked just a few minutes and suddenly everyone was speaking a different language!

As a broad concept, recognizing where entities meet and transition is crucial to navigating the world. Because interesting things often happen at these dividing lines, the implications are rich for the Troubleshooter. If you fix things long enough, you will eventually have to negotiate the following types of boundaries:

• Contractual: this is where you can see where the problem lies, but the responsibility for fixing it belongs to someone with whom you have a standing agreement. The example of the MPOE neatly illustrates this division: you pay the phone company every month with the expectation that they will provide a working service up to this boundary marker. Just like they will happily terminate your account for non-payment, you too must demand that they uphold their end of the deal. In that sense, you don’t ever “fix” the phone company’s technical problems, except by enforcing the letter of your pact.
• Business Relationship: this is often the flip side of the contractual dimension, where the maintenance of goodwill demands you fix something that is clearly not your responsibility. This is an incursion into your sovereign territory, but one that needs to be tolerated. We used to get on the phone all the time to help our clients ingest the data we produced, often troubleshooting their systems along the way. This wasn’t necessarily contractually mandated, but everything is on the table for the Big Client.
• Organizational: this is the case where a problem is internal to your organization, but fixing it requires resources outside your local sphere of influence. Instead of turning a wrench, these repairs require the tools of advocacy, seeking out those who have the power to help you and convincing them that they should.
• Legal: just because you can fix it, doesn’t mean the law allows you to do so. Some repairs can bring up interesting legal questions, like those involving reverse engineering. You may have signed an agreement that prevents tinkering, as with many closed source software products. The bottom line is that if you don’t own it, fixing it may require permission.
• Social: people can get territorial about…well, pretty much anything (see also: Milton’s red stapler from Office Space). If you’re cavalierly fixing problems in someone else’s domain, there can be social repercussions.

Border Skirmishes

When I was CTO of Discovery Mining, all these boundary types came together during my epic struggle to procure a high-speed data line connecting our office in the Presidio of San Francisco to one of our colocation facilities in Bayview (about 6 miles away as the crow flies). Getting this point-to-point circuit installed and tested was a series of maddening Kafkaesque vignettes that took me through a full range of emotions: optimism, confusion, anger, despair, numbed indifference, and finally anti-climatic triumph.

From start to finish, it took me 11 months to coordinate the installation of this line. 11 months. I thought about it every day during that period. This lengthy outcome was a severe violation of my expectations, given that I had seen the snazzy marketing page on the telecom provider’s website promoting the product. They made it seem like this was just something you ordered and then it arrived shortly thereafter, like getting a pizza delivered. Unfortunately, “layer 2 metropolitan gigabit ethernet connection” was not on Domino’s menu, or I would have gladly ordered it from them instead.

The cast of characters in my saga included a major telecom firm, a federal agency, a state regulatory commission, the building owner of the colocation facility, another company that we were sub-leasing space from, our Internet service provider (ISP), and us. Did I miss anyone? Within this group was a tangled web of business, legal, and technical relationships. In retrospect, half the battle was just figuring out who had jurisdiction to make a particular decision. Some territory was aggressively controlled while ironically, in other situations, it was nearly impossible to figure out who was in charge and could move my project along!

Witnessing the little skirmishes between these various entities as they asserted control over their territories was eye-opening. For example, take the meet-me-room, where our circuit was to be handed off from the telecom firm to our ISP. At the time, they were embroiled in a dispute with the colocation building owner over cross-connect fees and they absolutely refused to pay to connect our new line. It was a standoff that lasted for several weeks and neither side would budge on their demands. Also, getting this circuit somehow triggered a review of the contract by the California Public Utilities Commission. Weeks lost there too.

Although this line required a fairly simple physical connection (strands of fiber optic cable from point A to B), mostly already existing, the route unfortunately snaked through all these various fiefdoms. Some of these regimes were sympathetic allies and others were hostile territory. Not all travel looks like a photo shoot from an issue of Condé Nast.

During this frustrating process, there were also social boundaries that I could and could not cross. Since I was paying them, I could justify laying into our ISP for chronic mismanagement of the project and lack of communication about what was happening. So, I did.

On the other hand, getting fiber pulled to our office building was dependent on the whim of inaccessible bureaucrats with whom I had little influence. It was a mighty struggle getting them to answer any form of communication (phone messages, emails, smoke signals, etc.), with weeks sometimes lapsing between responses. With those people, it was cheerful and patient persistence at every turn, as I could never afford to be publicly angry with them.

Even after the circuit was turned up, we spent a few more months troubleshooting problems discovered by actually using it! There was one particularly embarrassing rollback that I had to stomach; it occurred after a failed switchover rained down chaos on our operations. That was the beginning of my education in networking arcana, with a deep-dive into the meaning of the three letters M-T-U.

Flipping the switch to fully utilize our new line for actual business was a day I’ll long remember. It occurred 399 days after my first enquiry. Who would have thought that ensuring a clear and stable path along a tiny sliver of glass, nearly weightless, would have been so difficult?

Bon Voyage! Remember Your Passport.

In life and business, we often rely on systems that are beyond our direct control. To get help, repair will involve trips to neighboring territories. Leaving your immediate locus of control requires human skills because your influence diminishes as you cross these border lines.

References:

• Header image: Jones, Suzi. Fence with Smooth Wire. 1978. July. Photograph. Retrieved from the Library of Congress, https://www.loc.gov/item/ncr000222/.