Packet Loss? What Packet Loss?Packet Loss? What Packet Loss?

When a systems engineers dismissed packet loss as a reason for poor voice quality, I had to get to the root cause to prove him wrong.

In working with an SMB client, I have once again discovered an example of a managed service provider (MSP) falling short on service. In this instance, however, the MSP actually did "roll the truck" and maintain a local presence, but still it couldn't figure out how to maintain consistent reliable telecommunications services for the client.

Here's what happened.

Called in about a voice problem, the MSP's systems engineer said packet loss "didn't exist" and so wasn't the problem. But in fact, the packet loss did exist. My key question was: Why didn't the systems engineer recognize the packet loss? To answer this, I had to show him and the customer the proof that the packet loss existed.

Using a PingInfoView tool, I pinged multiple host names and IP addresses to show the stats in one table. Once set, the pings revealed packet loss, as shown here:

Once I had identified packet loss, I needed to peel back each component to determine its root cause. With root cause, I could resolve the client's complaints regarding one-way audio and unreliable call control -- meaning, phones would not ring and calls would not complete.

In determining root cause, the first issue I discovered involved a firewall's inability to handle traffic effectively since little regard had been given to configuring the off-the-shelf solution for the client's environment. In addition, one firewall had a damaged Ethernet interface. This resulted from the client's use of local cable connections that did not have adequate grounding and surge protection. This type of problem evades MSPs because they don't see their customer networks as "powered networks."

We also identified an unmanaged LAN switch at a remote site that needed to be replaced -- not only was it old but also a root source of packet loss. Switches are not immune to electrical interference, either.

Networks are electrically powered, and until vendors and customers recognize that the local infrastructure requires protection and hardening against power surges and loss, they will continue being plagued by trouble. For providers, these issues will lead to customer churn.

Even with basic tools, the packet loss shown above appears to be everywhere -- a defective and undersized firewall and a bad switch can create a lot of smoke. Our misunderstanding became evident when we discovered that the firewall wasn't locked down so it could only accept traffic from the SIP trunk provider's IP sets.

Throwing in IP phones, workstations, and other endpoints onto the same local LAN was another basic mistake. Using virtual LANs to isolate traffic wasn't possible since the LAN switch was unmanaged. It just goes to show, dig, and dig a little more.

Follow Matt Brunk on Twitter and Google+!
@telecomworx
Matt Brunk on Google+