Know The Path Your Media Sessions Take
This will help you answer questions like, Is QoS properly configured along the path? Are markings maintained and is the desired queuing being done? and more.
This is my first blog for NoJitter, so a brief introduction is in order. I do network consulting at Chesapeake Netcraftsmen and my blogs tend to have an operational focus that is based on my real-world consulting experiences. I've been blogging for Infoblox and Netcraftsmen for several years, so you can look there to find additional posts that I've done over the years. You can find out more about me and Netcraftsmen on our Staff page.
Knowing the Media Path
It is important to know the paths over which your media sessions are communicating. Is QoS properly configured along the path? Are markings maintained and is the desired queuing being done? Are there links where errors or duplex mismatches are affecting voice, video, or session control? Is the path congested at some point, causing packet loss when a link in the path becomes oversubscribed? Is the session using the path that you expect it to take?
One of our customers once insisted that the network was causing high packet loss. They could see it in a video conferencing system. The video would intermittently be pixelated, indicating that the I-frames were being dropped.
The two video conferencing systems were in the same metropolitan area and had good connectivity. We couldn't find anything along the path between the video conferencing systems. We used traceroute to determine the path, then examined the links along the path. All the links were running clean.
All we could figure was that the lack of QoS was causing some packets to be dropped during periods of instantaneous buffer congestion. But the video was being impacted to an extent that seemed greater than what the lack of QoS would imply, given the interface loading along the path.
We eventually ran out of things to check in the network, so one day we took a look at the video conferencing system configurations. We found that the systems were using a media gateway that was located in the Internet, not a media gateway within the enterprise. D’oh! The packet loss was due to the media path going out into the Internet and back. Once the configurations were corrected, the video quality improved.
The lesson here was to check the configurations earlier in the troubleshooting session, to validate the path between the video systems. Traceroute isn't sufficient.
Another customer was experiencing slow applications, including voice and video to some remote sites that were connected via an MPLS carrier. Of course, the network, and the carrier in particular, were assumed to be the guilty parties.
While there was some congestion loss, as indicated by egress drops reported by router interfaces along the path, it wasn't as significant as what we were hearing about the impact on application performance. The drops, while they existed, were not a significant percentage of all traffic (< 0.0001% loss).
(See my past blogs on Application Analysis Using TCP Retransmissions, Part 1 and Application Analysis Using TCP Retransmissions, Part 2 and Carole Warner Reece’s blog Understanding Interface Errors and TCP Performance)
Upon investigation of other interfaces along the path, we found that one interface, between a distribution switch and a firewall, had high numbers of FCS errors and Runts. The interface was configured for full duplex. This is the signature of a duplex mismatch. Runts should be rare in a correctly operating network. Of course, we didn't have access to the firewall configurations, because the security team wanted to keep that information to themselves.
The combination of full duplex on one end and auto duplex detection on the other end of a link will result in a duplex mismatch--the duplex detection standard doesn't include a way for a hard-configured system to exchange information about its setting, and the system that is configured for 'auto' must go to half duplex when the negotiation fails. (See my past blog on Duplex Mismatch.) Therefore, one end is set to full while the other end of the link fails to negotiate and must go to half duplex.
Modern equipment from the last five years or so should have no problems with duplex negotiation. We find that many organizations are still living with the recommendations for duplex setting that are more than five years old, so we try to educate them on current best practices.
Know the path that your media streams are using. Monitor all links in the path. Use a configuration management system to verify that configurations are correct. A lot of different components must be configured and operating correctly for network applications, including voice and video, to work efficiently.