AV over IP Troubleshooting Guide
Audio-Visual over Internet Protocol (AV over IP) has revolutionized the industry by enabling flexible, scalable, and cost-effective distribution of audio and video content over standard network infrastructure. However, with this flexibility comes complexity, and troubleshooting AV over IP systems requires a deep understanding of networking principles, protocols, and specialized diagnostic techniques.
This comprehensive guide covers the most common AV over IP problems and their solutions, focusing on three major standards: NDI (Network Device Interface), SMPTE ST 2110, and SDVoE (Software Defined Video over Ethernet).
Understanding AV over IP Technologies
NDI (Network Device Interface)
NewTek's NDI is a software-defined approach that allows video systems to identify and communicate with one another over IP networks. NDI uses standard Ethernet networks and can adapt to varying bandwidth conditions through automatic quality adjustment.
Key NDI Characteristics:
- Automatic discovery and connection
- Adaptive bitrate streaming
- Low latency (typically 1-2 frames)
- Built-in audio embedding
- Supports both unicast and multicast transmission
SMPTE ST 2110
SMPTE ST 2110 is a suite of standards that defines the carriage, synchronization, and description of separate elementary essence streams over IP networks. Unlike SDI, ST 2110 separates video, audio, and ancillary data into individual streams.
Key ST 2110 Components:
- ST 2110-10: System timing and definitions
- ST 2110-20: Uncompressed active video
- ST 2110-21: Traffic shaping and delivery timing
- ST 2110-30: PCM digital audio
- ST 2110-40: Ancillary data
SDVoE (Software Defined Video over Ethernet)
SDVoE is a standard that enables the transmission of uncompressed or visually lossless compressed AV signals over standard Ethernet networks. It's designed for zero-frame latency and supports 4K60 4:4:4 signals.
Key SDVoE Features:
- Zero-frame latency
- Uncompressed or visually lossless compression
- Matrix switching capabilities
- Support for control and USB signals
- Centralized management
Common AV over IP Problems and Solutions
1. No Video Signal Issues
Problem: Black screen or no video display
Symptoms:
- Displays show no signal or black screen
- Audio may or may not be present
- Network traffic appears normal
Diagnostic Commands:
bash
[object Object],
ip ,[object Object], show
ethtool eth0
,[object Object],
tcpdump -i eth0 -c 100 host [source_ip]
,[object Object],
ip maddrs show eth0
,[object Object],
iftop -i eth0Solutions:
-
Verify Physical Connections:
- Check cable integrity with cable tester
- Ensure proper termination (568A or 568B standard)
- Test with known good cables
-
Check IP Configuration:
bash[object Object], ifconfig eth0 ,[object Object], ping [destination_ip] ,[object Object], route -n -
Validate VLAN Configuration:
bash[object Object], ,[object Object], /proc/net/vlan/config ,[object Object], tcpdump -i eth0 vlan [vlan_id]
2. Intermittent Signal Loss
Problem: Video drops out periodically
Symptoms:
- Video freezes or pixelates randomly
- Audio dropouts may accompany video issues
- Problems worsen during high network activity
Diagnostic Approach:
bash
[object Object],
ping -c 1000 -i 0.1 [destination_ip] | grep loss
,[object Object],
,[object Object], /proc/net/dev | grep eth0
,[object Object],
top -p $(pgrep -f ,[object Object],)
,[object Object],
journalctl -f | grep -i networkSolutions:
-
Buffer Optimization:
- Increase receive buffers:
echo 16777216 > /proc/sys/net/core/rmem_max - Increase send buffers:
echo 16777216 > /proc/sys/net/core/wmem_max
- Increase receive buffers:
-
Network Interface Tuning:
bash[object Object], ethtool -s eth0 autoneg off ,[object Object], ethtool -s eth0 speed 1000 duplex full ,[object Object], ethtool -G eth0 rx 4096 tx 4096
3. Quality and Compression Issues
Problem: Poor video quality or compression artifacts
Symptoms:
- Pixelation or blocking artifacts
- Color banding or posterization
- Blurry or soft image quality
Quality Assessment Commands:
bash
[object Object],
ffmpeg -i udp://[multicast_address]:[port] -t 10 -f null - 2>&1 | grep bitrate
,[object Object],
vlc -vvv udp://@[multicast_address]:[port] --intf dummy --extraintf logger
,[object Object],
hping3 -i u1000 -c 1000 [destination_ip] | awk ,[object Object], > jitter.logSolutions:
-
Bandwidth Optimization:
- Implement Quality of Service (QoS) policies
- Use dedicated VLANs for AV traffic
- Configure traffic shaping:
bash[object Object], tc qdisc add dev eth0 root handle 1: htb default 30 tc class add dev eth0 parent 1: classid 1:1 htb rate 900mbit tc class add dev eth0 parent 1:1 classid 1:10 htb rate 800mbit ceil 900mbit -
Codec Configuration:
- Adjust compression settings for available bandwidth
- Use appropriate profiles (baseline, main, high)
- Configure constant bitrate (CBR) for predictable bandwidth
Network Diagnostic Procedures
Comprehensive Network Health Check
1. Physical Layer Verification
bash
[object Object],
ethtool eth0 | grep -E ,[object Object],
,[object Object],
watch -n 1 ,[object Object],
,[object Object],
ethtool -S eth0 | grep -i error2. Network Layer Analysis
bash
[object Object],
ping -c 100 -i 0.01 [destination] > ping_results.txt
,[object Object],
ping -M ,[object Object], -s 1472 [destination]
,[object Object],
traceroute [destination]
mtr --report --report-cycles 100 [destination]3. Transport Layer Monitoring
bash
[object Object],
netstat -su | grep -i udp
,[object Object],
,[object Object], /proc/net/snmp | grep -i udp
,[object Object],
ss -tuln | grep [port_number]Advanced Network Analysis Tools
Using Wireshark for AV over IP Analysis
bash
[object Object],
tshark -i eth0 -f ,[object Object], -w av_capture.pcap
,[object Object],
tshark -r av_capture.pcap -Y ,[object Object], -T fields -e rtp.ssrc -e rtp.seq -e rtp.timestampCustom Monitoring Scripts
bash
[object Object],
,[object Object],
INTERFACE=,[object Object],
DEST_IP=,[object Object],
LOG_FILE=,[object Object],
,[object Object], ,[object Object],; ,[object Object],
TIMESTAMP=$(,[object Object], ,[object Object],)
,[object Object],
LINK_STATUS=$(ethtool ,[object Object], | grep ,[object Object], | awk ,[object Object],)
,[object Object],
RX_BYTES=$(,[object Object], /proc/net/dev | grep ,[object Object], | awk ,[object Object],)
,[object Object], 1
RX_BYTES_NEW=$(,[object Object], /proc/net/dev | grep ,[object Object], | awk ,[object Object],)
BANDWIDTH=$(( (,[object Object], - ,[object Object],) * ,[object Object], / ,[object Object], ))
,[object Object],
,[object Object], ,[object Object], >> ,[object Object],
,[object Object], 5
,[object Object],Bandwidth and Latency Troubleshooting
Calculating AV over IP Bandwidth Requirements
Uncompressed Video Bandwidth Formula
Bandwidth (bps) = Width × Height × Frame Rate × Bits per Pixel × Color Space Multiplier
Example for 1080p60 4:2:2 10-bit:
1920 × 1080 × 60 × 10 × 2.5 = 3.11 GbpsCompressed Video Considerations
- H.264: Typically 2-15 Mbps for HD, 15-50 Mbps for 4K
- H.265: ~50% bandwidth reduction compared to H.264
- JPEG2000: 50-200 Mbps for broadcast quality
Latency Optimization Techniques
Network-Level Optimizations
bash
[object Object],
,[object Object], 0 > /proc/sys/net/ipv4/tcp_timestamps
,[object Object], 1 > /proc/sys/net/ipv4/tcp_low_latency
,[object Object],
,[object Object], performance > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
,[object Object],
,[object Object], 2 > /proc/irq/[network_irq]/smp_affinityApplication-Level Tuning
bash
[object Object],
chrt -f 99 your_av_application
,[object Object],
taskset -c 0,1 your_av_application
,[object Object],
,[object Object], -l unlimitedBandwidth Monitoring and Analysis
Real-Time Bandwidth Monitoring
bash
[object Object],
vnstat -l -i eth0
,[object Object],
,[object Object],
iperf3 -s -p 5001
,[object Object],
iperf3 -c [server_ip] -p 5001 -t 60 -i 5Historical Bandwidth Analysis
bash
[object Object],
iperf3 -c [destination] -t 300 -J > bandwidth_test.json
,[object Object],
,[object Object], bandwidth_test.json | jq ,[object Object],Multicast Configuration Issues
Common Multicast Problems
1. IGMP Configuration Issues
Problem: Devices not receiving multicast streams Diagnostic Commands:
bash
[object Object],
,[object Object], /proc/sys/net/ipv4/conf/eth0/force_igmp_version
,[object Object],
tcpdump -i eth0 igmp
,[object Object],
,[object Object], /proc/net/igmpSolutions:
bash
[object Object],
,[object Object],
,[object Object],
,[object Object],
,[object Object],
,[object Object], 2 > /proc/sys/net/ipv4/conf/eth0/force_igmp_version
,[object Object],
ip maddr add 239.255.1.1 dev eth02. Multicast Routing Problems
Diagnostic Approach:
bash
[object Object],
,[object Object], /proc/net/ip_mr_cache
,[object Object],
,[object Object], /proc/net/ip_mr_vif
,[object Object],
socat UDP4-RECV:5004,ip-add-membership=239.255.1.1:eth0 -Switch Configuration for Multicast
Cisco Switch Configuration Example
! Enable multicast routing
ip multicast-routing
! Configure IGMP snooping
ip igmp snooping
ip igmp snooping vlan 100
! Configure multicast querier
ip igmp snooping querier
! Set IGMP query interval
ip igmp snooping query-interval 125
! Configure fast-leave processing
ip igmp snooping fast-leaveAdvanced Multicast Configuration
bash
[object Object],
interface GigabitEthernet0/1
ip multicast boundary 1
!
access-list 1 deny 239.255.0.0 0.0.255.255
access-list 1 permit any
,[object Object],
interface GigabitEthernet0/1
ip pim sparse-mode
,[object Object],
ip pim rp-address 192.168.1.1 239.255.0.0/16PTP Clock Synchronization
Understanding PTP in AV over IP
Precision Time Protocol (PTP) is crucial for maintaining synchronization across AV over IP systems, especially in SMPTE ST 2110 deployments.
PTP Diagnostic Commands
bash
[object Object],
systemctl status ptp4l
,[object Object],
ptp4l -f /etc/ptp4l.conf -m -i eth0
,[object Object],
journalctl -u ptp4l | grep ,[object Object],
,[object Object],
ethtool -T eth0PTP Configuration File Example
ini
[object Object],
,[object Object],
,[object Object], = G.,[object Object],.x
,[object Object], = ,[object Object],
,[object Object], = ,[object Object],
,[object Object], = ,[object Object],
,[object Object], = -,[object Object],
,[object Object], = -,[object Object],
,[object Object], = ,[object Object],x1
,[object Object], = ,[object Object],:,[object Object],:C2:,[object Object],:,[object Object],:,[object Object],E
,[object Object], = ,[object Object],:,[object Object],:C2:,[object Object],:,[object Object],:,[object Object],E
,[object Object], = ,[object Object],
,[object Object], = ,[object Object],
,[object Object], = ,[object Object],
,[object Object], = ,[object Object],
,[object Object], = ,[object Object],
,[object Object], = ,[object Object],
,[object Object],
,[object Object], = UDPv4
,[object Object], = E2EPTP Troubleshooting Steps
1. Verify PTP Hardware Support
bash
[object Object],
ethtool -T eth0
,[object Object],
,[object Object],
,[object Object],
,[object Object],2. Monitor PTP Performance
bash
[object Object],
,[object Object],
,[object Object], ,[object Object],; ,[object Object],
OFFSET=$(journalctl -u ptp4l --since ,[object Object], | grep ,[object Object], | ,[object Object], -1 | awk ,[object Object],)
DELAY=$(journalctl -u ptp4l --since ,[object Object], | grep ,[object Object], | ,[object Object], -1 | awk ,[object Object],)
,[object Object], ,[object Object],
,[object Object], 5
,[object Object],3. PTP Network Analysis
bash
[object Object],
tcpdump -i eth0 -s 1500 -w ptp_capture.pcap port 319 or port 320
,[object Object],
tshark -r ptp_capture.pcap -Y ,[object Object], -T fields -e ptp.messageType -e ptp.correctionSwitch Configuration Examples
Dedicated AV Network Switch Configuration
Cisco Catalyst Configuration
! Basic switch configuration for AV over IP
hostname AV-Switch-Core
! Enable necessary features
feature lldp
feature udld
feature interface-vlan
! Configure VLANs for AV traffic
vlan 100
name AV-Video-Production
vlan 101
name AV-Audio-Production
vlan 102
name AV-Control
! Configure trunk ports to other switches
interface Ethernet1/1-2
description Trunk to distribution switches
switchport mode trunk
switchport trunk allowed vlan 100-102
spanning-tree port type edge trunk
! Configure access ports for AV devices
interface Ethernet1/3-48
description AV Device Ports
switchport access vlan 100
spanning-tree port type edge
no shutdown
! Quality of Service configuration
policy-map type queuing AV-QOS
class type queuing c-out-q1
priority level 1
class type queuing c-out-q2
bandwidth remaining percent 80
random-detect minimum-threshold 80 packets maximum-threshold 1024 packets
interface Ethernet1/1-48
service-policy type queuing output AV-QOS
! IGMP snooping configuration
ip igmp snooping
ip igmp snooping vlan 100-101
ip igmp snooping querierAdvanced Switch Features for AV
! Storm control for broadcast protection
interface Ethernet1/3-48
storm-control broadcast level 10.00
storm-control multicast level 50.00
storm-control action shutdown
! Port security (optional, use with caution in AV environments)
interface Ethernet1/10
switchport port-security
switchport port-security maximum 2
switchport port-security mac-address sticky
switchport port-security violation restrict
! Jumbo frame support for high-bandwidth AV
system jumbomtu 9216
interface Ethernet1/1-48
mtu 9216Network Optimization for Different AV Standards
NDI Optimized Configuration
! NDI discovery multicast group: 239.255.42.99
ip igmp snooping vlan 100 mrouter interface Ethernet1/1
! Optimize for NDI discovery
interface Ethernet1/3-48
ip igmp fast-leave
ip igmp query-interval 60
! Buffer optimization for NDI
interface Ethernet1/3-48
priority-queue out
mls qos trust dscpST 2110 Optimized Configuration
! PTP configuration for ST 2110
ptp mode boundary
ptp domain 127
ptp priority1 128
ptp priority2 128
interface Ethernet1/3-48
ptp enable
ptp announce interval 0
ptp sync interval -4
ptp delay-req interval -4
! Traffic shaping for ST 2110
policy-map AV-ST2110
class AV-VIDEO
shape average 1500000000 bps
set dscp af41
class AV-AUDIO
shape average 10000000 bps
set dscp af31Performance Monitoring Tools
Network Performance Monitoring
1. Bandwidth Monitoring Tools
bash
[object Object],
iftop -i eth0 -n -P
,[object Object],
nload eth0
,[object Object],
vnstat -i eth0 -h ,[object Object],
vnstat -i eth0 -d ,[object Object],2. Latency and Jitter Analysis
bash
[object Object],
,[object Object],
TARGET_IP=,[object Object],
LOG_FILE=,[object Object],
,[object Object], ,[object Object],; ,[object Object],
TIMESTAMP=$(,[object Object], ,[object Object],)
PING_RESULT=$(ping -c 1 -W 1 ,[object Object], | grep ,[object Object], | awk ,[object Object], | ,[object Object], -d,[object Object], -f2)
,[object Object], [ -n ,[object Object], ]; ,[object Object],
,[object Object], ,[object Object], >> ,[object Object],
,[object Object],
,[object Object], 1
,[object Object],
,[object Object],
awk -F,[object Object], ,[object Object], ,[object Object], | awk ,[object Object],3. Packet Loss Detection
bash
[object Object],
,[object Object],
TARGET=,[object Object], ,[object Object],
PORT=,[object Object],
INTERFACE=,[object Object],
,[object Object],
tcpdump -i ,[object Object], -c 1000 host ,[object Object], and port ,[object Object], | \
awk ,[object Object],
,[object Object],
watch -n 5 ,[object Object],Application-Specific Monitoring
NDI Performance Monitoring
bash
[object Object],
,[object Object],
,[object Object],
NDI_DISCOVERY_PORT=,[object Object],
NDI_DATA_PORT_RANGE=,[object Object],
,[object Object], ,[object Object],
,[object Object], ,[object Object],
tcpdump -i eth0 port ,[object Object], -c 10
,[object Object], ,[object Object],
netstat -an | grep -E ,[object Object],SMPTE ST 2110 Stream Analysis
bash
[object Object],
,[object Object],
STREAM_IP=,[object Object],
STREAM_PORT=,[object Object],
,[object Object], ,[object Object],
,[object Object],
tshark -i eth0 -f ,[object Object], -T fields \
-e frame.time_relative \
-e rtp.seq \
-e rtp.timestamp \
-e frame.len \
-E header=y \
-E separator=, > st2110_analysis.csv
,[object Object],
python3 << ,[object Object],Custom Monitoring Dashboard
Web-Based Monitoring Interface
html
[object Object],
,[object Object],
,[object Object],
,[object Object],AV over IP Monitor,[object Object],
,[object Object],[object Object],
,[object Object],[object Object],[object Object],
,[object Object],
,[object Object],
,[object Object],AV over IP Network Monitor,[object Object],
,[object Object],
,[object Object],
,[object Object],Bandwidth Utilization,[object Object],
,[object Object],[object Object],
,[object Object],
,[object Object],
,[object Object],Network Latency,[object Object],
,[object Object],[object Object],
,[object Object],
,[object Object],
,[object Object],Active Streams,[object Object],
,[object Object],[object Object],
,[object Object],
,[object Object],
,[object Object],Error Statistics,[object Object],
,[object Object],[object Object],
,[object Object],
,[object Object],
,[object Object],[object Object],[object Object],
,[object Object],
,[object Object],Conclusion
Troubleshooting AV over IP systems requires a systematic approach combining traditional networking knowledge with AV-specific expertise. The key to successful troubleshooting lies in:
- Understanding the technology stack - Know whether you're dealing with NDI, ST 2110, SDVoE, or other protocols
- Systematic diagnosis - Start with physical layer issues and work up through the network stack
- Proper monitoring tools - Use both general networking tools and AV-specific analyzers
- Documentation - Keep detailed records of configurations and changes
- Proactive monitoring - Implement continuous monitoring to catch issues before they impact operations
By following the procedures and using the tools outlined in this guide, AV professionals can effectively diagnose and resolve the majority of AV over IP issues encountered in real-world deployments.
Remember that AV over IP troubleshooting often requires collaboration between AV engineers and network administrators. Ensure clear communication channels exist between these teams, and that both understand the unique requirements of professional AV applications.
For complex issues that persist after following this guide, consider engaging with AV over IP specialists or the equipment manufacturers' technical support teams, armed with the diagnostic data collected using these procedures.
End of documentation