Synchronize multiple players

 

This page is related to Multiscreen synchronization.

Introduction

Multiple SpinetiX players can ensure seamless synchronization of content across multiple screens when they are time-synchronized, meaning that they have the same date/time information. This is achieved through the Network Time Protocol (NTP). Network Time Protocol (NTP), a networking protocol for clock synchronization between computer systems over packet-switched, variable-latency data networks. In operation since before 1985, NTP is one of the oldest Internet protocols in current use.

Each player features a quartz-crystal internal clock that is subject to the clock drift effect – to compensate this effect, one or more NTP servers, having a good accuracy (as in few milliseconds), must be provided. NTP can usually maintain time within tens of milliseconds over Internet and can achieve better than one millisecond accuracy in local area networks under ideal conditions.

There are three ways to time-sync your players:

• Using public NTP servers
• Using local NTP servers
• Using one player as NTP server

Note:

It is recommended to calibrate the player's internal clock before using it in a synchronized setup, because a non-calibrated player may behave very differently comparing to others that have been calibrated.

Using public NTP servers

It is possible to use public NTP servers to synchronize multiple players as long as they are fully reliable (low stratum value, reach rate close to 100%), because otherwise they could negatively influence the synchronization of players; when in doubt, go with local network NTP servers or use one player as NTP server for the other players alternatives. Network port UDP 123 must be open for connecting to external NTP servers.

External NTP servers can be found on these lists:

• Public NTP Secondary (stratum 2) Time Servers
• Public NTP Primary (stratum 1) Time Servers (better, but might not all be open for public usage)

Note:

The pool.ntp.org NTP servers configured by default on the players should not be used when playing synchronized content because they are not always well synchronized between them and not fully reliable; they are solely fine for getting time updates under regular usage.

Using local NTP servers

The best way to synchronize multiple players is to use one or more local NTP servers – which could be a computer, a managed switch, a router, an embedded device, etc.

• All major Linux distributions normally have the NTP daemon preinstalled.
• Managed switches and routers often have an NTP server included.
• For Windows-based servers, the built-in Windows Time Service (W32Time) cannot be used for this application – you can use instead the NTP distribution from Meinberg.
• For local networks which have a permanent Internet connection to an ISP, the easiest solution is to use the ISP's NTP server. Check with the ISP if it provides a NTP server to its customers, and make sure that your firewall is properly configured to let NTP traffic through (UDP port 123 inbound and outbound).
• Several companies also sell embedded, standalone, NTP servers which synchronize to GPS, longwave/shortwave time signals (e.g., DCF77, WWVB), GSM, CDMA and/or other NTP servers - here are some examples: Veracity, Microchip, Meinberg, EndRun Technologies, FEI-Zyfer, TimeTools, Galleon. Note that SpinetiX has not validated and does not endorse any of their products, the list is provided solely as a convenience.

Note:

No matter what NTP server is used, it should be synchronized to a reliable source of time, as in other NTP servers or local clocks (GSM, DCF77, etc.).

Configuration

Configure a local NTP server

To change the NTP configuration, follow these steps:

1. Open Control Center and go to Network → NTP page.
   • For legacy players, go to Date/Time page → Time Sources tab.
2. Remove all the default pool.org NTP servers since they are not fully reliable for synchronized content.
3. Add preferably three or more NTP servers, ideally with different upstream servers.
   • Having only one NTP server is also acceptable, although it can be a single point of failure, but certainly, do not use 2 NTP servers because this may lead to non-synchronized playback (each player will select one or the other at random, since there is no way to tell which source is better).
4. Click on the "Apply" button.

Rules

1. The same NTP configuration is used for all the players.
2. At least one NTP server is reachable, has its stratum value less than 16 (recommended values are: 1,2,3) and is selected as peer.
   • If you see messages in the NTP statistics concerning "outliers", then these servers are being seen as unreliable and not used.
3. The NTP servers are either always running or they are up and fully initialized by the time the players boot. For the latter, use "Pause device at startup ..." option to force the player to wait a certain period for the NTP source to be ready.

Using a player as NTP server

When none of the local NTP server options provided above can be used, one of the players can be used as primary NTP server for the others, without a significant performance reduction in the vast majority of practical applications. It is strongly recommended to use fixed IP addresses for the players in such setups.

Note:

When no NTP is used, the internal clock of the "primary" player may slowly drift with respect to actual time – this time drift is passed onto the other players so that all of them stay synchronized; the only inconvenient is that the time displayed on the screen might up to 1 min per month different than the actual time.

Configuration

To configure the players for such a setup, follow these steps:

1. Designate one player as the NTP server for the other players – this will be the "primary" and all the other units will be "followers".
2. Make sure that the internal clock of the "primary" player is calibrated.
3. Configure the "primary" player:
   1. Open Control Center and go to Network → NTP page.
      • For legacy players, go to Date/Time page → Time Sources tab.
   2. Remove all the default pool.org NTP servers since they are not fully reliable for synchronized content.
   3. Select up to five players from the "followers" group for monitoring by the "primary" player. If you have more than 5 "followers", simply choose 5 of them.
   4. For each of these, enter their IP address or hostname address as Server 1 / ... / Server 5, and enable the "Monitor only" option.
      • This is necessary for the "primary" player to keep the synchronization with the rest of the players in case of a restart, by updating its time from one of the monitored "followers".
      • Enabling the "Monitor only" option avoids creating loops that could make the synchronization unstable. Note that the monitored followers appear as rejected on the "NTP Statistics" sectopm.
      • It is not recommended adding external NTP servers (with or without checking the "Monitor only" option), unless they are always reachable or network interruptions do not last longer than about 30 minutes.
   5. Set the "Pause device at startup ..." option to 0 seconds (in case it was previously changed).
   6. Click on the "Apply" button.
4. Configure all the "follower" players (including the ones that are not monitored by the "primary" one):
   1. Open Control Center and go to Network → NTP page.
      • For legacy players, go to Date/Time page → Time Sources tab.
   2. Remove all the default pool.org NTP servers.
   3. Set the IP or hostname address of the "primary" player as the only NTP server (i.e., as "Server 1").
   4. Make sure that the "Monitor only" option is disabled.
   5. Set the "Pause device at startup ..." option to "90 s" (seconds)
      • This is necessary in case all the players are restarting in the same time, so that the "primary" player can fully initialize its NTP service.
   6. Click on the "Apply" button.

Troubleshooting

If you are reading this, then something is not working properly and the players are not in sync. To troubleshoot this, follow these steps:

1. Recheck the NTP configuration on all players.
   • Make sure that the NTP server addresses are correct. Note that pool.ntp.org servers must NOT be used.
   • If a player is used as NTP server for others, make sure that its IP address (or hostname) is set as the only entry in the NTP server configuration and the "Pause device at startup ..." option is set to "90 s" (seconds) on all the "followers".
2. Check the NTP statistics of each player for warning messages.
   • If there is a warning message about the local clock being not calibrated, or being calibrated with a dummy value of zero, then calibrate the internal clock of that player.
   • If there is a warning message about no NTP peer(s) being selected, then make sure that the NTP servers are physically accessible and check your network configuration – most common causes are firewalls preventing NTP network traffic (UDP port 123).
   • If there is a warning message saying that the value of the saved correction is unusually large, then verify that the selected NTP server is working reliably. If possible, change the that NTP server with a better one.
3. Check that the selected NTP peer is well reached.

   For this, check the NTP statistics of each player for the reach level of the selected peer. If the reach level of the peer NTP server (other than loopback) is lower than 75%, then either the NTP server is not physically accessible (is it often rebooting?) or NTP requests are being "lost". For more details, read the "Low NTP server reach" section below.

4. Check that the players are well NTP synchronized.

   For this, read "How to check the NTP synchronization" section below.

If all the above tests are fine, then the players are in sync and the desynchronization is most probably due to the content not being synchronized.

Low NTP server reach

Players getting out of sync could be due to a low reach of the NTP server. The percentage of successful attempts out of the player's last 8 attempts to reach the NTP server can be found within the NTP statistics displayed in Control Center – this should be as close as possible to 100%. If the reach percentage is low, then either the NTP server is not physically accessible or NTP requests are being "lost".

The latter is not so obvious to detect, unless some network traffic monitoring is done. But the following might give you an idea where to look first. An interesting case was brought to our attention, involving a local configuration with a player acting as NTP server for the other players – all players being connected on the same switch. Still, the "followers" were showing low reaching percentages and some not even considering the "primary" player as their NTP source.

The cause was the configuration of the aging time for the MAC table of the switch, set by default to 300 seconds.

The solution in such cases is to configure this switch setting to a higher value (our recommendation is between 10 to 30 minutes), at least higher than the ARP aging time. Note that using indefinite disables the MAC aging, so use this only if the switch has a fairly static number of end devices; otherwise the table will eventually fill up.

For more details, see the following articles:

• Cisco Nexus 3000 Series NX-OS Layer 2 Switching Configuration Guide
• Juniper EX Series mac-table-aging-time
• Netgear GS700AT Series Smart Switch Software Administration Manual

How to check the NTP synchronization

To check if multiple players are well NTP synchronized, you can run the ntpdate command from a Linux machine (if using Windows, use the NTP distribution from Meinberg), followed by the -q option and the list of addresses to check against the NTP time of the reference, like this:

ntpdate -q Address_1 Address_2 Address_3

• The Address_i could be either the IP or the hostname of the player.
• From the command results, the data that need to be verified is the offset (in seconds). The value itself is not important (because it's the offset comparing to the NTP time of the Linux machine), what is important is the relative difference between the offset of each player – this needs to be less than 20ms for the players to be well NTP synchronized.
• Note that any device at Stratum 16 is not considered to be synchronized.