Synchronize multiple HMPs

From SpinetiX Support Wiki

Jump to: navigation, search

Introduction

The first requirement for creating synchronization is to time-synchronize multiple HMP devices to the same NTP source(s) having a good accuracy (as in few milliseconds).

Because each HMP features a quartz-crystal internal clock that is subject to the clock drift effect, one or more NTP servers must be provided to compensate this effect. NTP can usually maintain time within tens of milliseconds over Internet and can achieve better than one millisecond accuracy in local area networks under ideal conditions.

It is recommended that the internal clock of each HMP is calibrated before using the player in a synchronized setup, because a non-calibrated player may behave very differently comparing to others that have been calibrated.

Using public NTP servers

It is possible to use public NTP servers to synchronize multiple HMP devices as long as they are fully reliable (low stratum value, reach rate close to 100%), because otherwise they could negatively influence the synchronization of players; when in doubt, go with local network NTP servers or use an HMP as NTP server for the other players alternatives.

For external NTP servers, check out the following links:


Note Note:
The default pool.ntp.org NTP servers should not be used as NTP sources when synchronizing HMP devices for playing synchronized content because they are not always well synchronized between them and not fully reliable; they are solely fine for getting time updates under regular usage.

Using local NTP servers

The best way to synchronize multiple HMPs is to use one or more local NTP servers - which could be a computer, a managed switch, a router, an embedded device etc.

  • All major Linux distributions normally have the NTP daemon pre-installed.
  • Managed switches and routers often have an NTP server included.
  • For Windows based servers, use the NTP distribution from Meinberg. Note that the built-in Windows Time Service cannot be used.
  • For local networks which have a permanent Internet connection to an ISP the easiest solution is to use the ISP's NTP server. Check with the ISP if it provides a NTP server to its customers and make sure that your firewall is properly configured to let NTP traffic through (UDP port 123 inbound and outbound).
  • Several companies also sell embedded, standalone, NTP servers which synchronize to GPS, longwave/shortwave time signals (e.g., DCF77, WWVB), GSM, CDMA and/or other NTP servers - here are some examples: Veracity, Symmetricom, Meinberg, EndRun Technologies, Zyfer, TimeTools, Galleon. Note that SpinetiX has not validated and does not endorse any of their products, the list is provided solely as a convenience.


Note Note: No matter what NTP server is used, it should be synchronized to a reliable source of time, as in other NTP servers or local clocks (GSM, DCF77 etc.).

Configuration

To change the NTP configuration follow these steps:

  1. Open Control Center and go to:
  2. Remove all the default pool.org NTP servers since they are not fully reliable for synchronized content.
  3. Add three or more NTP servers, ideally with different upstream servers.
    • Having only one NTP server is also acceptable, though it can be a single point of failure, but certainly, do not use 2 NTP servers because this may lead to non-synchronized playback (each HMP will select one or the other at random, since there is no way to tell which source is better).
  4. Click on the "Apply" button.

Rules

  1. The same NTP configuration is used for all the players.
  2. At least one NTP server is reachable, has its stratum value less than 16 (recommended values are: 1,2,3) and is selected as peer.
    • If you see messages in the NTP statistics concerning "outliers" then these servers are being seen as unreliable and their data is not used.
  3. The NTP servers are either always running or they are up and fully initialized by the time the HMP units boot.
    • For the latter, note that you can use "Pause device at startup ..." option to force the HMP to wait a certain period of time for the NTP source to be ready.

Using an HMP as NTP server

When none of the local NTP server options provided above can be used, one of the HMP devices can be used as master NTP server for the others, without a significant performance reduction in the vast majority of practical applications. It is strongly recommended to use fixed IP addresses for the players in such setups.


Note Note:
When no NTP is used, the internal clock of the "master" player may slowly drift with respect to actual time - this time drift is passed onto the other players so that all of them stay synchronized; the only inconvenient is that the time displayed on the screen might up to 1 min per month different than the actual time.

Configuration

To configure the HMPs for such a setup, follow these steps:

  1. Designate one HMP as the NTP server for the other players - this will be the "master" and all the other units will be "slaves".
  2. Make sure that the internal clock of the "master" HMP is calibrated.
  3. Configure the "master" HMP:
    1. Open Control Center and go to:
    2. Remove all the default pool.org NTP servers since they are not fully reliable for synchronized content.
    3. Select up to five HMPs from the "slaves" group for monitoring by the "master" HMP. If you have more than 5 "slaves", simply choose 5 of them.
    4. For each of these, enter their IP address as Server 1 / ... / Server 5, and enable the "Monitor only" option.
      • This is necessary for the "master" HMP to keep the synchronization with the rest of the players in case of a restart, by updating its time from one of the monitored "slaves".
      • Enabling the "Monitor only" option avoids creating loops that could make the synchronization unstable. Note that the monitored slaves will appear as rejected on the "NTP Statistics" tab.
      • It is not recommended to add external NTP servers (with or without checking the "Monitor only" option), unless they are always reachable or network interruptions do not last longer than about 30 minutes.
    5. Set the "Pause device at startup ..." option to 0 seconds (in case it was previously changed).
    6. Click on the "Apply" button.
  4. Configure all the "slave" HMP units (including the ones that are not monitored by the "master"):
    1. Open Control Center and go to:
    2. Remove all the default pool.org NTP servers.
    3. Set the IP address of the "master" HMP as the only NTP server (i.e. as "Server 1").
    4. Make sure that the "Monitor only" option is disabled.
    5. Set the "Pause device at startup ..." option to "90 s" (seconds)
      • This is necessary in case all the players are restarting in the same time, so that the "master" HMP can start and fully initialize its NTP service.
    6. Click on the "Apply" button.

Troubleshooting

If you are reading this, then something is not working properly and the players are not in sync. To troubleshoot this, follow these steps:

  1. Recheck the NTP configuration on all players.
    • Make sure that the NTP server addresses are correct. Note that pool.ntp.org servers must NOT be used.
    • If an HMP is used as NTP server for others, make sure that its IP address (or hostname) is set as the only entry in the NTP server configuration and the "Pause device at startup ..." option is set to "90 s" (seconds) on all the "slaves".
  2. Check the NTP statistics of each player for warning messages.
    • If there is a warning message about the local clock being not calibrated, or being calibrated with a dummy value of zero, then calibrate the internal clock of that HMP.
    • If there is a warning message about no NTP peer(s) being selected, then make sure that the NTP servers are physically accessible and check your network configuration - most common causes are firewalls preventing NTP network traffic (UDP port 123).
    • If there is a warning message saying that the value of the saved correction is unusually large, then verify that the selected NTP server is working reliably. If possible, change the that NTP server with a better one.
  3. Check that the selected NTP peer is well reached.
    For this, check the NTP statistics of each player for the reach level of the selected peer. If the reach level of the peer NTP server (other than loopback) is lower than 75%, then either the NTP server is not physically accessible (is it often rebooting?) or NTP requests are being "lost". For more details, read the "Low NTP server reach" section below.
  4. Check that the players are well NTP synchronized.
    For this, read "How to check the NTP synchronization" section below.

If all the above tests are fine, then the players are in sync and the desynchronization is most probably due to the content not being synchronized.

Low NTP server reach

Players getting out of sync could be due to a low reach of the NTP server. The percentage of successful attempts out of the player's last 8 attempts to reach the NTP server can be found within the NTP statistics displayed in Control Center - this should be as close as possible to 100%. If the reach percentage is low, then either the NTP server is not physically accessible or NTP requests are being "lost".

The latter is not so obvious to detect, unless some network traffic monitoring is done. But the following might give you an idea where to look first. An interesting case was brought to our attention, involving a local configuration with an HMP acting as NTP server for the other players - all players being connected on the same switch. Still, the "slaves" were showing low reaching percentages and some not even considering the "master" HMP as their NTP source.

The cause was the configuration of the aging time for the MAC table of the switch, set by default to 300 seconds.

The solution in such cases is to configure this switch setting to a higher value - our recommendation is between 10 to 30 minutes - at least higher than the ARP aging time. Note that using indefinite disables the MAC aging, so use this only if the switch has a fairly static number of end devices; otherwise the table will eventually fill up.

For more details, see the following articles:

How to check the NTP synchronization

To check if multiple players are well NTP synchronized, you can run the ntpdate command from a Linux machine (if using Windows, use the NTP distribution from Meinberg), followed by the -q option and the list of addresses to check against the NTP time of the reference, like this:

ntpdate -q Address_1 Address_2 Address_3
  • The Address_i could be either the IP or the hostname of the player.
  • From the command results, the data that need to be verified is the offset (in seconds). The value itself is not important (because it's the offset comparing to the NTP time of the Linux machine), what is important is the relative difference between the offset of each player - this needs to be less than 20ms for the players to be well NTP synchronized.
  • Note that any device at Stratum 16 is not considered to be synchronized.
This page was last modified on 3 October 2017, at 11:04.