Pull Mode

From SpinetiX Support Wiki

Jump to: navigation, search
Version: 2.0 (revision history)
Release date: January 8, 2015

Summary information and/or updates to the documentation can be found on this page.

Introduction

Pull Mode is a file transfer mechanism of the player, mainly used for updating the player content in particular scenarios, such as:

  • The players are located in private networks protected by firewall/NAT and direct network access is not avaialble to push the content updates.
  • The players are deployed in particular environments (e.g., slow network connection, 4G) where resilience to network failures or network usage optimization are mandatory.
  • The content must be retrieved from different locations/servers, sometimes managed by different persons, and merged on the player storage.
  • The content is generated by third-party web applications (CMS) and pulled by the players on demand.

To use Pull Mode with the player, you need to set up a content server and configure the player accordingly from Control Center. For advanced uses, you must configure the player to load an iCalendar file (.ics) written using the syntax described below.

Features

Pull Mode main feature is smart content update, yet it can also be used to upload certain files from the player onto a remote server, or as RPC bridge.

Smart content update

This feature offers many options for automatic update of the content from remote web servers, at predefined intervals (hourly, daily etc.), according to a custom schedule, or on demand.

When activated, the player will do the following:

  1. Connect to the specified content server and get the list of files present on that location.
  2. Compare that with the content present on its local storage.
  3. Remove from its local storage all files that are not present on the web server.
  4. Download any new/modified file onto the local storage. Unmodified files are not touched.
  5. When downloading has finished, the project's index.svg is restarted.
Note Notes:
  • The player will copy the content to its local storage, making the system resilient to network failures. Once the content has been downloaded, the server is no longer needed until the next update check. The downloading process was designed to work even for very poor connections - there is an initial 10 minutes time-out to establish the connection and later another 10 minutes time-out during the transmission of data.
  • The smart content pull ensures downloading only new or modified files, not everything else, thus saving network traffic / bandwidth - the Etag and/or Last-Modified HTTP directives are used to determine if a file was modified or not. See Caching page for more details.
  • The manual configuration allows for content download from one location. See also Scheduled Download feature of HMP400/W, HMP350, and HMP300 models, which allows configuring multiple locations.
  • For advanced users, the publish action offers even more options for fine-tuning the content download, for instance, the files can be deleted after finishing the download with the -deletelast option.

Files upload

This feature offers the possibility to transfer files, mainly logs (see Scheduled Upload), from the player onto a specified content server.

Configuration

To use Pull Mode with the player, you need to set up a content server and configure the player accordingly from Control Center. For advanced uses, a iCalendar (ics) file using a special syntax must be written and configured on the player.

In most cases, the server location would be protected and you'll need to configure the credentials to be used by the player to access that location.

HMP400/W, HMP350, HMP300

On these player models, the manual settings have been grouped under Content > Scheduled Download, allowing for content retrival from a server with a click of a button or in a scheduled manner (daily or hourly). See how to configure Scheduled Download on the player.

The advanced Pull Mode functionality, driven by an iCalendar (.ics) file, can be enabled from Control Center > Advanced Applications > Pull Mode - two options are available:

  • From uploaded iCalendar file (ics)
  • From remote iCalendar file (ics)
Note Notes:

HMP200, HMP130, HMP100

Pull Mode can be enabled from HMP Control Center > Content Settings > Pull Mode tab, by selecting one of the three available options:

  • Manual settings
    Enable this option to manually configure the HMP to: Update the content (daily or hourly) and to upload logs (daily or hourly).
  • From uploaded iCalendar file (ics)
  • From remote iCalendar file (ics)
Note Notes:

Content server

Pull Mode's implies using a public or private content server to host the content to be downloaded by the player via HTTP protocol. Any standard web server can be used, with or without support for WebDAV protocol. Most popular ones, like Apache or Microsoft IIS, support WebDAV, though usually it must be manually enabled; there are also some public WebDAV servers that can be used, for instance box or myDrive. Otherwise, some extra steps might be required for content servers without WebDAV support.

Note Notes:
  • Write access rights on the web server are not required, unless the upload action is used.
  • The port used by Pull Mode is the one you have specified when entering the project source / log destination - in the absence of such port, then the default port of the scheme is used, meaning 80 for http and 443 for https.
  • Although a player can technically be used as a WebDAV server, this configuration is not recommended because the it can only respond to maximum two concurrent clients (this includes player web interface, Control Center, Fusion) and the performance on the host player is degraded while the file reconciliation is taking place.
  • See also the generic remarks when using Web servers.

Content server with WebDAV support

Web Distributed Authoring and Versioning (WebDAV) is an extension of the Hypertext Transfer Protocol (HTTP) that allows clients to perform remote content authoring operations: create, change, move, and delete. The most important features of the WebDAV protocol include the maintenance of properties (such as the creation / modification date), namespace management, collections (creation, removal, and listing of various resources), and overwrite protection.

The player uses WebDAV's PROPFIND requests to retrieve first the collection structure (also known as directory hierarchy) of the remote server location and then the properties (such as the creation / modification date) of each file, so that only new or modified files are downloaded.

Most popular web servers, like Apache or Microsoft IIS, support WebDAV, though usually it must be manually enabled; there are also some public WebDAV servers that can be used. Here are some examples of web servers that can be used for Pull Mode:

Web server Notes
Apache Apache HTTP Server is a free and open-source cross-platform web server, developed and maintained by an open community of developers under the auspices of the Apache Software Foundation. For details on how to enable mod_dav module on Apache HTTP Server Version 2.2, see the official Apache Module mod_dav page.
Microsoft IIS Internet Information Services (IIS) is included with all Professional versions of Windows, so it can be used for testing, usually without further installation. However, by default, IIS is heavily locked down with regard to WebDAV.
  • IIS 7 introduced a new WebDAV stack which, among other changes, restricts WebDAV functions for unauthenticated (anonymous) users to PROPFIND.
  • PUT, MKCOL, PROPPATCH, COPY, MOVE, DELETE, and WebDAV-based GET requests all require authentication (see this Microsoft page for more details).
  • Ensure that SVG is listed in the allowed MIME types, and directory browsing is enabled if you are using log uploads.
  • If using basic authentication, ensure that only basic is enabled in the IIS site configuration, otherwise, the player may not be able to connect.

See also:

Box Box (formerly Box.net), based in Redwood City, California, is a cloud content management and file sharing service for businesses. The company uses a freemium business model to provide cloud storage and file hosting for personal accounts and businesses.

To use this WebDAV-enabled public server with Pull Mode, check out this tutorial: Use box.com as content server.

Note that as of October 25, 2019, Box has ended WebDAV support.

MyDrive MyDrive is a free and secure online storage service, based in Switzerland. It offers 512 MB space as a freemium service.

To use this WebDAV-enabled, public server with Pull Mode, check out this tutorial: Use myDrive as content server.

NAS devices Network-attached storage (NAS) is a file-level computer data storage server connected to a computer network, providing data access to a group of clients. NAS systems are networked appliances which contain one or more storage drives, often arranged into logical, redundant storage containers or RAID. SpinetiX does not recommend specific NAS servers, but mid-range manufacturers such as Synology, QNAP, and Netgear have performance NAS appliances which support WebDAV and are of a much more practical form factor than rack servers. They usually avoid unnecessary license costs.

Two tutorials are available: Use Synology NAS as content server and Use QNAP NAS as content server.

Content server without WebDAV support

Content servers without WebDAV support can be used with Pull Mode, yet an additional step might be required to allow the player to get the list of files present on that location. Standard HTTP commands don't allow listing the content of a plain HTTP web server location for security reasons - the solution is to use an XML repository descriptor file and configure the player to point to this XML file instead of the root of the repository.

When using the Publish Location feature of Elementi M / X such an XML file (named "spx-listing.xml") is automatically generated within the root of the repository. Otherwise, either use a script that generates it (see sample PHP scripts below) or create it manually, following the instructions detailed on the "XML repository descriptor file" page.

Advanced uses

Control Pull Mode with an ics file

For accurate control over the Pull Mode's actions to be performed by the player, create an iCalendar file (.ics) according to the API specification and configure it on the player. The ics file can be directly uploaded on the player from Control Center or it can be uploaded on a remote web server from which the player is instructed to retrieved it.

This ics file can be created using Elementi, Outlook, Google Calendar or manually within a text editor. It can be used to:

  • Update the player's content according to a predefined schedule, including using complex recurring rules, from one or more server locations.
  • Upload more player's logs, as well as snapshot images and the player's report onto a designated server.
  • Enable the communication with a RPC Concentrator and / or execute RPC commands.

Dynamic pull

Note  
The add_pull_action RPC command should be used to trigger a pull action dynamically, as it is more flexible than the method below.

In some rare scenarios, you may want to configure Pull mode to checks for updates very regularly (e.g. 5 mins). In a normal setup, this would consume many network resources and put unnecessary load on the player. To avoid this:

  • Configure the player to use a dummy ICS file on your remote server for the Pull schedule.
  • Set Check calendar every : to a small duration (1m or 5m for instance).
    The player will contact the server every time to check if the ICS file has been modified. It will use ETag and HEAD requests to check for a new file (see Caching page).
  • By default the server will return an empty ICS file.
  • To trigger a content Pull, change the ICS file on the server to add a single event taking place between some short time in the past, to a time in the future long enough for the Pull to complete (2 hours for instance).
    The event location should contain either the URI to the WebDAV server hosting the project, or to the XML describing the list of file to download if you are not using a WebDAV server.
  • This way, the player does not need to check the its entire storage at the interval specified, unless the ics file has changed.

Pull Mode and Fusion

Applies to HMP200, HMP130, HMP100.

When Fusion mode is enabled on the device, you can still use Pull mode (firmware 2.2.3 or later is required), thus allowing a mix of Fusion and externally managed content, however that content cannot be directly accessed from Fusion like any other media. Instead you need to reference such files within a custom skin, a custom slide template, an hypermedia project or any combination of these.

The default destination for a Download operation is no longer the root folder, but a sub-folder named "publish". In case an ics file is used to control the Pull Mode, then make sure to specify the destination folder like this: -dest publish.

Sample PHP scripts

Also check the RPC server-side application samples for remote managing of the player.

Files pull

Version: 1.0 (revision history)
Release date: 2013-06-13

SpinetiX is providing an automatic generation of XML description listing the content of a files folder (written in PHP) in order to make it easier for you to develop your own HTTP based Pull Mode server.

Note Note:
Make sure to check the README file before getting started!

Database pull

Version: 1.0 (revision history)
Release date: 2013-06-13

SpinetiX is providing a server side application for managing and testing the Pull Mode functionality of the players. Its purpose is to serve as an example and guide implementation for a simple Pull Mode server based on an SQLite database.

Note Note:
Make sure to check the README file before getting started!

API specification

This section details the syntax to use within the iCalendar file (.ics) used for advanced uses of Pull Mode.

Event

Each action of Pull mode is described by an event (item, meeting) within the ICS file, and this event must have a certain format.

The following iCalendar fields are used to define the action:

  • SUMMARY (also referred to as: Title or Subject) - specifies the action to be performed, as one of the following:
    • publish → to download a project (or just some files) from a server to the player,
    • upload → to upload different files from the player to a server,
    • rpc → to enable sending RPC statuses to a remote server (RPC Concentrator),
    • inline-rpc → to trigger an RPC command. See more details on Inline RPC.
  • URL (also referred to as: URI, Location, Where) → specifies the URI of the action.
  • DTSTART and DTEND (also referred to as: From & To, Start Time & End time) → specifies the starting and ending date and time for the event (or for the first event of a series if the recurring rule is used).
  • RRULE → specifies the recurrence rule applying to this event.
  • DESCRIPTION → specifies additional options for the upload or rpc actions.
  • Other fields, such as: UID, VTIMEZONE, PRIORITY etc., can also be used within the ics file.
Note Note:
In parentheses, there are the alternative names used within different applications for the above-mentioned iCalendar fields.

Duration of an event

The duration of an event represents the time period between DTSTART and DTEND, during which the player will try to execute the specified action (with other words, it's the validity period of the action, not the duration of the action itself, which can take far less or even more than the duration of the event).

  • If the action is not successful for any reasons (for instance, the server cannot be contacted), the player will retry the same action until the DTEND is reached; at that point the action is discarded even if the action has not been executed.
  • Once the action is successful, the event is removed from the Pull mode queue (event if the DTEND has not been reached).
Note Note:
If the player is restarted, the ICS file controlling the Pull mode will be reprocessed, which means that some past events might be executed again if their end-date is after the current time.

Recurrence rule

The RRULE property defines a repeating pattern for recurring events. It is used in computing the recurrence set of instances for that event, generated by considering the initial DTSTART property along with the RRULE, RDATE, and EXDATE properties contained within the recurring component. This property should not be specified more than once.

Here is an example of an event with a duration of 5m 30s, that is repeating every 15 minutes:

DTSTART:20150101T000000 
DTEND:20150101T000530
RRULE:FREQ=MINUTELY;INTERVAL=15;WKST=SU
Recurring events limits

When using recurrence rules, make sure that the following limits are respected:

  1. No more than 100 events are generated for any interval of 8 hours!
    This means that the repeating interval should not be less than 5 minutes if there is only one event/action, while if there are two events, the repeating interval for each event should not be less than 10 minutes.
  2. Avoid overlapping an event by the other events from its series!
    This means that if an action needs to be executed every two hours, the duration of the event should not be greater than two hours - it could be exactly 2h, or it could be less (e.g., 1h 59m 59s).
    DTSTART:20150101T000000 
    DTEND:20150101T015959
    RRULE:FREQ=HOURLY;INTERVAL=2;WKST=SU
    

Publish action

The "publish" action controls when the content of the device should be updated from a remote server. At the end of a successful publish-action, the content of the device is an exact copy of the content on the server and the main index.svg file is reloaded.

The following iCalendar fields are used to define the "publish" action:

  • SUMMARY:publish
  • URL:server_location
    This must point either to a remote location (folder) on an WebDAV-enabled server or to an XML repository descriptor file in the case of a standard HTTP server.
  • DESCRIPTION:parameters
    Optional field to specify additional parameters for the "publish" action, each separated by the newline character (\n), from the following:
    • -deletelast
      By default, the removable files are deleted at the beginning of the "publish" action to save space on the player's storage. When this option is specified, the files are deleted after the new/modified ones have been uploaded on the player.
    • -dest folder
      This option changes the publishing destination from the content root folder (the default behavior) to an inner folder, and the main index.svg file is no longer reloaded. The folder path entered as destination must be URI encoded and must not start by "/" or "\" – if the specified folder does not exist on the player, it will be created.
    • -norecurs
      Prevents recursion during the publishing process; only the root folder or the one specified with -dest option is modified, and no other sub-folders are created or deleted.
    • -statusinterval n_seconds (since 3.0.3 release)
      Sends an update every n seconds to the RPC concentrator, if one is configured on the player.
    • -statusnrfiles n_files (since 3.0.3 release)
      Sends an update every n files to the RPC concentrator, if one is configured on the player.
    • -statussize n_bytes (since 3.1.0 release)
      Sends an update every n bytes to the RPC concentrator, if one is configured on the player.
    • -feedback uri [fulllisting]
      If there's no RPC Concentrator configured, use this option to have a feedback update sent to the provided URI at the end of the publishing action. The HTTP POST call contains the following items: start time, number of uploaded files, number of errors, status type ("final"), and, if fulllisting is present, the content of the XML repository descriptor file.
Note Notes:
  • If -statusinterval, -statusnrfiles, or -statussize options are used together, a pull_status notification is sent when all the conditions are reached.
  • On legacy players, when Fusion interface is enabled, you must use -dest publish, otherwise the publishing won't be executed.
Example

To instruct the player to check for updates/download a project from SpinetiX server, everyday starting at 6 AM (and up to 7 AM), the ICS file would look like this:

BEGIN:VCALENDAR 
VERSION:2.0 
PRODID:-//Spinetix.com/NONSGML Spinetix Control Center//EN 
BEGIN:VEVENT 
UID:14d64d53-a926-4a71-848d-c78452b195fe 
DTSTART:20190101T060000 
DTEND:20190101T070000
RRULE:FREQ=DAILY;WKST=SU  
SUMMARY:publish 
URL: http://demo.spinetix.com/project/default/publish.xml
DESCRIPTION:-deletelast
END:VEVENT 
END:VCALENDAR
Note Note:
See also the publish screen feature.

Upload action

The upload action can be used to upload different files (logs, snapshot images, report files) from the player to a remote server (usually a WebDAV enabled server, though a standard HTTP server can be used as well). This action doesn't affect in any way the content on the player.

The upload action uses the following fields:

  • SUMMARY:upload
  • URL:<server_location>
    It must point to a server location (e.g., http://webdav.spinetix.com/logs/) where the player should upload the specified files.
  • DESCRIPTION:[<options>\n]<file>[ <destination_name>]
    Contains one or more file upload instructions, eventually using a custom destination name and / or preceded by one or more options for the upload action. Each file upload instruction must be separated by "\n" (newline character).
    <file>
    This can be an URI (e.g., http://localhost/status/snapshot) pointing to the file or one of the following shortcuts:
    • -accounting : to upload the accounting log files.
    • -alllogs : to upload all the logs, both current (plain text) and past (archived).
    • -content : to upload an XML repository descriptor file containing the list of files present on the player content server.
    • -currentlogs : to upload the current logs (plain text).
    • -log <name> : to upload a specific log file, being any of the regular logs (e.g., -log accounting.log, -log uploader.log etc.), but also the player's info status file (-log info) or the system log (-log syslog). Added in 3.0.0 release.
    • -logs : to upload the archived logs from the previous days; a check is done to make sure all past logs are uploaded. Note that at most 7 days of logs are kept onto the player.
    • -pastlogs : to upload the past logs (archived); no checks are done to find out whether any of the logs have already been uploaded on the server.
    • -report : to upload a standard report file of the player.
    • -snapshot : to upload the player snapshot.
    <destination_name>
    Specifies the naming format for the uploaded file through the following special placeholders:
    • %D : modification date of the uploaded file
    • %H : modification time of the uploaded file
    • %d : date of the upload
    • %h : time of the upload
    • %n : name of the source file
    • %s : serial number of the player
    • %x : extension of the source file
    If missing, the default file naming format is set to %n-%s-%d-%h.%x.
    <options>
    One or more options can be specified for the following batch of files to upload, each option separated by "\n" (newline character).
    • -buffer true : to force the buffering of files sent to the server to ensure that the Content-Length header is valid (necessary for any upload to Amazon S3 or mydrive.ch for instance). Added in 3.0.6 release.
    • -data <key> <value> : to specify a key=value pair to be added to the POST request. This option can be specified multiple times in case more key/value pairs need to be added. Note: using -data reset will remove all the key / value pairs previously set. Added in 3.0.6 release.
    • -format <destination_name_syntax> : to specify a global file naming format, using the syntax specified above, for files specified using URIs; this doesn't apply to shortcuts.
    • -header <name> <value> : to specify additional HTTP headers to be added to the PUT / POST request. This option can be used multiple times in case more headers need to be added. Note: using -header reset will remove all the headers previously set. Added in 3.0.6 release.
    • -method post : to set the upload method as POST (make sense in the case of a regular HTTP server), instead of the default PUT method.
Example
BEGIN:VCALENDAR 
VERSION:2.0 
PRODID:-//Spinetix.com/NONSGML Spinetix Control Center//EN 
BEGIN:VEVENT 
UID:14d64d53-a926-4a71-848d-c78234b195fg 
DTSTART:20190101T000000 
DTEND:20190101T013000
RRULE:FREQ=HOURLY;INTERVAL=2;WKST=SU
SUMMARY:upload 
URL: http://webdav.spinetix.com/logs/ 
DESCRIPTION:-buffer true\n-currentlogs\n-snapshot %s-%n.jpg
END:VEVENT 
END:VCALENDAR

This command instructs the player to upload every two hours the player's current log files and snapshot (using a custom name format) onto a server. The option -buffer true applies to all uploads.

RPC action

The RPC action allow configuring an RPC Concentrator through the ICS file controlling the Pull Mode. This feature was added in firmware 3.0.0.

The following iCalendar fields are used to define the RPC action:

  • SUMMARY:rpc
  • URL:RPC_Concentrator_uri
    Specifies the URI of the RPC Concentrator.
  • DESCRIPTION:parameters
    Optional field to specify additional parameters for the RPC Concentrator, each separated by the newline character (\n), such as:
    • -polling n_seconds
      The polling frequency, in seconds, at which the "ready" notification is sent to the RPC concentrator – typical values are between 10s and 120s. If omitted, it defaults to one hour.
    • -notification
      Indicates that no RPC commands are accepted within the server's response to the player's "ready" notification.
    • -info n_seconds flags_object
      Indicates that the player must send an "info" notification at the predefined interval in seconds, rounded to the next multiple of the pooling time. If the “flags” JSON object is omitted, it defaults to using the "all" flag.
    • -restarted flags_object
      Indicates that the "restarted" notification must follow the provided "flags" (JSON object) – the default flags are "reason" and "status".
The optional "flags_object" parameter must be provided as a JSON object, such as: { "all": true }.
For more details about these flags, see the get_info command in the RPC API technical documentation.

Inline RPC action

The Inline-RPC action allow using RPC commands inside the ICS file controlling the Pull Mode. This feature was added in firmware 3.0.0.

The following iCalendar fields are used to define the inline RPC command:

  • SUMMARY:inline-rpc
  • DTSTART:datetime and DTEND:datetime
    Specifies the starting and ending date and time for the event (or for the first event of a series if the recurring rule is used).
  • RRULE
    Specifies the recurrence rule(s) applying to this event.
  • DESCRIPTION
    Specifies the RPC command to be executed.
Note Notes:
  • The ICS file can be uploaded to the player or referenced from a web server, as well as included within a custom configuration file.
  • Contrary to all the other events controlled by ICS files, an inline RPC command will only be executed if the player is turned on at the start time of the event!

Daily player restart

For instance, to trigger a restart of the player each night at 1 AM, add an event like this:

BEGIN:VCALENDAR 
VERSION:2.0 
PRODID:-//Spinetix.com/NONSGML Spinetix Control Center//EN 
BEGIN:VEVENT 
UID:14d64d53-a926-4a71-848d-c78452b345fe 
DTSTART:20220801T010000 
DTEND:20220801T013000 
SUMMARY:inline-rpc 
DESCRIPTION: {"method":"restart","params":[],"id":1} 
RRULE:FREQ=DAILY;WKST=SU 
END:VEVENT 
END:VCALENDAR

Troubleshooting

Common problems

Note  
Errors regarding Pull Mode operations are stored in the uploader.log file.
  • The error "Could not read status line: connection was closed by server " means that the server is reachable but it is returning an error on the path being used. Check the capitalisation, spelling and syntax of the URL is correct.
  • 404: File Not Found errors: Ensure that the not-found file is added as a valid MIME type for the WebDAV server you are using.
    • In most cases that's caused by IIS server not allowing the SVG files (the MIME type is: image/svg+xml).
  • IIS returns "unauthorised" errors: Check that the user IIS_IUSR is added with Read permissions on the Virtual Directory you are using.
  • Spaces in the path in Pull Mode may cause problems. Use underscores instead of spaces.

Enable extended logging

The extended logging of Pull Mode actions should only be enabled when asked by SpinetiX Support.

For HMP400/W, HMP350, and HMP300 devices, you need to apply the following config file:

<?xml version="1.0"?>
<configuration version="2.0">  
    <pullmode-logs-level>trace</pullmode-logs-level>    
</configuration>
Note Note:
Replace trace by info when done.

For HMP200, HMP130, and HMP100 devices, follow this procedure:

  1. Open HMP Control Center.
  2. On the "Status" page, find the HMP picture and click over the SpinetiX logo (in case of an HMP200) or the blue button (in case of an HMP130 / HMP100).
  3. A confirmation dialog appears - click "Yes" to enable the "Advanced settings" menu on the left side.
  4. Go to "Content Settings" page > "Pull Mode" tab > "Log settings" section.
  5. Set the "Log level" option to "Trace".
  6. Click the "Apply" button.
  7. Wait for the problematic action to be executed by the HMP, then generate a report and send it to SpinetiX Support for further analysis.
Note Note:
Once the issue has been identified / solved, make sure to set the log level back to "Info".
This page was last modified on 22 December 2023, at 17:44.