Pull Mode

From SpinetiX Support Wiki

(Redirected from Pull mode)
Jump to: navigation, search
"Documentation"
SpinetiX Technical Documentation - Version 2.0 (2015-01-08)

Summary information and/or updates to the documentation can be found on this page.

Introduction

Pull Mode is a content transfer mechanism of the Hyper Media Player, mainly used for updating the player content in particular scenarios, where:

  • Players are located in private networks protected by firewall / NAT and direct network access is not allowed for updating the content.
  • Players are deployed in particular environments (slow network connection, 4G) where resilience to network failures or network usage optimization are mandatory.
  • Content is generated by third-party web applications / CMS.
  • Content must be retrieved from different locations / servers, eventually managed by different persons, and merged into one.

Pull Mode main feature is smart content update, yet it can also be used to upload certain files from the player onto a remote server, or as RPC bridge.

To use Pull Mode with the HMP, you need to set up a content server and configure the player accordingly from Control Center. For advanced uses, a iCalendar (ics) file using a special syntax must be written and configured on the player.

Features

Smart content update

This feature offers many options for automatic update of the content from remote web servers, at predefined intervals (hourly, daily etc.) or on demand.

When activated, the HMP will do the following:

  1. Connect to the specified content server.
  2. Get the list of files present on that location and compare it with the content present on its local storage.
  3. Remove from its local storage all files that are no longer present on the web server.
  4. Download all the new and modified files onto the local storage. Unmodified files will not be touched.
  5. When finishing downloading, the project's index.svg is restarted.


Note Notes:
  • Content on HMPs behind firewall or NAT can be updated practically from anywhere.
  • The HMP will copy the content to its local storage, making the system resilient to network failures. Once the content has been downloaded, the server is no longer needed until the next update check. The downloading process was designed to work even for very poor connections - there is an initial 10 minutes time-out to establish the connection and later another 10 minutes time-out during the transmission of data.
  • The smart content pull ensures downloading only new or modified files, not everything else, thus saving network traffic / bandwidth - the Etag and/or Last-Modified HTTP directives are used to determine if a file was modified or not. See Caching page for more details.
  • The manual configuration allows for content download from one location. See also Scheduled Download feature of HMP350 and HMP300 models, which allows also for configuring multiple locations.
  • For advanced users, the publish action offers even more options for fine-tuning the content download - for instance, the order of the last two steps above can be changed using the -deletelast option.

Files upload

This feature offers the possibility to transfer files, mainly logs (see Scheduled Upload), from the player onto a specified content server.

Configuration

To use Pull Mode with the HMP, you need to set up a content server and configure the player accordingly from Control Center. For advanced uses, a iCalendar (ics) file using a special syntax must be written and configured on the player.

In most cases, the server location would be protected and you'll need to configure the credentials to be used by the player to access that location.

HMP350 and HMP300

On these player models, the manual settings have been grouped under Content > Scheduled Download, allowing for content retrival from a server with a click of a button or in a scheduled manner (daily or hourly). See how to configure Scheduled Download on the player.

The advanced Pull Mode functionality, driven by an iCalendar (.ics) file, can be enabled from HMP Control Center > Advanced Applications > Pull Mode - two options are available:

  • From uploaded iCalendar file (ics)
  • From remote iCalendar file (ics)


Note Notes:

HMP200, HMP130, HMP100

Pull Mode can be enabled from HMP Control Center > Content Settings > Pull Mode tab, by selecting one of the three available options:

  • Manual settings
    Enable this option to manually configure the HMP to: Update the content (daily or hourly) and to upload logs (daily or hourly).
  • From uploaded iCalendar file (ics)
  • From remote iCalendar file (ics)


Note Notes:

Content server

Pull Mode's implies using a public or private content server to host the content to be downloaded by the player via HTTP protocol. Any standard file server can be used, with or without support for WebDAV protocol. Most popular web servers, like Apache or Microsoft IIS, support WebDAV, though usually it must be manually enabled; there are also some public WebDAV servers that can be used, for instance box or myDrive. Otherwise, some extra steps might be required for content servers without WebDAV support.


Note Notes:
  • Write access rights on the web server are not required, unless the upload action is used.
  • The port used by Pull Mode is the one you have specified when entering the project source / log destination - in the absence of such port, then the default port of the scheme is used, meaning 80 for http and 443 for https.
  • Although an HMP can technically be used as a WebDAV server, this configuration is not recommended because the HMP can only respond to maximum two concurrent clients (this includes player web interface, HMP Control Center, Fusion) and the performance on the host HMP is degraded while the file reconciliation is taking place.
  • See also the generic remarks when using Web servers.

Content server with WebDAV support

Web Distributed Authoring and Versioning (WebDAV) is an extension of the Hypertext Transfer Protocol (HTTP) that allows clients to perform remote content authoring operations: create, change, move, and delete. The most important features of the WebDAV protocol include the maintenance of properties (such as the creation / modification date), namespace management, collections (creation, removal, and listing of various resources), and overwrite protection.

The HMP will use WebDAV's PROPFIND requests to retrieve first the collection structure (also known as directory hierarchy) of the remote server location and then the properties (such as the creation / modification date) of each file, so that only new or modified files are downloaded.

Most popular web servers, like Apache or Microsoft IIS, support WebDAV, though usually it must be manually enabled; there are also some public WebDAV servers that can be used. Here are some examples of web servers that can be used for Pull Mode:

Web server Notes
Apache Apache HTTP Server is a free and open-source cross-platform web server, developed and maintained by an open community of developers under the auspices of the Apache Software Foundation. For details on how to enable mod_dav module on Apache HTTP Server Version 2.2, see the official Apache Module mod_dav page.
Microsoft IIS Internet Information Services (IIS) is included with all Professional versions of Windows, so it can be used for testing, usually without further installation. However, by default, IIS is heavily locked down with regard to WebDAV.
  • IIS 7 introduced a new WebDAV stack which, among other changes, restricts WebDAV functions for unauthenticated (anonymous) users to PROPFIND.
  • PUT, MKCOL, PROPPATCH, COPY, MOVE, DELETE, and WebDAV-based GET requests all require authentication (see this Microsoft page for more details).
  • Ensure that SVG is listed in the allowed MIME types, and directory browsing is enabled if you are using log uploads.
  • If using basic authentication, in the IIS site configuration ensure that only basic is enabled. Enabling multiple types will result in the HMP not being able to connect.

See also:

Box Box (formerly Box.net), based in Redwood City, California, is a cloud content management and file sharing service for businesses. The company uses a freemium business model to provide cloud storage and file hosting for personal accounts and businesses.

To use this WebDAV-enabled public server with Pull Mode, check out this tutorial: Use box.com as content server.

Note that Box will be disabling WebDAV access on Jan 31, 2019 so any WebDAV-based access will stop working at that time.

MyDrive MyDrive is a free and secure online storage service, based in Switzerland. It offers 100MB space as a freemium service.

To use this WebDAV-enabled, public server with Pull Mode, check out this tutorial: Use myDrive as content server.

NAS devices Network-attached storage (NAS) is a file-level computer data storage server connected to a computer network providing data access to a group of clients. NAS systems are networked appliances which contain one or more storage drives, often arranged into logical, redundant storage containers or RAID. SpinetiX does not recommend specific NAS servers, but mid-range manufacturers such as Synology, QNAP and Netgear have performance NAS appliances which support WebDAV and are of a much more practical form factor than rack servers. They usually avoid unnecessary license costs.

Two tutorials are available: Use Synology NAS as content server and Use QNAP NAS as content server.

Content server without WebDAV support

Content servers without WebDAV support can be used with Pull Mode, yet an additional step might be required to allow the player to get the list of files present on that location. Standard HTTP commands don't allow listing the content of a plain HTTP web server location for security reasons - the solution is to use an XML repository descriptor file and configure the player to point to this XML file instead of the root of the repository.

When using the Publish Location feature of Elementi M / X such an XML file (named "spx-listing.xml") is automatically generated within the root of the repository. Otherwise, either use a script that generates it (see sample PHP scripts below) or create it manually, following the instructions detailed on the "XML repository descriptor file" page.

Advanced uses

Control Pull Mode with an ics file

For accurate control over the Pull Mode's actions to be performed by the player, create an iCalendar file (.ics) according to the API specification and configure it on the HMP. The ics file can be directly uploaded on the HMP (from Control Center) or it can be uploaded on a remote web server from which the HMP will be instructed to retrieved it.

This ics file can be created using Elementi, Outlook, Google Calendar or manually within a text editor. It can be used to:

  • Update the player's content according to a predefined schedule, including using complex recurring rules, from one or more server locations.
  • Upload more player's logs, as well as snapshot images and the player's report onto a designated server.
  • Enable the communication with a RPC Concentrator and / or execute RPC commands.

Dynamic pull

The add_pull_action RPC command should be used to trigger a pull action dynamically. It is more flexible than the method described bellow.

In some rare scenarios, you may want to configure Pull mode to checks for updates very regularly (e.g. 5 mins). In a normal setup, this would consume many network resources and put unnecessary load on the HMP. To avoid this:

  • Configure the HMP to use a dummy ICS file on your remote server for the Pull schedule.
  • Set Check calendar every : to a small duration (1m or 5m for instance).
    The player will contact the server every time to check if the ICS file has been modified. It will use ETag and HEAD requests to check for a new file (see Caching page).
  • By default the server will return an empty ICS file.
  • To trigger a content Pull, change the ICS file on the server to add a single event taking place between some short time in the past, to a time in the future long enough for the Pull to complete (2 hours for instance).
    The event location should contain either the URI to the WebDAV server hosting the project, or to the XML describing the list of file to download if you are not using a WebDAV server.
  • This way, the player does not need to check the its entire storage at the interval specified, unless the ics file has changed.

Pull Mode and Fusion

Applies to HMP200, HMP130, HMP100.

When Fusion mode is enabled on the device, you can still use Pull mode (firmware 2.2.3 or later is required), thus allowing a mix of Fusion and externally managed content, however that content cannot be directly accessed from Fusion like any other media. Instead you need to reference such files within a custom skin, a custom slide template, an hypermedia project or any combination of these.

The default destination for a Download operation is no longer the root folder, but a sub-folder named "publish". In case an ics file is used to control the Pull Mode, then make sure to specify the destination folder like this: -dest publish.

Sample PHP scripts

Also check the RPC server-side application samples for remote managing of the HMP.

Files pull

SpinetiX is providing an automatic generation of XML description listing the content of a files folder (written in PHP) in order to make it easier for you to develop your own HTTP based Pull Mode server.

Note Note:
Make sure to check the README file before getting started!

Database pull

SpinetiX is providing a server side application for managing and testing Pull Mode functionality of HMP devices. Its purpose is to serve as an example and guide implementation for a simple Pull Mode server for HMP devices based on a Database. All Pull Mode server activity is done through an SQLite database.

Note Note:
Make sure to check the README file before getting started!

API specification

This section details the syntax to use within the iCalendar file (.ics) used for advanced uses of Pull Mode.

Event

Each action of Pull mode is described by an event (item, meeting) within the ics file and this event must have a certain format.

The following iCalendar fields are used to define the action (in parenthesis the alternative names used within applications for these fields):

  • SUMMARY (also referred to as: Title or Subject) - specifies the action to be performed and can take one of the following values:
    • publish - to publish a project (or just some files) from a server to the HMP (a.k download),
    • upload - to upload different files from the HMP to a server,
    • rpc - to enable sending RPC statuses to a remote server (RPC Concentrator).
    • inline-rpc - to trigger an RPC command.
  • URL (also referred to as: URI, Location, Where) - specifies the URI of the action.
  • DTSTART and DTEND (also referred to as: From & To, Start Time & End time) - specifies the starting and ending date and time for the event (or for the first event of a series if the recurring rule is used).
  • RRULE - specifies the recurrence rule(s) applying to this event.
  • DESCRIPTION - specifies additional options for the upload or rpc actions.
  • Other fields, such as: UID, VTIMEZONE, PRIORITY etc., can also be used within the ics file.

Duration of an event

The duration of an event represents the time period between DTSTART and DTEND, during which the player will try to execute the specified action (with other words, it's the validity period of the action, not the duration of the action itself, which can take far less or even more that the duration of the event).

  • If the action is not successful for any reasons (for instance, the server cannot be contacted), the player will retry the same action until the DTEND is reached; at that point the action is discarded even if the action has not been executed.
  • Once the action is successful, the event is removed from the Pull mode queue (event if the DTEND has not been reached).
Note Note:
If the player is restarted, the ics file controlling the Pull mode will be reprocessed, which means that some past events might be executed again if their DTEND is still in the future.

Recurrence rules

When using recurrence rules, make sure that no more than 100 events are generated for any interval of 8 hours!

  • For instance, if there is only one event / action, the repeating interval should not be less than 5 minutes, while if there are two events, the repeating interval for each event should not be less than 10 minutes.

A special care should be taken to avoid overlapping an event by the other events from its series. For instance, for an action than needs to be executed every two hours, the duration of the event should not be greater than two hours - it could be exactly 2h or it could be less.

  • Example #1 shows an event with a duration of 1h 59m 59s, that is repeating every 2 hours.
  • Example #2 shows an event with a duration of 5m 30s, that is repeating every 15 minutes.

Example #1

DTSTART:20150101T000000 
  DTEND:20150101T015959
  RRULE:FREQ=HOURLY;INTERVAL=2;WKST=SU

Example #2

DTSTART:20150101T000000 
  DTEND:20150101T000530
  RRULE:FREQ=MINUTELY;INTERVAL=15;WKST=SU

Publish action

The publish action controls when the content of the device should be updated from a remote server. At the end of a successful publish action, the content of the device is an exact copy of the content on the server and the main index.svg file is reloaded.

The publish action uses the following fields:

  • SUMMARY
    This must be set to publish.
  • URL
    This must point either to a remote location (folder) on an WebDAV-enabled server or to an XML repository descriptor file in the case of a standard HTTP server.
  • DESCRIPTION
    Optional field that can be used to specify additional settings for the publish action, each separated by "\n" (newline character), from the following:
    • -deletelast
      By default, the files that need to be removed are deleted at the beginning of the publish action to save space on the HMP storage. When this option is specified, the file are deleted after the new / modified ones have been uploaded on the HMP.
    • -dest folder
      This option changes the destination of the publish from the content root folder (the default behavior) to an inner folder and the main index.svg file is no longer reloaded. The path entered as destination folder must be URI encoded and must not start by "/" or "\" - if the specified folder does not exist on the HMP storage, it will be created. Note that when Fusion is enabled, the publish action must be set like this: -dest publish, otherwise it won't be executed.
    • -norecurs
      Prevents recursion during the publish process; only the root folder (or the folder specified with -dest option) is modified and no other sub-folders are created or deleted.
    • -statusinterval n_seconds (since 3.0.3 release)
      Sends an update every n seconds to the RPC concentrator (if one is configured on the HMP).
    • -statusnrfiles n_files (since 3.0.3 release)
      Sends an update every n files to the RPC concentrator (if one is configured on the HMP).
    • -statussize n_bytes (since 3.1.0 release)
      Sends an update every n bytes to the RPC concentrator (if one is configured on the HMP).
    Note Note:
    If -statusinterval, -statusnrfiles or -statussize options are used together, an update status is (re)sent when each of the these conditions is reached.


Example:
To instruct the player to download / check for updates a project from SpinetiX server, each day starting at 6 AM (and up to 7 AM), the ICS file would look like this:

BEGIN:VCALENDAR 
VERSION:2.0 
PRODID:-//Spinetix.com/NONSGML Spinetix Control Center//EN 
BEGIN:VEVENT 
UID:14d64d53-a926-4a71-848d-c78452b195fe 
DTSTART:20130101T060000 
DTEND:20130101T070000
RRULE:FREQ=DAILY;WKST=SU  
SUMMARY:publish 
URL: http://demo.spinetix.com/project/default/publish.xml
DESCRIPTION:-deletelast
END:VEVENT 
END:VCALENDAR
Note Note:
See also the publish screen feature.

Upload action

The upload action can be used to upload different files (logs, snapshot images, report files) from the HMP to a remote server (usually a WebDAV enabled server, though a standard HTTP server can be used as well). This action doesn't affect in any way the content on the HMP.

The upload action uses the following fields:

  • SUMMARY
    It must be set to upload.
  • URL
    It must point to the remote location where the files will be uploaded.
  • DESCRIPTION
    It contains the list of files to be uploaded and, if needed, additional options for the upload action, separated by "\n" (newline character).
    To specify files use any of the following:
    • <file_name> or <file_name> <destination_name>
      Specifies the file (file_name) to be uploaded on the server using the specified name (destination_name), if any, or the default file naming format %n-%s-%d-%h.%x" (see -format option for more details).
    • -accounting
      Triggers the upload of the accounting log file.
    • -alllogs
      Triggers the upload of all the logs, both current (plain text) and past (archived).
    • -content
      Triggers the upload of the XML repository descriptor file, containing the list of all the content files present on the HMP.
    • -currentlogs
      Triggers the upload of the current logs (plain text).
    • -log <name> (since 3.0.0 release)
      Triggers the upload of a specific log file, which can be any of the regular logs (e.g., "accounting.log", "uploader.log" etc.), but also the HMP info file ("-log info") or the system log ("-log syslog").
    • -logs
      Triggers the upload of the archived logs from the previous day. Pull Mode retains a history of uploaded logs and will make sure all logs from the past are uploaded. Note that at most 7 days of logs are kept inside the player.
    • -pastlogs
      Triggers the upload of the past logs (archived). No checks are done for finding out if any of the logs have already been uploaded or not.
    • -report
      Triggers the upload of a standard report file.
    • -snapshot
      Triggers the upload of the current snapshot. By default, the file name is set to "snapshot-%s-%d-%h.jpg" (see -format option for more details).
    To specify additional options use any of the following:
    • -buffer true (since 3.0.6 release)
      Forces the buffering of any file sent to the server to ensure that the Content-Lenght header is valid (necessary for any upload to Amazon S3 for instance).
    • -data <key> <value> (since 3.0.6 release)
      Specifies a key=value pair to be added to the POST request. This option can be specified multiple times in case more key/value pairs need to be added. Note: using -data reset will remove all the key / value pairs previously set.
    • -format <syntax_string>
      Specifies the file naming format for the uploaded files, where syntax_string is built using the following special placeholders:
      %D : modification date of the uploaded file
      %H : modification time of the uploaded file
      %d : date of the upload
      %h : time of the upload
      %n : name of the source file
      %s : serial number of the HMP
      %x : extension of the source file
    • -header <name> <value> (since 3.0.6 release)
      Specifies additional HTTP headers to be added to the PUT / POST request. This option can be specified multiple times in case more headers need to be added. Note: using -header reset will remove all the headers previously set.
    • -method post
      Specifies the upload method as POST, instead of the default PUT. A post command would make sense in the case of a regular HTTP server.

Example:
To instruct the player to upload two log files to a server using a custom file name format, every two hours, the ics file would look like this:

BEGIN:VCALENDAR 
VERSION:2.0 
PRODID:-//Spinetix.com/NONSGML Spinetix Control Center//EN 
BEGIN:VEVENT 
UID:14d64d53-a926-4a71-848d-c78234b195fg 
DTSTART:20130101T000000 
DTEND:20130101T010000
RRULE:FREQ=HOURLY;INTERVAL=2;WKST=SU
SUMMARY:upload 
URL: http://webdav.spinetix.com/logs/ 
DESCRIPTION:-format %s/%d-%n.%x\n-accounting\nhttp://localhost/log/warn.log 
END:VEVENT 
END:VCALENDAR

Troubleshooting

Common problems

Errors regarding Pull Mode operations are stored in the uploader.log file.
  • The error "Could not read status line: connection was closed by server " means that the server is reachable but it is returning an error on the path being used. Check the capitalisation, spelling and syntax of the URL is correct.
  • 404: File Not Found errors: Ensure that the not-found file is added as a valid MIME type for the WebDAV server you are using.
    • In most cases that's caused by IIS server not allowing the SVG files (the MIME type is: image/svg+xml).
  • IIS returns "unauthorised" errors: Check that the user IIS_IUSR is added with Read permissions on the Virtual Directory you are using.
  • Spaces in the path in Pull Mode may cause problems. Use underscores instead of spaces.

Enable extended logging

The extended logging of Pull Mode actions should only be enabled when asked by SpinetiX Support.

For HMP350 and HMP300 devices, you need to apply the following config file:

<?xml version="1.0"?>
<configuration version="2.0">  
    <pullmode-logs-level>trace</pullmode-logs-level>    
</configuration>
Note Note:
Replace trace by info when done.


For HMP200, HMP130, and HMP100 devices, follow this procedure:

  1. Open HMP Control Center.
  2. On the "Status" page, find the HMP picture and click over the SpinetiX logo (in case of an HMP200) or the blue button (in case of an HMP130 / HMP100).
  3. A confirmation dialog appears - click "Yes" to enable the "Advanced settings" menu on the left side.
  4. Go to "Content Settings" page > "Pull Mode" tab > "Log settings" section.
  5. Set the "Log level" option to "Trace".
  6. Click the "Apply" button.
  7. Wait for the problematic action to be executed by the HMP, then generate a report and send it to SpinetiX Support for further analysis.


Note Note:
Once the issue has been identified / solved, make sure to set the log level back to "Info".
This page was last modified on 29 October 2018, at 19:24.