Pull Mode

From SpinetiX Support Wiki

(Redirected from Pull mode)
Jump to: navigation, search

Contents

"Documentation"
SpinetiX Technical Documentation - Version 3.1.0 (2014-10-13)

Summary information and/or updates to the documentation can be found on this page.

Introduction

Pull Mode is one of the content management methods of the HMP whereby the content stored on a remote web server is periodically pulled by the HMP onto its local storage. Basically, when using the Pull Mode, the HMP is instructed to download content from a server at specific intervals and each time the HMP will do the following:

  1. Connect to the specified web server.
  2. Compare the content on the web server with the content present on the local storage.
  3. Remove from the local storage all files that are no longer present on the web server.
  4. Download the new files and the modified ones to the local storage.
    • The Etag and/or Last-Modified HTTP directives are used to determine if a file was modified or not - see Caching page for more details.
    • Unmodified files will not be downloaded.

Note: The order of the last two steps can be changed using the option -deletelast .

Advantages

  • Updates of the content can be programmed at predefined times or under request (using RPC).
  • The HMP will copy the content to its local storage, making the system resilient to network failures. Moreover, the uploading process was designed to work even for very poor connections. There is an initial 10 minutes time-out to establish the connection and later another 10 minutes time-out during the transmission of data.
  • Content on HMPs behind firewall or NAT can be updated practically from anywhere.
  • The Pull Mode mechanism can also be used to upload logs and snapshot images to a remote web server for monitoring purposes.

Pull Mode and Fusion

When Fusion mode is enabled on the device, you can still use Pull mode (firmware 2.2.3 or later is required), thus allowing a mix of Fusion and externally managed content, however that content cannot be directly accessed from Fusion like any other media. Instead you need to reference such files within a custom skin, a custom slide template, an hypermedia project or any combination of these.

The default destination for a Pull operation is no longer the root folder, but a sub-folder named "publish". In case an ics file is used to control the Pull Mode, then make sure to specify the destination folder like this: -dest publish.

Bandwidth management

* See full article Bandwidth management.

Configure Pull Mode on HMP

Content Settings - Pull Mode tab

Pull Mode can be enabled from HMP Control Center > Content Settings > Pull Mode tab, by selecting one of the three available options:

  • Manual settings
    Use this option to manually configure the HMP to:
    • pull content from a web server at a predefined time, either daily or hourly*;
    • upload its logs (all of them or just the accounting ones) to a web serverat a predefined time, either daily or hourly*;
    • contact an RPC Concentrator at a predefined pooling interval.
      To prevent the HMP from executing RPC commands from the RPC Concentrator, enable the "Notification only" option.
    *The hourly option is available since 3.1.1.
  • From remote iCalendar file (ics)
    Use this option to control the Pull Mode using an ics file that you have uploaded onto a web server.
    Use the option labeled "Check calendar every" to specify the maximum duration after which the HMP must contact the server to find out if the calendar has been updated in the meantime. Note that if the current calendar contains an event during this interval, the HMP automatically checks for updates before executing that event (and restarts the timer).

Examples:

To schedule a daily update of the content from the web server, follow these steps:

  1. Select "Manual settings" option.
  2. Enable "Automatically upload project to the HMP..." option.
  3. Enter the source address from where to pull the content.
  4. Enter the daily update time.

To schedule a daily upload of the HMP logs onto the web server, follow these steps:

  1. Select "Manual settings" option.
  2. Enable "Automatically upload logs from the HMP..." option.
  3. Enter the destination address where to upload the logs.
  4. Enter the daily upload time.
  5. If you want only the accounting logs to be uploaded, then select "Accounting logs only" option.

Web servers

Pull Mode publish action can be used with any standard web server, with or without support for WebDAV.

  • A WebDAV compatible web server is recommended and most popular web servers, like Apache or Microsoft IIS, do support WebDAV, though usually it must be manually enabled. There are also some public WebDAV servers that can be used, for instance myDrive.ch.
  • If WebDAV cannot be used, the HMP can pull from a plain HTTP web server as well, but since there is no standard method of listing the content of a repository using standard HTTP commands, an XML repository descriptor file must be available on the web server.
    • Starting with version 3.1.0, Elementi M and Elementi X automatically generate this file ("spx-listing.xml") within the project during publishing. See Publish Location page for more details about this feature.

Notes:

  • Write access rights on the web server are required only if the upload action is used.
  • The port used by Pull Mode is the one you have specified when entering the project source / log destination - in the absence of such port, then the default port of the scheme is used, meaning 80 for http and 443 for https.
  • Although an HMP can technically be used as a WebDAV server, this configuration is not recommended because the HMP can only respond to maximum two concurrent clients (this includes HMP Control Center and Fusion) and the performance on the host HMP will be degraded while the file reconciliation is taking place.
  • See also the generic remarks when using Web servers.

Apache

For details on how to enable mod_dav module on Apache HTTP Server Version 2.2, see the official Apache Module mod_dav page.

Microsoft IIS

  • Internet Information Services (IIS) is included with all Professional versions of Windows, so it can be used for testing, usually without further installation. However, by default, IIS is heavily locked down with regard to WebDAV.
  • For a full guide to setting up Microsoft IIS 7 as a Pull Mode source, see the Configuring IIS as server for Pull mode tutorial.
  • IIS 7 introduced a new WebDAV stack which, among other changes, restricts WebDAV functions for unauthenticated (anonymous) users to PROPFIND.
  • PUT, MKCOL, PROPPATCH, COPY, MOVE, DELETE, and WebDAV-based GET requests all require authentication (see this Microsoft page for more details).
  • Ensure that SVG is listed in the allowed MIME types, and directory browsing is enabled if you are using log uploads.
  • If using basic authentication, in the IIS site configuration ensure that only basic is enabled. Enabling multiple types will result in the HMP not being able to connect.
  • See #Troubleshooting and common errors in Pull mode below for more tips. Also see the Web servers page for more general information.

MyDrive.ch

To use the WebDAV-enabled public server myDrive.ch with Pull Mode, check out Tutorial:Using myDrive as content server.

Box.com

To use the WebDAV-enabled public server box.com with Pull Mode, follow these instructions:

  1. Create an account on box.com
  2. Create a Publish Location and then publish your project from Elementi. Alternatively, you can upload your project manually or using a WebDAV client.
  3. Open HMP Control Center > Network Settings page > Credentials tab.
  4. Add https://dav.box.com/dav/ as Server URI and your box.com username and password.
  5. Go to the Content Settings page > Pull Mode tab and set the project source to: https://dav.box.com/dav/ProjectName/.

Dropbox

To use the Dropbox public server with Pull Mode, check out Tutorial:Using Dropbox as content server.

NAS devices

It is often overkill to use a full enterprise-class, rack-mounted server to host content for Pull Mode. SpinetiX does not recommend specific NAS servers, but mid-range manufacturers such as Synology, QNAP and Netgear have performance NAS appliances which support WebDAV and are of a much more practical form factor than rack servers. They usually avoid unnecessary license costs.

For more details about using NAS devices with Pull Mode, check out these tutorials:

Sample PHP scripts

Also check the RPC management samples for the management of the HMP

File pull

SpinetiX is providing an automatic generation of XML description listing the content of a files folder (written in PHP) in order to make it easier for you to develop your own HTTP based Pull mode server.

  • Make sure you read the README file before getting started!

Database pull

SpinetiX is providing a server side application for managing and testing Pull mode functionality of HMP devices. Its purpose is to serve as an example and guide implementation for a simple Pull mode server for HMP devices based on a Database.

All pull mode server activity is done through an SQLite database.

  • Make sure you read the README file before getting started!

Control the Pull Mode with an ics file

An iCalendar file (.ics) can be used to accurately control the actions performed by the HMP when working in Pull mode, for instance to:

  • Update the content of the HMP more (or less) than once per day and even from different locations.
  • Schedule the update using complex recurring rules.
  • Upload different files to a server: logs, snapshot images, report file.
  • Enable the communication with a remote RPC server.

The ics file for controlling the Pull mode actions can be created using different methods: Elementi, HMD, Outlook, Google Calendar or manually with any text editor.

  • Usually, it's easier to create the ics file within an application. In this case, if you want to see to contents of the ics file, simply open it within a text editor.

The ics file can be directly uploaded on the HMP (from Control Center) or it can be uploaded on a remote web server from which the HMP will be instructed to retrieved it.

Events and actions

Each action of Pull mode is described by an event (item, meeting) within the calendar file and this event must have a certain format.

The following iCalendar fields are used to define the action (in parenthesis the alternative names used within applications for these fields):

  • SUMMARY (also referred to as: Title or Subject) - specifies the action to be performed and can take one of the following values:
    • publish - to publish a project (or just some files) from a server to the HMP,
    • upload - to upload different files from the HMP to a server,
    • rpc - to enable sending RPC statuses to a remote server (RPC Concentrator).
    • inline-rpc - to trigger an RPC command.
  • URL (also referred to as: URI, Location, Where) - specifies the URI of the action.
  • DTSTART and DTEND (also referred to as: From & To, Start Time & End time) - specifies the starting and ending date and time for the event (or for the first event of a series if the recurring rule is used).
  • RRULE - specifies the recurrence rule(s) applying to this event.
  • DESCRIPTION - specifies additional options for the upload or rpc actions.
  • Other fields, such as: UID, VTIMEZONE, PRIORITY etc., can also be used within the ics file.

Duration of an event

The duration of an event represents the time period between DTSTART and DTEND, during which the player will try to execute the specified action (with other words, it's the validity period of the action, not the duration of the action itself, which can take far less or even more that the duration of the event).

  • If the action is not successful for any reasons (for instance, the server cannot be contacted), the player will retry the same action until the DTEND is reached; at that point the action is discarded even if the action has not been executed.
  • Once the action is successful, the event is removed from the Pull mode queue (event if the DTEND has not been reached).

Note Note: If the player is restarted, the ics file controlling the Pull mode will be reprocessed, which means that some past events might be executed again if their DTEND is still in the future.

Recurrence rules

When using recurrence rules, make sure that no more than 100 events are generated for any interval of 8 hours, for instance:

  • In case of only one event / action, the repeating interval should not be less than 5 minutes.
  • In case of two events, the repeating interval for each event should not be less than 10 minutes.

Note Note: A special care should be taken to avoid overlapping an event by the other events from its series.

  • For instance, for an action than needs to be executed every two hours, the duration of the event should not be more than two hours, like in the following example:
DTSTART:20130101T000000 
DTEND:20130101T010000
RRULE:FREQ=HOURLY;INTERVAL=2;WKST=SU

Publish action

The publish action controls when the content of the device should be updated from a remote server. At the end of a successful publish action, the content of the device is an exact copy of the content on the server.

The publish action uses the following fields:

  • SUMMARY
    It must be set to publish.
  • URL
    It must point to a remote location (folder) on an WebDAV server or to an xml file with the repository description for a standard HTTP server.
  • DESCRIPTION
    (Optional) It can be used to specify additional options for the publish action, separated by "\n" (newline character), from the following:
    • -deletelast
      By default, the files that need to be removed are deleted at the beginning of the publish action to save space on the HMP storage. When this option is specified, the file are deleted after the new / modified ones have been uploaded on the HMP.
    • -dest folder
      By default, the files are published into the root folder - you can change this by using this option along with a destination folder (it must be URI encoded and must not start by "/" or "\") . If the folder does not exist on the HMP storage, it will be created.
      Note that when Fusion is enabled, the publish can only be done within the folder "publish".
    • -norecurs
      Prevents recursion during the publish process; only the root folder (or the folder specified with -dest option) is modified and no other sub-folders are created or deleted.
    • -statusinterval n_seconds (since 3.0.3 release)
      Sends an update every n seconds to the RPC concentrator (if one is configured on the HMP).
    • -statusnrfiles n_files (since 3.0.3 release)
      Sends an update every n files to the RPC concentrator (if one is configured on the HMP).
    • -statussize n_bytes (since 3.1.0 release)
      Sends an update every n bytes to the RPC concentrator (if one is configured on the HMP).
    Note Note: If -statusinterval, -statusnrfiles or -statussize options are used together, an update status is (re)sent every time any of the these conditions is reached.

Example:
To instruct the player to download / check for updates a project from SpinetiX server, each day starting at 6 AM (and up to 7 AM), the ics file would look like this:

BEGIN:VCALENDAR 
VERSION:2.0 
PRODID:-//Spinetix.com/NONSGML Spinetix Control Center//EN 
BEGIN:VEVENT 
UID:14d64d53-a926-4a71-848d-c78452b195fe 
DTSTART:20130101T060000 
DTEND:20130101T070000
RRULE:FREQ=DAILY;WKST=SU  
SUMMARY:publish URL: http://demo.spinetix.com/project/default/publish.xml
DESCRIPTION:-deletelast
END:VEVENT 
END:VCALENDAR

Note Note: See also the publish_screen feature.

Upload action

The upload action can be used to upload different files (logs, snapshot images, report files) from the HMP to a remote server (usually a WebDAV enabled server, though a standard HTTP server can be used as well). This action doesn't affect in any way the content on the HMP.

The upload action uses the following fields:

  • SUMMARY
    It must be set to upload.
  • URL
    It must point to the remote location where the files will be uploaded.
  • DESCRIPTION
    It contains the list of files to be uploaded and, if needed, additional options for the upload action, separated by "\n" (newline character).
    To specify files use any of the following:
    • <file_name> or <file_name> <destination_name>
      Specifies the file (file_name) to be uploaded on the server using the specified name (destination_name), if any, or the default file naming format %n-%s-%d-%h.%x" (see -format option for more details).
    • -accounting
      Triggers the upload of the accounting log file.
    • -alllogs
      Triggers the upload of all the logs, both current (plain text) and past (archived).
    • -content
      Triggers the upload of the XML repository descriptor file, containing the list of all the content files present on the HMP.
    • -currentlogs
      Triggers the upload of the current logs (plain text).
    • -log <name> (since 3.0.0 release)
      Triggers the upload of a specific log file, which can be any of the regular logs (e.g., "accounting.log", "uploader.log" etc.), but also the HMP info file ("-log info") or the system log ("-log syslog").
    • -logs
      Triggers the upload of the archived logs from the previous day. Pull Mode retains a history of uploaded logs and will make sure all logs from the past are uploaded. Note that at most 7 days of logs are kept inside the player.
    • -pastlogs
      Triggers the upload of the past logs (archived). No checks are done for finding out if any of the logs have already been uploaded or not.
    • -report
      Triggers the upload of a standard report file.
    • -snapshot
      Triggers the upload of the current snapshot. By default, the file name is set to "snapshot-%s-%d-%h.jpg" (see -format option for more details).
    To specify additional options use any of the following:
    • -buffer true (since 3.0.6 release)
      Forces the buffering of any file sent to the server to ensure that the Content-Lenght header is valid (necessary for any upload to Amazon S3 for instance).
    • -data <key> <value> (since 3.0.6 release)
      Specifies a key=value pair to be added to the POST request. This option can be specified multiple times in case more key/value pairs need to be added. Note: using -data reset will remove all the key / value pairs previously set.
    • -format <syntax_string>
      Specifies the file naming format for the uploaded files, where syntax_string is built using the following special placeholders:
      %D : modification date of the uploaded file
      %H : modification time of the uploaded file
      %d : date of the upload
      %h : time of the upload
      %n : name of the source file
      %s : serial number of the HMP
      %x : extension of the source file
    • -header <name> <value> (since 3.0.6 release)
      Specifies additional HTTP headers to be added to the PUT / POST request. This option can be specified multiple times in case more headers need to be added. Note: using -header reset will remove all the headers previously set.
    • -method post
      Specifies the upload method as POST, instead of the default PUT. A post command would make sense in the case of a regular HTTP server.

Example:
To instruct the player to upload two log files to a server using a custom file name format, every two hours, the ics file would look like this:

BEGIN:VCALENDAR 
VERSION:2.0 
PRODID:-//Spinetix.com/NONSGML Spinetix Control Center//EN 
BEGIN:VEVENT 
UID:14d64d53-a926-4a71-848d-c78234b195fg 
DTSTART:20130101T000000 
DTEND:20130101T010000
RRULE:FREQ=HOURLY;INTERVAL=2;WKST=SU
SUMMARY:upload URL: http://webdav.spinetix.com/logs/ 
DESCRIPTION:-format %s/%d-%n.%x\n-accounting\nhttp://localhost/log/warn.log 
END:VEVENT 
END:VCALENDAR

Using Pull mode in a pseudo-push scenario

"Info"
Starting from firmware 2.2.3 the add_pull_action() RPC command should be used to trigger a pull dynamically. It is more flexible than the method described bellow.

In some rare scenarios, you may want to configure Pull mode to checks for updates very regularly (e.g. 5 mins). In a normal setup, this would consume many network resources and put unnecessary load on the HMP. To avoid this:

  • Configure the HMP to use a dummy ICS file on your remote server for the Pull schedule.
  • Configure the Check calendar every : to a small duration (1m or 5m for instance).
    The player will contact the server every time to check if the ICS file has been modified. It will use ETag and HEAD requests to check for a new file (see Caching page).
  • By default the server will return an empty ICS file.
  • To trigger a content Pull, change the ICS file on the server to add a single event taking place between some short time in the past, to a time in the future long enough for the Pull to complete (2 hours for instance).
    The event location should contain either the URI to the WebDAV server hosting the project, or to the XML describing the list of file to download if you are not using a WebDAV server.
  • This way, the player does not need to check the its entire storage at the interval specified, unless the ics file has changed.

Troubleshooting

Common problems

"Info"
Errors regarding Pull Mode operations are stored in the uploader.log file.
  • The error "Could not read status line: connection was closed by server " means that the server is reachable but it is returning an error on the path being used. Check the capitalisation, spelling and syntax of the URL is correct.
  • 404: File Not Found errors: Ensure that the not-found file is added as a valid MIME type for the WebDAV server you are using.
    • In most cases that's caused by IIS server not allowing the SVG files (the MIME type is: image/svg+xml).
  • IIS returns "unauthorised" errors: Check that the user IIS_IUSR is added with Read permissions on the Virtual Directory you are using.
  • Spaces in the path in Pull Mode may cause problems. Use underscores instead of spaces.

Enable extended logging

The extended logging of Pull Mode actions should only be enabled when asked by SpinetiX Support. To do so, follow this procedure:

  1. Open HMP Control Center.
  2. On the "Status" page, find the HMP picture and click where the SpinetiX logo (in case of an HMP200) or the blue button (in case of an HMP130 / HMP100) is displayed.
  3. A confirmation dialog will appear; click on "Yes" to enable the "Advanced settings" menu on the left side.
  4. Go to Content Settings > Pull Mode tab > Log settings section.
  5. Set the "Log level" option to "Trace".
  6. Click the "Apply" button.
  7. Wait for the problematic action to be executed by the HMP, then generate a report and send it to SpinetiX Support for further analysis.

Note Note: Once the issue has been identified / solved, make sure to set the log level back to "Info".