Squarebox

Before you start

In order to use this Archiware P5 archiving plugin you need:

CatDV Server 8.0.3 or later

CatDV Pegasus 13.0 or later

CatDV Plugin licence 'ARCAW' (Rest API licence with multiple sessions)

To trigger Archiware file transfers from the Worker you need:

Archiware Worker Plugin 2.1.2, included in this installation as ArchiwareWorker.catdv

In addition, to run the archive service standalone (outside the server):

CatDV Service Control Panel 1.2 or later

New Installation

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Set server properties required by the plugin:

Open 'Server Config' from the control panel

Enter required properties into the CatDV Server tab, 'Other' field.

See 'Plugin server properties' below and 'Running archive service standalone', if applicable.

Restart the CatDV server from the control panel

Open the CatDV client (or log the client back in to the server)

Configure one or more service accounts. The first account will always be the default account, used to verify that Archiware is available and used as a fallback when necessary. See 'Manage service command' below for information on the details and settings for a service account:

In the client, run Tools->Manage Archiware P5 Archive Service.

On the accounts tab enter the service account details. Mandatory details are flagged with an asterisk. Note that once a service account has been used to archive a clip, it will no longer be possible to update its identifier.

Click 'Add'.

The service status can be monitored on the management and job queue screens.

If using media store service mappings to automatically map clips to archive locations, they need to be set up. See 'Media store service mappings' below.

Optionally configure archiving, processing and UI settings for the plugin via the corresponding tabs on Tools->Manage Archiware P5 Archive Service.

If running the archive service standalone (configured via a server property in step two), then configure and start the archive service via the service control panel. See 'Running archive service standalone'.

Verify the service setup: In the client, run Tools->Manage Archiware P5 Archive Service again. The service status for the default account should be 'Running (online)'. The status may be 'Running (offline)' if Archiware P5 is not currently accessible.

IF the Worker IS NOT being installed:

Delete the Archiware worker plugin file with extension '.catdv' from the Archiware plugin directory (see first step).

IF the Worker IS being installed:

Move the Archiware worker plugin file with extension '.catdv' FROM the plugin directory installed in the first step TO the worker plugins directory:

e.g. on a Mac: /Library/Application Support/Square Box/Extensions

e.g. on Windows: %ProgramData%\Square Box\Extensions

e.g. on Linux: /usr/local/catdvWorker/extensions

If the worker is running, re-connect to the server (Edit Config / CatDV Server / Reconnect) so that the archive plugin fields are available.

Verify that the Archiware worker actions are available. It may be necessary to quit and re-open the worker to pick up the new worker actions.

Upgrade Installation

Make a note of the current plugin version from this document.

If running the archive service standalone, use the service control panel to stop the service.

Stop the CatDV server from the control panel.

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Remove or move the directory / files for the prior plugin from the plugins directory.

Carry out any 'Upgrade' instructions for each plugin version listed in the Release Notes (last section in this document) above the last installed version, working back from the oldest applicable version.

Restart the CatDV server from the control panel.

Read through the 'Changes' for each plugin version listed in the Release Notes (last section in this document) above the last installed version, and go to Tools->Manage Archiware Archive Service to verify that the details / settings for each account and the archiving / processing / UI settings for the plugin are correct for this installation. See 'Manage service command' below for more details.

If running the archive service standalone (configured via a server property), then update and start the archive service via the service control panel. See 'Running archive service standalone'.

Open the CatDV client (or log the client back in to the server)

Verify the service setup: In the client, run Tools->Manage Archiware P5 Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if Archiware or the default account client is not currently accessible.

Update (or delete) the Archiware worker plugin by following the instructions from last step for a new installation, making sure to move / remove the .catdv file for the prior version of the plugin from the extensions directory.

Plugin server properties

When running the archive service "InPlugin", the following server properties must be set:

catdv.archiware_archive.licence_code = <generated licence code for plugin>

When running the archive service "Standalone", the following server properties must ALSO be set:

catdv.archiware_archive.service_mode = Standalone

When running the archive service standalone, the properties for the archive service must include:

catdv.archiware_archive.licence_code = <generated licence code for plugin>

catdv.archiware_archive.service_mode = Standalone

catdv.rest_api.host = <CatDV server hostname>

catdv.rest_api.port = <CatDV API port>

catdv.rest_api.user = <CatDV API user>

catdv.rest_api.password = <CatDV API user password> (** exclude if blank, for dev)

Typical rest API values for a development machine would be (password property omitted for a blank password):

localhost, 8080, Administrator

In addition, the following optional property may be set to turn on debug logging on both the server and standalone service:

catdv.archiware_archive.debug = true

Running archive service standalone

By default, the service that handles archive jobs runs inside the plugin. To run it as a standalone process, ensure that the service mode property is set to Standalone in the server when installing the plugin (see second step of Installation):

catdv.archiware_archive.service_mode = Standalone

The following is required to run the plugin's archive service on a separate machine from the CatDV server:

CatDV Service Control Panel 1.2 or later

To configure and start the standalone archive service using the service control panel:

Verify that at least one service account should be configured in the client / web plugin UI

Install the CatDV Service Control Panel

Open the CatDV Service Control Panel

Click 'Service Config' to open the service configuration

Enter license details on the Licensing tab

Enter required and optional service properties on the Settings tab. See 'Plugin server properties' below.

On the Installation tab, note the install location ('Server Install Path')

On the file system, navigate to the install location and copy the plugin files into the plugins sub-directory.

Click 'Start' to start the archive service.

To update the standalone archive service:

The service should have been stopped via the Service Control Panel 'Stop' prior to stopping the CatDV Server.

Open the CatDV Service Control Panel

Add / edit service properties on the Settings tab if applicable.

Go to the install location (see Installation tab) and replace the plugin files in the plugins sub-directory with those for the latest version of the plugin.

Click 'Start' to start the archive service.

Windows UNC Paths

Archiware does not support Windows UNC paths. To enable the plugin to handle media with UNC paths, each volume must be mounted as a lettered drive accessible to the server hosting Archiware. In addition, a Media Store must be set up to map each volume (or a path(s) within the volume) to the mounted drive path. The plugin itself will use the UNC path to access such media but will pass the corresponding drive path to Archiware for archive and restore.

For example, if the volume \\CatDV-mini-Z2\MySAN is mounted on the Archiware server as drive E:, then as a minimum a Media Store with the following paths would be required:

\\CatDV-mini-Z2\MySAN

E:

Alternatively, one or more media stores could be used to map to locations within the volume:

\\CatDV-mini-Z2\MySAN\Nature

E:\Nature

Using the plugin

The plugin comprises various commands that are available from the Tools menu in the client.

File transfers are initiated by the schedule copy / move / restore commands but are carried out by a separate process. This means that the client does not automatically reflect updates to a clip on completion of a transfer. Use 'Server->Refresh Window' in the client to pick up any changes during / after transfer.

NB - If you will be using the plugin to archive media with Windows UNC paths, see the preceding section regarding the additional configuration required, as Archiware does not support them.

The plugin includes the following commands:

Command

Description

Manage Archiware P5 Archive Service

View full status details of the archive service and manage the service account(s) required to access the Archiware P5 archives. If a connection to Archiware P5 cannot be made an error is displayed and the values are not saved

View Archiware P5 Archive Service Job Queue

Lists jobs and filter by status (Pending, Complete, Failed, All). It provides the capability to view job contents, view job results and cancel waiting or queued jobs.

Schedule copy to Archiware P5

Adds a bulk copy job for the selected clips (the bulk job contains a child job for each individual clip which can be monitored separately). Copy jobs can be scheduled when no archive service is running / online but will only be run when the default archive service is online (running and Archiware P5 is accessible). When the copy job is run, the source media associated with each clip is copied from the file system to Archiware P5 and the original file is preserved.

Schedule move to Archiware P5

As copy but on successful transfer of the file to storage the job attempts to delete each original file. If for some reason the deletion fails, this does not affect the job status (or archive status of the clip) but "Purge failed" is recorded in the applicable job data.

Schedule restore from Archiware P5

Adds a bulk restore job for the selected clips (the bulk job contains a child job for each individual clip which can be monitored separately). Restores can be scheduled when no archive service is running / online but will only be run when the default archive service is online (running and Archiware P5 is accessible). When the restore job is run, the source media associated with each clip is copied from Archiware P5 to the file system.

Purge files archived to Archiware P5

Deletes the source media associated with the selected clips if they have been successfully archived.

Managing Service Accounts

The accounts tab on the ‘Manage Archiware P5 Archive Service’ command provides facilities to manage the service account(s) used to connect to Archiware P5 archives. A single account is usually sufficient to operate the plugin. However, if required, there is an add-on which enables multiple accounts to be set up. This is enabled as an add-on via a special server config property.

When there are multiple accounts, the account used may be determined automatically via ‘Media Store Service Path Mappings’ (see later section) or selected by the user on archive / restore if allow override of Account ID is turned on from the UI tab of the manage service command (see next section).

An account which is in use, i.e. has been used to archive files, must be unlocked before it can be updated or deleted. This is to ensure that accidental changes cannot be made to an account such that files can no longer be restored. Note it is not possible to change the account identifier on an in-use account.

The accounts tab provides the following operations:

Button

Description

Clear

Clears the current selection / details so that only default values are filled in.

Set Default

Updates the default account to the current selected account.

Add

Creates a new account with the specified details.

Unlock

Unlocks an in-use account in preparation for update / delete.

Update

Updates the selected account with the specified details.

If an account is in use, it must be unlocked before it can be updated. Please take care when updating an account which is in use, as changing some settings could break the restore of files archived with that account. For example, if the account key is switched to one which does not have the same access or if any encryption settings are changed.

Note it is not possible to change the account identifier on an in-use account.

Delete

Deletes the selected account.

Please take care when deleting an account which is in use, as it will no longer be possible to restore any clips arched with that account. This feature is intended for removing test accounts and any applicable clips should be deleted or restored and re-archived.

Manage service command

The following can be configured from the Manage Service command in the web or desktop UI for the plugin:

ACCOUNTS TAB / DETAILS (for an account):

Field

Description

Account Identifier

Identifying name for this service account. This may contain only alpha-numerics and hyphens. Note it is not possible to change the account identifier on an in-use account.

Server Host

Archiware P5 server machine, e.g. 'localhost' or an IP address

Server Port

Server port on which Archiware CLI is running, e.g. 9501

Server User

Name of user on the server host

Server Password

Password of user on the server host

Client

Archiware P5 client machine, e.g. 'localhost'

Default Plan

The plan that will be used as the archive fallback for this account by default. This is not applicable when a media store service mapping applies. Otherwise, if the UI tab settings allow the user to override the plan then this will only be used as the default the first time the user does an archive. Subsequent archives by the same user will default to the last value they entered.

NSDChat Command

The location of the NSDChat command on the machine on which the CatDV Server is running. This defaults to the typical location for a Windows or Unix OS, as appropriate.

NSDChat Command Timeout

The time in seconds before an NSDChat command is timed out and the plugin puts the CatDV job into a waiting / polling state pending completion of the Archiware job. Defaults to 15 seconds.

NB - an account which is in use, i.e. has been used to archive files, must be unlocked before it can be updated or deleted. This is to ensure that accidental changes cannot be made to an account such that files can no longer be restored.

ARCHIVING TAB:

Purge directories

Determines whether or not empty directories are deleted when purging (or moving) files.

PROCESSING TAB:

Concurrent job limit

Determines the number of transfers that the archive service will attempt to run concurrently. The default value is 4.

Number Delayed retries

The number of times a waiting job is retried at an interval which increases exponentially. The default value is 10, which is equivalent to roughly 34 hours elapsed running time of the archive service.

Maximum retry delay (time period in milliseconds)

Limits the maximum time period between retries of waiting jobs. The default value is 0, which represents no maximum value.

Max jobs running delay (time period in milliseconds)

Determines the time period to wait before attempting to run another job when the maximum number of transfers are already running. The default value is 0.

This would typically be increased (either when a typical transfer is likely to be slow or when the concurrent transfer limit is greater than 1) in order to reduce hits to the database to query jobs whilst max transfers are running.

Loop delay (time period in milliseconds)

Determines the frequency with which the archive service checks the Job queue and attempts to run outstanding Jobs. The default value is 30000, equivalent to 30 seconds

Retry running job delay (time period in milliseconds)

Determines the time period after which a Job which is running will be restarted if it has not been updated during that period. Defaults to a value equivalent to 1 hour

Stalled job delay

Determines the time period since a running job's last progress update, after which it is's status is flagged as stalled. Defaults to a value equivalent to 1 minute

UI TAB:

Restrict command access

Restricts the specified plugin commands to sys admin users. Can be used to hide 'config' commands only (i.e. Manage Service), ‘config / archive’ (i.e. to restrict copy or move to archive but not restore) or 'all' commands. Default is ‘none’.

Days to display

The number of days into the past for which jobs are listed in the job queue.

Any job updated in this time period is listed. The default value is 10.

Max no jobs to display

The maximum number of jobs which will be listed in the job queue. This overrides days_to_display. The default value is 1000.

Allow overrides

Enables the facility for users to override one or more parameters at the point of archive or restore.

Media Store Service Path Mappings

Media Stores can be used to automatically map clips to their archive locations by adding a service path to a media store. The service path determines the service account, plan and (optional) folder path for the archived files. For Archiware P5 a service path has the format:

<serviceType>://<accountIdentifier>/<plan>

or

<serviceType>://<accountIdentifier>/<plan>/<folderPrefix>

For example, if a Media Store were set up with the following two entries:

/Volumes/dance-videos/drafts

archiware://squarebox-test/10001/dance

Then a clip located at:

/Volumes/dance-videos/drafts/greatest-show/this-is-me

Would be archived:

using the credentials from the 'squarebox-test' account

into the plan '10001'

with a path of '/dance/greatest-show/this-is-me'

NB - If media store service mappings are not used then the archive location always mirrors the source file path of the first file in the batch, as it is a constraint of Archiware that files archived together must have the same 'index root', i.e. 'folder path' in Archiware's index.

The 'folder path' can be set or overridden on archive if 'location' is set as an allowed override. See Manage Service Command / UI TAB.

Archive details pane

The plugin automatically creates a panel entitled "Archiware P5" containing the clip metadata which describes it's Archiware P5 archive state, including:

Field

Description

squarebox.catdv.archive.Archiware.serviceType

Type of service responsible for file transfer

squarebox.catdv.archive.Archiware.serviceName

Name of service responsible for file transfer

squarebox.catdv.archive.Archiware.status

Archive status

squarebox.catdv.archive.Archiware.location

Location of file in storage

squarebox.catdv.archive.Archiware.restoreLocation

Location file will be (if job pending) or was last restored to

squarebox.catdv.archive.Archiware.date

Date (timestamp) of latest change in archive status

squarebox.catdv.archive.Archiware.dateLastArchived

Date last archived

squarebox.catdv.archive.Archiware.dateLastRestored

Date last restored

squarebox.catdv.archive.Archiware.numArchives

The number of times the clip has been successfully archived

squarebox.catdv.archive.Archiware.archiveKey

Identifier of file in storage

squarebox.catdv.archive.Archiware.batchID

Identifies the batch of files with which the clip was archived

squarebox.catdv.archive.Archiware.jobID

Identifier of current / latest archive job

squarebox.catdv.archive.Archiware.parentJobID

n/a for Archiware P5 (related to bulk archives)

squarebox.catdv.archive.Archiware.userId

ID of user initiating current / latest transfer

squarebox.catdv.archive.Archiware.historyJson

Record of all archive activity

squarebox.catdv.archive.Archiware.purgeError

Details of purge failure

squarebox.catdv.archive.Archiware.accountIdentifier

Identifier of the Archiware P5 Storage Account in CatDV

squarebox.catdv.archive.Archiware.client

Archiware client to transfer file to

squarebox.catdv.archive.Archiware.plan

Archiware plan to transfer file to

squarebox.catdv.archive.Archiware.handle

Handle of archived file on Archiware

squarebox.catdv.archive.Archiware.volume

Volume of archived file on Archiware

squarebox.catdv.archive.Archiware.barcode

Tape barcode of volume of archived file on Archiware, if applicable

Known Issues

If the archive location is overridden via a media store service path mapping, no folder hierarchy will be preserved within that location. This is due to a constraint of Archiware.

If clips in the same archive batch have media store mappings which resolve to a different 'folder path', they will all be placed in the 'folder path' for the first clip in the batch. This is due to a constraint of Archiware and needs some consideration.

If you will be using the plugin to archive media with Windows UNC paths, see the section 'Windows UNC Paths' regarding the additional configuration required, as Archiware does not support them.

Licence Code

IMPORTANT: You may install and use this software only in accordance with the terms of the CatDV Server 8 license agreement.

Square Box Systems Ltd.

September 2019

Release Notes

2.2.2betap1 (2019-09-10)

- Fix for notification changes to expedite processing of newly queued archive jobs

2.2.2beta (2019-09-06)

- Fixed label for NSDChat command timeout in manage service command

- Add workaround support for Windows UNC paths (see README)

2.2.0beta (2019-08-02)

- Fix cancel job(s) to attempt to stop[scheduled]/cancel[running] corresponding job(s) on Archiware

- Retry service start if rest API is not ready

- Modify generation of clip level archive status to incorporate both current and pending status

e.g. 'Archived [Copy running] to ..." etc

- Use queued archive job notifications to expedite processing of newly queued archive jobs

- Send archive job status notifications for all applicable changes in archive job status

- Restore by default to current media file path from clip, rather than from last archived parameters, so that target

restore location can be updated when e.g. media paths are updated with a different volume name

2.1.4betap1 (2019-07-11)

- Add configurable service account field for nsdchat command timeout

- Reduce the number of distinct nsdchat server session ids created (NB potentially un-resolved for high volume purge)

- Modify text breaks for account update confirmation dialog, to fix web UI wrapping issue

2.1.4beta (2019-07-04)

- Correct required server version in README

- Restrict update of in use accounts via unlock/update/confirm

- Change deletion of in use accounts to unlock/delete/confirm direct from accounts tab

- Update service fields / panel creation to include all fields in field group but exclude some from the panel

- Fix 'column ... already exists' error adding an account

- Fix for UnsupportedOperationException in Collections::UnmodifiableMap.put() on plugin init

- Fix NPE on cancel job, when associated clip is not found

2.1.3beta (2019-06-19)

- update README

2.1.2beta (2019-05-31)

- Switch slow jobs into waiting status (showing archiware job status) after a timeout period

- Allow account id override on restore, to facilitate restore using QLS data

- Automatically add required archiware metadata keys to archiware index, when adding account

- Record archiware volume and barcode in clip archive parameters (for tapes they may be the same)

2.1.1beta (2019-05-24)

- Fix windows path format when adding an entry to an archive selection

- Fix nsdchat geterror - command is case sensitive on Windows

- Extract error details from 'job xmlticket' when Archiware job completion is warning/exception/failure

- Update logging

2.1.0betap1 (2019-05-22)

- Fix display of json based history for web UI

2.1.0beta (2019-05-22)

- Submit files to Archiware in batches for archive and restore

- Capture nsdchat server details and always use session ids to avoid spurious errors due to session clashes

- Change fallback location mapping for Archiware to 'mirror', to exclude the timestamp and related colon in path issue.

- Fix Archiware restore path for Windows

- Add additional archive history field in JSON format

- Temporarily disable in use check for archive account updates

2.1.0prototype2

- Escape backslashes and curly braces in archiware params that are wrapped in curly braces

- Use nsdchat getError command in place of reading from archiware log file (*log file account param removed*)

- Include missing apache commons lang library in plugin release

2.1.0prototype

- Initial prototype version

Copyright © Square Box Systems Ltd. 2002-2019. All rights reserved.