Squarebox

Before you start

In order to use this Archiware P5 archiving plugin you need:

CatDV Server 8.0.1 or later

CatDV Pegasus 13.0 or later

CatDV Plugin licence 'ARCAW' (Rest API licence with multiple sessions)

To trigger Archiware file transfers from the Worker you need:

Archiware Worker Plugin 2.1.2, included in this installation as ArchiwareWorker.catdv

In addition, to run the archive service standalone (outside the server):

CatDV Service Control Panel 1.2 or later

Installation

NEW

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Set server properties required by the plugin:

Open 'Server Config' from the control panel

Enter required properties into the CatDV Server tab, 'Other' field.

See 'Plugin server properties' below and 'Running archive service standalone', if applicable.

Restart the CatDV server from the control panel

Open the CatDV client (or log the client back in to the server)

Configure one or more service accounts. The first account will always be the default account, used to verify that Archiware is available and used as a fallback when necessary. See 'Manage service command' below for information on the details and settings for a service account:

In the client, run Tools->Manage Archiware P5 Archive Service

On the accounts tab enter the service account details. Mandatory details are flagged with a *. Note that once a service account has been used to archive a clip, it will no longer be possible to update it's details.

Click 'Add'.

The service status can be monitored on the management and job queue screens.

If using media store service mappings to automatically map clips to archive locations, they need to be set up. See 'Media store service mappings' below.

Optionally configure archiving, processing and UI settings for the plugin via the corresponding tabs on Tools->Manage Archiware P5 Archive Service.

If running the archive service standalone (configured via a server property in step two), then configure and start the archive service via the service control panel. See 'Running archive service standalone'.

Verify the service setup:

In the client, run Tools->Manage Archiware P5 Archive Service again. The service status for the default account should be 'Running (online)'. The status may be 'Running (offline)' if Archiware P5 is not currently accessible.

Install (or delete) the Archiware worker plugin:

IF the Worker IS NOT installed:

Delete the Archiware worker plugin file with extension '.catdv' from the archiware plugin directory (see step 1).

IF the Worker IS installed:

Move the Archiware worker plugin file with extension '.catdv' FROM the plugin directory installed in step 1 TO the worker plugins directory:

e.g. on a Mac: /Library/Application Support/Square Box/Extensions

e.g. on Windows: %ProgramData%\Square Box\Extensions

e.g. on Linux: /usr/local/catdvWorker/extensions

If the worker is running, re-connect to the server (Edit Config / CatDV Server / Reconnect) so that the archive plugin fields are available.

Verify that the Archiware worker actions are available. **It may be necessary to quit and re-open the worker to pick up the new worker actions.

UPDATE

Make a note of the current plugin version - from Archiware_README.txt / Archiware_ReleaseNotes.txt in the Archiware plugin directory or from the server log ar start up when the plugin is loaded

If running the archive service standalone, use the service control panel to stop the service.

Stop the CatDV server from the control panel

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Remove or move the directory / files for the prior plugin from the plugins directory.

Carry out any 'Upgrade' instructions for each plugin version listed in the Archiware_ReleaseNotes.txt (from this zip) above the last installed version, working back from the oldest applicable version

Restart the CatDV server from the control panel

Read through the 'Changes' for each plugin version listed in the Archiware_ReleaseNotes.txt (from this zip) above the last installed version, and go to Tools->Manage Amazon S3 Archive Service to verify that the details / settings for each account and the archiving / processing / UI settings for the plugin are correct for this installation. See 'Manage service command' below for more details.

If running the archive service standalone (configured via a server property in step two), then update and start the archive service via the service control panel. See 'Running archive service standalone'.

Open the CatDV client (or log the client back in to the server)

Verify the service setup: In the client, run Tools->Manage Archiware P5 Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if Archiware or the default account client is not currently accessible.

Update (or delete) the Archiware worker plugin by following the instructions from step 8 for a new installation, making sure to move / remove the .catdv file for the prior version of the plugin from the extensions directory.

Running archive service standalone

By default, the service that handles archive jobs runs inside the plugin. To run it as a standalone process, ensure that the service mode property is set to Standalone in the server when installing the plugin (see step 2 of Installation):

catdv.archiware_archive.service_mode = Standalone

The following is required to run the plugin's archive service on a separate machine from the CatDV server:

CatDV Service Control Panel 1.2 or later

** NB At least one service account should be configured in the client / web plugin UI before it is started.

To configure and start the standalone archive service using the service control panel:

Install the CatDV Service Control Panel

Open the CatDV Service Control Panel

Click 'Service Config' to open the service configuration

Enter license details on the Licensing tab

Enter required and optional service properties on the Settings tab. See 'Plugin server properties' below.

On the Installation tab, note the install location ('Server Install Path')

On the file system, navigate to the install location and copy the plugin files into the plugins sub-directory.

Click 'Start' to start the archive service.

To update the standalone archive service:

* The service should have been stopped via the Service Control Panel 'Stop' prior to stopping the CatDV Server.

Open the CatDV Service Control Panel

Add / edit service properties on the Settings tab if applicable.

Go to the install location (see Installation tab) and replace the plugin files in the plugins sub-directory with those for the latest version of the plugin.

Click 'Start' to start the archive service.

Plugin server properties

When running the archive service "InPlugin", the following server properties must be set:

catdv.archiware_archive.licence_code = <generated licence code for plugin>

When running the archive service "Standalone", the following server properties must ALSO be set:

catdv.archiware_archive.service_mode = Standalone

*NB the default value for this property is: InPlugin

When running the archive service standalone, the properties for the archive service must include:

catdv.archiware_archive.licence_code = <generated licence code for plugin>

catdv.archiware_archive.service_mode = Standalone

catdv.rest_api.host = <CatDV server hostname>

catdv.rest_api.port = <CatDV API port>

catdv.rest_api.user = <CatDV API user>

catdv.rest_api.password = <CatDV API user password> (** exclude if blank, for dev environment)

Typical rest API values for a development machine would be (password property omitted for a blank password):

localhost, 8080, Administrator

In addition, the following optional property may be set to turn on debug logging:

catdv.archiware_archive.debug = true

Media Store service mappings

Media Stores can be used to automatically map clips to their archive locations by adding a service path to a media store. The service path determines the service account, plan and (optional) folder path for the archived files For Archiware P5 a service path has the format:

<serviceType>://<accountIdentifier>/<plan>

or

<serviceType>://<accountIdentifier>/<plan>/<folderPrefix>

For example, if a Media Store were set up with the following two entries:

/Volumes/dance-videos/drafts

archiware://squarebox-test/10001/dance

Then a clip located at:

/Volumes/dance-videos/drafts/greatest-show/this-is-me

...would be archived using the credentials from the 'squarebox-test' account into the plan '10001' with a path of '/dance/greatest-show/this-is-me'

** If media store service mappings are not used then the archive location always mirrors the src file path of the first file in the batch (it is a constraint of Archiware that all files archived together must have the same 'index root', i.e. 'folder path' location in Archiware's index.

** The 'folder path' can be set or overridden on archive if 'location' is set as an allowed override for archive (see Manage Service Command / UI TAB)

Manage service command

The following can be configured from the Manage Service command in the web or desktop UI for the plugin:

ACCOUNTS TAB / DETAILS (for an account):

Account Identifier:

Identifying name for this service account. This may contain only alphanumerics and hyphens.

Server Host:

Archiware P5 server machine, e.g. 'localhost' or an IP address

Server Port:

Server port on which Archiware CLI is running, e.g. 9501

Server User:

Name of user on the server host

Server Password:

Password of user on the server host

Client:

Archiware P5 client machine, e.g. 'localhost'

Default Plan:

The plan that will be used as the archive fallback for this account by default.

This is not applicable when a media store service mapping applies. Otherwise, if the UI tab settings allow the user to override the plan then this will only be used as the default the first time the user does an archive. Subsequent archives by the same user will default to the last value they entered.

NSDChat Command:

The location of the NSDChat command on the machine on which the CatDV Server is running.

This defaults to the typical location for a Windows or Unix OS, as appropriate.

** NB an account which is in use, i.e. has been used to archive files, must be unlocked before it can be updated or deleted. This is to ensure that accidental changes can not be made to an account such that files can no longer be restored.

Note it is not possible to change the account identifier on an in use account.

To update an in use account:

select the account on the accounts tab

click 'Unlock'

change details and click 'Update'

click 'Update account' to confirm

To delete an in use account:

select the account on the accounts tab

click 'Unlock'

change details and click 'Delete'

click 'Delete account' to confirm

ARCHIVING TAB:

Purge directories:

Determines whether or not empty directories are deleted when purging (or moving) files.

PROCESSING TAB:

Concurrent job limit

Determines the number of transfers that the archive service will attempt to run concurrently. The default value is 4.

Number delayed retries

The number of times a waiting job is retried at an interval which increases exponentially. The default value is 10, which is equivalent to roughly 34 hours elapsed running time of the archive service.

Maximum retry delay (time period in milliseconds):

Limits the maximum time period between retries of waiting jobs. The default value is 0, which represents no maximum value.

Max jobs running delay (time period in milliseconds):

Determines the time period to wait before attempting to run another job when the maximum number of transfers are already running. The default value is 0.

This would typically be increased (either when a typical transfer is likely to be slow or when the concurrent transfer limit is greater than 1) in order to reduce hits to the database to query jobs whilst max transfers are running.

Default value is 0.

Loop delay (time period in milliseconds):

Determines the frequency with which the archive service checks the Job queue and attempts to run outstanding Jobs. The default value is 30000, equivalent to 30 seconds.

Retry running job delay (time period in milliseconds):

Determines the time period after which a Job which is running will be restarted if it has not been updated during that period. Defaults to a value equivalent to 1 hour.

Stalled job delay:

Determines the time period since a running job's last progress update, after which it is's status is flagged as stalled. Defaults to a value equivalent to 1 minute.

UI TAB:

Restrict command access:

Restricts the specified plugin commands to sys admin users. Can be used to hide 'config' commands only (i.e. Manage Service), or 'all' commands.

Default is 'none';

NB - this only works with CatDV client versions from 12 onwards.

Days to display:

The number of days into the past for which jobs are listed in the job queue.

Any job updated in this time period is listed. The default value is 10.

Max no jobs to display:

The maximum number of jobs which will be listed in the job queue. This overrides days_to_display. The default value is 1000.

Allow overrides:

Enables the facility for users to override one or more parameters at the point of archive or restore.

Using the plugin

The plugin comprises various commands that are available from the Tools menu in the client.

File transfers are initiated by the schedule copy / move / restore commands but are carried out by a separate process. This means that the client does not automatically reflect updates to a clip on completion of a transfer. Use 'Server->Refresh Window' in the client to pick up any changes during / after transfer.

The plugin includes the following commands:

* Manage Archiware P5 Archive Service - View full status details of the archive service and manage the service account(s) required to access the Archiware P5 archives. These include client and default container name. If a connection to Archiware P5 cannot be made an error is displayed and the values are not saved.

* View Archiware P5 Archive Service Job Queue - list of jobs that can be filtered by status (Pending, Complete, Failed, All). It provides the capability to view job contents, view job results and cancel waiting or queued jobs.

* Schedule Copy to Archiware P5 - adds a bulk copy job for the selected clips (the bulk job contains a child job for each individual clip which can be monitored separately). Copy jobs can be scheduled when no archive service is running / online but will only be run when the default archive service is online (running and Archiware P5 is accessible). When the copy job is run, the source media associated with each clip is copied from the file system to Archiware P5 and the original file is preserved.

* Schedule move to Archiware P5 - as copy but on successful transfer of the file to storage the job attempts to delete the original file. If for some reason the deletion fails this does not affect the job status (or archive status of the clip) but Purge failed" is recorded in the job data.

* Schedule Restore from Archiware P5 - adds a bulk restore job for the selected clips (the bulk job contains a child job for each individual clip which can be monitored separately). Restores can be scheduled when no archive service is running / online but will only be run when the default archive service is online (running and Archiware P5 is accessible). When the restore job is run, the source media associated with each clip is copied from Archiware P5 to the file system.

* Purge files archived to Archiware P5 - deletes the source media associated with the selected clips if they have been successfully archived.

Archive details pane

The plugin automatically creates a panel entitled "Archiware P5" containing the clip metadata which describes it's Archiware P5 archive state, including:

squarebox.catdv.archive.Archiware.serviceType - Type of service responsible for file transfer

squarebox.catdv.archive.Archiware.serviceName - Name of service responsible for file transfer

squarebox.catdv.archive.Archiware.status - Archive status

squarebox.catdv.archive.Archiware.location - Location of file in storage

squarebox.catdv.archive.Archiware.restoreLocation - Location file will be / was last restored to

squarebox.catdv.archive.Archiware.date - Date (timestamp) of latest change in archive status

squarebox.catdv.archive.Archiware.dateLastArchived - Date last archived

squarebox.catdv.archive.Archiware.dateLastRestored - Date last restored

squarebox.catdv.archive.Archiware.numArchives - The number of times the clip has been successfully archived

squarebox.catdv.archive.Archiware.archiveKey - Identifier of file in storage

squarebox.catdv.archive.Archiware.batchID - Identifies the batch of files with which the clip was archived

squarebox.catdv.archive.Archiware.jobID - Identifier of current / latest archive job

squarebox.catdv.archive.Archiware.parentJobID - n/a for Archiware P5 (related to bulk archives)

squarebox.catdv.archive.Archiware.userId - ID of user initiating current / latest transfer

squarebox.catdv.archive.Archiware.historyJson - Record of all archive activity

squarebox.catdv.archive.Archiware.purgeError - Details of purge failure

squarebox.catdv.archive.Archiware.accountIdentifier - Identifier of the Archiware P5 Storage Account

squarebox.catdv.archive.Archiware.client - Archiware client to transfer file to

squarebox.catdv.archive.Archiware.plan - Archiware plan to transfer file to

squarebox.catdv.archive.Archiware.handle - Handle of archived file on Archiware

squarebox.catdv.archive.Archiware.volume - Volume of archived file on Archiware

squarebox.catdv.archive.Archiware.barcode - Tape barcode of volume of archived file on Archiware, if applicable

Known Issues

If clips in the same archive batch have media store mappings which resolve to a different 'folder path', they will all be placed in the 'folder path' for the first clip in the batch. This is due to a constraint of Archiware and needs some consideration.

Licence Code

IMPORTANT: You may install and use this software only in accordance with the

terms of the CatDV Server 7.3 license agreement.

Square Box Systems Ltd.

June 2019

Copyright © Square Box Systems Ltd. 2002-2019. All rights reserved.