Squarebox

Before you start

In order to use this Archiware P5 archiving plugin you need:

CatDV Server 8.0.7p3 or later OR 9.0.1p14 or later 

CatDV Pegasus 13.0.5 or later

CatDV Plugin licence 'ARCAW' (Rest API licence with multiple sessions)

Archiware 5.6 or later (currently we do Not support Archiware V6)

To trigger Archiware file transfers from the Worker you need:

Archiware Worker Plugin 2.2.3, included in this installation as ArchiwareWorker.catdv

In addition, to run the archive service standalone (outside the server):

CatDV Service Control Panel 1.3.1 or later

New Installation

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Set server properties required by the plugin:

Open 'Server Config' from the control panel

Enter required properties into the CatDV Server tab, 'Other' field.

See Plugin server properties.

Restart the CatDV server from the control panel

Open the CatDV client (or log the client back in to the server)

Configure one or more service accounts. The first account will initially be the default account, used to verify that Archiware is available and used as a fallback when necessary. See Manage service command for information on the details and settings for a service account:

In the client, run Tools->Manage Archiware P5 Archive Service.

On the accounts tab enter the service account details. Mandatory details are flagged with an asterisk. Note that once a service account has been used to archive a clip, it will no longer be possible to update its identifier.

Click 'Add'.

Configure service settings (See Manage service command)

Optionally configure archiving, processing and UI settings for the plugin via the corresponding tabs on Tools->Manage Archiware P5 Archive Service.

Consider File Verification on Archive and decide whether the ‘Verify after archive’ setting should be set. Note that it is generally recommended not to set plan(s) in Archiware to verify automatically.

Set up Media Store Service Path Mappings to automatically map drives/folders to archive locations. If you plan to always use a single service account and a single plan, you may skip this step and use default location mapping. However, it is still recommended to use this approach if you don’t want the file ‘paths’ on Archiware to include the path of the storage volume.

If you will be using Windows UNC Paths, set up drives and create or amend media stores as required, to enable Archiware to access the storage.

If Running archive service standalone, then configure and start the archive service via the service control panel.

Verify the service setup: In the client, run Tools->Manage Archiware P5 Archive Service again. The service status for the default account should be 'Running (online)'. The status may be 'Running (offline)' if Archiware P5 is not currently accessible.

IF the Worker IS NOT being installed:

Delete ArchiwareWorker.catdv from the Archiware plugin directory (see first step).

IF the Worker IS being installed:

Move ArchiwareWorker.catdv FROM the Archiware plugin directory (see first step) TO the worker plugins directory:

e.g. on a Mac: /Library/Application Support/Square Box/Extensions

e.g. on Windows: %ProgramData%\Square Box\Extensions

e.g. on Linux: /usr/local/catdvWorker/extensions

If the worker is running, re-connect to the server so that the archive plugin fields are available.

Verify that the Archiware worker actions are available. It may be necessary to quit and re-open the worker to pick up the new worker actions.

Upgrade Installation

Make a note of the current plugin version from this document (or see latest version in Release Notes).

If running the archive service standalone, use the service control panel to stop the service.

Stop the CatDV server from the control panel.

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Remove or move the directory / files for the prior plugin from the plugins directory.

Carry out any 'Upgrade' instructions for each plugin version listed in the Release Notes above the last installed version, working back from the oldest applicable version.

Restart the CatDV server from the control panel.

Read through the 'Changes' for each plugin version listed in the Release Notes above the last installed version, and go to Tools->Manage Archiware Archive Service to verify that the details / settings for each account and the archiving / processing / UI settings for the plugin are correct for this installation. See Manage service command for more details.

If Running archive service standalone, then update and start the archive service via the service control panel.

Open the CatDV client (or log the client back in to the server)

Verify the service setup: In the client, run Tools->Manage Archiware P5 Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if Archiware is not currently accessible.

Update (or delete) the Archiware worker plugin by following the instructions for the worker plugin in a new installation. If there is an older Archiware worker plugin file in the extensions directory with a versioned name, delete it.

Plugin server properties

When running the archive service "InPlugin", the following server properties must be set:

catdv.archiware_archive.licence_code = <generated licence code for plugin>

When running the archive service standalone, the properties for the archive service must include:

catdv.archiware_archive.licence_code = <generated licence code for plugin>

catdv.rest_api.host = <CatDV server hostname>

catdv.rest_api.port = <CatDV API port>

catdv.rest_api.user = <CatDV API user>

catdv.rest_api.password = <CatDV API user password> (** exclude if blank, for dev)

Typical rest API values for a development machine would be (password property omitted for a blank password):

localhost, 8080, Administrator

In addition, the following optional property may be set to turn on debug logging on both the server and standalone service:

catdv.archiware_archive.debug = true

Running archive service standalone

By default, the service that handles archive jobs runs inside the CatDV Server. The following is required to run the plugin's archive service on a separate machine from the CatDV server:

CatDV Service Control Panel 1.3 or later

To configure and start the standalone archive service using the service control panel:

Open the Manage service command in the client / web plugin UI

Verify that at least one service account has been configured

In the Processing tab, set the ‘Service mode’ to ‘standalone’ and click ‘Save’

Re-open the Manage Service Command and verify that the service is ‘Stopped’. If not, restart the CatDV server.

Install the CatDV Service Control Panel

Open the CatDV Service Control Panel

Click 'Service Config' to open the service configuration

Fill in license details on the Licensing tab

Optionally, fill in details of a REST endpoint on the REST Service tab

Fill in server connection details on the Server Connection tab (connection can be checked on the control panel)

Enter required service properties on the Settings tab. See Plugin server properties. (**Note the server connection details are duplicated here for now, pending a plugin update to remove them)

On the Installation tab, note the install location ('Server Install Path')

On the file system, navigate to the install location and copy the plugin files into the plugins sub-directory.

Click 'Start' to start the archive service.

To update the standalone archive service:

The service should have been stopped via the Service Control Panel 'Stop' prior to stopping the CatDV Server.

Open the CatDV Service Control Panel

Add / edit service properties on the Settings tab if applicable.

Go to the install location (see Installation tab) and replace the plugin files in the plugins sub-directory with those for the latest version of the plugin.

Click 'Start' to start the archive service.

Archiware Configuration

File Verification on Archive

Archiware P5 is responsible for moving files to and from the file system to its’ storage and is therefore responsible for data integrity. According to the Archiware manual, “It is not required to start a separate verification of the written data since the tape drives detect errors automatically while writing anyway”. However, if this is not considered sufficient to ensure that data integrity is fully preserved, and therefore that no files can be corrupted and lost, the plugin provides a mechanism to trigger Archiware to verify the checksum on newly archived files.

Archiware P5 calculates a checksum on each source file when archiving and saves it in the index along with the file's metadata. The checksum mechanism which is used can be set in the archive pool settings - MD5 is the default. The checksums are only used if a verify job is either configured to run automatically in the Archive Plan (for each archive job that completes) or is run by the plugin. As part of the verify job the files are read from tape, the checksums are recalculated and then compared to the values stored in the index.

Outside of P5, these checksums do not make sense, as they are not applied to the file on disk but on internal data blocks, including all attributes, streams and forks of the files. There is no way to get a file size or checksum that can be compared to the file size or checksum that regards the data segment of the file only.

When using the Archiware plugin, it is not recommended to configure a verify job to run automatically in the Archive Plan. This is because the plugin cannot easily determine whether or not a file has been auto-verified. If auto-verify is used in conjunction with the plugin move or purge commands, it is possible that file verification could fail after the plugin has deleted the source file(s).

If file verification is required, it should be turned on via the ‘Verify after archive’ setting on the ‘Archiving’ tab of Tools->Manage Archiware P5 Archive Service. In this case, the plugin will itself trigger a verify job for each completed archive job and ensure that files are not removed until they have been verified. Note that it is not possible to run a verify job on an archive job which partially completed, in which case files will need to be re-submitted.

IMPORTANT: When ‘Verify after archive’ is on, the plugin will only submit one archive or verify job to run on Archiware at a time, as the P5 manual states:

“When verification is enabled, all the tapes written by the archive job are required once more for verification. Removing the media will cause the job to wait for tapes. When verification is enabled, do not run multiple archive jobs to the same media pool at the same time since that might cause a deadlock.”

In addition, the plugin will automatically set a long timeout duration on an archive job, to try and ensure that the archive and verify jobs run contiguously, in order to avoid unnecessary swapping of tapes.

Worker Configuration

Since ‘verify after archive’ limits active archive jobs to one, there is no value to running archive tasks across multiple workers, or even multiple threads. Therefore, any copy/move to archive task should be configured so that it is only running on a single worker and with “Limit number of tasks” to “At most 1”. The poll interval can be increased if necessary, to improve throughput of small files.

Metadata

The plugin automatically adds three custom metadata fields to Archiware the first time an Archiware account is added:

user_file_size

user_file_date

user_file_orig_path

Media Store Service Path Mappings

Media Stores should be used to automatically map file storage to archive locations by adding a service path to each media store. The service path determines the service account, plan and (optional) folder path for the archived files. Service accounts are set up via the Manage Service screen (see Managing Service Accounts) and are given an identifier when they are created.

For Archiware P5 a service path has the format:

<serviceType>://<accountIdentifier>/<plan>

or

<serviceType>://<accountIdentifier>/<plan>/<folderPrefix>

For example, if a Media Store were set up with the following two entries:

/Volumes/dance-videos/drafts

archiware://squarebox-test/10001/dance

Then a clip located at:

/Volumes/dance-videos/drafts/greatest-show/this-is-me

Would be archived:

using the credentials from the 'squarebox-test' account

into the plan '10001'

with a path of '/dance/greatest-show/this-is-me'

Each Media Store is intended to represent a distinct segment of physical storage so, to avoid ambiguity, no path should be present in multiple media stores and paths should not overlap (e.g. there should not co-exist media stores for /Volumes/dance-videos/drafts and /Volumes/dance-videos/drafts/special).

If you want to avoid having duplicate index entries for files with the same path and name from different drives, make sure they are mapped to either distinct Archiware plans or distinct folders within the same Archiware plan.

The 'folder path' can be set or overridden on archive if 'location' is set as an allowed override. See Manage Service Command (UI TAB).

IMPORTANT: When the ‘folder path’ for a batch of files sent to Archiware is overridden in any way, it is a constraint of Archiware that ALL files in the batch must have the same ‘folder path’. This means that in this case file hierarchy will not be preserved in the Archiware index and that sending files to Archiware with a mixture of different mapped paths, or a mixture of default and mapped paths, will have unpredictable results. Depending on the order in which the files in the batch are processed, they will either all by located each at their default ‘mirror’ path, or all at one of the mapped paths – the first one processed. The order in which the files are processed is not predictable.

Restore path mapping

If a restore location is not writable when a restore job is run, the plugin will use Media Stores to attempt to find an alternative writable restore location. If there is a Media Store with a path matching the restore location, the plugin will check whether any of the other paths in that Media Store are writable and substitute the root directory of the restore location accordingly.

The restore location can be overridden when a restore job is scheduled, if the option to override it is turned on. In earlier versions of the plugin, in this case no file hierarchy was preserved – all files were written directly to the specified directory. Now the plugin will use Media Stores to try and preserve file hierarchy for restored files. If any Media Store path is found that matches the default restore location, it will be used to substitute the root directory of the default restore location with the override.

Note that if many files are restored together to an overridden location, some files may have hierarchy preserved and some may not, depending on whether matching Media Store paths are found.

This scenario is a little more complicated for the Archiware plugin where Archiware is responsible for writing restored files and typically runs on a separate server to the plugin service. A path which is writable from the server running the plugin service may not be writable from the server running Archiware. In this case there must be a corresponding path with an Archiware target, writable from the Archiware server, which the plugin will pass to Archiware.

Path Mapping for Mixed Server OS

If the Archiware server is running under a different OS from the plugin service, it is necessary to map the paths that it uses to read and write files. The path to be used by Archiware must have a custom path type with a target of ‘Archiware Server’.

To set up the custom path type:

From CatDV web admin ‘Media Stores’

Click ‘Manage Path Types’ on bottom left

Click ‘Add’

Enter name (e.g. ‘Archiware Server’), media type ‘Original (hi-res)’, target ‘Archiware Server’

Note ideally there should only be one path of this type in a media store, otherwise all must be accessible to Archiware and any could be used at random.

Each Media Store is intended to represent a distinct segment of physical storage so, to avoid ambiguity, no path should be present in multiple media stores and paths should not overlap (e.g. there should not co-exist media stores for /Volumes/dance-videos and /Volumes/dance-videos/drafts).

Example path mapping where the plug service is running on Windows and the Archiware service is running on Unix:

Media Store: Gamma

Path: X:, type ‘Original/Hi-res’

Path: \volumes\gamma, type ‘Original/Hi-res’

Path: \volumes\gamma, type ‘Archiware Server’

Path Mapping for Windows UNC Paths

Archiware does not support Windows UNC paths. To enable the plugin to handle media with UNC paths, each volume must be mounted as a lettered drive accessible to the server hosting Archiware. In addition, a Media Store must be set up (or amended if applicable) to map each UNC volume (or path(s) within the volume) to a lettered drive path. The plugin itself will use the UNC path to access applicable media but will pass the corresponding drive path to Archiware for archive and restore.

As in the preceding section, the path to be used by Archiware must have a custom path type with a target of ‘Archiware Server’. See preceding section for details of how to set this up.

Each Media Store is intended to represent a distinct segment of physical storage so, to avoid ambiguity, no path should be present in multiple media stores and paths should not overlap (e.g. there should not co-exist media stores for /Volumes/dance-videos and /Volumes/dance-videos/drafts).

Example UNC path mapping where the volume is mounted on the Archiware server as drive E:

Media Store: Ingest

Path: \\CatDV-mini-Z2\MySAN, type ‘Original/Hi-res’

Path: E:, type ‘Archiware Server’

Example UNC path mapping for a sub-dir where the volume is accessible via two subnets and mounted on the Archiware server as drive X:

Media Store: Ingest

Path: \\176.45.4.9\Ingest, type ‘Original/Hi-res’

Path: \\176.32.12.3\Ingest, type ‘Original/Hi-res’

Path: X:\Ingest, type ‘Archiware Server’

Using the plugin

The plugin comprises various commands that are available from the Tools menu in the client.

File transfers are initiated by the schedule copy / move / restore commands but are carried out by a separate process. This means that the client does not automatically reflect updates to a clip on completion of a transfer. Use Server->Refresh Window in the client to pick up any changes during / after transfer.

NB - If you will be using the plugin to archive media with Windows UNC paths, see the preceding section regarding the additional configuration required, as Archiware does not support them.

The plugin includes the following commands:

Command

Description

Manage Archiware P5 Archive Service

View full status details of the archive service and manage the service account(s) required to access the Archiware P5 archives. If a connection to Archiware P5 cannot be made an error is displayed and the values are not saved

View Archiware P5 Archive Service Job Queue

Lists jobs and filter by status (Pending, Complete, Failed, All). It provides the capability to view job contents, view job results and cancel waiting or queued jobs.

Schedule copy to Archiware P5

Adds a bulk copy job for the selected clips (the bulk job contains a child job for each individual clip which can be monitored separately). Copy jobs can be scheduled when no archive service is running / online but will only be run when the default archive service is online (running and Archiware P5 is accessible). When the copy job is run, the source media associated with each clip is copied from the file system to Archiware P5 and the original file is preserved.

Depending on ‘Allow overrides’ in manage service, it is possible to override:

account id – replaces default or media store mapped account

plan – replaces default or media store mapped plan

location – replaces default or media store mapped archive location

Schedule move to Archiware P5

As copy but on successful transfer of the file to storage the job attempts to delete each original file. If for some reason the deletion fails, this does not affect the job status (or archive status of the clip) but "Purge failed" is recorded in the applicable job data.

Schedule restore from Archiware P5

Adds a bulk restore job for the selected clips (the bulk job contains a child job for each individual clip which can be monitored separately). Restores can be scheduled when no archive service is running / online but will only be run when the default archive service is online (running and Archiware P5 is accessible). When the restore job is run, the source media associated with each clip is copied from Archiware P5 to the file system.

Depending on ‘Allow overrides’ in manage service, it is possible to override:

account id – replaces default or media store mapped account. This option will rarely be required

location – replaces default restore location (archive source), hierarchy is preserved for files with a corresponding Media Store path

If you wish to use a UNC path for the restore location, see Windows UNC Paths.

Purge files archived to Archiware P5

Deletes the source media associated with the selected clips if they have been successfully archived.

Managing Service Accounts

The accounts tab on Tools->Manage Archiware P5 Archive Service provides facilities to manage the service account(s) used to connect to Archiware P5 archives. A single account is usually sufficient to operate the plugin. However, if required, there is an add-on which enables multiple accounts to be set up. This is enabled as an add-on via a special server config property.

When there are multiple accounts, the account used may be determined automatically via Media Store Service Path Mappings or selected by the user on archive / restore if allow override of Account ID is turned on from the UI tab . See Manage service command (UI Tab).

An account which is in use, i.e. has been used to archive files, must be unlocked before it can be updated or deleted. This is to ensure that accidental changes cannot be made to an account such that files can no longer be restored. Note it is not possible to change the account identifier on an in-use account.

The accounts tab provides the following operations:

Button

Description

Clear

Clears the current selection / details so that only default values are filled in.

Set Default

Updates the default account to the current selected account.

Add

Creates a new account with the specified details.

Unlock

Unlocks an in-use account in preparation for update / delete.

Update

Updates the selected account with the specified details.

If an account is in use, it must be unlocked before it can be updated. Please take care when updating an account which is in use, as changing some settings could break the restore of files archived with that account.

Note it is not possible to change the account identifier on an in-use account.

Delete

Deletes the selected account.

Please take care when deleting an account which is in use, as it will no longer be possible to restore any clips arched with that account. This feature is intended for removing test accounts and any applicable clips should be deleted or restored and re-archived.

Manage service command

The following can be configured from Tools->Manage Archiware P5 Archive Service in the web or desktop UI for the plugin:

ACCOUNTS TAB / DETAILS (for an account):

Field

Description

Account Identifier

Identifying name for this service account. This may contain only alpha-numerics and hyphens. Note it is not possible to change the account identifier on an in-use account.

Server Host

Archiware P5 server machine, e.g. 'localhost' or an IP address

Server Port

Server port on which Archiware CLI is running, e.g. 9501

Server User

Name of user on the server host

Server Password

Password of user on the server host

Client

Archiware P5 client machine, e.g. 'localhost'

Default Plan

The plan that will be used as the archive fallback for this account by default. This is not applicable when a media store service mapping applies. Otherwise, if the UI tab settings allow the user to override the plan then this will only be used as the default the first time the user does an archive. Subsequent archives by the same user will default to the last value they entered.

NSDChat Command

The location of the NSDChat command on the machine on which the CatDV Server is running. This defaults to the typical location for a Windows or Unix OS, as appropriate.

NSDChat Command Timeout

The time in seconds before an NSDChat command is timed out and the plugin puts the CatDV job into a waiting / polling state pending completion of the Archiware job.

Defaults to 60 seconds.

NB - an account which is in use, i.e. has been used to archive files, must be unlocked before it can be updated or deleted. This is to ensure that accidental changes cannot be made to an account such that files can no longer be restored.

ARCHIVING TAB:

Verify after archive

Flag which triggers post archive checksum verification.

See File verification on archive.

Exclude existing files from restore

Skip files that already exist when scheduling restore jobs. Default is “yes” for new installs.

Restore via temp file

Flag which determines whether files are restored via a temp .$$$ file or directly

Purge directories

Determines whether or not empty directories are deleted when purging (or moving) files.

PROCESSING TAB:

Service mode

NB Always ensure that the standalone service is NOT running prior to changing this value.

Determines whether the job processing service runs in-server (as a thread in the CatDV Server) or standalone (via the Service Control Panel). The initial/default value is in-server.

When this value is switched, the plugin will attempt to start or stop the in-server service as appropriate. If this does not appear to have been successful, as reported on the Overview tab, then it will be necessary to restart the CatDV server to apply the change.

Max files per job

Automatically split files across multiple jobs if the number of files submitted for archive or restore to a single account/plan exceeds this value. Default value is 400. A zero value indicates no limit.

Concurrent job limit

Determines the number of transfers that the archive service will attempt to run concurrently. The default value is 5.

Number delayed retries

The number of times a waiting job is retried at an interval which increases exponentially. The default value is 10, which is equivalent to roughly 34 hours elapsed running time of the archive service.

Maximum retry delay (time period)

Limits the maximum duration between retries of waiting jobs. The default value is blank.

Loop delay (time period)

Determines the frequency with which the archive service checks the Job queue and attempts to run outstanding Jobs. Defaults to 30s.

Retry running job delay (time period)

Determines the duration after which a Job which is running will be restarted if it has not been updated during that period. Defaults to 1h.

Stalled job delay

Determines the duration since a running job's last progress update, after which its’ status is flagged as stalled. Defaults to 1m.

Archiware job completion check

On submit – check every

The frequency with which the plugin initially checks whether a job submitted to Archiware is complete, until the ‘check for’ duration has elapsed. Default is 1s.

Note that if ‘check for’ > 30s and ‘check every’ < 5s, after 30s has elapsed ‘check every’ is automatically increased to 5s.

On submit – check for

Duration the plugin initially waits for a job submitted to Archiware to complete, before the plugin job goes into a waiting state with a periodic check for completion. Default is 5s.

A completion check retry does not affect the overall number of retries for a genuine failure, as it causes the number of retries to be incremented. Furthermore, the retry delay after a completion check is always capped to ‘Retries – check every (max)’, whereas the retry delay for a genuine failure is only capped if the general ‘Maximum retry delay’ is set.

If for some reason a job becomes genuinely ‘stuck’ on Archiware it will need to be cancelled manually from the plugin (which will attempt to also cancel the Archiware job) or Archiware (which will cause the plugin job to fail on the next retry).

Note that this value is not honoured for archives if ‘Verify after archive’ is set. In that case a large delay of 5m is automatically set, to minimise the possibility of tapes being swapped out for another archive job during the delay between archive and verify that results from the archive job timing out.

Retries – check every (max)

The maximum duration before retrying a job which is waiting for completion on Archiware. This still applies regardless of whether a general maximum retry delay has been set. Default is 1h.

UI TAB:

Restrict command access

Restricts the specified plugin commands to sys admin users. Can be used to hide 'config' commands only (i.e. Manage Service), ‘config / archive’ (i.e. to restrict copy and move to archive but not restore and purge), ‘config / archive / purge’ (i.e. to restrict copy, move and purge but not restore) or 'all' commands. Default is ‘none’.

Days to display

The number of days into the past for which jobs are listed in the job queue.

Any job updated in this time period is listed. The default value is 10.

Max no jobs to display

The maximum number of jobs which will be listed in the job queue. This overrides days_to_display. The default value is 1000.

Allow overrides

Enables users to override one or more parameters at the point of archive or restore.

**Any duration can be set in h (hours), m (minutes), s (seconds) or any combination in that order, e.g. 2h, 2h 30m, 5m, 1m 30s, 1s.

Archive details pane

The plugin automatically creates a panel entitled "Archiware P5" containing the clip metadata which describes it's Archiware P5 archive state, including:

Field

Description

squarebox.catdv.archive.Archiware.serviceType

Type of service responsible for file transfer

squarebox.catdv.archive.Archiware.serviceName

Name of service responsible for file transfer

squarebox.catdv.archive.Archiware.status

Archive status

squarebox.catdv.archive.Archiware.location

Location of file in storage

squarebox.catdv.archive.Archiware.restoreLocation

Location file will be (if job pending) or was last restored to

squarebox.catdv.archive.Archiware.date

Date (timestamp) of latest change in archive status

squarebox.catdv.archive.Archiware.dateLastArchived

Date last archived

squarebox.catdv.archive.Archiware.dateLastRestored

Date last restored

squarebox.catdv.archive.Archiware.numArchives

The number of times the clip has been successfully archived

squarebox.catdv.archive.Archiware.archiveKey

Identifier of file in storage

squarebox.catdv.archive.Archiware.batchID

Identifies the batch of files with which the clip was archived

squarebox.catdv.archive.Archiware.jobID

Identifier of current / latest archive job

squarebox.catdv.archive.Archiware.parentJobID

n/a for Archiware P5 (related to bulk archives)

squarebox.catdv.archive.Archiware.userId

ID of user initiating current / latest transfer

squarebox.catdv.archive.Archiware.historyJson

Record of all archive activity

squarebox.catdv.archive.Archiware.purgeError

Details of purge failure

squarebox.catdv.archive.Archiware.accountIdentifier

Identifier of the Archiware P5 Storage Account in CatDV

squarebox.catdv.archive.Archiware.client

Archiware client to transfer file to

squarebox.catdv.archive.Archiware.plan

Archiware plan to transfer file to

squarebox.catdv.archive.Archiware.handle

Handle of archived file on Archiware

squarebox.catdv.archive.Archiware.pool

Name of media pool associated with plan

squarebox.catdv.archive.Archiware.volume

Volume of archived file on Archiware

squarebox.catdv.archive.Archiware.volumeLabel

Volume label of archived File on Archiware

squarebox.catdv.archive.Archiware.barcode

Tape barcode of volume of archived file on Archiware, if applicable

Troubleshooting

Archiware NOT accessible / IO Error executing Archiware command. When paired with an exception in the log “java.io.IOException: Cannot run program "/usr/local/aw/bin/nsdchat": error=316, spawn failed” this is likely to be because a file in the bundled Java runtime needs to be made executable: <Server/Service Dir>/jre/lib/jspawnhelper

Known Issues

If the archive location is overridden via a media store service path mapping, no folder hierarchy will be preserved within that location. This is due to a constraint of Archiware.

If clips in the same archive batch have media store mappings which resolve to a different 'folder path', they will all be placed in the 'folder path' for the first clip in the batch. This is due to a constraint of Archiware and needs some consideration.

If you will be using the plugin to archive media with Windows UNC paths, see Windows UNC Paths regarding the additional configuration required, as Archiware does not support them.

Licence Code

IMPORTANT: You may install and use this software only in accordance with the terms of the CatDV Server 8 license agreement.

Square Box Systems Ltd.

May 2020

Release Notes

2.2.7p1 (2020-06-29)

Upgrade

1. - CatDV Server 8.0.7p3 or later OR 9.0.1p14 or later 

- If using worker, update worker plugin to 2.2.3 (included with this plugin)

Changes

- Fix for conflicting clip edits in worker archive plugin actions, following meta-clip status update

- Fix to try and ensure that duplicate clips are excluded from restores, based on media path rather than file path

- Fix to ensure that volume / volume label / Pool / Barcode are preserved after restore

- Improve clip level archive status to include barcode, so that archive tape field is also populated appropriately

2.2.7 (2020-05-29)

- Update summary archive status on meta-clips when meta-clip member archive status changes

2.2.6p7 (2020-05-20)

- Fix for purge of restored file where mapped media path has changed since file was archived

2.2.6p6 (2020-05-13)

- Additional manage service option to restrict access to purge command along with archive commands

2.2.6p5 (2020-05-04)

Upgrade

- If using worker, update Archiware worker plugin to 2.2.2 (included with this plugin)

Changes

- Add option to exclude existing files from restore jobs

- Fix default html format for json history field, to resolve issue of archive pane flipping / redrawing

- Fix purge of empty complex media folders from bulk jobs

2.2.6p4 (2020-04-02)

Upgrade

- If running service standalone, update CatDV Service Control Panel to 1.3.1

Changes

- Fix for issue restoring files ingested with non-windows paths on Windows.

- Make manage service and job queue windows non-modal in desktop UI (*supported from Pegasus 13.0.9)

2.2.6p3 (2020-03-31)

- Fatally fail building Archiware archive / restore job if an included clip is not found

- Additional debug for generation of restore location

- Fix field type for archive fields ‘media file path’ and ‘last archived params’

- Update log of archive history to include location restored to, rather than location restored from

2.2.6p2 (2020-03-19)

- Fix to ensure restore jobs can run alongside an archive job when verify after archive is on

- Fix to constrain total concurrent transfers to include restore jobs

2.2.6p1 (2020-03-5)

- Fix check which prevents multiple active Archiware jobs when ‘verify after archive’ is enabled.

- Fatally terminate plugin job if Archiware job is terminated or cancelled.

- Ignore trailing slashes in media store paths

- Fix worker plugin commands so that they don’t update the allowed overrides settings for the service to all be enabled

- Update worker plugin to 2.2.1 (only for debug output / version consistency)

2.2.6 (2020-02-11)

Upgrade (whilst CatDV server is down)

- backup the service table from the DB if it contains multiple rows

- run catdvarchiwareplugin2.2.6.sql against the CatDV DB to update the service UID to the new combined value

**Note that if there are multiple service rows with ‘serviceType’ Archiware, in which case only the one most recently updated should be in use, this script will update the UID on the service with the most recent lastModifiedDate and delete the other(s).

- remove occurrences of catdv.archiware_archive.service_mode from the server and service properties, via their respective control panels.

Changes

- significant update to plugin initialisation to prevent it blocking the server thread on start-up and simplify switching between in-server or standalone service for job processing. The same UID is now used for in-server and standalone services and the service mode is switched via a UI service config option, rather than system property. See Manage service command (Processing Tab – Service mode).

2.2.5p6 (2020-03-04)

- Fix check which prevents multiple active Archiware jobs when ‘verify after archive’ is enabled.

2.2.5p5 (2020-01-31)

Updates

- If running service standalone, update CatDV Service Control Panel to 1.3

Changes

- Fix standalone service to run with updated Service Control Panel

- Fix Archiware plugin to always map file paths used by Archiware, not just windows UNC paths, to cater for running Archiware on a different OS to the CatDV

- Fix for file names containing curly braces

- Fatally fail jobs where the associated clip is not found (deleted since job created) and include ‘Clip not found’ in status details.

- Fix to prevent job exceeding max retries

2.2.5p4 (2020-01-15)

Update

- Update worker plugin, so ‘Fix volume data’ is available

Changes

- Fix to ensure archive volume barcode field is always populated on archive

- Add archive fields for volume label and pool

- New worker task ‘Fix volume data’ which can be used to ensure pool/volume/volume label/barcode are populated for archived files

- Increase default NSDChat command timeout to 60s, as adding user defined keys on account creation is slow in some large installations

- Make feedback message more prominent for account operations on the manage service command

- On job queue, preserve job result fields on refresh, to prevent output field jumping every few seconds

2.2.5p3 (2019-12-17)

- Improve speed of scheduling bulk jobs

- Include not ready message in status of incomplete bulk job to which child jobs are being added

- Workaround for issue where json history field has acquired spurious ‘r’ characters after the comma separators. If the history field fails to parse, it will attempt to skip any spurious ‘r’ chars so that it will load successfully.

2.2.5p2 (2019-12-13)

- Additional debug around purge clip add json history

2.2.5p1 (2019-12-09)

- Add config option to Archiware plugin to switch whether files are restored via a temporary .$$$ file

2.2.5 (2019-12-04)

- Add new service config value ‘Max files per job’ on processing tab. Automatically split files across multiple jobs if the number of files submitted for archive or restore to a single account/plan exceeds this value (default 400)

- Fix waiting job to start a new Archiware job on retry, where existing Archiware job no longer exists

- Fix cancel waiting job where Archiware job no longer exists

- Add totalled file size for bulk jobs to job queue

- Potential fix for purging files with special characters, which fails in some environments

- Potential fix for partial job completion on Windows

2.2.4betap4 (2019-11-26)

- All service config duration values are now displayed, and may be entered, in h (hours), m (minutes), s (seconds) or any combination in that order, e.g. 2h, 2h 30m, 5m, 1m 30s, 1s.

- Rename, group together and update descriptions for Archiware config values that relate to checking for Archiware job completion.

‘Job completion delay’ => ‘On submit – check every’

‘Job completion timeout’ => ‘On submit – check for’

‘Job timeout retry delay’ => ‘Retries – check every (max)’

2.2.4betap3 (2019-11-22)

- Eliminate concept of ‘timeout’ for status of jobs submitted to Archiware such that the plugin is waiting for completion

2.2.4betap2 (2019-11-21)

- Additional debug output for Archiware cancel job

2.2.4betap1 (2019-11-18)

- Add mechanism to auto-increment number of retries and auto-cap retry delay when jobs time out waiting for Archiware

2.2.4beta (2019-11-15)

- Implement restore path mapping

- Save running job report in job result output when Archiware job times out

- Limit jobs running on Archiware to the value of ‘concurrent job limit’, whilst always running jobs which previously timed out

2.2.3betap4 (2019-11-11)

- Fix issue causing a timed out restore job to fail for a Windows UNC path

2.2.3betap3 (2019-11-7)

- Fix issue causing a timed out job to be re-submitted, rather than checked for completion, such that it eventually fails with max retries

- Fix issue where restore will fail if there is a temporary restore file left from a previous failure

- Fix issue where restore will fail when media has a UNC path AND the mapped path is not accessible to plugin on server

- Add workaround for issue with Archiware error feedback, so that some files from a partially failed job can complete

- Correct error message on a partial copy/move failure when 'verify after archive' is turned off

- Correct logging for a postponed job when 'verify after archive' is turned on

- Reduce repeated attempts to run postponed job when 'verify after archive' is turned on

2.2.3betap2 (2019-10-31)

Upgrade

- Any media store path(s) previously used/added to represent the mounted drive equivalent of a Windows UNC path must now have a custom path type with a target of ‘Archiware Server’ (requires server 8.0.4p4 or later). If the mounted drive path was in use prior to the plugin installation, this will require it to be added to the media store again with the new path type. See Windows UNC Paths.

Changes

- Require server version 8.0.4 for fix to #190925-110544-112, issue with table row auto-selection on web

- Require server version 8.0.4 for fix to #190927-110649-832, cannot multi-select expandable rows on web

- Fix job queue to ensure that job results and details are cleared when the job table empties due to refresh or cancel

- Update documentation to ensure that file verification and Windows UNC paths are covered during installation

- Update documentation for Media Store Service Path Mapping

2.2.3betap1 (2019-10-16)

- Fix to enable multiple UNC paths in a Media Store used to provide Windows UNC to drive path mapping.

- Fix to eliminate case sensitivity in Windows UNC to drive path mapping.

- Potential fix to accommodate space(s) in Archiware account password.

- Change job processing to improve feedback on individual file errors and allow partial restores to complete.

- Add ‘Verify after archive’ flag to service, which triggers post archive checksum verification.

See section ‘Archiware Configuration – File verification on archive’.

2.2.3beta (2019-09-27)

- Fix Job Queue so that an attempt to cancel a failed/complete job does not report as a job cancelled - Fix likely cause of unexpected service account re-ordering in manage service command

- Make job completion delay and job completion timeout configurable on manage service processing tab

- Fix occasional erroneous display of unlock warning for on update/delete for accounts which are not in use

2.2.2betap1 (2019-09-10)

- Fix allow overrides checkboxes on manage service command in line with server change for ticket #181106-091441-139

- Fix for notification changes to expedite processing of newly queued archive jobs

2.2.2beta (2019-09-06)

- Fixed label for NSDChat command timeout in manage service command

- Add workaround support for Windows UNC paths (see README)

2.2.0beta (2019-08-02)

- Fix cancel job(s) to attempt to stop[scheduled]/cancel[running] corresponding job(s) on Archiware

- Retry service start if rest API is not ready

- Modify generation of clip level archive status to incorporate both current and pending status

e.g. 'Archived [Copy running] to ..." etc

- Use queued archive job notifications to expedite processing of newly queued archive jobs

- Send archive job status notifications for all applicable changes in archive job status

- Restore by default to current media file path from clip, rather than from last archived parameters, so that target

restore location can be updated when e.g. media paths are updated with a different volume name

2.1.4betap1 (2019-07-11)

- Add configurable service account field for nsdchat command timeout

- Reduce the number of distinct nsdchat server session ids created (NB potentially un-resolved for high volume purge)

- Modify text breaks for account update confirmation dialog, to fix web UI wrapping issue

2.1.4beta (2019-07-04)

- Correct required server version in README

- Restrict update of in use accounts via unlock/update/confirm

- Change deletion of in use accounts to unlock/delete/confirm direct from accounts tab

- Update service fields / panel creation to include all fields in field group but exclude some from the panel

- Fix 'column ... already exists' error adding an account

- Fix for UnsupportedOperationException in Collections::UnmodifiableMap.put() on plugin init

- Fix NPE on cancel job, when associated clip is not found

2.1.3beta (2019-06-19)

- update README

2.1.2beta (2019-05-31)

- Switch slow jobs into waiting status (showing archiware job status) after a timeout period

- Allow account id override on restore, to facilitate restore using QLS data

- Automatically add required archiware metadata keys to archiware index, when adding account

- Record archiware volume and barcode in clip archive parameters (for tapes they may be the same)

2.1.1beta (2019-05-24)

- Fix windows path format when adding an entry to an archive selection

- Fix nsdchat geterror - command is case sensitive on Windows

- Extract error details from 'job xmlticket' when Archiware job completion is warning/exception/failure

- Update logging

2.1.0betap1 (2019-05-22)

- Fix display of json based history for web UI

2.1.0beta (2019-05-22)

- Submit files to Archiware in batches for archive and restore

- Capture nsdchat server details and always use session ids to avoid spurious errors due to session clashes

- Change fallback location mapping for Archiware to 'mirror', to exclude the timestamp and related colon in path issue.

- Fix Archiware restore path for Windows

- Add additional archive history field in JSON format

- Temporarily disable in use check for archive account updates

2.1.0prototype2

- Escape backslashes and curly braces in archiware params that are wrapped in curly braces

- Use nsdchat getError command in place of reading from archiware log file (*log file account param removed*)

- Include missing apache commons lang library in plugin release

2.1.0prototype

- Initial prototype version

Copyright © Square Box Systems Ltd. 2002-2019. All rights reserved.