Squarebox

Before you start

In order to use this BlackPearl archiving plugin you need:

CatDV Server 8.0.7p3 or later OR 9.0.1p14 or later 

CatDV Pegasus 13.0.5 or later

CatDV Plugin licence 'ARCBP' (Rest API licence with multiple sessions)

BlackPearl 5.0

To trigger BlackPearl file transfers from the Worker you need:

BlackPearl Worker Plugin 2.2.3, included with this plugin

In addition, to run the archive service standalone (outside the server):

CatDV Service Control Panel 1.3.1 or later

New Installation

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Set server properties required by the plugin:

Open 'Server Config' from the control panel

Enter required properties into the CatDV Server tab, 'Other' field.

See Plugin server properties

Restart the CatDV server from the control panel

Open the CatDV client (or log the client back in to the server)

See BlackPearl Simulator setup for details of how to set up a local BlackPearl simulator for development purposes. Otherwise, you should receive assistance from Spectra Logic to set up your BlackPearl with a user(s) that can access the web management interface and an initial bucket(s) as required.

See File Overwrite / Versioning for consideration of whether to stick with the default location mapping of ‘batchMirror’; where file paths on the BlackPearl are prepended with a date/time stamp); or to use BlackPearl versioning (keep latest / keep multiple) in conjunction with location mapping of ‘mirror’, where file paths on the BlackPearl mirror those on the file system.

Configure one or more service accounts. The first account will initially be the default account, used to verify that the BlackPearl is available and used as a fallback when necessary. See Manage service command for information on the details and settings for a service account:

In the client, run Tools->Manage BlackPearl Archive Service

On the accounts tab, enter the service account details. Mandatory details are flagged with an asterisk. Note that once a service account has been used to archive a clip, it will no longer be possible to update its identifier.

NB - THE END POINT ADDRESS IS ALWAYS DIFFERENT TO THE ADDRESS OF THE BLACK PEARL MANAGEMENT CONSOLE. IF YOU ONLY HAVE THE ADDRESS OF THE MANAGEMENT CONSOLE IT WILL NOT WORK.

Click 'Add'.

Configure service settings (See Manage service command)

Optionally configure archiving, processing and UI settings for the plugin via the corresponding tabs on Tools-> Manage BlackPearl Archive Service

Set up Media Store Service Path Mappings to automatically map drives/folders to archive locations. If you plan to always use a single service account and a single plan, you may skip this step and use default location mapping. However, it is still recommended to use this approach if you don’t want the file ‘paths’ on BlackPearl to include the path of the storage volume.

If Running archive service standalone, then configure and start the archive service via the service control panel.

Verify the service setup: In the client, run Tools->Manage BlackPearl Archive Service again. The service status for the default account should be 'Running (online)'. The status may be 'Running (offline)' if the BlackPearl is not currently accessible.

IF the Worker IS NOT being installed:

Delete BlackPearlWorker.catdv from the BlackPearl plugin directory (see first step).

IF the Worker IS being installed:

Move the BlackPearlWorker.catdv FROM the BlackPearl plugin directory (see first step) TO the worker plugins directory:

e.g. on a Mac: /Library/Application Support/Square Box/Extensions

e.g. on Windows: %ProgramData%\Square Box\Extensions

e.g. on Linux: /usr/local/catdvWorker/extensions

If the worker is running, re-connect to the server (Edit Config / CatDV Server / Reconnect) so that the archive plugin fields are available.

Verify that the BlackPearl worker actions are available. It may be necessary to quit and re-open the worker to pick up the new worker actions.

Upgrade Installation

Make a note of the current plugin version from this document (or see latest version in Release Notes).

If running the archive service standalone, use the service control panel to stop the service.

Stop the CatDV server from the control panel

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Remove or move the directory / files for the prior plugin from the plugins directory.

Carry out any 'Upgrade' instructions for each plugin version listed in the Release Notes above the last installed version, working back from the oldest applicable version

Restart the CatDV server from the control panel

Read through the 'Changes' for each plugin version listed in the Release Notes above the last installed version, and go to Tools->Manage BlackPearl Archive Service to verify that the details / settings for each account and the archiving / processing / UI settings for the plugin are correct for this installation. See Manage service command for more details.

If Running archive service standalone, then update and start the archive service via the service control panel.

Open the CatDV client (or log the client back in to the server)

Verify the service setup: In the client, run Tools->Manage BlackPearl Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if the BlackPearl is not currently accessible.

Update (or delete) the BlackPearl worker plugin by following the instructions for the worker plugin in new installation. If there is an older BlackPearl worker plugin file in the extensions directory with a versioned name, delete it.

Plugin server properties

When running the archive service "InPlugin", the following server properties must be set:

catdv.blackpearl_archive.licence_code = <generated licence code for plugin>

When running the archive service standalone, the properties for the archive service must include:

catdv.blackpearl_archive.licence_code = <generated licence code for plugin>

catdv.rest_api.host = <CatDV server hostname>

catdv.rest_api.port = <CatDV API port>

catdv.rest_api.user = <CatDV API user>

catdv.rest_api.password = <CatDV API user password> (** exclude if blank, for dev)

Typical rest API values for a development machine would be (password property omitted for a blank password):

localhost, 8080, Administrator

In addition, the following optional property may be set to turn on debug logging on both the server and standalone service:

catdv.blackpearl_archive.debug = true

The following property only needs to be set if the plugin is installed as a sub-directory where the name has been changed from the one extracted from the installation zip. In this case it should be set to the full path of the plugin sub-directory.

catdv.blackpearl_archive.plugin_dir = /path/to/plugin/files

Running archive service standalone

By default, the service that handles archive jobs runs inside the CatDV Server. The following is required to run the plugin's archive service on a separate machine from the CatDV server:

CatDV Service Control Panel 1.3 or later

To configure and start the standalone archive service using the service control panel:

Open the Manage service command in the client / web plugin UI

Verify that at least one service account has been configured

In the Processing tab, set the ‘Service mode’ to ‘standalone’ and click ‘Save’

Re-open the Manage Service Command and verify that the service is ‘Stopped’. If not, restart the CatDV server.

Install the CatDV Service Control Panel

Open the CatDV Service Control Panel

Click 'Service Config' to open the service configuration

Fill in license details on the Licensing tab

Optionally, fill in details of a REST endpoint on the REST Service tab

Fill in server connection details on the Server Connection tab (connection can be checked on the control panel)

Enter required service properties on the Settings tab. See Plugin server properties. (**Note the server connection details are duplicated here for now, pending a plugin update to remove them)

On the Installation tab, note the install location ('Server Install Path')

On the file system, navigate to the install location and copy the plugin files into the plugins sub-directory.

Click 'Start' to start the archive service.

To update the standalone archive service:

The service should have been stopped via the Service Control Panel 'Stop' prior to stopping the CatDV Server.

Open the CatDV Service Control Panel

Add / edit service properties on the Settings tab if applicable.

Go to the install location (see Installation tab) and replace the plugin files in the plugins sub-directory with those for the latest version of the plugin.

Click 'Start' to start the archive service.

BlackPearl Configuration

File Overwrite / Versioning

Each BlackPearl bucket has a data policy applied, for example ‘Single Copy on Tape’ or ‘Dual Copy on Tape’. By default, the pre-configured data policies typically have versioning turned off, which means any attempt to write to the same path on the BlackPearl more than once will generate a 409 OBJECT_ALREADY_EXISTS error. This is why the default location mapping for files is ‘batchMirror’.

This version of the BlackPearl plugin supports file overwrite where location mapping is not configured as ‘batchMirror’. For file overwrite to work, the ‘Versioning’ setting on any applicable data policies needs to be set accordingly. To change the Versioning setting for a data policy via the BlackPearl management dashboard:

Choose Configuration -> Advanced Bucket Management -> Storage & Policy Management from the top menu

In the Data Policies list, select the applicable policy e.g. 'Single Copy on Tape'

Choose Action -> Edit from the menu at the top LHS of the page

Under Versioning check 'Keep Latest' or 'Keep Multiple Versions' and click Save

Simulator Setup

You can download the latest version of the simulator from: https://developer.spectralogic.com/downloads/

The Simulator Installation and Usage Instructions can be found here: https://developer.spectralogic.com/installing-the-blackpearl-simulator-as-a-vm/

Follow the above instructions to:

Deploy as either a VMWare OVA or a Virtual Disk (VMDK)

Power on and configure the simulator

To create a bucket owned by the administrator / user you added:

Click 'Configuration' and 'Buckets' from the dashboard top menu bar

Click 'Action' and 'New', enter your bucket name, select the name of the applicable user and click 'Create'.

To force the simulator to 'write' files from cache to tape more quickly:

Start up the simulator

In the web interface go to Configuration > Services

Double click on the “S3” service. Then go to Action > Activate Data Path Back End. Then click on the “Activate” button in the dialog to confirm

Connection Details

The values required to connect to the BlackPearl archive service are:

endpoint - remote address of the BlackPearl, e.g. for the simulator: 192.168.56.101:8080

access key - access key from S3 credentials of BlackPearl user

secret key - secret key from S3 credentials of BlackPearl user

default bucket - name of an accessible bucket on the BlackPearl which can be used to verify the connection,

used as the default bucket the first time a user attempts to copy / move files to the BlackPearl

To get the S3 credentials for a BlackPearl user:

Open the admin console for the BlackPearl

Click 'Configuration->Users'

Select the user

Click 'Action->Show S3 Credentials'

Media Store Service Path Mappings

Media Stores should be used to automatically map file storage to archive locations by adding a service path to each media store. The service path determines the service account, bucket and (optional) folder path for the archived files. Service accounts are set up via the Manage Service screen (see Managing Service Accounts) and are given an identifier when they are created.

For BlackPearl a service path has the format:

<serviceType>://<accountIdentifier>/<plan>

or

<serviceType>://<accountIdentifier>/<plan>/<folderPrefix>

For example, if a Media Store were set up with the following two entries:

/Volumes/dance-videos/drafts

blackpearl://squarebox-test/media-one/dance

Then a clip located at:

/Volumes/dance-videos/drafts/greatest-show/this-is-me

Would be archived:

using the credentials from the 'squarebox-test' account

into the bucket 'media-one'

with a path of '/dance/greatest-show/this-is-me'

Each Media Store is intended to represent a distinct segment of physical storage so, to avoid ambiguity, no path should be present in multiple media stores and paths should not overlap (e.g. there should not co-exist media stores for /Volumes/dance-videos/drafts and /Volumes/dance-videos/drafts/special).

To avoid having duplicate index entries for files with the same path and name from different drives, make sure they are mapped to either distinct BlackPearl buckets or distinct folders within the same BlackPearl bucket.

The 'folder path' can be set or overridden on archive if 'location' is set as an allowed override. See Manage Service Command (UI TAB).

Restore Path Mapping

If a restore location is not writable when a restore job is run, the plugin will use Media Stores to attempt to find an alternative writable restore location. If there is a Media Store with a path matching the restore location, the plugin will check whether any of the other paths in that Media Store are writable and substitute the root directory of the restore location accordingly.

The restore location can be overridden when a restore job is scheduled, if the option to override it is turned on. In earlier versions of the plugin, in this case no file hierarchy was preserved – all files were written directly to the specified directory. Now the plugin will use Media Stores to try and preserve file hierarchy for restored files. If any Media Store path is found that matches the default restore location, it will be used to substitute the root directory of the default restore location with the override.

Note that if many files are restored together to an overridden location, some files may have hierarchy preserved and some may not, depending on whether matching Media Store paths are found.

Using the plugin

The plugin comprises various commands that are available from the Tools menu in the client.

File transfers are initiated by the schedule copy / move / restore commands but are carried out by a separate process. This means that the client does not automatically reflect updates to a clip on completion of a transfer. Use 'Server->Refresh Window' in the client to pick up any changes during or after transfer.

The plugin includes the following commands:

Command

Description

Manage BlackPearl Archive Service

View full status details of the archive service and manage the service account(s) required to access the BlackPearl. These include end point, access key, secret key and default bucket name. If a connection to the BlackPearl cannot be made an error is displayed and the values are not saved.

View BlackPearl Archive Service Job Queue

Lists jobs and filter by status (Pending, Complete, Failed, All). It provides the capability to view job contents, view job results and cancel waiting or queued jobs.

Schedule copy to BlackPearl

Adds a bulk copy job for the selected clips (the bulk job contains a child job for each individual clip which can be monitored separately). Copy jobs can be scheduled when no archive service is running / online but will only be run when the default archive service is online (running and the BlackPearl is accessible). When the copy job is run, the source media associated with each clip is copied from the file system to the BlackPearl and the original file is preserved.

Depending on ‘Allow overrides’ in manage service, it is possible to override:

account id – replaces default or media store mapped account

bucket – replaces default or media store mapped bucket

location – replaces default or media store mapped archive location

Schedule move to BlackPearl

As copy but on successful transfer of the file to storage the job attempts to delete the original file. If for some reason the deletion fails, this does not affect the job status (or archive status of the clip) but "Purge failed" is recorded in the applicable job data.

Schedule restore from BlackPearl

Adds a bulk restore job for the selected clips (the bulk job contains a child job for each individual clip which can be monitored separately). Restores can be scheduled when no archive service is running / online but will only be run when the default archive service is online (running and the BlackPearl is accessible). When the restore job is run, the source media associated with each clip is copied from the BlackPearl to the file system.

Depending on ‘Allow overrides’ in manage service, it is possible to override:

account id – replaces default or media store mapped account. This option will rarely be required

location – replaces default restore location (archive source), hierarchy is preserved for files with a corresponding Media Store Path

Purge local files copied to BlackPearl

Deletes local copies of source media associated with the selected clips, if they have been successfully archived.

NB files deleted from archive cannot be deleted via this command but can be deleted via the delete command below with the ‘Delete local media’ option on, even if the files have already been deleted from archive.

Delete archived files from BlackPearl

Deletes the archived copies of source media associated with the selected clips, if they have been successfully archived. Includes options to purge the media and proxy from local storage. The clip records are not deleted, so history is retained, although they can be removed subsequently if required.

If the BlackPearl has been set up to propagate copies of the media to multiple locations (e.g. tape, backup tape, S3, etc), ALL copies will be deleted.

If the BlackPearl bucket a clip resides in has versioning on (keep latest or keep all) then nothing will actually be deleted but no version will be marked as latest. Therefore, this implementation will only reduce S3 costs if versioning is None.

IMPORTANT: This operation is irreversible. The UI enforces confirmation but care must be taken when setting the worker up to trigger this operation.

View BlackPearl Tape Details

Retrieves and displays details of the tape(s) on which the selected clip(s) reside, including the barcode, type, serial number and ID of each tape.

NB tape details cannot be stored against clips, as the tape location may change if a tape is detected as bad, or if tapes are recompacted to make space, etc.

Managing Service Accounts

The accounts tab on Tools->Manage BlackPearl Archive Service provides facilities to manage the service account(s) used to connect to a BlackPearl. A single account is usually sufficient to operate the plugin. However, if required, there is an add-on which enables multiple accounts to be set up. This is enabled as an add-on via a special server config property.

When there are multiple accounts, the account used may be determined automatically via Media Store Service Path Mappings or selected by the user on archive / restore if allow override of Account ID is turned on from the UI tab . See Manage service command (UI Tab).

An account which is in use, i.e. has been used to archive files, must be unlocked before it can be updated or deleted. This is to ensure that accidental changes cannot be made to an account such that files can no longer be restored. Note it is not possible to change the account identifier on an in-use account.

IMPORTANT: Account updates should be a rare occurrence. If the plugin service is running in the server these may be picked up without a server restart but, depending on which values were updated, this may cause some existing jobs to fail. If the plugin service is running standalone, the service must be re-started in order to pick up the account changes.

The accounts tab provides the following operations:

Button

Description

Clear

Clears the current selection / details so that only default values are filled in.

Set Default

Updates the default account to the current selected account.

Add

Creates a new account with the specified details.

Unlock

Unlocks an in-use account in preparation for update / delete.

Update

Updates the selected account with the specified details.

If an account is in use, it must be unlocked before it can be updated. Please take care when updating an account which is in use, as changing some settings could break the restore of files archived with that account.

Note it is not possible to change the account identifier on an in-use account.

Delete

Deletes the selected account.

Please take care when deleting an account which is in use, as it will no longer be possible to restore any clips arched with that account. This feature is intended for removing test accounts and any applicable clips should be deleted or restored and re-archived.

Manage Service Command

The following can be configured from Tools->Manage BlackPearl Archive Service in the web or desktop UI for the plugin:

ACCOUNTS TAB / DETAILS (for an account):

Field

Description

Account Identifier

Identifying name for this service account. This may contain only alpha-numerics and hyphens. Note it is not possible to change the account identifier on an in-use account.

Endpoint

Remote address of the BlackPearl, e.g. for the simulator: 192.168.56.101:8080

Access Key

Access key from S3 credentials of BlackPearl user

Secret Key

Secret key from S3 credentials of BlackPearl user

Default Bucket Name

The bucket that will be used as the archive fallback for this account by default. This is not applicable when a media store service mapping applies. Otherwise, if the UI tab settings allow the user to override the bucket then this will only be used as the default the first time the user does an archive. Subsequent archives by the same user will default to the last value they entered.

No. write transfer retries

The number of times the SpectraLogic BlackPearl interface retries a write transfer before it assumes that a cache full condition has occurred. The default value is 5.

ARCHIVING TAB:

Location mapping

Note that this is mainly for legacy purposes as the preferred approach is now to use the ‘mirror’ option in conjunction with Media Store Service Path Mappings.

Fallback method for generating the archive file location on the BlackPearl if no media store service mapping applies. The default is 'batchMirror' as that was the only option recommended prior to BlackPearl 5.0, which supports File overwrite / versioning.

Valid values are:

batchMirror: The file(s) selected for archive are batched together in a date and time stamped directory (format /yyyy-mm-dd/hh:mm:ss.mmm) and mirror their source file path(s) within the batch directory. This has the effect of versioning uploads to BlackPearl as a new copy is written for each transfer.

mirror: The archive location always mirrors the source file path which means a file is replaced each time it is archived. Use media store mappings to cater for files with the same path on different file systems.

mediaStoreRelative: Generates relative file paths from the relevant media store(s). If no matching media store path is found, the archive location is generated using the 'mirror' approach.

mediaStoreAbsolute: As 'mediaStoreRelative' but also prepends the path with the name of the media store.

Exclude existing files from restore

Skip files that already exist when scheduling restore jobs. Default is “yes” for new installs.

Purge directories

Determines whether or not empty directories are deleted when purging (or moving) files.

PROCESSING TAB:

Service mode

NB Always ensure that the standalone service is NOT running prior to changing this value.

Determines whether the job processing service runs in-server (as a thread in the CatDV Server) or standalone (via the Service Control Panel). The initial/default value is in-server.

When this value is switched, the plugin will attempt to start or stop the in-server service as appropriate. If this does not appear to have been successful, as reported on the Overview tab, then it will be necessary to restart the CatDV server to apply the change.

Concurrent job limit

Determines the number of transfers that the archive service will attempt to run concurrently. The default value is 5.

Number delayed retries

The number of times a waiting job is retried at an interval which increases exponentially. The default value is 10, which is equivalent to roughly 34 hours elapsed running time of the archive service.

Maximum retry delay (time period)

Limits the maximum duration between retries of waiting jobs. The default value is blank.

Loop delay (time period)

Determines the frequency with which the archive service checks the Job queue and attempts to run outstanding Jobs. Defaults to 30s.

Retry running job delay (time period)

Determines the duration after which a Job which is running will be resumed (if possible) or restarted if it has not been updated during that period. Defaults to 1h.

Stalled job delay

Determines the duration since a running job's last progress update, after which its’ status is flagged as stalled. Defaults to 1m.

UI TAB:

Restrict command access

Restricts the specified plugin commands to sys admin users. Can be used to hide 'config' commands only (i.e. Manage Service and Delete archived files), ‘config / archive’ (i.e. to restrict copy and move to archive but not restore and purge), ‘config / archive / purge’ (i.e. to restrict copy, move and purge but not restore) or 'all' commands. Default is ‘none’.

Days to display

The number of days into the past for which jobs are listed in the job queue.

Any job updated in this time period is listed. The default value is 10.

Max no jobs to display

The maximum number of jobs which will be listed in the job queue. This overrides days_to_display. The default value is 1000.

Allow overrides

Enables the facility for users to override one or more parameters at the point of archive or restore.

**Any duration can be set in h (hours), m (minutes), s (seconds) or any combination in that order, e.g. 2h, 2h 30m, 5m, 1m 30s, 1s.

Archive details pane

The plugin automatically creates a panel entitle "BlackPearl Archive" containing the clip metadata which describes it's BlackPearl archive state, including:

Field

Description

squarebox.catdv.archive.BlackPearl.serviceType

Type of service responsible for file transfer

squarebox.catdv.archive.BlackPearl.serviceName

Name of service responsible for file transfer

squarebox.catdv.archive.BlackPearl.status

Archive status

squarebox.catdv.archive.BlackPearl.location

Location of file in storage

squarebox.catdv.archive.BlackPearl.restoreLocation

Location file will be (if job pending) or was last restored to

squarebox.catdv.archive.BlackPearl.date

Date of latest change in archive status

squarebox.catdv.archive.BlackPearl.dateLastArchived

Date last archived

squarebox.catdv.archive.BlackPearl.dateLastRestored

Date last restored

squarebox.catdv.archive.BlackPearl.numArchives

The number of times the clip has been successfully archived

squarebox.catdv.archive.BlackPearl.archiveKey

Identifier of file in storage

squarebox.catdv.archive.BlackPearl.batchID

Identifies the batch of files with which the clip was archived

squarebox.catdv.archive.BlackPearl.jobID

Identifier of current / latest archive job

squarebox.catdv.archive.BlackPearl.parentJobID

Identifier of current / latest bulk job which includes this clip

squarebox.catdv.archive.BlackPearl.userId

ID of user initiating current / latest transfer

squarebox.catdv.archive.BlackPearl.historyJson

Record of all archive activity

squarebox.catdv.archive.BlackPearl.history

Record of all archive activity (legacy, prior to addition of json history)

squarebox.catdv.archive.BlackPearl.purgeError

Details of purge failure

squarebox.catdv.archive.BlackPearl.accountIdentifier

Identifier of the BlackPearl service account in CatDV

squarebox.catdv.archive.BlackPearl.endpoint

Endpoint identifying the BlackPearl

squarebox.catdv.archive.BlackPearl.bucketName

Name of bucket to transfer file to / from

Known Issues

Restore can overwrite read only files.

Licence Code

IMPORTANT: You may install and use this software only in accordance with the terms of the CatDV Server 7.3 license agreement.

Square Box Systems Ltd.

June 2020

Release notes

2.2.8p1 (2020-06-29)

Upgrade

1. - CatDV Server 8.0.7p3 or later OR 9.0.1p14 or later 

- If using worker, update worker plugin to 2.2.3 (included with this plugin)

Changes

- Fix for conflicting clip edits in worker archive plugin actions, following meta-clip status update

- Fix to try and ensure that duplicate clips are excluded from restores, based on media path rather than file path

- Fix for worker delete action delete local media / proxy, to work around worker checkbox issue

- Disallow purge of files that have been deleted from archive (and not re-archived).

2.2.8 (2020-05-29)

- Update summary archive status on meta-clips when meta-clip member archive status changes

- Fix for purge of restored file where mapped media path has changed since file was archived

2.2.7 (2020-05-13)

Upgrade

- If using worker, update worker plugin to 2.2.2 (included with this plugin)

Changes

- Add capability to “Delete archived files” from UI and worker

- Additional manage service option to restrict access to purge command along with archive commands

- Add option to exclude existing files from restore jobs

- Fix default html format for json history field, to resolve issue of archive pane flipping / redrawing

2.2.6 (2020-04-16)

Upgrade (READ IN FULL FIRST)

- Now require CatDV Server 8.0.4 or later and CatDV Pegasus 13.0.5 or later (for transfer progress notifications)

- If running service standalone, update CatDV Service Control Panel to 1.3.1

- If required, update worker plugin to 2.2.1

- If required, need to set an undocumented config property to enable multiple accounts, as it is an add on product

- backup the service table from the DB if it contains multiple rows

- If the worker is configured to make use of restore location, be aware that file hierarchy will now be preserved relative to a hi-res Media Store path, IF any exists which matches the default restore location (media file path from clip’s archive data).

- run catdvblackpearlplugin2.2.6.sql against the CatDV DB to update the service UID to the new combined value

**Note that if there are multiple service rows with ‘serviceType’ BlackPearl, in which case only the one most recently updated should be in use, this script will update the UID on the service with the most recent lastModifiedDate and delete the other(s).

- The plugin will automatically migrate the connection details from the service definition currently in use (whether in server or standalone) to be the initial 'default' service account. There is a ONE TIME opportunity to customise the identifier of this account by setting the following configuration parameter in the server BEFORE re-starting for the first time after the upgrade, as the identifier of an account which is 'in use' cannot be changed (NB - The specified account identifier may contain only alphanumerics and hyphens):

catdv.blackpearl_archive.migration_account_identifier = <my-account-identifier>

With a single account and no media store service mappings, the plugin will continue to operate as before, using the default account and configured location mapping for all archives. With multiple accounts and corresponding media store service mappings, any clips for which no service mapping is found will fall back to use the configured location mapping, as before.

- The plugin will automatically migrate any optional config values from the server properties to the service definition and they should subsequently be edited using the web or desktop Manage Service UI. You should be able to confirm that these have been set appropriately by going into the UI and comparing the values to the ones in the server and/or service panel config.

**NB Only server property values previously read and loaded into the DB by the 1.x plugin will be preserved. If properties need to be changed as part of the migration these must be applied in the UI afterwards.

**NB For a standalone service, any additional properties set only in the service config will need to be set manually in the UI.

- After this migration, all optional service config properties should be deleted from the config for the server and (if applicable) service control panel. The following properties are not optional and should not be removed if present:

catdv.blackpearl_archive.licence_code

catdv.blackpearl_archive.debug

- In the BlackPearl panel, ensure "BlackPearl Archive History" (identifier squarebox.catdv.archive.BlackPearl.history) is below the new "BlackPearl Archive History (json)" (identifier squarebox.catdv.archive.BlackPearl.historyjson). Ensure these fields are the last two in the panel as the json history field displays as a table.

Changes (MAJOR UPDATE)

- Add support for multiple archive accounts

- Enhancements to manage service command, enabling most configuration to be done via the UI

- Support for automatic mapping from clips to archive locations, via Media Store service mappings which include a service identifier, account identifier, bucket name and (optional) 'folder' e.g. blackpearl://squarebox-test-2/media-one/dance

- Implement restore path mapping

- significant update to plugin initialisation to prevent it blocking the server thread on start-up and simplify switching between in-server or standalone service for job processing. The same UID is now used for in-server and standalone services and the service mode is switched via a UI service config option, rather than system property.

- New plugin command on the tools menu to retrieve and display the details of the tape(s) on which selected clip(s) reside.

- Add ability to restrict access to (config and) archive but allow restore

- Send job progress notifications, using new notification system

- Add json version of archive history to clip archive metadata

- Update service fields / panel creation to include all fields in field group but exclude some from the panel

- Modify generation of clip level archive status to incorporate both current and pending status e.g. 'Archived [Copy running] to ..." etc

- Use queued archive job notifications to expedite processing of newly queued archive jobs

- Send archive job status notifications for all applicable changes in archive job status

- On job queue ensure job details/results fields are cleared when job table empties due to refresh or cancel

- Fix to prevent job exceeding maxRetries

- Fix purge of empty complex media folders from bulk jobs

- Fix to ensure that on restore, restore location is checked for update since last archive, rather than the media location.

1.5.2 (2020-03-31)

- Complete jobs as soon as files have reached the BlackPearl cache, rather than waiting for transfer to final archive location, as per the best practise documented by SpectraLogic for BlackPearl 5.x.

1.5.1 (2020-01-27)

Upgrade

- Requires BlackPearl 5.x as SDK updated to 5.0.4

- Requires server 8.0.4 for supporting changes and fixes for operating on multiple table rows

Changes

- Update to DS3 Java SDK 5.0.4, which supports file overwrite (see File Overwrite / Versioning)

- Update job queue to support multi-select cancel

- Display file size in job queue for new jobs, where possible

- Fix to prevent job exceeding max retries

- Fatally fail jobs where the associated clip is not found (deleted since job created), and include ‘Clip not found’ in status details.

1.5.0 (2019-10-31)

- Run another job (if available) immediately after completion / failure of a running job

- Changes to purge (these apply to move to archive as well as direct purge) - clear date last restored, log 'File purged' to clip archive params history, add option of "Yes to all" confirmation for purging multiple altered files.

- Restore by default to current media file path from clip, rather than from last archived parameters, so that target

restore location can be updated when e.g. media paths are updated with a different volume name

- Fix NPE on cancel job, when associated clip is not found

1.4.19p2 (2019-12-16)

- Improve speed / blocking of scheduling bulk jobs - use updateBulkJobReadyToRun to avoid un-necessarily cascading

the status update down to child jobs and clip archive metadata.

- Include not ready message in the status description of a job which is not ready to run

- Run another job (if available) immediately after completion / failure of a running job

- Additional debug output for service loop

1.4.19p1 (2019-05-10)

- Potential fix for ticket #190509-104746-987

1.4.19 (2019-03-22)

- Fix blocking mechanism for ensuring that multiple processes cannot update a job's status simultaneously

- Put archive jobs "Transferring to Final Archive Location" on a different retry schedule to general failures to reduce the lag between job completion on BlackPearl and in CatDV. After the first hour, check job status every hour up to 48 hours, before timing out.

1.4.18 (2019-03-05)

- prevent spurious 'Network outage'/NPE error when no restore directory can be extracted from the restore location

1.4.17 (2019-02-27)

- ensure that a missing file date / path in an archived file's metadata does not cause purge or restore to fail

- update thread handling when processing jobs, to ensure a completed job will not be retried before it's status has been fully updated

1.4.16 (2019-02-05)

- Display 'Retry' for status of unsuccessful job results if the job is being / will be retried

- Fix to enable all spectra plugin files to be in the plugin sub-directory.

1.4.15 (2019-01-31)

- Improve performance of Job Queue command for large lists of jobs

- Fix to cause job creation to fail if no source media are updated with the archive details for the job

- Add plugin command that can be triggered via the rest API to generate clip data for a file archived outside CatDV

- Add config param 'catdv.blackpearl_archive.max_retry_delay' to limit the delay period between retries of waiting jobs.

- Trim trailing path separators from archive / restore location overrides

1.4.12

- Add config param 'catdv.blackpearl_archive.purge_directories' to turn off purging of directories when moving / purging files.

- Update README to cover installing and updating the archive worker plugin, now included as part of the plugin installation.

1.4.11p5 (2019-01-25)

- Fix for issue of jobs stuck in running / stalled state

1.4.11p4 (2019-01-16)

- Fix to ensure plugin calls server rest API as /catdv/api/... instead of /api/... to prevent failed calls due to redirection of POST requests in some environments

1.4.11 (2018-10-03)

Upgrade

- If the 'catdv.blackpearl_archive.allow_override' server property is explicitly set to include 'location:Location:archive', modify it to 'archiveLocation:Location:archive'

Changes

- Add option to restore files to a specified location / directory (can be enabled as an override for restore)

1.4.9 (2018-09-25)

Upgrade

- This is a significant update from the latest release of the BlackPearl plugin 1.4.1p<xx>. If upgrading from that version please ensure you execute the Upgrade step from 1.4.2beta. If upgrading from an earlier version, apply all upgrade steps between that version and this version.

Changes

- Additional fix for move command failing to purge files

1.4.8betap7 (2018-09-14)

- Fix for move command failing to purge files

- Fix for error message on cancel job contains NullPointerException

- Fix for issue which causes a new FieldGroup for an archive plugin to be created each time the server is restarted

- Fixed bug causing purge commands to fail from the web UI

- Improve the error reporting when attempting to archive offline files

1.4.8betap6 (2018-07-25)

- Fixed bucket name bug in utility for generating a CatDV XML file from the contents of a BlackPearl bucket

1.4.8betap5 (2018-07-25)

- Updated utility for generating a CatDV XML file from the contents of a BlackPearl bucket

1.4.8betap4 (2018-07-16)

- Correct job type for BlackPearl worker restore command to a bulk job type.

1.4.8betap3 (2018-07-10)

- Add utility for generating a CatDV XML file from the contents of a BlackPearl bucket.

1.4.8betap2

- Fix location override for archives triggered from worker

- Record archive failure in clip archive status if clip has never been archived

1.4.8betap1 (2018-06-20)

- Update README

- Fix to eliminate spurious exception starting / stopping standalone service

1.4.8beta (2018-06-19)

- Manage service command: preserve new connection details even if they are not valid / updated, to ease entry

- Manage service command: obfuscate the secret key value

** Merged from 1.4.1p11

- Fix for NPE when setting up panel definitions (findFieldGroupByIdentifierPrefix)

- Include sfl4j-simple library for BlackPearl logging

1.4.7beta (2018-06-07)

- Additional location mapping option to determine location on archive from Media Store paths, includes mediaStoreRelative and mediaStoreAbsolute.

- Fix to ensure that location on archive does not contain '\' separators, regardless of location mapping in use

- Fix for NPE when clip has media file but no metadata (introduced by fix in 1.4.5, merged from patch 1.4.1betap8 )

- Fix for bug which causes cancel job to fail when clip associated with job is not found

- Integrate with ServicePlugin framework

1.4.5 (2018-05-03)

- Fix for rest.NotFoundException scheduling jobs: change method for checking job exists, to avoid server incompatibility

1.4.5beta

- Move packages in order to split out re-usable plugin SDK classes

- Fix to prevent NPE - on archiving / restoring, skip clips that have no source media (i.e. sequences)

- Fix for location override in WorkerBackupCommand

- Update BlackPearl ds3 Java SDK libraries to 4.1.0, along with updates to libs they depend on, to suport BlackPearl 4.1

** version 1.4.1betap9 also has the updated libraries

1.4.4beta

- Updated httpclient and httpcore libraries (required to support other archive plugin)

1.4.3beta (2018-02)

- Add config param 'catdv.blackpearl_archive.allow_override' to enable override of archive location from the UI.

- Add config param 'catdv.blackpearl_archive.location_mapping' to enable batching of file archive locations to be turned off - i.e. subsequent archives replace the existing file(s), rather than creating new 'date / time stamped' copies.

- Fix to ensure server config values that affect the UI are picked up on startup when archive service is standalone.

- Fix exponential back off timing for archive job retries

- Add date last restored as an archive parameter

1.4.2betap1 (2018-01-22)

- Fix for archive service start

1.4.2beta (2018-01-09)

Upgrade

- run catdvarchiveplugins1.4.2.sql against the CatDV DB to rename a job data field and to update the textual job and clip archive status values for consistency

Changes

- Add capability to run plugin archive service standalone, via config param catdv.blackpearl_archive.service_mode

- Provision for providing more detailed status information for jobs and (archiving) files

- Add detailed job status and job data to Job Details pane in the Service Job Queue UI

- New config param 'catdv.blackpearl_archive.max_jobs_to_display' for configuring the maximum number of jobs which will be listed in the job queue. This overrides days_to_display and defaults to 1000

- Increased default value for 'catdv.blackpearl_archive.loop_delay' to 30 seconds

- Increased default value for 'catdv.blackpearl_archive.concurrent_transfer_limit' to 4

- Increased default value for 'catdv.blackpearl_archive.days_to_display' to 10

1.4.1p11

- Fix for NPE when setting up panel definitions (findFieldGroupByIdentifierPrefix)

- Include sfl4j-simple library for BlackPearl logging

1.4.1p10 (2019-05-09)

- Fix for NPE when clip has media file but no metadata (introduced by fix in 1.4.1betap8)

1.4.1betap9 (2018-04-05)

- Update BlackPearl ds3 Java SDK libraries to 4.1.0, along with updates to libs they depend on, to suport BlackPearl 4.1

1.4.1betap8 (2018-02-19)

- Fix to prevent NPE - on archiving / restoring, skip clips that have no source media (i.e. sequences)

1.4.1betap7 (2018-02-05)

- Add date last restored as an archive parameter

1.4.1betap6 (2017-12-22)

- Fixes for worker plugin commands

1.4.1betap5 (2017-12-20)

- Create parent directories if required when restoring files

- Return json response message from WorkerPurgeCommand

- Include metaclipID where applicable in error details for worker backup / restore / purge commands

1.4.1betap4 (2017-12-12)

- Fix bulk backup and restore commands to capture and report unexpected errors queuing child jobs

- Update worker backup and restore commands to return a JSON representation of the archive result in the command response.

1.4.1betap3 (2017-12-01)

- Fix to ignore clips with duplicate mapped file paths, before processing to queue jobs

1.4.1betap2

- New hidden versions of backup (copy/move) and purge commands for use by worker

1.4.1betap1 (2017-11-16)

- Fix infinite loop caused by an error during the creation of a bulk job, where it is stuck in a non-runnable state

- Fix to ensure archive metadata for source media with matching media paths are updated simultaneously

- Additional service logging, new config param 'catdv.blackpearl_archive.debug' now turns on/off most logging after initial plugin startup

1.4.1beta (2017-06-09)

Upgrade

- remove 'catdv.rest_api.*' config properties in Server Config on CatDV control panel

Changes

- major update to job queue command UI, utilising new features provided by version 3 of plugin framework

- make API calls in process (inside server) from plugin commands and from archive service when running inside server (depends on version 3 of plugin framework)

1.3.5beta (2017-06-09)

Upgrade

- change config property 'catdv.rest_api.client_key' to 'catdv.blackpearl_archive.licence_code' in Server Config on CatDV control panel

- run catdvarchiveplugins1.3.5.sql against the CatDV DB to fix the field type for archive date fields

Changes

- improve job processing loop so that job priorities are always respected: only take one job from the queue at a time, when waiting job delays have elapsed re-queue jobs rather than processing waiting jobs directly (note re-queued jobs of equal priority will be processed before newer jobs, as the lowest job id will be processed first)

- ensure that transfers to the Black Pearl never block the job processing loop, keeping the connection status up to date

- enable processing of multiple concurrent transfers plus config param max_jobs_running_delay for optimisation

- improve handling when the Black Pearl is unavailable: after a short while the service status updates to "Running (offline)"

- restart orphaned "in progress" jobs (e.g. from a server restart) more quickly

- Option to enter bucket name when copying / moving files to BlackPearl. The bucket name on the Manage Service screen is now only the default value filled in the first time a user attempts to copy or move files. Once the user has entered a bucket, a subsequent move / copy will default to the last value entered. The bucket warning and "Apply bucket to queued jobs" checkbox are no longer on the Manage Service screen as changes there now only affect the default bucket for a new user of the archive plugin.

- Option to specify destination location 'path' when copying / moving files to BlackPearl from the worker.

- Add config param "catdv.blackpearl_archive.restrict_command_access" which can be used to hide 'all' plugin commands or 'config' commands (currently Manage Service). See README for details.

- switch to using statusCode as primary status value (except job queries which requires latest server)

- add support for pausing a service (processing transfers)

- improve plugin licence handling

- update BlackPearl SDK for Java library to 3.4.0

- fix archive date fields in desktop client

- fix NPE when clip has no metadata

1.3.4

Changes

- to support HGST/StrongBox archives, add new config param catdv.s3archive_service.signer_override and upgrade to aws-java-sdk-1.11.52 (along with dependencies: joda-time-2.9.5.jar, httpcore-4.4.5.jar, httpclient-4.5.2.jar)

1.3.3

Upgrade

- generate a rest api licence code registered to the customer for client 'ARCBP' and add to the configured properties:

catdv.rest_api.client_key=<generated_licence_code>

- Run catdvblackpearlplugin1.3.3.sql against the CatDV DB to set status codes for existing Jobs

- check in web admin that there are not two BlackPearl archive panels. if there are, delete the one that is not referenced by the data field of the service table entry for 'BlackPearl'

Changes

- use a separate license pool for each archive plugin's access to the Rest API

- fixes for archiving of complex media

- complex media archive status reflects the archive status of it's contents (clip media 'Archive Details' field now updated along with the archive values stored in clip media metadata)

- auto-divide clips into multiple bulk jobs when restoring from multiple buckets

- rename package squarebox.util to squarebox.archiveplugin.util to avoid name clashes in the server

- add more detail to successful output for job results

- change location of files on the BlackPearl archive: <batch id>/<actual file path>

- the batch id is the date/time that a selection of files was scheduled for archive to the BlackPearl

- propagate changes to archive metadata to all source media with the same value for mediaPath

- set status code on archive Jobs along with text status

- purge empty directories along with files when using the 'Schedule move... ' or 'Purge files...' commands

- the top two directory levels are always preserved even if empty, e.g. '/volumes/media/'

- automatically create field definition(s) for any new archive metadata and add them to the plugin's archive panel

- throw a meaningful error if any of the properties required by the RestAPI are not set

1.3.2p2 (2016-10-19)

- updated BlackPearl API to version ds3-sdk-3.2.6 which contains a bug fix for the restore of files larger than 100GB.

** if you are using the BlackPearl developer simulator you may need to upgrade to the latest version

1.3.2p1 (2016-09-21)

- patch for NullPointerException when scheduling transfers for clips with no existing metadata

1.3.2 (2016-09-08)

Upgrade

- Run catdvblackpearlplugin1.3.2.sql against the CatDV DB to update the Job types

Changes

- fix for archiving mapped paths

- changed plugin command names and job types for clarity

1.3.1 (2016-08-10)

Upgrade

- Run catdvblackpearlplugin1.3.1.sql against the CatDV DB to convert archive dates in DB from timestamps to ISO date strings

- If an archive panel was previously set up manually, this should be removed, as one is now auto-created

- Remove 'catdvrestapi.jar' from the plugins directory, as this is now rolled into 'catdvarchiveplugin.jar'

Changes

- Store archive dates in clip metadata as ISO date strings, so they are user readable without processing

- Setting up field definitions / panel: fix to ensure service name is used rather than a hardcoded value

- Job queue: Display dates in ISO date format

- Job queue: Always re-open in 'All jobs' mode, IF screen is dismissed using 'Close' button

- Clip archive metadata: fix archive history newlines, date format and prepend rather than append entries

- Changing service endpoint: cancel queued restores as well as interrupted jobs

- Changing service endpoint: fix to ensure clip metadata and job descriptions are always updated correctly for queued jobs

- Changing bucket name (for queued jobs): fix so only applied to queued archives, not restores

- Changing bucket name (for queued jobs): fix to ensure clip metadata and job descriptions are always update correctly for queued archives

1.2

- Retain last entered values in Manage Service command until valid values have been saved

- Eliminate config property catdv.blackpearl_archive.classpath

- Automatically set up archive panel / field definitions when service starts

- Reinstate feature for updating the bucket name on queued jobs

- Enhance fix clip state command

1.1

- Disable non-functioning feature for updating the bucket name on queued jobs

1.0

- Initial production version

Copyright © Square Box Systems Ltd. 2002-2019. All rights reserved.