Squarebox

Before you start

In order to use this BlackPearl archiving plugin you need:

CatDV Server 7.3 or later

CatDV Pegasus 12.0b15 or later

CatDV Plugin licence 'ARCBP' (Rest API licence with multiple sessions)

BlackPearl 3.0 - 4.1

To trigger BlackPearl file transfers from the Worker you need:

BlackPearl Worker Plugin 1.4.11, included in this installation as

BlackPearlWorker1.4.11.catdv

In addition, to run the archive service standalone (outside the server):

CatDV Service Control Panel 1.2 or later

Installation

New

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Set server properties required by the plugin:

Open 'Server Config' from the control panel

Enter required properties into the CatDV Server tab, 'Other' field. See 'Plugin server properties' below and 'Running archive service standalone', if applicable.

Enter optional properties if desired. See 'Plugin server properties' below.

Restart the CatDV server from the control panel

Open the CatDV client (or log the client back in to the server)

Configure the parameters to access the BlackPearl:

In the client, run Tools->Manage BlackPearl Archive Service, enter end point, access key, secret key and bucket name and click 'save and start'. If the values entered are valid and a connection to the BlackPearl can be established, the archive service will start up automatically. The service status can be monitored on the management and job queue screens.

NB - THE END POINT ADDRESS IS ALWAYS DIFFERENT TO THE ADDRESS OF THE BLACK PEARL MANAGEMENT CONSOLE. IF YOU ONLY HAVE THE ADDRESS OF THE MANAGEMENT CONSOLE IT WILL NOT WORK.

See 'BlackPearl Simulator setup' for details of how to set up a local BlackPearl simulator for development purposes. Otherwise, you should receive assistance from SpectraLogic to set up your BlackPearl with a user(s) that can access the web management interface and an initial bucket(s) as required.

See 'BlackPearl configuration values' for details of the parameters that need to be set up to access theBlackPearl.

If running the archive service standalone (configured via a server property in step two), then configure and start the archive service via the service control panel. See 'Running archive service standalone'.

Verify the service setup: In the client, run Tools->Manage BlackPearl Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if the BlackPearl is not currently accessible.

Install (or delete) the BlackPearl worker plugin:

IF the Worker IS NOT installed: Delete the BlackPearl worker plugin file with extension '.catdv' from the BlackPearl plugin directory (see step 1).

IF the Worker IS installed: Move the BlackPearl worker plugin file with extension '.catdv' FROM the BlackPearl plugin directory installed in step 1 TO the worker plugins directory:

e.g. on a Mac: /Library/Application Support/Square Box/Extensions

e.g. on Windows: %ProgramData%\Square Box\Extensions

e.g. on Linux: /usr/local/catdvWorker/extensions

If the worker is running, re-connect to the server (Edit Config / CatDV Server / Reconnect) so that the archive plugin fields are available.

Verify that the BlackPearl worker actions are available. **It may be necessary to quit and re-open the worker to pick up the new worker actions.

Update

Make a note of the current plugin version - from BlackPearl_README.txt / BlackPearl_ReleaseNotes.txt in the plugin directory or from the server log at start up when the plugin is loaded

If running the archive service standalone, use the service control panel to stop the service.

Stop the CatDV server from the control panel

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Remove or move the directory / files for the prior plugin from the plugins directory.

Carry out any 'Upgrade' instructions for each plugin version listed in the BlackPearl_ReleaseNotes.txt (from this zip) above the last installed version, working back from the oldest applicable version

Read through the 'Changes' for each plugin version listed in the BlackPearl_ReleaseNotes.txt (from this zip) above the last installed version, to check whether any new config params have been added which could usefully be customised for this installation

Restart the CatDV server from the control panel

If running the archive service standalone (configured via a server property in step two), then update and start the archive service via the service control panel. See 'Running archive service standalone'.

Open the CatDV client (or log the client back in to the server)

Verify the service setup: In the client, run Tools->Manage BlackPearl Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if the BlackPearl is not currently accessible.

Update (or delete) the BlackPearl worker plugin by following the instructions from step 8 for a new installation, making sure to move / remove the .catdv file for the prior version of the plugin from the extensions directory.

Running archive service standalone

By default, the service that handles archive jobs runs inside the plugin. To run it as a standalone process,

ensure that the service mode property is set to Standalone in the server when installing the plugin (see step 2

of Installation):

catdv.blackpearl_archive.service_mode = Standalone

The following is required to run the plugin's archive service on a separate machine from the CatDV server:

CatDV Service Control Panel 1.1.1 or later

NB The archive service connection must be configured in the client / web plugin UI before it can be started.

To configure and start the standalone archive service using the service control panel:

Install the CatDV Service Control Panel

Open the CatDV Service Control Panel

Click 'Service Config' to open the service configuration

Enter license details on the Licensing tab

Enter required and optional service properties on the Settings tab. See 'Plugin server properties' below.

On the Installation tab, note the install location ('Server Install Path')

On the file system, navigate to the install location and copy the plugin files into the plugins sub-directory.

Click 'Start' to start the archive service.

To update the standalone archive service:

The service should have been stopped via the Service Control Panel 'Stop' prior to stopping the CatDV Server.

Open the CatDV Service Control Panel

Add / edit service properties on the Settings tab if applicable.

Go to the install location (see Installation tab) and replace the plugin files in the plugins sub-directory with those for the latest version of the plugin.

Click 'Start' to start the archive service.

Plugin server properties

When running the archive service "InPlugin", the following server properties must be set:

catdv.blackpearl_archive.licence_code = <generated licence code for plugin>

When running the archive service standalone, the following server properties must ALSO be set:

catdv.blackpearl_archive.service_mode = Standalone

When running the archive service standalone, the properties for the archive service must include:

catdv.blackpearl_archive.licence_code = <generated licence code for plugin>
catdv.blackpearl_archive.service_mode = Standalone
catdv.rest_api.host = <CatDV server hostname>
catdv.rest_api.port = <CatDV API port>
catdv.rest_api.user = <CatDV API user>
catdv.rest_api.password = <CatDV API user password>         (** exclude if blank, for dev environment)

Typical rest API values for a development machine would be (password property omitted for a blank password):

localhost, 8080, Administrator

In addition, the following optional properties can also be set:

catdv.service.api_timeout = <time_period_in_milliseconds>

Determines the timeout period after which the current connection to theCatDV API will be assumed to have timed out and therefore be replaced with a new connection (this is to pre-empt exceptions due to timeouts).

catdv.blackpearl_archive.service_mode = InPlugin / Standalone

Determines whether the job processing service for the plugin runs inside the plugin or as a standalone process outside the CatDV server. The default value is "InPlugin".

catdv.blackpearl_archive.restrict_command_access = NOT SET / config / all

If set, restricts the specified plugin commands to sys admin users. Can be used to hide 'config' commands only (i.e. Manage Service), or 'all' commands.

NB - this only works with CatDV client versions from 12 onwards.

catdv.blackpearl_archive.allow_override

Enables the facility to override one or more parameters at the point of archive or restore. The comma-separated string value may include:

archiveLocation:Location:archive

restoreLocation:Location:restore

blackpearl.bucket_name:Bucket name:archive

The middle item is the UI field label and may be modified.

The default value is "blackpearl.bucket_name:Bucket name:archive" Set to 'none' to allow no overrides.

catdv.blackpearl_archive.location_mapping

Method for generating the archive file location on the BlackPearl. Valid values are:

batchMirror: The file(s) selected for archive are batched together in a date and time stamped directory (format /yyyy-mm-dd/hh:mm:ss.mmm) and mirror their src file path(s) within the batch directory. This has the effect of versioning uploads to the BlackPearl as a new copy is written for each transfer.

The default is 'batchMirror', which is the only recommended option until the release of BlackPearl 5.0.

The following options which are available for other archive types are not currently recommended for the BlackPearl as it does not permit a file to be archived to the same location multiple times. Any attempt to archive a file more than once will result in an error. This constraint will no longer apply with BlackPearl 5.0.

mirror: The archive location always mirrors the src file path which means a file is replaced each time it is archived. **NB The downside of this approach is that it does not cater for files with the same path on different file systems.

mediaStore:

mediaStoreRelative: Generates relative file paths from the relevant media store(s).

If no matching media store path is found, the archive location is generated using the 'mirror' approach.

mediaStoreAbsolute: As 'mediaStoreRelative' but also prepends the path with the name of the media store.

catdv.blackpearl_archive.days_to_display = <number_of_days>

The number of days into the past for which jobs are listed in the job queue.

Any job updated in this time period is listed. The default value is 10.

catdv.blackpearl_archive.max_jobs_to_display = <number_of_jobs>

The maximum number of jobs which will be listed in the job queue. This overrides days_to_display. The default value is 1000.

catdv.blackpearl_archive.loop_delay = <time_period_in_milliseconds>

Determines the frequency with which the archive service checks the Job queue and attempts to run outstanding Jobs. The default value is 30000, equivalent to 30 seconds.

catdv.blackpearl_archive.max_jobs_running_delay = <time_period_in_milliseconds>

Determines the time period to wait before attempting to run another job when the maximum number of transfers are already running. The default value is 0. This would typically be increased (either when a typical transfer is likely to be slow or when the concurrent transfer limit is greater than 1) in order to reduce hits to the database to query jobs whilst max transfers are running.

catdv.blackpearl_archive.num_delayed_job_retries

The number of times a waiting job is retried at an interval which increases exponentially. The default value is 10, which is equivalent to roughly 34 hours elapsed running time of the archive service.

catdv.blackpearl_archive.max_retry_delay = <time_period_in_milliseconds>

Limits the maximum time period between retries of waiting jobs. The default value is 0, which represents no maximum value.

catdv.blackpearl_archive.num_write_transfer_retries

The number of times the SpectraLogic BlackPearl interface retries a write transfer before it assumes that a cache full condition has occurred. The default value is 5.

catdv.blackpearl_archive.retry_in_progress_job_delay

Determines the time period after which a Job which is running will be resumed (if possible) or restarted if it has not been updated during that period. Defaults to a value equivalent to 1 hour.

catdv.blackpearl_archive.concurrent_transfer_limit

Determines the number of transfers that the archive service will attempt to run concurrently. The default value is 4.

catdv.blackpearl_archive.plugin_dir

Only needs to be set if the plugin is installed as a sub-directory where the name has been changed from the one extracted from the installation zip. In this case it should be set to the full path of the plugin sub-directory.

BlackPearl Simulator Setup

You can download the latest version of the simulator from:

https://developer.spectralogic.com/downloads/

The Simulator Installation and Usage Instructions can be found here:

https://developer.spectralogic.com/sim-install/

Follow these instructions to:

Set up the BlackPearl Simulator

Configure the simulator

Configure a user

Get the access key / secret key (S3 credentials) for that user

Create a bucket owned by the user you configured:

Click 'Configuration' and 'Buckets' from the dashboard top menu bar

Click 'Action' and 'New', enter your bucket name, select the name of the user you configured and click 'Create'.

To force the simulator to 'write' files from cache to tape more quickly:

Start up the simulator

In the web interface go to Configuration > Services

Double click on the “S3” service. Then go to Action > Activate Data Path Back End. Then click on the “Activate” button in the dialog to confirm

BlackPearl configuration values

The values required to run the BlackPearl archive service are:

endpoint - remote address of the BlackPearl, e.g. for the simulator: 192.168.56.101:8080

access key - access key from S3 credentials of BlackPearl user

secret key - secret key from S3 credentials of BlackPearl user

default bucket - name of an accessible bucket on the BlackPearl which can be used to verify the connection,

used as the default bucket the first time a user attempts to copy / move files to the BlackPearl

To get the S3 credentials for a BlackPearl user:

Open the admin console for the BlackPearl

Click 'Configuration->Users'

Select the user

Click 'Action->Show S3 Credentials'

Using the plugin

The plugin comprises various commands that are available from the Tools menu in the client.

File transfers are initiated by the schedule copy / move / restore commands but are carried out by a separate process. This means that the client does not automatically reflect updates to a clip on completion of a transfer. Use 'Server->Refresh Window' in the client to pick up any changes during / after transfer.

The plugin includes the following commands:

* Manage BlackPearl Archive Service - View full status details of the archive service and enter / update the configuration values required to access the BlackPearl. These include end point, access key, secret key and default bucket name. If a connection to the BlackPearl cannot be made an error is displayed and the values are not saved.

* View BlackPearl Archive Service Job Queue - list of jobs that can be filtered by status (Pending, Complete, Failed, All). It provides the capability to view job contents, view job results and cancel waiting or queued jobs.

* Schedule Copy to BlackPearl - adds a bulk copy job for the selected clips (the bulk job contains a child job for each individual clip which can be monitored separately). Copy jobs can be scheduled when the archive service is not running or is offline but will only be run when the archive service is online (running and the BlackPearl is accessible). When the copy job is run, the source media associated with each clip is copied from the file system to the BlackPearl and the original file is preserved.

* Schedule move to BlackPearl - as copy but on successful transfer of the file to storage the job attempts to delete the original file. If for some reason the deletion fails this does not affect the job status (or archive status of the clip) but "Purge failed" is recorded in the job data.

* Schedule Restore from BlackPearl - adds a bulk restore job for the selected clips (the bulk job contains a child job for each individual clip which can be monitored separately). Restores can be scheduled when the archive service is not running or is offline but will only be run when the archive service is online (running and the BlackPearl is accessible). When the restore job is run, the source media associated with each clip is copied from the BlackPearl to the file system.

* Purge Files Archived to BlackPearl - deletes the source media associated with the selected clips if they have been successfully archived.

Archive details pane

The plugin automatically creates a panel entitle "BlackPearl Archive" containing the clip metadata which describes it's BlackPearl archive state, including:

squarebox.catdv.archive.BlackPearl.serviceType - Type of service responsible for file transfer

squarebox.catdv.archive.BlackPearl.serviceName - Name of service responsible for file transfer

squarebox.catdv.archive.BlackPearl.status - Archive status

squarebox.catdv.archive.BlackPearl.location - Location of file in storage

squarebox.catdv.archive.BlackPearl.date - Date (timestamp) of latest change in archive status

squarebox.catdv.archive.BlackPearl.dateLastArchived - Date last archived

squarebox.catdv.archive.BlackPearl.dateLastRestored - Date last restored

squarebox.catdv.archive.BlackPearl.numArchives - The number of times the clip has been successfully archived

squarebox.catdv.archive.BlackPearl.archiveKey - Identifier of file in storage

squarebox.catdv.archive.BlackPearl.batchID - Identifies the batch of files with which the clip was archived

squarebox.catdv.archive.BlackPearl.jobID - Identifier of current / latest archive job

squarebox.catdv.archive.BlackPearl.parentJobID - Identifier of current / latest bulk job which includes this clip.

squarebox.catdv.archive.BlackPearl.userId - ID of user initiating current / latest transfer

squarebox.catdv.archive.BlackPearl.history - Record of all archive activity

squarebox.catdv.archive.BlackPearl.endpoint - Endpoint identifying the BlackPearl

squarebox.catdv.archive.BlackPearl.bucketName - Name of bucket to transfer file to / from

Known Issues

Restore can overwrite read only files.

Licence Code

IMPORTANT: You may install and use this software only in accordance with the terms of the CatDV Server 7.1.1 license agreement.

Square Box Systems Ltd.

October 2017

Release notes

1.4.19p1 (2019-05-10)

- Potential fix for ticket #190509-104746-987

1.4.19 (2019-03-22)

- Fix blocking mechanism for ensuring that multiple processes cannot update a job's status simultaneously

- Put archive jobs "Transferring to Final Archive Location" on a different retry schedule to general failures to reduce the lag between job completion on BlackPearl and in CatDV. After the first hour, check job status every hour up to 48 hours, before timing out.

1.4.18 (2019-03-05)

- prevent spurious 'Network outage'/NPE error when no restore directory can be extracted from the restore location

1.4.17 (2019-02-27)

- ensure that a missing file date / path in an archived file's metadata does not cause purge or restore to fail

- update thread handling when processing jobs, to ensure a completed job will not be retried before it's status has been fully updated

1.4.16 (2019-02-05)

- Display 'Retry' for status of unsuccessful job results if the job is being / will be retried

- Fix to enable all spectra plugin files to be in the plugin sub-directory.

1.4.15 (2019-01-31)

- Improve performance of Job Queue command for large lists of jobs

- Fix to cause job creation to fail if no source media are updated with the archive details for the job

- Add plugin command that can be triggered via the rest API to generate clip data for a file archived outside CatDV

- Add config param 'catdv.blackpearl_archive.max_retry_delay' to limit the delay period between retries of waiting jobs.

- Trim trailing path separators from archive / restore location overrides

1.4.12

- Add config param 'catdv.blackpearl_archive.purge_directories' to turn off purging of directories when moving / purging files.

- Update README to cover installing and updating the archive worker plugin, now included as part of the plugin installation.

1.4.11p5 (2019-01-25)

- Fix for issue of jobs stuck in running / stalled state

1.4.11p4 (2019-01-16)

- Fix to ensure plugin calls server rest API as /catdv/api/... instead of /api/... to prevent failed calls due to redirection of POST requests in some environments

1.4.11 (2018-10-03)

Upgrade

- If the 'catdv.blackpearl_archive.allow_override' server property is explicitly set to include 'location:Location:archive', modify it to 'archiveLocation:Location:archive'

Changes

- Add option to restore files to a specified location / directory (can be enabled as an override for restore)

1.4.9 (2018-09-25)

Upgrade

- This is a significant update from the latest release of the BlackPearl plugin 1.4.1p<xx>. If upgrading from that version please ensure you execute the Upgrade step from 1.4.2beta. If upgrading from an earlier version, apply all upgrade steps between that version and this version.

Changes

- Additional fix for move command failing to purge files

1.4.8betap7 (2018-09-14)

- Fix for move command failing to purge files

- Fix for error message on cancel job contains NullPointerException

- Fix for issue which causes a new FieldGroup for an archive plugin to be created each time the server is restarted

- Fixed bug causing purge commands to fail from the web UI

- Improve the error reporting when attempting to archive offline files

1.4.8betap6 (2018-07-25)

- Fixed bucket name bug in utility for generating a CatDV XML file from the contents of a BlackPearl bucket

1.4.8betap5 (2018-07-25)

- Updated utility for generating a CatDV XML file from the contents of a BlackPearl bucket

1.4.8betap4 (2018-07-16)

- Correct job type for BlackPearl worker restore command to a bulk job type.

1.4.8betap3 (2018-07-10)

- Add utility for generating a CatDV XML file from the contents of a BlackPearl bucket.

1.4.8betap2

- Fix location override for archives triggered from worker

- Record archive failure in clip archive status if clip has never been archived

1.4.8betap1 (2018-06-20)

- Update README

- Fix to eliminate spurious exception starting / stopping standalone service

1.4.8beta (2018-06-19)

- Manage service command: preserve new connection details even if they are not valid / updated, to ease entry

- Manage service command: obfuscate the secret key value

** Merged from 1.4.1p11

- Fix for NPE when setting up panel definitions (findFieldGroupByIdentifierPrefix)

- Include sfl4j-simple library for BlackPearl logging

1.4.7beta (2018-06-07)

- Additional location mapping option to determine location on archive from Media Store paths, includes mediaStoreRelative and mediaStoreAbsolute.

- Fix to ensure that location on archive does not contain '\' separators, regardless of location mapping in use

- Fix for NPE when clip has media file but no metadata (introduced by fix in 1.4.5, merged from patch 1.4.1betap8 )

- Fix for bug which causes cancel job to fail when clip associated with job is not found

- Integrate with ServicePlugin framework

1.4.5 (2018-05-03)

- Fix for rest.NotFoundException scheduling jobs: change method for checking job exists, to avoid server incompatibility

1.4.5beta

- Move packages in order to split out re-usable plugin SDK classes

- Fix to prevent NPE - on archiving / restoring, skip clips that have no source media (i.e. sequences)

- Fix for location override in WorkerBackupCommand

- Update BlackPearl ds3 Java SDK libraries to 4.1.0, along with updates to libs they depend on, to suport BlackPearl 4.1

** version 1.4.1betap9 also has the updated libraries

1.4.4beta

- Updated httpclient and httpcore libraries (required to support other archive plugin)

1.4.3beta (2018-02)

- Add config param 'catdv.blackpearl_archive.allow_override' to enable override of archive location from the UI.

- Add config param 'catdv.blackpearl_archive.location_mapping' to enable batching of file archive locations to be turned off - i.e. subsequent archives replace the existing file(s), rather than creating new 'date / time stamped' copies.

- Fix to ensure server config values that affect the UI are picked up on startup when archive service is standalone.

- Fix exponential back off timing for archive job retries

- Add date last restored as an archive parameter

1.4.2betap1 (2018-01-22)

- Fix for archive service start

1.4.2beta (2018-01-09)

** Upgrade **

- run catdvarchiveplugins1.4.2.sql against the CatDV DB to rename a job data field and to update the textua job and clip archive status values for consistency

Changes

- Add capability to run plugin archive service standalone, via config param catdv.s3archive_service.service_mode

- Provision for providing more detailed status information for jobs and (archiving) files

- Add detailed job status and job data to Job Details pane in the Service Job Queue UI

- New config param 'catdv.blackpearl_archive.max_jobs_to_display' for configuring the maximum number of jobs which will be listed in the job queue. This overrides days_to_display and defaults to 1000

- Increased default value for 'catdv.blackpearl_archive.loop_delay' to 30 seconds

- Increased default value for 'catdv.s3archive_service.concurrent_transfer_limit' to 4

- Increased default value for 'catdv.s3archive_service.days_to_display' to 10

1.4.1p11

- Fix for NPE when setting up panel definitions (findFieldGroupByIdentifierPrefix)

- Include sfl4j-simple library for BlackPearl logging

1.4.1p10 (2019-05-09)

- Fix for NPE when clip has media file but no metadata (introduced by fix in 1.4.1betap8)

1.4.1betap9 (2018-04-05)

- Update BlackPearl ds3 Java SDK libraries to 4.1.0, along with updates to libs they depend on, to suport BlackPearl 4.1

1.4.1betap8 (2018-02-19)

- Fix to prevent NPE - on archiving / restoring, skip clips that have no source media (i.e. sequences)

1.4.1betap7 (2018-02-05)

- Add date last restored as an archive parameter

1.4.1betap6 (2017-12-22)

- Fixes for worker plugin commands

1.4.1betap5 (2017-12-20)

- Create parent directories if required when restoring files

- Return json response message from WorkerPurgeCommand

- Include metaclipID where applicable in error details for worker backup / restore / purge commands

1.4.1betap4 (2017-12-12)

- Fix bulk backup and restore commands to capture and report unexpected errors queuing child jobs

- Update worker backup and restore commands to return a JSON representation of the archive result in the command response.

1.4.1betap3 (2017-12-01)

- Fix to ignore clips with duplicate mapped file paths, before processing to queue jobs

1.4.1betap2

- New hidden versions of backup (copy/move) and purge commands for use by worker

1.4.1betap1 (2017-11-16)

- Fix infinite loop caused by an error during the creation of a bulk job, where it is stuck in a non-runnable state

- Fix to ensure archive metadata for source media with matching media paths are updated simultaneously

- Additional service logging, new config param 'catdv.blackpearl_archive.debug' now turns on/off most logging after initial plugin startup

1.4.1beta (2017-06-09)

Upgrade

- remove 'catdv.rest_api.*' config properties in Server Config on CatDV control panel

Changes

- major update to job queue command UI, utilising new features provided by version 3 of plugin framework

- make API calls in process (inside server) from plugin commands and from archive service when running inside server (depends on version 3 of plugin framework)

1.3.5beta (2017-06-09)

Upgrade

- change config property 'catdv.rest_api.client_key' to 'catdv.blackpearl_archive.licence_code' in Server Config on CatDV control panel

- run catdvarchiveplugins1.3.5.sql against the CatDV DB to fix the field type for archive date fields

Changes

- improve job processing loop so that job priorities are always respected: only take one job from the queue at a time, when waiting job delays have elapsed re-queue jobs rather than processing waiting jobs directly (note re-queued jobs of equal priority will be processed before newer jobs, as the lowest job id will be processed first)

- ensure that transfers to the Black Pearl never block the job processing loop, keeping the connection status up to date

- enable processing of multiple concurrent transfers plus config param max_jobs_running_delay for optimisation

- improve handling when the Black Pearl is unavailable: after a short while the service status updates to "Running (offline)"

- restart orphaned "in progress" jobs (e.g. from a server restart) more quickly

- Option to enter bucket name when copying / moving files to BlackPearl. The bucket name on the Manage Service screen is now only the default value filled in the first time a user attempts to copy or move files. Once the user has entered a bucket, a subsequent move / copy will default to the last value entered. The bucket warning and "Apply bucket to queued jobs" checkbox are no longer on the Manage Service screen as changes there now only affect the default bucket for a new user of the archive plugin.

- Option to specify destination location 'path' when copying / moving files to BlackPearl from the worker.

- Add config param "catdv.blackpearl_archive.restrict_command_access" which can be used to hide 'all' plugin commands or 'config' commands (currently Manage Service). See README for details.

- switch to using statusCode as primary status value (except job queries which requires latest server)

- add support for pausing a service (processing transfers)

- improve plugin licence handling

- update BlackPearl SDK for Java library to 3.4.0

- fix archive date fields in desktop client

- fix NPE when clip has no metadata

1.3.4

Changes

- to support HGST/StrongBox archives, add new config param catdv.s3archive_service.signer_override and upgrade to aws-java-sdk-1.11.52 (along with dependencies: joda-time-2.9.5.jar, httpcore-4.4.5.jar, httpclient-4.5.2.jar)

1.3.3

Upgrade

- generate a rest api licence code registered to the customer for client 'ARCBP' and add to the configured properties:

catdv.rest_api.client_key=<generated_licence_code>

- Run catdvblackpearlplugin1.3.3.sql against the CatDV DB to set status codes for existing Jobs

- check in web admin that there are not two BlackPearl archive panels. if there are, delete the one that is not referenced by the data field of the service table entry for 'BlackPearl'

Changes

- use a separate license pool for each archive plugin's access to the Rest API

- fixes for archiving of complex media

- complex media archive status reflects the archive status of it's contents (clip media 'Archive Details' field now updated along with the archive values stored in clip media metadata)

- auto-divide clips into multiple bulk jobs when restoring from multiple buckets

- rename package squarebox.util to squarebox.archiveplugin.util to avoid name clashes in the server

- add more detail to successful output for job results

- change location of files on the BlackPearl archive: <batch id>/<actual file path>

- the batch id is the date/time that a selection of files was scheduled for archive to the BlackPearl

- propagate changes to archive metadata to all source media with the same value for mediaPath

- set status code on archive Jobs along with text status

- purge empty directories along with files when using the 'Schedule move... ' or 'Purge files...' commands

- the top two directory levels are always preserved even if empty, e.g. '/volumes/media/'

- automatically create field definition(s) for any new archive metadata and add them to the plugin's archive panel

- throw a meaningful error if any of the properties required by the RestAPI are not set

1.3.2p2 (2016-10-19)

- updated BlackPearl API to version ds3-sdk-3.2.6 which contains a bug fix for the restore of files larger than 100GB.

** if you are using the BlackPearl developer simulator you may need to upgrade to the latest version

1.3.2p1 (2016-09-21)

- patch for NullPointerException when scheduling transfers for clips with no existing metadata

1.3.2 (2016-09-08)

Upgrade

- Run catdvblackpearlplugin1.3.2.sql against the CatDV DB to update the Job types

Changes

- fix for archiving mapped paths

- changed plugin command names and job types for clarity

1.3.1 (2016-08-10)

Upgrade

- Run catdvblackpearlplugin1.3.1.sql against the CatDV DB to convert archive dates in DB from timestamps to ISO date strings

- If an archive panel was previously set up manually, this should be removed, as one is now auto-created

- Remove 'catdvrestapi.jar' from the plugins directory, as this is now rolled into 'catdvarchiveplugin.jar'

Changes

- Store archive dates in clip metadata as ISO date strings, so they are user readable without processing

- Setting up field definitions / panel: fix to ensure service name is used rather than a hardcoded value

- Job queue: Display dates in ISO date format

- Job queue: Always re-open in 'All jobs' mode, IF screen is dismissed using 'Close' button

- Clip archive metadata: fix archive history newlines, date format and prepend rather than append entries

- Changing service endpoint: cancel queued restores as well as interrupted jobs

- Changing service endpoint: fix to ensure clip metadata and job descriptions are always updated correctly for queued jobs

- Changing bucket name (for queued jobs): fix so only applied to queued archives, not restores

- Changing bucket name (for queued jobs): fix to ensure clip metadata and job descriptions are always update correctly for queued archives

1.2

- Retain last entered values in Manage Service command until valid values have been saved

- Eliminate config property catdv.blackpearl_archive.classpath

- Automatically set up archive panel / field definitions when service starts

- Reinstate feature for updating the bucket name on queued jobs

- Enhance fix clip state command

1.1

Disable non-functioning feature for updating the bucket name on queued jobs

1.0

Initial production version

Copyright © Square Box Systems Ltd. 2002-2019. All rights reserved.