Squarebox

Before you start

In order to use this BackBlaze archiving plugin you need:

CatDV Server 7.3 or later

CatDV Pegasus 12.0b15 or later

CatDV Plugin license 'ARCBB' (Rest API license with multiple sessions)

To trigger BackBlaze file transfers from the Worker you need:

BackBlaze Worker Plugin 1.5, included in this installation as BackBlazeWorker.catdv

In addition, to run the archive service standalone (outside the server):

CatDV Service Control Panel 1.2 or later

New Installation

IMPORTANT: If you don't already have a BackBlaze account or you don't know the access keys see 'BackBlaze setup' for further details.

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on Unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Set optional server properties for the plugin, if desired:

Open 'Server Config' from the control panel

Enter required properties into the CatDV Server tab, 'Other' field. See 'Plugin server properties' below and 'Running archive service standalone', if applicable.

Enter optional properties if desired. See 'Plugin server properties' below.

Restart the CatDV server from the control panel

Open the CatDV client (or log the client back in to the server)

Configure the parameters to access BackBlaze:

In the client, run Tools->Manage BackBlaze Archive Service, enter account id, application key and bucket name and click 'save' or 'save and start'. If the values entered are valid and a connection to BackBlaze can be established, then when the service mode is 'InPlugin' the archive service will be started automatically.

The service status can be monitored on the management and job queue screens.

If running the archive service standalone (configured via a server property in step two), then configure and start the archive service via the service control panel. See 'Running archive service standalone'.

Verify the service setup: In the client, run Tools->Manage BackBlaze Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if BackBlaze is not currently accessible.

IF the Worker IS NOT being installed

Delete the BackBlaze worker plugin file with extension '.catdv' from the BackBlaze plugin directory (see first step)

IF the Worker IS being installed:

Move the BackBlaze worker plugin file with extension '.catdv' FROM the BackBlaze plugin directory installed in first step TO the worker plugins directory:

e.g. on a Mac: /Library/Application Support/Square Box/Extensions e.g. on Windows: %ProgramData%\Square Box\Extensions e.g. on Linux: /usr/local/catdvWorker/extensions

If the worker is running, re-connect to the server (Edit Config / CatDV Server / Reconnect) so that the archive plugin fields are available.

Verify that the BackBlaze worker actions are available. It may be necessary to quit and re-open the worker to pick up the new worker actions.

Upgrade Installation

Make a note of the current plugin version from this document (or the latest version in the ‘Release Notes’)

If running the archive service standalone, use the service control panel to stop the service.

Stop the CatDV server from the control panel

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Remove or move the directory / files for the prior plugin from the plugins directory.

Carry out any 'Upgrade' instructions for each plugin version listed in the Release Notes (last section of this document) above the last installed version, working back from the oldest applicable version

Read through the 'Changes' for each plugin version listed in the Release Notes (last section of this document) above the last installed version, to check whether any new config params have been added which could usefully be customised for this installation

Start the CatDV server from the control panel

If running the archive service standalone (configured via a server property in second step), then update and start the archive service via the service control panel. See 'Running archive service standalone'.

Open the CatDV client (or log the client back in to the server)

Verify the service setup: In the client, run Tools->Manage BackBlaze Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if BackBlaze is not currently accessible.

Update (or delete) the BackBlaze worker plugin by following the instructions from the last step for a new installation, making sure to move / remove the .catdv file for the prior version of the plugin from the extensions directory.

Running archive service standalone

By default, the service that handles archive jobs runs inside the plugin. To run it as a standalone process, ensure that the service mode property is set to Standalone in the server when installing the plugin (see second step of Installation):

catdv.backblaze_archive.service_mode = Standalone

The following is required to run the plugin's archive service on a separate machine from the CatDV server:

CatDV Service Control Panel 1.2 or later

To configure and start the standalone archive service using the service control panel:

Verify that the archive service connection has been configured in the client / web plugin UI

Install the CatDV Service Control Panel

Open the CatDV Service Control Panel

Click 'Service Config' to open the service configuration

Enter license details on the Licensing tab

Enter required and optional service properties on the Settings tab. See 'Plugin server properties' below.

On the Installation tab, note the install location ('Server Install Path')

On the file system, navigate to the install location and copy the plugin files into the plugins sub-directory.

Click 'Start' to start the archive service.

To update the standalone archive service:

The service should have been stopped via the Service Control Panel 'Stop' prior to stopping the CatDV Server.

Open the CatDV Service Control Panel

Add / edit service properties on the Settings tab if applicable.

Go to the install location (see Installation tab) and replace the plugin files in the plugins sub-directory with those for the latest version of the plugin.

Click 'Start' to start the archive service.

Plugin server properties

When running the archive service "InPlugin", the following server properties must be set:

catdv.backblaze_archive.licence_code = <generated licence code for plugin>

When running the archive service standalone, the following server properties must ALSO be set:

catdv.backblaze_archive.service_mode = Standalone

When running the archive service standalone, the property file for the archive service must include:

catdv.backblaze_archive.licence_code = <generated licence code for plugin>

catdv.backblaze_archive.service_mode = Standalone

catdv.rest_api.host = <CatDV server hostname>

catdv.rest_api.port = <CatDV API port>

catdv.rest_api.user = <CatDV API user>

catdv.rest_api.password = <CatDV API user password> (** exclude if blank, for dev)

Typical rest API values for a development machine would be (password property omitted for a blank password):

catdv.rest_api.host = localhost

catdv.rest_api.port = 8080

catdv.rest_api.user = administrator

In addition, the following optional properties can also be set:

Property

Description

catdv.service.api_timeout = <time_period_in_milliseconds>

Determines the timeout period after which the current connection to the CatDV API will be assumed to have timed out and therefore be replaced with a new connection (this is to pre-empt exceptions due to timeouts).

catdv.backblaze_archive.service_mode = InPlugin / Standalone

Determines whether the job processing service for the plugin runs inside the plugin or as a standalone process outside the CatDV server. The default value is "InPlugin".

catdv.backblaze_archive.restrict_command_access = NOT SET / config / all

If set, restricts the specified plugin commands to sys admin users. Can be used to hide 'config' commands only (i.e. Manage Service), or 'all' commands.

NB - this only works with CatDV client versions from 12 onwards.

catdv.backblaze_archive.allow_overridde

Enables the facility to override one or more parameters at the point of archive or restore. The comma-separated string value may include:

archiveLocation:Location:archive

restoreLocation:Location:restore

backblaze.bucket_name:Bucket name:archive

The middle item is the UI field label and may be modified. The default value is "backblaze.bucket_name:Bucket name:archive". Set to 'none' to allow no overrides.

catdv.backblaze_archive.location_mapping

Method for generating the archive file location on B2. The default is 'batchMirror'. Valid values are:

batchMirror: The file(s) selected for archive are batched together in a date and time stamped directory (format /yyyy-mm-dd/hh:mm:ss.mmm) and mirror their source file path(s) within the batch directory. This has the effect of versioning uploads to B2 as a new copy is written.

mirror: The archive location always mirrors the source file path which means a file is replaced each time it is archived. The downside of this approach is that it does not cater for files with the same path on different file systems.

mediaStoreRelative: Generates relative file paths from the relevant media store(s). If no matching media store path is found, the archive location is generated using the 'mirror' approach.

mediaStoreAbsolute: As 'mediaStoreRelative' but also prepends the path with the name of the media store.

catdv.backblaze_archive.days_to_display = <number_of_days>

The number of days into the past for which jobs are listed in the job queue. Any job updated in this time period is listed. The default value is 10.

catdv.backblaze_archive.max_jobs_to_display = <number_of_jobs>

The maximum number of jobs which will be listed in the job queue. This overrides days_to_display. The default value is 1000.

catdv.backblaze_archive.loop_delay = <time_period_in_milliseconds>

Determines the frequency with which the archive service checks the Job queue and attempts to run outstanding Jobs. The default value is 30000, equivalent to 30 seconds.

catdv.backblaze_archive.max_jobs_running_delay = <time_period_in_milliseconds>

Determines the time period to wait before attempting to run another job when the maximum number of transfers are already running. The default value is 0. This would typically be increased (either when a typical transfer is likely to be slow or when the concurrent transfer limit is greater than 1) in order to reduce hits to the database to query jobs whilst max transfers are running. Default value is 0.

catdv.backblaze_archive.retry_in_progress_job_delay = <time_period_in_milliseconds>

Determines the time period after which a Job which is running will be restarted if it has not been updated during that period. Defaults to a value equivalent to 1 hour.

catdv.backblaze_archive.num_delayed_job_retries

The number of times an interrupted job is retried at an interval which increases exponentially. The default value is 10, which is equivalent to roughly 34 hours elapsed running time of the archive service.

catdv.backblaze_archive.max_retry_delay = <time_period_in_milliseconds>

Limits the maximum time period between retries of waiting jobs. The default value is 0, which represents no maximum value.

catdv.backblaze_archive.concurrent_transfer_limit

Determines the number of transfers that the BackBlaze transfer manager will attempt to run concurrently. The default value is 4.

BackBlaze Setup / Configuration values

In order to use the plugin, a BackBlaze account is required. If you need to set up a BackBlaze account for testing purposes, got to https://www.backblaze.com/b2/sign-up.html to create an account with 10GB free.

For details of pricing see: https://help.backblaze.com/hc/en-us/articles/217667478-Understanding-B2-Pricing-Structure

Once you are logged into a BackBlaze account, the 'My Account' link will take you to the BackBlaze console: https://secure.backblaze.com/b2_buckets.htm

In order for the plugin to copy to / move to / restore from your BackBlaze account, you will need to enable B2, set up an application key and create at least one bucket. The account has a master application key which has complete access.

NB - B2 has an API in development which will enable application keys to be created with restricted capabilities. Eventually, it will be possible to create and manage application keys from the BackBlaze console but initially this will only be possible via the API.

To enable B2 cloud storage:

To enable B2 on an existing BackBlaze account, first login to the BackBlaze account with the registered email address and password.

Once logged in, select the 'My Settings' link from the left side navigation.

At the bottom of the 'My Settings' page B2 can be enabled under the 'Enabled Products' section.

To get the master application key:

From the 'My Account' page (https://secure.backblaze.com/b2_buckets.htm), click on the link that reads 'Show Account ID & Application Key".

To create a bucket:

Go to the 'My Account' page (https://secure.backblaze.com/b2_buckets.htm)

Click the link 'Create a bucket'

Enter the bucket name, choose private or public and click 'Create a Bucket'. Files in a private bucket can only by accessed via the console or using the application key. Files in a public bucket can be downloaded by anyone, but the downloads will be charged against your account.

Note bucket names must be globally unique across all B2 cloud storage accounts

The values required to configure the service in the BackBlaze plugin are:

Account ID - account id from a BackBlaze account

Application key - application key from a BackBlaze account

Default bucket - name of an accessible bucket on BackBlaze which can be used to verify the connection, used as the default bucket the first time a user attempts to copy / move files to BackBlaze

Using the plugin

The plugin comprises various commands that are available from the Tools menu in the client.

File transfers are initiated by the schedule copy / move / restore commands but are carried out by a separate process. This means that the client does not automatically reflect updates to a clip on completion of a transfer. Use 'Server->Refresh Window' in the client to pick up any changes during / after transfer.

The plugin includes the following commands:

Command

Description

Manage BackBlaze Archive Service

Displays full status details of the archive service and enables the user to enter / update the configuration values required to access the B2 storage. These include account id, application key and default bucket name. If a connection to BackBlaze cannot be made an error is displayed and the values are not saved.

View BackBlaze Archive Service Job Queue

Lists jobs that can be filtered by status (Pending, Complete, Failed, All). It provides the capability to view job contents, view job results and cancel waiting or queued jobs.

Schedule copy to BackBlaze

Adds a copy job for each clip selected (if multiple clips reference the same source media only one job is created). Copy jobs can be scheduled when the archive service is not running or is offline but will only be run when the archive service is online (running and BackBlaze is accessible). When the copy job is run, the source media associated with the clip is copied from the file system to BackBlaze and the original file is preserved.

Schedule move to BackBlaze

As copy but on successful transfer of the file to storage the job attempts to delete the original file. If for some reason the deletion fails, this does not affect the job status (or archive status of the clip) but "Purge failed" is recorded in the job data.

Schedule restore from BackBlaze

Adds a restore job for each clip selected (if multiple clips reference the same source media only one job is created). Restores can be scheduled when the archive service is not running or is offline but will only be run when the archive service is online (running and BackBlaze is accessible). When the restore job is run, the source media associated with the clip is copied from BackBlaze to the file system.

Purge files archived to BackBlaze

Deletes the source media associated with the selected clips if they have been successfully archived.

Archive details pane

The plugin automatically creates a panel entitled "BackBlaze Archive" containing the clip metadata which describes it's BackBlaze archive state, including:

Field

Description

squarebox.catdv.archive.BackBlaze.serviceType

Type of service responsible for file transfer

squarebox.catdv.archive.BackBlaze.serviceName

Name of service responsible for file transfer

squarebox.catdv.archive.BackBlaze.status

Archive status

squarebox.catdv.archive.BackBlaze.location

Location of file in storage

squarebox.catdv.archive.BackBlaze.date

Date (timestamp) of latest change in archive status

squarebox.catdv.archive.BackBlaze.dateLastArchived

Date last archived

squarebox.catdv.archive.BackBlaze.numArchives

The number of times the clip has been successfully archived

squarebox.catdv.archive.BackBlaze.archiveKey

Identifier of file in storage

squarebox.catdv.archive.BackBlaze.batchID

Identifies the batch of files with which the clip was archived

squarebox.catdv.archive.BackBlaze.jobID

Identifier of current / latest archive job

squarebox.catdv.archive.BackBlaze.parentJobID

n/a for BackBlaze (related to bulk archives)

squarebox.catdv.archive.BackBlaze.userId

ID of user initiating current / latest transfer

squarebox.catdv.archive.BackBlaze.history

Record of all archive activity

squarebox.catdv.archive.BackBlaze.bucketName

Name of bucket to transfer file to / from

Known Issues

Restore jobs don't give progress updates. This is a limitation of the B2 Java SDK.

A network outage causes the archive service to block for 2 minutes. This is a limitation of the B2 Java SDK;

License Code

IMPORTANT: You may install and use this software only in accordance with the terms of the CatDV Server 7.3 license agreement.

Square Box Systems Ltd.

July 2019

Release notes

1.5.0p2 (2019-07-29)

Upgrade - run catdvarchiveplugins1.5.sql against the CatDV DB to fix the field type for archive date fields - If the 'catdv.backblaze_archive.allow_override' server property is explicitly set to include 'location:Location:archive', modify it to archiveLocation:Location:archive' Changes - Restore by default to current media file path from clip, rather than from last archived parameters, so that target restore location can be updated when e.g. media paths are updated with a different volume name - Fix NPE on cancel job, when associated clip is not found - Changes to purge (apply to move to archive as well as direct purge): clear date last restored, log 'File purged' to clip archive params history, add option of "Yes to all" confirmation for purging multiple altered files. - Fix blocking mechanism for ensuring that multiple processes cannot update a job's status simultaneously - Prevent spurious 'Network outage'/NPE error when no restore directory can be extracted from the restore location - Display 'Retry' for status of unsuccessful job results if the job is being / will be retried - Fix NPE listing jobs if user doesn't have permission to access selected clip - Improve performance of Job Queue command for large lists of jobs - Fix to cause job creation to fail if no source media are updated with the archive details for the job - Fix for issue of jobs stuck in running / stalled state - Trim trailing path separators from archive / restore location overrides - Fix to ensure plugin calls server rest API as /catdv/api/... instead of /api/... to prevent failed calls due to redirection of POST requests in some environments - Add plugin command that can be triggered via the rest API to generate clip data for a file archived outside CatDV - Add config param 'catdv.backblaze_archive.max_retry_delay' to limit the delay period between retries of waiting jobs. - Add config param 'catdv.backblaze_archive.purge_directories' to turn off purging of directories when moving / purging files. - Update README to cover installing and updating the archive worker plugin, now included as part of the plugin installation. - Add option to restore files to a specified location / directory (can be enabled as an override for restore)

1.4.9 (2018-09-14)

- Fix for error message on cancel job contains NullPointerException

- Fix for issue which causes a new FieldGroup for an archive plugin to be created each time the server is restarted

- Improve the error reporting when attempting to archive offline files

1.4.8betap4 (2018-07-25)

- Fixed bug causing purge commands to fail from the web UI

1.4.8betap3 (2018-07-11)

- Fix build / jar files

1.4.8betap2

- Fix location override for archives triggered from worker

- Record archive failure in clip archive status if clip has never been archived

1.4.8beta

- Integrate with ServicePlugin framework

- Manage service command: preserve new connection details even if they are not valid / updated, to ease entry

- Manage service command: obfuscate the application key value

1.4.6p1 (2018-05-29)

- Minimize class C BackBlaze transactions

1.4.6 (2018-05-09)

- Additional location mapping option to determine location on archive from Media Store paths, includes mediaStoreRelative and mediaStoreAbsolute.

- Fix to ensure that location on archive does not contain '\' separators, regardless of location mapping in use

- Fix for NPE when clip has media file but no metadata (introduced by fix in 1.4.5)

1.4.5 (2018-05-03)

- Fix for rest.NotFoundException scheduling jobs: change method for checking job exists, to avoid server incompatibility

1.4.5beta (2018-04-5)

- Changed verify BackBlaze connection to use downloadByName with invalid bucket/file name, to reduce api calls/cost.

- Update rest API wrapper to use version 7 of the API rather than version 5

- Move packages in order to split out re-usable plugin SDK classes

- Fix to prevent NPE - on archiving / restoring, skip clips that have no source media (i.e. sequences)

- Fix for location override in WorkerBackupCommand

1.4.4beta

- Initial beta version (2018-03-29)

Copyright © Square Box Systems Ltd. 2002-2019. All rights reserved.