Squarebox

Before you start

In order to use this BackBlaze archiving plugin you need:

CatDV Server 7.3 or later

CatDV Pegasus 12.0b15 or later

CatDV Plugin licence 'ARCBB' (Rest API licence with multiple sessions)

To trigger BackBlaze file transfers from the Worker you need:

BackBlaze Worker Plugin 1.4.8beta

In addition, to run the archive service standalone (outside the server):

CatDV Service Control Panel 1.2 or later

Installation

NEW

Copy all the files included in this zip to the plugin directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

*NB - if you already have catdvplugin.jar you don't need to overwrite it.

Set optional server properties for the plugin, if desired:

Open 'Server Config' from the control panel

Enter required properties into the CatDV Server tab, 'Other' field. See 'Plugin server properties' below and 'Running archive service standalone', if applicable.

Enter optional properties if desired. See 'Plugin server properties' below.

Restart the CatDV server from the control panel

Open the CatDV client (or log the client back in to the server)

Configure the parameters to access BackBlaze:

In the client, run Tools->Manage BackBlaze Archive Service, enter account id, application key and bucket name and click 'save' or 'save and start'. If the values entered are valid and a connection to BackBlaze can be established, then when the service mode is 'InPlugin' the archive service will be started automatically.

The service status can be monitored on the management and job queue screens.

*NB - If you don't already have an BackBlaze account or you don't know the access keys see 'BackBlaze setup' for details of how to set them up.

If running the archive service standalone (configured via a server property in step two), then configure and start the archive service via the service control panel. See 'Running archive service standalone'.

Verify the service setup: In the client, run Tools->Manage BackBlaze Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if BackBlaze is not currently accessible.

UPDATE

Make a note of the current plugin version - from BackBlaze_README.txt / BackBlaze_ReleaseNotes.txt in the plugin directory or from the server log at start up when the plugin is loaded

If running the archive service standalone, use the service control panel to stop the service.

Stop the CatDV server from the control panel

Copy all the files included in this zip to the plugin directory:

e.g. on unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

*NB - if you already have catdvplugin.jar you don't need to overwrite it.

Carry out any 'Upgrade' instructions for each plugin version listed in the BackBlaze_ReleaseNotes.txt (from this zip) above the last installed version, working back from the oldest applicable version

Read through the 'Changes' for each plugin version listed in the BackBlaze_ReleaseNotes.txt (from this zip) above the last installed version, to check whether any new config params have been added which could usefully be customised for this installation

Start the CatDV server from the control panel

If running the archive service standalone (configured via a server property in step two), then update and start the archive service via the service control panel. See 'Running archive service standalone'.

Open the CatDV client (or log the client back in to the server)

Verify the service setup:

In the client, run Tools->Manage BackBlaze Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if BackBlaze is not currently accessible.

Running archive service standalone

By default, the service that handles archive jobs runs inside the plugin. To run it as a standalone process,

ensure that the service mode property is set to Standalone in the server when installing the plugin (see step 2

of Installation):

catdv.backblaze_archive.service_mode = Standalone

The following is required to run the plugin's archive service on a separate machine from the CatDV server:

CatDV Service Control Panel 1.1.1 or later

** NB The archive service connection must be configured in the client / web plugin UI before it can be started.

To configure and start the standalone archive service using the service control panel:

Install the CatDV Service Control Panel

Open the CatDV Service Control Panel

Click 'Service Config' to open the service configuration

Enter license details on the Licensing tab

Enter required and optional service properties on the Settings tab. See 'Plugin server properties' below.

On the Installation tab, note the install location ('Server Install Path')

On the file system, navigate to the install location and copy the plugin files into the plugins sub-directory.

Click 'Start' to start the archive service.

To update the standalone archive service:

* The service should have been stopped via the Service Control Panel 'Stop' prior to stopping the CatDV Server.

Open the CatDV Service Control Panel

Add / edit service properties on the Settings tab if applicable.

Go to the install location (see Installation tab) and replace the plugin files in the plugins sub-directory with those for the latest version of the plugin.

Click 'Start' to start the archive service.

Plugin server properties

When running the archive service "InPlugin", the following server properties must be set:

catdv.backblaze_archive.licence_code = <generated licence code for plugin>

When running the archive service standalone, the following server properties must ALSO be set:

catdv.backblaze_archive.service_mode = Standalone

When running the archive service standalone, the property file for the archive service must include:

catdv.backblaze_archive.licence_code = <generated licence code for plugin>

catdv.backblaze_archive.service_mode = Standalone

catdv.rest_api.host = <CatDV server hostname>

catdv.rest_api.port = <CatDV API port>

catdv.rest_api.user = <CatDV API user>

catdv.rest_api.password = <CatDV API user password> (** exclude if blank, for dev environment)

Typical rest API values for a development machine would be (password property omitted for a blank password):

localhost, 8080, Administrator

In addition, the following optional properties can also be set:

catdv.service.api_timeout = <time_period_in_milliseconds>

Determines the timeout period after which the current connection to the CatDV API will be assumed to have timed out and therefore be replaced with a new connection (this is to pre-empt exceptions due to timeouts).

catdv.backblaze_archive.service_mode = InPlugin / Standalne

Determines whether the job processing service for the plugin runs insidethe plugin or as a standalone process outside the CatDV server. The default value is "InPlugin".

catdv.backblaze_archive.restrict_command_access = NOT SET / config / all

If set, restricts the specified plugin commands to sys admin users. Can be used to hide 'config' commands only (i.e. Manage Service), or 'all' commands.

NB - this only works with CatDV client versions from 12 onwards.

catdv.backblaze_archive.allow_override

Enables the facility to override one or more parameters at the point of archive or restore. The comma-separated string value may include:

location:Location:archive

backblaze.bucket_name:Bucket name:archive

The middle item is the UI field label and may be modified.

The default value is "backblaze.bucket_name:Bucket name:archive"

Set to 'none' to allow no overrides.

catdv.backblaze_archive.location_mapping

Method for generating the archive file location on B2. Valid values are:

batchMirror : The file(s) selected for archive are batched together in a date and time stamped directory (format /yyyy-mm-dd/hh:mm:ss.mmm) and mirror their src file path(s) within the batch directory. This has the effect of versioning uploads to B2 as a new copy is written for each transfer.

mirror: The archive location always mirrors the src file path which meansa file is replaced each time it is archived. **NB The downside of this approach is that it does not cater for files with the same path on different file systems.

mediaStore:

mediaStoreRelative: Generates relative file paths from the relevant media store(s). If no matching media store path is found, the archive location is generated using the 'mirror' approach.

mediaStoreAbsolute: As 'mediaStoreRelative' but also prepends the path with thename of the media store.

The default is 'batchMirror'.

catdv.backblaze_archive.days_to_display = <number_of_days>

The number of days into the past for which jobs are listed in the job queue. Any job updated in this time period is listed. The default value is 10.

catdv.backblaze_archive.max_jobs_to_display = <number_of_jobs>

The maximum number of jobs which will be listed in the job queue. This overrides days_to_display. The default value is 1000.

catdv.backblaze_archive.loop_delay = <time_period_in_milliseconds>

Determines the frequency with which the archive service checks the Job queue and attempts to run outstanding Jobs. The default value is 30000, equivalent to 30 seconds.

catdv.backblaze_archive.max_jobs_running_delay = <time_period_in_milliseconds>

Determines the time period to wait before attempting to run another job when the maximum number of transfers are already in progress. The default value is 0. This would typically be increased (either when a typical transfer is likely to be slow or when the concurrent transfer limit is greater than 1) in order to reduce hits to the database to query jobs whilst max transfers are in progress.

Default value is 0.

catdv.backblaze_archive.retry_in_progress_job_delay = <time_period_in_milliseconds>

Determines the time period after which a Job which is in progress will be restarted if it has not been updated during that period. Defaults to a value equivalent to 1 hour.

catdv.backblaze_archive.num_delayed_job_retries

The number of times an interrupted job is retried at an interval which increases exponentially. The default value is 10, which is equivalent to roughly 34 hours elapsed running time of the archive service.

catdv.backblaze_archive.concurrent_transfer_limit

Determines the number of transfers that the BackBlaze transfer manager will attempt to run concurrently. The default value is 4.

BackBlaze Setup / Configuration values

In order to use the plugin, a BackBlaze account is required. If you need to set up a BackBlaze account for testing purposes, got to https://www.backblaze.com/b2/sign-up.html to create an account with 10GB free.

NB - for details of pricing see: https://help.backblaze.com/hc/en-us/articles/217667478-Understanding-B2-Pricing-Structure

Once you are logged into a BackBlaze account, the 'My Account' link will take you to the BackBlaze console: https://secure.backblaze.com/b2_buckets.htm

In order for the plugin to copy to / move to / restore from your BackBlaze account, you will need to enable B2, set up an application key and create at least one bucket. The account has a master application key which has complete access.

**NB - B2 has an API in development which will enable application keys to be created with restricted capabilities. Eventually, it will be possible to create and manage applicationkeys from the BackBlaze console but initially this will only be possible via the API.

To enable B2 cloud storage:

To enable B2 on an existing Backblaze account, first login to the Backblaze account with the registered email address and password. Once logged in, select the 'My Settings' link from the left side navigation. At the bottom of the 'My Settings' page B2 can be enabled under the 'Enabled Products' section.

To get the master application key:

From the 'My Account' page (https://secure.backblaze.com/b2_buckets.htm), click on the link that reads 'Show Account ID & Application Key".

To create a bucket:

Go to the 'My Account' page (https://secure.backblaze.com/b2_buckets.htm)

Click the link 'Create a bucket'

Enter the bucket name, choose private or public and click 'Create a Bucket'. Files in a private bucket can only by accessed via the console or using the application key. Files in a public bucket can be downloaded by anyone, but the downloads will be charged against your account.

**NB - bucket names must be globally unique across all B2 cloud storage accounts

The values required to configure the service in the BackBlaze plugin are:

Account ID - account id from a BackBlaze account

Application key - application key from a BackBlaze account

Default bucket - name of an accessible bucket on BackBlaze which can be used to verify the connection, used as the default bucket the first time a user attempts to copy / move files to BackBlaze

Using the plugin

The plugin comprises various commands that are available from the Tools menu in the client.

File transfers are initiated by the schedule copy / move / restore commands but are carried out by a separate process. This means that the client does not automatically reflect updates to a clip on completion of a transfer. Use 'Server->Refresh Window' in the client to pick up any changes during / after transfer.

The plugin includes the following commands:

* Manage BackBlaze Archive Service - View full status details of the archive service and enter / update the configuration values required to access the B2 storage. These include account id, application key and default bucket name. If a connection to BackBlaze cannot be made an error is displayed and the values are not saved.

* View BackBlaze Archive Service Job Queue - list of jobs that can be filtered by status (Pending, Complete, Failed, All). It provides the capability to view job contents, view job results and cancel interrupted or queued jobs.

* Schedule copy to BackBlaze - adds a copy job for each clip selected (if multiple clips reference the same source media only one job is created). Copy jobs can be scheduled when the archive service is not running or is offline but will only be run when the archive service is online (running and BackBlaze is accessible). When the copy job is run, the source media associated with the clip is copied from the file system to BackBlaze and the original file is preserved.

* Schedule move to BackBlaze - as copy but on successful transfer of the file to storage the job attempts to delete the original file. If for some reason the deletion fails this does not affect the job status (or archive status of the clip) but "Purge failed" is recorded in the job data.

* Schedule restore from BackBlaze - adds a restore job for each clip selected (if multiple clips reference the same source media only one job is created). Restores can be scheduled when the archive service is not running or is offline but will only be run when the archive service is online (running and BackBlaze is accessible). When the restore job is run, the source media associated with the clip is copied from BackBlaze to the file system.

* Purge files archived to BackBlaze - deletes the source media associated with the selected clips if they have been successfully archived.

Archive details pane

The plugin automatically creates a panel entitled "BackBlaze Archive" containing the clip metadata which describes it's BackBlaze archive state, including:

squarebox.catdv.archive.BackBlaze.serviceType - Type of service responsible for file transfer

squarebox.catdv.archive.BackBlaze.serviceName - Name of service responsible for file transfer

squarebox.catdv.archive.BackBlaze.status - Archive status

squarebox.catdv.archive.BackBlaze.location - Location of file in storage

squarebox.catdv.archive.BackBlaze.date - Date (timestamp) of latest change in archive status

squarebox.catdv.archive.BackBlaze.dateLastArchived - Date last archived

squarebox.catdv.archive.BackBlaze.numArchives - The number of times the clip has been successfully archived

squarebox.catdv.archive.BackBlaze.archiveKey - Identifier of file in storage

squarebox.catdv.archive.BackBlaze.batchID - Identifies the batch of files with which the clip was archived

squarebox.catdv.archive.BackBlaze.jobID - Identifier of current / latest archive job

squarebox.catdv.archive.BackBlaze.parentJobID - n/a for BackBlaze (related to bulk archives)

squarebox.catdv.archive.BackBlaze.userId - ID of user initiating current / latest transfer

squarebox.catdv.archive.BackBlaze.history - Record of all archive activity

squarebox.catdv.archive.BackBlaze.bucketName - Name of bucket to transfer file to / from

Known Issues

Restore jobs don't give progress updates. This is a limitation of the B2 Java SDK.

A network outage causes the archive service to block for 2 minutes. This is a limitation of the B2 Java SDK;

Attempting to cancel an 'In progress' job incorrectly gives a "NullPointerException" error due to a client bug. (In Progress jobs cannot be cancelled).

License Code

IMPORTANT: You may install and use this software only in accordance with the terms of the CatDV Server 7.2.5 license agreement.

Square Box Systems Ltd.

March 2018

Release notes

1.4.9 (2018-09-14)

- Fix for error message on cancel job contains NullPointerException

- Fix for issue which causes a new FieldGroup for an archive plugin to be created each time the server is restarted

- Improve the error reporting when attempting to archive offline files

1.4.8betap4 (2018-07-25)

- Fixed bug causing purge commands to fail from the web UI

1.4.8betap3 (2018-07-11)

- Fix build / jar files

1.4.8betap2

- Fix location override for archives triggered from worker

- Record archive failure in clip archive status if clip has never been archived

1.4.8beta

- Integrate with ServicePlugin framework

- Manage service command: preserve new connection details even if they are not valid / updated, to ease entry

- Manage service command: obfuscate the application key value

1.4.6p1 (2018-05-29)

- Minimize class C BackBlaze transactions

1.4.6 (2018-05-09)

- Additional location mapping option to determine location on archive from Media Store paths, includes mediaStoreRelative and mediaStoreAbsolute.

- Fix to ensure that location on archive does not contain '\' separators, regardless of location mapping in use

- Fix for NPE when clip has media file but no metadata (introduced by fix in 1.4.5)

1.4.5 (2018-05-03)

- Fix for rest.NotFoundException scheduling jobs: change method for checking job exists, to avoid server incompatibility

1.4.5beta (2018-04-5)

- Changed verify BackBlaze connection to use downloadByName with invalid bucket/file name, to reduce api calls/cost.

- Update rest API wrapper to use version 7 of the API rather than version 5

- Move packages in order to split out re-usable plugin SDK classes

- Fix to prevent NPE - on archiving / restoring, skip clips that have no source media (i.e. sequences)

- Fix for location override in WorkerBackupCommand

1.4.4beta

- Initial beta version (2018-03-29)

Copyright © Square Box Systems Ltd. 2002-2019. All rights reserved.