Virtalis Reach Automated Backup Guide

Document style

In this document variables primed for substitution before execution are highlighted as such

1 <MY_VARIABLE>

For example 

1docker login <MY_ID> <MY_PASSWORD> --verbosity debug

becomes    

1docker login 123 password_123 --verbosity debug

In this document commands to execute in a shell are shown as code and each block of code is designed to be a single command that can be copy and pasted

1These are commands to be entered in a shell in your clusters administration console 2This is another block of code \ 3that uses "\" to escape newlines \ 4and can be copy and pasted straight into your console 

Overview

Virtalis Reach comes with an optional automated backup system allowing an Admin to restore to an earlier snapshot in the event of a disaster. We will install Velero to back up the state of your Kubernetes Cluster and uses a custom built solution which leverages Restic to back up the persistent data imported into Virtalis Reach.

Alternatively you can consider using your own backup solution, a good option is the PersistentVolumeSnapshot which creates a snapshot of a persistent volume at a point in time, the biggest caveat is that they’re only supported on a number of platforms like Azure and AWS.

If you opt in for a different solution to the one we provide, you have to be mindful of the fact that not all databases used by Virtalis Reach support live backups, this means that the databases have to be taken offline before backing up.

You should consider creating regular backups of your buckets which hold the
backed up data in case of failure, this will be done through your cloud
provider or manually if you host your own bucket.

Here is a list of databases in use by Reach that can only be backed up while offline:

  • Minio

  • Neo4j

Pre-installation

Creating a storage location

Recommended

Follow the “Create S3 Bucket” and “Set permissions for Velero” sections from and make sure that you create the following 2 buckets under your s3 bucket:

  • reach-restic

  • reach-velero

 GitHub - vmware-tanzu/velero-plugin-for-aws: Plugins to support Velero on AWS 

Export the address and port of the bucket you have created

1export S3_BUCKET_ADDRESS=<address> 2#i.e S3_BUCKET_ADDRESS=192.168.1.3, S3_BUCKET_ADDRESS=mydomain.com 3export S3_BUCKET_PORT=<port> 4export S3_BUCKET_PROTOCOL=<http or https> 5export S3_BUCKET_REGION=<region> 

Not recommended - create an s3 bucket on the same cluster, alongside Virtalis Reach

Customize the persistence.size if the total size of your data exceeds 256gb and change the storage class REACH_SC if needed

1export REACH_SC=local-path 2kubectl create ns reach-backup 3 4#check if pwgen is installed for the next step 5command -v pwgen 6 7kubectl create secret generic reach-s3-backup -n reach-backup \ 8--from-literal='access-key'=$(pwgen 30 1 -s | tr -d '\n') \ 9--from-literal='secret-key'=$(pwgen 30 1 -s | tr -d '\n') 10 11 12helm upgrade --install reach-s3-backup bitnami/minio -n reach-backup --version 3.6.1 \ 13--set persistence.storageClass=$REACH_SC \ 14--set persistence.size=256Gi \ 15--set mode=standalone \ 16--set resources.requests.memory='150Mi' \ 17--set resources.requests.cpu='250m' \ 18--set resources.limits.memory='500Mi' \ 19--set resources.limits.cpu='500m' \ 20--set disableWebUI=true \ 21--set useCredentialsFile=true \ 22--set volumePermissions.enabled=true \ 23--set defaultBuckets="reach-velero reach-restic" \ 24--set global.minio.existingSecret=reach-s3-backup 25 26 27cat <<EOF > credentials-velero 28[default] 29aws_access_key_id=$(kubectl get secret reach-s3-backup -n reach-backup -o jsonpath="{.data.access-key}" | base64 --decode) 30aws_secret_access_key=$(kubectl get secret reach-s3-backup -n reach-backup -o jsonpath="{.data.secret-key}" | base64 --decode) 31EOF 

Export the address and port of the bucket you have created

1export S3_BUCKET_ADDRESS=reach-s3-backup-minio.reach-backup.svc.cluster.local 2export S3_BUCKET_PORT=9000 3export S3_BUCKET_PROTOCOL=http 4export S3_BUCKET_REGION=local 

Set up deployment shell

For the duration of this installation, you have to navigate to the k8s folder that is downloadable by following the Reach Installation Guide

Make scripts executable

1sudo chmod +x trigger-database-restore.sh trigger-database-backup.sh install-backup-restore.sh

Export out the following variables

1export ACR_REGISTRY_NAME=virtaliscustomer

Export the address of the reach-restic bucket

1export REPO_URL=s3:$S3_BUCKET_PROTOCOL://$S3_BUCKET_ADDRESS:$S3_BUCKET_PORT/reach-restic

Substitute the variable values and export them

1export REACH_NAMESPACE=<name of kubernetes namespace Virtalis Reach is deployed in>

Optional configuration variables

1export MANAGED_TAG=<custom image tag for Virtalis Reach services>

 

Velero Installation

The consequent steps will assume you named your Velero bucket “reach-velero”

Add the VMware helm repository and update 

1helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts 2helm repo update 

Install Velero

1helm install velero vmware-tanzu/velero \ 2--namespace velero \ 3--create-namespace \ 4--set-file credentials.secretContents.cloud=./credentials-velero \ 5--set configuration.provider=aws \ 6--set configuration.backupStorageLocation.name=reach-velero \ 7--set configuration.backupStorageLocation.bucket=reach-velero \ 8--set configuration.backupStorageLocation.config.region=$S3_BUCKET_REGION \ 9--set configuration.backupStorageLocation.config.s3Url=$S3_BUCKET_PROTOCOL://$S3_BUCKET_ADDRESS:$S3_BUCKET_PORT \ 10--set configuration.backupStorageLocation.config.publicUrl=$S3_BUCKET_PROTOCOL://$S3_BUCKET_ADDRESS:$S3_BUCKET_PORT \ 11--set configuration.backupStorageLocation.config.s3ForcePathStyle=true \ 12--set initContainers[0].name=velero-plugin-for-aws \ 13--set initContainers[0].image=velero/velero-plugin-for-aws:v1.1.0 \ 14--set initContainers[0].volumeMounts[0].mountPath=/target \ 15--set initContainers[0].volumeMounts[0].name=plugins \ 16--set snapshotsEnabled=false \ 17--version 2.23.1 \ 18--set deployRestic=true 

Install the velero cli client

1wget https://github.com/vmware-tanzu/velero/releases/download/v1.5.3/velero-v1.5.3-linux-amd64.tar.gz 2tar -xzvf velero-v1.5.3-linux-amd64.tar.gz 3rm -f velero-v1.5.3-linux-amd64.tar.gz 4sudo mv $(pwd)/velero-v1.5.3-linux-amd64/velero /usr/bin/ 5sudo chmod +x /usr/bin/velero 

Manually create a single backup to verify that the connection to the aws bucket is working

1velero backup create test-backup-1 \ 2--storage-location=reach-velero --include-namespaces $REACH_NAMESPACE 

Watch the status of the backup until it’s finished, this should show up as complete if everything was set up correctly

1watch -n2 velero backup get

Create a scheduled backup

1velero create schedule cluster-backup --schedule="45 23 * * 6" \ 2--storage-location=reach-velero --include-namespaces $REACH_NAMESPACE 

This schedule will run a backup every Saturday at 23:45PM

Restic integration

The custom restic integration uses Kubernetes jobs to mount the data, encrypt it, and send it to a bucket. Kubernetes CustomResourceDefinitions are used to store the information about the restic repositories as well as any created backups.

By default the scheduled data backup runs on every Friday at 23:45PM, this can be modified by editing the cronSchedule field in all values.yaml files located in backup-restore/helmCharts/<release_name>/ with the exception of common-lib.

All the performed backups are offline backups therefore Reach will be unavailable for that period as a number of databases have to be taken down.

Installation

Create an AWS bucket with the name “reach-restic” by following the same guide from the Velero section

Replace the keys and create a secret containing the reach-restic bucket credentials

1kubectl create secret generic reach-restic-bucket-creds -n "$REACH_NAMESPACE" \ 2--from-literal='AWS_ACCESS_KEY'='<ACCESS_KEY>' \ 3--from-literal='AWS_SECRET_KEY'='<SECRET_KEY>' 

If you instead opted in to deploy an s3 bucket on the same cluster, run this instead

1kubectl create secret generic reach-restic-bucket-creds -n "$REACH_NAMESPACE" \ 2--from-literal='AWS_ACCESS_KEY'=$(kubectl get secret reach-s3-backup -n reach-backup -o jsonpath="{.data.access-key}" | base64 --decode) \ 3--from-literal='AWS_SECRET_KEY'=$(kubectl get secret reach-s3-backup -n reach-backup -o jsonpath="{.data.secret-key}" | base64 --decode) 

Export the address of the reach-restic bucket

1export REPO_URL=s3:$S3_BUCKET_PROTOCOL://$S3_BUCKET_ADDRESS:$S3_BUCKET_PORT/reach-restic

Run the installation

1./install-backup-restore.sh

Check if all the -init-repository- jobs have completed

1kubectl get pods -n $REACH_NAMESPACE | grep init-repository

Query the list of repositories

1kubectl get repository -n $REACH_NAMESPACE

The output should look something like this with the status of all repositories showing as Initialized

1NAME                    STATUS        SIZE   CREATIONDATE 2artifact-binary-store   Initialized   0B   2021-03-01T10:21:53Z 3artifact-store          Initialized   0B   2021-03-01T10:21:57Z 4job-db                  Initialized   0B   2021-03-01T10:21:58Z 5keycloak-db             Initialized   0B   2021-03-01T10:21:58Z 6vrdb-binary-store       Initialized   0B   2021-03-01T10:21:58Z 7vrdb-store              Initialized   0B   2021-03-01T10:22:00Z 

Once you are happy to move on, delete the completed job pods

1kubectl delete jobs -n $REACH_NAMESPACE -l app=backup-restore-init-repository

Trigger a manual backup

1./trigger-database-backup.sh

After a while, all the -triggered-backup- jobs should show up as Completed

1kubectl get pods -n "$REACH_NAMESPACE" | grep triggered-backup

Query the list of snapshots

1kubectl get snapshot -n "$REACH_NAMESPACE"

The output should look something like this with the status of all snapshots showing as Completed

1NAME                                     STATUS      ID         CREATIONDATE 2artifact-binary-store-minio-1614596088   Completed   62e1a3a3   2021-03-01T10:54:48Z 3artifact-store-neo4j-core-1614596081     Completed   6ae45b16   2021-03-01T10:54:41Z 4job-db-mysql-master-1614596084           Completed   944321c7   2021-03-01T10:54:44Z 5keycloak-db-mysql-master-1614596084      Completed   468e81a1   2021-03-01T10:54:44Z 6vrdb-binary-store-minio-1614596088       Completed   729ae219   2021-03-01T10:54:48Z 7vrdb-store-neo4j-core-1614596083         Completed   1c2e7bee   2021-03-01T10:54:43Z 

Once you are happy to move on, delete the completed job pods

1kubectl delete jobs -n $REACH_NAMESPACE -l app=backup-restore-triggered-backup

Restoring data

Plan your restoration by gathering a list of the snapshot IDs you will be restoring from and export them

Begin by querying the list of repositories

1kubectl get repo -n "$REACH_NAMESPACE" 2NAME                    STATUS        SIZE   CREATIONDATE 3artifact-binary-store   Initialized   12K    2021-07-02T12:03:26Z 4artifact-store          Initialized   527M   2021-07-02T12:03:29Z 5comment-db              Initialized   180M   2021-07-02T12:03:37Z 6job-db                  Initialized   181M   2021-07-02T12:03:43Z 7keycloak-db             Initialized   193M   2021-07-02T12:03:43Z 8vrdb-binary-store       Initialized   12K    2021-07-02T12:03:46Z 9vrdb-store              Initialized   527M   2021-07-02T12:02:44Z 

Run a dry run of the restore script to gather a list of the variables you have to export

1DRY_RUN=true ./trigger-database-restore.sh

Sample output

1Error: ARTIFACT_BINARY_STORE_RESTORE_ID has not been exported. Please run 'kubectl get snapshot -n develop -l repository=artifact-binary-store' to see a list of available snapshots. 2Error: ARTIFACT_STORE_RESTORE_ID has not been exported. Please run 'kubectl get snapshot -n develop -l repository=artifact-store' to see a list of available snapshots. 3Error: COMMENT_DB_RESTORE_ID has not been exported. Please run 'kubectl get snapshot -n develop -l repository=comment-db' to see a list of available snapshots. 4Error: JOB_DB_RESTORE_ID has not been exported. Please run 'kubectl get snapshot -n develop -l repository=job-db' to see a list of available snapshots. 5Error: KEYCLOAK_DB_RESTORE_ID has not been exported. Please run 'kubectl get snapshot -n develop -l repository=keycloak-db' to see a list of available snapshots. 6Error: VRDB_BINARY_STORE_RESTORE_ID has not been exported. Please run 'kubectl get snapshot -n develop -l repository=vrdb-binary-store' to see a list of available snapshots. 7Error: VRDB_STORE_RESTORE_ID has not been exported. Please run 'kubectl get snapshot -n develop -l repository=vrdb-store' to see a list of available snapshots. 

Query available snapshots or use the commands returned in the output above to query by specific repositories

1kubectl get snapshot -n "$REACH_NAMESPACE"

This should return a list of available snapshots

1NAME                                     STATUS      ID         CREATIONDATE 2artifact-binary-store-minio-1625227568   Completed   4a291256   2021-07-02T12:06:08Z 3artifact-store-neo4j-core-1625227565     Completed   41ddd05d   2021-07-02T12:06:05Z 4comment-db-mysql-master-1625227573       Completed   e72b49c5   2021-07-02T12:06:13Z 5job-db-mysql-master-1625227578           Completed   eb5329f4   2021-07-02T12:06:18Z 6keycloak-db-mysql-master-1625227574      Completed   9199e4d5   2021-07-02T12:06:14Z 7vrdb-binary-store-minio-1625227572       Completed   cf035aa4   2021-07-02T12:06:12Z 8vrdb-store-neo4j-core-1625227581         Completed   08d366f9   2021-07-02T12:06:21Z 

It’s strongly advised to restore all the backed up data using snapshots from the same day to avoid any missing/inaccessible data.

Note down the replica count for nginx before scaling it down

1kubectl get deploy -n ingress-nginx 2export NGINX_REPLICAS=<CURRENT_REPLICA_COUNT> 

Scale down the nginx ingress service to prevent people from accessing Reach during the restoration process

1kubectl scale deploy --replicas=0 ingress-nginx-ingress-controller -n ingress-nginx

Run the restore

1./trigger-database-restore.sh

Unset exported out restore id’s

1charts=( $(ls backup-restore/helmCharts/) ); for chart in "${charts[@]}"; do if [ $chart == "common-lib" ]; then continue; fi; id_var="$(echo ${chart^^} | sed 's/-/_/g')_RESTORE_ID"; unset ${id_var}; done

After a while, all the -triggered-restore- jobs should show up as Completed

1kubectl get pods -n "$REACH_NAMESPACE" | grep triggered-restore

Once you are happy to move on, delete the completed job pods

1kubectl delete jobs -n $REACH_NAMESPACE -l app=backup-restore-triggered-restore

Watch and wait for all pods that are running to be Ready

1watch -n2 kubectl get pods -n "$REACH_NAMESPACE"

Scale back nginx

1kubectl scale deploy --replicas="$NGINX_REPLICAS" \ 2ingress-nginx-ingress-controller -n ingress-nginx 

Verify that everything is working by testing standard functionality like
importing a file or viewing a visualization

T
Tom is the author of this solution article.

Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.