Skip to content

Backup and Restore#

Back up and restore applications on KubeStack+

Prerequisite#

You need Velero CLI setup, follow the velero-cli doc

Service viewpoint#

Backup#

KubeStack+ divides backups into two:

  1. Customer backups
  2. Platform Tools backups

These Backups are stored on the Cloud Provider under your account.

Customer Backups#

There are 3 default backup schedules for customers that are deployed with offered Velero tool:

Schedule Backup Frequency Backup Retention Backup Scope
Daily Customer Workload Backup Every 24 hrs Last 7 Backup(s) Objects + PV Snapshots
Hourly Full Cluster Object Backup Every hour Last 24 Backup(s) Objects
Weekly Full Backup Once Every week Last 4 Backup(s) Objects + PV Snapshots

Platform Tools backups#

Stakater only takes backups of the managed applications:

Tool Backup Frequency Backup Retention
Harbor Every 24 hrs Last 3 Backup(s)
Prometheus (Cluster) Every 24 hrs Last 3 Backup(s)
Prometheus (Applications) Every 24 hrs Last 3 Backup(s)
Elasticsearch Every 24 hrs Last 3 Backup(s)
OpenBao Every 24 hrs Last 3 Backup(s)
CodeReadyWorkspaces Every 24 hrs Last 3 Backup(s)
RHSSO (Keycloak) Every 24 hrs Last 3 Backup(s)
ArgoCD Every 24 hrs Last 3 Backup(s)

If you want to change backup frequency/retention times for your specific needs, contact support.

Restore#

Resources can be restored on demand. Please contact support and specify the following

  • Time to restore back to.
  • Namespaces to include/exclude from backup
  • Resources to include/exclude from backup
  • labelSelector to filter objects to restore
  • Whether to include cluster resources or not
  • Whether to restore PVs or not (Not applicable on Object Only Backups)

Technical viewpoint#

KubeStack+ uses managed Velero operator to provision the Velero server. This backup and restore process can be used for both disaster recovery and cluster migration.

Backup#

Using Schedules and Read-Only Backup Storage Locations & Volume Snapshot Locations.

First things first, Velero needs a BackupStorageLocation where backups can be stored These are including the backup destination storage information

  • BackupStorageLocation CR ex:
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: openshift-velero
  spec:
    config:
      region: eu-central-1
    objectStorage:
      bucket: *********
    provider: aws
  • VolumeSnapshotlocation CR (Not needed when using CSI Plugin for backups) ex:
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  name: default
  namespace: openshift-velero
  spec:
    config:
      region: eu-central-1
    provider: aws

Then schedule the backup by using the backup locations and specifying the various filters for backup target.

  • Create Schedule CR ex:
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: backup-schedule
  namespace: openshift-velero
  spec:
    schedule: 0 */6 * * *
    template:
      labelSelector:
        matchExpressions:
        * key: app
          operator: In
          values:
        * nexus
      includeClusterResources: true
      includedNamespaces:
      * '*'
      includedResources:
      * '*'
      snapshotVolumes: true
      storageLocation: default
      ttl: 24h0m0s
      volumeSnapshotLocations:
      * default

Or you can specify a single Backup using a backup CR.

  • Create a Backup CR ex:
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: data-backup
  namespace: openshift-velero
spec:
  includedNamespaces:
  * openshift-stakater-3scale
  includedResources:
  * '*'
  storageLocation: default
  ttl: 72h0m0s
  volumeSnapshotLocations:
  * default

Different Schedule Definitions and their meanings#

Changing some parameters in Schedule CR, changes the backup behavior

  • snapshotVolumes: (boolean)

    • true (makes Velero to take native volume snapshots)
    • false (no native Velero snapshots taken)
    • Not Specify (Defaults to auto behavior define within Velero) Note: You don't specify this parameter with Velero CSI plugin

  • includeClusterResources: (boolean)

    • true (includes all cluster level resource, snapshotVolumes needs to be true to take PV snapshots)
    • false (exclude cluster level resources, including PVs. So there will be no snapshots)

  • Excluding both snapshotVolumes and includeClusterResources will have default behavior of taking snapshots for only included namespaces PVs.

Backup target Filters#

You can use resource filtering options to backup specific resources. Typical ones are following;

  • labelSelector: Specify the labels for the backup target resources MatchExpression has several operators such as In, NotIn, Exists and DoesNotExist.
  • includeClusterResources: (boolean) Set true to include cluster level resources like PV etc.
  • includedNamespaces: Specify the namespaces in which backup target resources are included
  • includedResources: Specify the resource types for the backup target resources
  • excludeNamespaces: Specify the namespaces to exclude from backup
  • excludeResources: Specify the resource types to exclude from backup NOTE: You have to select the resource filters properly. For example, if the target application has cluster-scope resources, then you cannot use --include-namespaces only.

backup destination#

  • snapshotVolumes: (boolean) Set true for doing volume snapshots
  • storageLocation: BackupStorageLocation CR name
  • volumeSnapshotLocations: VolumeSnapshotlocation CR name

retention policy#

  • ttl: The backup retention period

Restore#

Restore on the same cluster or the other cluster in the case of cluster broken. Update your backup storage location to read-only mode (this prevents backup objects from being created or deleted in the backup storage location during the restore process):

kubectl patch backupstoragelocation <STORAGE LOCATION NAME> \
    --namespace openshift-velero \
    --type merge \
    --patch '{"spec":{"accessMode":"ReadOnly"}}'

Create a restore with your most recent Velero Backup:

velero restore create --from-backup <BACKUP NAME>

When ready, revert your backup storage location to read-write mode:

kubectl patch backupstoragelocation <STORAGE LOCATION NAME> \
   --namespace openshift-velero \
   --type merge \
   --patch '{"spec":{"accessMode":"ReadWrite"}}'

restore target filter#

If you want to restore specific resources, you can use resource filtering. For example, you can restore resources in web namespace:

velero restore create --from-backup <BACKUP NAME> --include-namespaces web --namespace openshift-velero

Note

You have to select the resource filters properly. For example, if the target application has cluster-scope resources, then you cannot use --include-namespaces only.