Backup and Restore the United Manufacturing Hub

This page describes how to back up the following:

• All Node-RED flows
• All Grafana dashboards
• The Helm values used for installing the united-manufacturing-hub release
• All the contents of the United Manufacturing Hub database (factoryinsight and umh_v2)
• The Management Console Companion’s settings

It does not back up:

• Additional databases other than the United Manufacturing Hub default database
• TimescaleDB continuous aggregates: Follow the official documentation to learn how.
• TimescaleDB policies: Follow the official documentation to learn how.
• Everything else not included in the previous list

This procedure only works on Windows.

Before you begin

Download the backup scripts and extract the content in a folder of your choice.

For this task, you need to have PostgreSQL installed on your machine.

You also need to have enough space on your machine to store the backup. To check the size of the database, ssh into the system and follow the steps below:

sudo $(which kubectl) exec -it $(sudo $(which kubectl) get pods --kubeconfig /etc/rancher/k3s/k3s.yaml -n united-manufacturing-hub -l app.kubernetes.io/component=timescaledb -o jsonpath="{.items[0].metadata.name}") --kubeconfig /etc/rancher/k3s/k3s.yaml -n united-manufacturing-hub -- psql -U postgres

This command will open a psql shell connected to the default postgres database.

Run the following command to get the size of the database:

SELECT pg_size_pretty(pg_database_size('umh_v2')) AS "umh_v2", pg_size_pretty(pg_database_size('factoryinsight')) AS "factoryinsight";

Backup

Generate Grafana API Key

Create a Grafana API Token for an admin user by following these steps:

1. Open the Grafana UI in your browser and log in with an admin user.
2. Click on the Configuration icon in the left sidebar and select API Keys.
3. Give the API key a name and change its role to Admin.
4. Optionally set an expiration date.
5. Click Add.
6. Copy the generated API key and save it for later.

Stop workloads

To prevent data inconsistencies, you need to temporarily stop the MQTT and Kafka Brokers.

Access the instance’s shell and execute the following commands:

sudo $(which kubectl) scale statefulset united-manufacturing-hub-kafka --replicas=0 -n united-manufacturing-hub --kubeconfig /etc/rancher/k3s/k3s.yaml
sudo $(which kubectl) scale statefulset united-manufacturing-hub-hivemqce --replicas=0 -n united-manufacturing-hub --kubeconfig /etc/rancher/k3s/k3s.yaml

Copy kubeconfig file

To run the backup script, you’ll first need to obtain a copy of the Kubernetes configuration file from your instance. This is essential for providing the script with access to the instance.

1. In the shell of your instance, execute the following command to display the Kubernetes configuration:

   sudo cat /etc/rancher/k3s/k3s.yaml
   

   Make sure to copy the entire output to your clipboard.

   This tutorial is based on the assumption that your kubeconfig file is located at /etc/rancher/k3s/k3s.yaml. Depending on your setup, the actual file location might be different.

2. Open a text editor, like Notepad, on your local machine and paste the copied content.

3. In the pasted content, find the server field. It usually defaults to https://127.0.0.1:6443. Replace this with your instance’s IP address

   server: https://<INSTANCE_IP>:6443
   

4. Save the file as k3s.yaml inside the backup folder you downloaded earlier.

Backup using the script

The backup script is located inside the folder you downloaded earlier.

1. Open a terminal and navigate inside the folder.

   cd <FOLDER_PATH>
   

2. Run the script:

   .\backup.ps1 -IP <IP_OF_THE_SERVER> -GrafanaToken <GRAFANA_API_KEY> -KubeconfigPath .\k3s.yaml
   

   You can find a list of all available parameters down below.

   If OutputPath is not set, the backup will be stored in the current folder.

This script might take a while to finish, depending on the size of your database and your connection speed.

If the connection is interrupted, there is currently no option to resume the process, therefore you will need to start again.

Here is a list of all available parameters:

Restore

Each component of the United Manufacturing Hub can be restored separately, in order to allow for more flexibility and to reduce the damage in case of a failure.

Copy kubeconfig file

To run the backup script, you’ll first need to obtain a copy of the Kubernetes configuration file from your instance. This is essential for providing the script with access to the instance.

1. In the shell of your instance, execute the following command to display the Kubernetes configuration:

   sudo cat /etc/rancher/k3s/k3s.yaml
   

   Make sure to copy the entire output to your clipboard.

   This tutorial is based on the assumption that your kubeconfig file is located at /etc/rancher/k3s/k3s.yaml. Depending on your setup, the actual file location might be different.

2. Open a text editor, like Notepad, on your local machine and paste the copied content.

3. In the pasted content, find the server field. It usually defaults to https://127.0.0.1:6443. Replace this with your instance’s IP address

   server: https://<INSTANCE_IP>:6443
   

4. Save the file as k3s.yaml inside the backup folder you downloaded earlier.

Cluster configuration

To restore the Kubernetes cluster, execute the .\restore-helm.ps1 script with the following parameters:

.\restore-helm.ps1 -KubeconfigPath .\k3s.yaml -BackupPath <PATH_TO_BACKUP_FOLDER>

Verify that the cluster is up and running by opening UMHLens / OpenLens and checking if the workloads are running.

Grafana dashboards

To restore the Grafana dashboards, you first need to create a Grafana API Key for an admin user in the new cluster by following these steps:

1. Open the Grafana UI in your browser and log in with an admin user.
2. Click on the Configuration icon in the left sidebar and select API Keys.
3. Give the API key a name and change its role to Admin.
4. Optionally set an expiration date.
5. Click Add.
6. Copy the generated API key and save it for later.

Then, on your local machine, execute the .\restore-grafana.ps1 script with the following parameters:

.\restore-grafana.ps1 -FullUrl http://<IP_OF_THE_SERVER>:8080 -Token <GRAFANA_API_KEY> -BackupPath <PATH_TO_BACKUP_FOLDER>

Restore Node-RED flows

To restore the Node-RED flows, execute the .\restore-nodered.ps1 script with the following parameters:

.\restore-nodered.ps1 -KubeconfigPath .\k3s.yaml -BackupPath <PATH_TO_BACKUP_FOLDER>

Restore the database

1. Check the database password by running the following command in your instance’s shell:

   sudo $(which kubectl) get secret united-manufacturing-hub-credentials --kubeconfig /etc/rancher/k3s/k3s.yaml -n united-manufacturing-hub -o jsonpath="{.data.PATRONI_SUPERUSER_PASSWORD}" | base64 --decode; echo
   

2. Execute the .\restore-timescale.ps1 and .\restore-timescale-v2.ps1 script with the following parameters to restore factoryinsight and umh_v2 databases:

   .\restore-timescale.ps1 -Ip <IP_OF_THE_SERVER> -BackupPath <PATH_TO_BACKUP_FOLDER> -PatroniSuperUserPassword <DATABASE_PASSWORD>
   .\restore-timescale-v2.ps1 -Ip <IP_OF_THE_SERVER> -BackupPath <PATH_TO_BACKUP_FOLDER> -PatroniSuperUserPassword <DATABASE_PASSWORD>
   

Restore the Management Console Companion

Execute the .\restore-companion.ps1 script with the following parameters to restore the companion:

.\restore-companion.ps1 -KubeconfigPath .\k3s.yaml -BackupPath <FULL_PATH_TO_BACKUP_FOLDER>

Troubleshooting

Unable to connect to the server: x509: certificate signed …

This issue may occur when the device’s IP address changes from DHCP to static after installation. A quick solution is skipping TLS validation. If you want to enable insecure-skip-tls-verify option, run the following command on the instance’s shell before copying kubeconfig on the server:

sudo $(which kubectl) config set-cluster default --insecure-skip-tls-verify=true --kubeconfig /etc/rancher/k3s/k3s.yaml

What’s next

• Take a look at the UMH-Backup repository
• Learn how to manually backup and restore the database
• Read how to import and export Node-RED flows via the UI