Ubuntu Opensim Server Backup.

• 

• 

• 

• 

• 

• 

This article looks at Ubuntu Opensim Server Backup. It covers how to do to so adequately while being affordable.

Previous and Related Articles

The guide is written to accompany Grid backups and oar files on the Fire And Ice Grid blog. Additionally, it follows on from Setup Ubuntu 20.04 for Opensimulator, Opensim with Robust multiple services on Ubuntu, Automated Opensim Startup and Shutdown and Securing An Opensim Ubuntu Server. Most of the article is suitable to use on any small scale server setup. However, automatic OAR generation is specific to opensimulator.

Ubuntu Opensim Server Backup Principles

The 3-2-1 backup rule is a standard which many adhere to. Firstly it states to keep three coppies of all data. Secondly that the data should be stored in two locations. Finally that one of those locatoins needs to be off site. Importantly, one of these backups must be available quickly.

The 3-2-1 rule is a little dated, however, the principle still applies. Firstly, keep several copies of all data. Then secondly store each backup in a different datacentre. Finally, ensure that the data centres used are not operated by the same company.

The primary server should only have access to the first backup store. Then the VPS server needs SSH access to the primary server to push files quickly if required. Both the VPS and the office server duplicate the first backup independently. Similarly, it is essential to ensure that neither the VPS nor the primary server has access to the office location.

• Keep a minimum of 3 Copies of all data
• One copy of the data must be available quickly
• Disconnected servers for backup replication
• Use Multiple data centres
• More than one company should host the data centres
• Encrypt data before transmission
• Use incremental backups and subsequently reduce time and bandwidth

Opensim Specific Backup Information

No single solution is adequate to restore an opensimulator grid from a catastrophic event. Regions usually restore correctly from OAR files, but exceptions do exist. Inventory archives can rebuild user inventories, but grid owners can not use them. To restore users friends lists, groups and other details; the Robust database needs saving and restoring. In most cases, the Asset folder is also required (if FSAssets is used). Some scripts can cause objects to break after OAR restoration. As a result, a copy of the simulator folders needs saving along with the simulators database. This also means keeping each simulator in its own folder, and with its own database is a good idea

Opensim Inventory Archive

Inventory archives are less useful than they could be. To make an inventory archive, the user’s password has to be available. Individuals running their own simulators will find this helpful, but grid owners are unable to use this feature. Full details are on the opensimulator Inventory Archives wiki page.

Opensim Region Backup – Automatic OAR

Full details of the specification for are on the Auto Backup page of the opensimulator wiki. To use auto-backup add a few lines to the OpenSim.ini configuration file. The path needs to set the first backup location or duplicated there afterwards. Exceptions exist with individual scripts however so these can not be the whole solution.

[AutoBackupModule]
AutoBackupModuleEnabled = true
 
; enabled
AutoBackup = true
 
; daily = 1440 minutes, 2880 = 2 days, 3 backups per week, restarts at 7 days
AutoBackupInterval = 2880
 
; Always run
AutoBackupBusyCheck = true
 
;Skip the backup if more than this many avatars are present
AutoBackupThreshold = 1
 
; we do want assets
AutoBackupSkipAssets = false
 
; save 2 weeks data (7 files per region over 14 days)
AutoBackupKeepFilesForDays = 14
 
; 
AutoBackupNaming = Time
 
; save into a bin/Autobackup folder
 
AutoBackupDir = "/**folder path for the OAR files**"

The auto-backup directory can be inside your mounted G-Suite drive.

Opensim Server Backup – Keeping The Cost Down

There are many storage providers. Their prices vary dramatically along with their specification. The example in this article actually saves data to the secondary store (G-Suite) first. The VPS server then uses Rsync to clone the G-Suite drive to the VPS. Another machine in the office clones the secondary store again.

The VPS can push data to the central server at speed if required. The VPS needs SSH keypair access to the primary server. However, the main server must not have access to the VPS. Equally, neither the main server nor the VPS can access the office backup. Working this way protects two backups from main server breaches.

Use a VPS (virtual private server) for the primary Backup.

VPS servers are often cheaper for small servers than cloud storage such as Google Cloud, Microsoft Azure and Amazon S3. A VPS server can also act independently of the primary server. Thus it provides storage and the disconnected replication of backups. Contabo offer competitively priced VPS.

Google G-Suite (Google Drive) – Secondary backup store.

Googles G-Suite drive space is very low-cost cloud storage. Each user costs around £7 per month and gets 1 TB of drive space. In contrast, a G-Suite account with more than 4 users gets unlimited space. However, there are restrictions on the total daily bandwidth and speed to consider. It is too slow for a primary backup store.

Office / Home high capacity storage drive

Keep one copy of the data in your own office. This copy of the backup data will is the last resort backup. To add an even higher level of security make this a drive which can be unplugged when not in use.

Backup Software

• google-drive-ocamlfuse – mount google drive to a local folder
• SystemD (used to start ocamlfuse at startup)
• Restic
• MySql dump/plump
• Percona Xtra Backup
• Rsync
• SSH key pairs
• Bash Scripts
• Crontab – to automate the bash scripts

Mounting a G-Suite Google Drive – Google Drive Ocamlfuse

Google Drive Ocamlfuse is a command-line client for mounting google drives on Linux. Ocamlfuse can use team drive space, multiple accounts and headerless installs. To help automate the backups, it is a good idea to save the automatic OAR backups to a folder in google drive.

Start ocamlfuse when the server boots – use systemd

Create a systemd service file using Nano or another text editor. Next, Change the name at the end of the line if you wish to name the service differently.

sudo nano /etc/systemd/system/gdrive-opensim

Copy-paste the content below. All places where a double asterisk (** **) surrounds something, replace the content with system-specific information.

[Unit]
Description= Ocamlfuse startup and keep alive
After=syslog.target network.target

[Service]
Type=simple
User=**YourUserName**
Group=**YourUserGroup**
WorkingDirectory=/home/**YourUserName**
ExecStart=/home/**YourUserName**/google-drive-ocamlfuse -label **DriveLable** /**PathToYourMountPoint**/
ExecStop=/home/**YourUserName**/fusermount -u /**PathToYourMountPoint**/
RemainAfterExit=yes
KillMode=none
Restart=always
Environment=USER=**YourUserName** HOME=/home/**YourUserName**

[Install]
WantedBy=multi-user.target

Save and exit the file

CTRL + O
ENTER
CONTROL + X

Finally enable and start the new service

sudo systemctl enable gdrive-opensim.service
sudo systemctl start gdrive-opensim.service

Ubuntu Opensim Server Backup – Restic folder backup

Restic is a command-line backup system using high-level encryption. It also supports environmental variables so everything can be automated, and almost any storage space is compatible. Data encryption happens on the local machine before transmission. Each snapshot restic takes only saves the extra data not stored by the previous jobs.

Install and setup Restic

To install restic on Ubuntu use the command below, for other operating systems please see the main restic installation page.

apt-get install restic

Decide on a file structure. Eg.

GoogleDrives/
└── Opensim
    └── GridBackups
        ├── MySqlBackup
        ├── Server1
        ├── Server2
        └── Server3

Create a Restic environment file and add the information in the example below.

nano .restic-opensim-environment

Add in the following, changing the path, user name and password to match the system being used.

export RESTIC_PASSWORD="**MyPassword**"
export RESTIC_REPOSITORY="/home/**YourUserName**/GoogleDrives/Opensim/GridBackups/Server1"

Make sure no other users can read this file

chmod go-rwx .restic-opensim-environment

Initiate the restic repository with these commands

cd ~/
source .restic-opensim-environment
restic init --repo /home/**YourUserName**/GoogleDrives/Opensim/GridBackups/Server1

Wait for confirmation.

Backup using Restic

To do a backup execute the following commands. Rember to change **YourUser** to your user name and **Opensim** to the location of the opensim directory.

cd ~/
source .restic-opensim-environment
restic backup /home/**YourUser**/**Opensim**

Over time the size of the restic backup folder will grow without stopping. Keep the data volume down by combining “forget” and “prune. To keep the last two backups use the command below. Alternatively, adjust it to suit your requirements.

restic forget --keep-last 2 --prune

Adding this command to a bash script allows automation of the backup.

Ubuntu Opensim Server Backup – MySql Databases

MySql dump is compatible with all versions of MySql and MariaDB. Users of MySQL V5 or higher should consider using MySql Pump instead. Firstly create a new file using nano or another similar text editor. Secondly, paste in the structure below. Thirdly adjust the unique details (the ones with a ** at the start and end). Finally, save and exit. MySql dump locks the database during a backup. It is worth considering Percona Xtra backup discussed below for more extensive Robust databases. While the database is locked, users will not be able to log in.

[client]
user = **DatabaseUserName**
password = **DatabaseUserPassword**

Ensure only your user can read the file

chmod go-rwx .MySql-opensim.cnf

Create a database backup using the command below.

mysqldump --defaults-extra-file=/home/**YourUserName**/.MySql-opensim.cnf --quick -u **DatabaseUserName** **DatabaseName** | bzip2 > **DatabaseName**.sql.bz2"

Backing up a large database file directly to Google drive can cause a timeout issue. To avoid this issue, back up the database to a local drive first, then move the file to Google drive.”

cd ~/
mkdir MySqlBackupTemp

Ubuntu Opensim Server Backup – MySql Databases – Percona

Percona Xtra backup is an alternative to using MySql dump/plump. It comes with the advantage that during backup the databases are not locked. Additionally, the backup process is substantially quicker than using MySql Dump. However, there is a cost when restoring. Percona makes binary backups of the MySql data folder. In order to restore a database into this folder, the MySql service must be stopped. How problematic this is, depends on various factors. It is worth considering when choosing the backup method.

Ubuntu Opensim Server Backup – Rsync

Rsync is a utility for efficiently transferring and synchronising files. Files can be local or on a different server using SSH. To install rsync, use the following command in terminal.

sudo apt-get install rsync

Rsync – Duplicating The First Backup

On each server you wish to duplicate the first back on, setup Google Drive Ocamlfuse and make a folder to synchronise the backup to.

cd ~/
mkdir GridBackupClone
rsync -azP --delete ~/GoogleDrives/Opensim/GridBackups ~/GridBackupClone

Rsync – Synchronize other folders

This example shows rsync to synchronise the contents of the default www folders with a folder called WwwBackup inside the Opensim folder. Additionally, note that “–delete” is used, consequently, if files are removed from the source, they are also removed from the destination.

rsync -azP --delete /var/www ~/Opensim/WwwBackup

Bash Script to automate backing up all database and folders

The penultimate stage is to create a bash script to launch the entire process. Starting with the MySql databases and then moving on to synchronising other folders into the one being backed up, and creation of a snapshot. Finally, when everything else is finished, a prune will be carried out.

#!/usr/bin/env bash

#!/usr/bin/env bash
#######################
#Start User Variables#
MySqlTempFolderName="MySqlTempBackup"
MySqlFinalFolderPath="GoogleDrives/Opensim/GridBackups/MySqlBackup/"
MySqlDetailsFile=".MySql-opensim.cnf"
ResticEnviroment=".restic-opensim-environment"
FolderToBackup="Opensim"
NumberOfBackupsToKeep="4"
FolderForRsyncToBackup="/var/www"
FolderForRsynToBackupTo="OpenSim/WwwBackup"
#Databases is a list of all Mysql Databases to be backed up. 
DATABASES=( Robust Simulator1 Simulator2 Simulator3 Simulator4 )

#End User Variables
#####################


BackupMySql()
{
    cd  "$HOME"/"$MySqlTempFolderName" || return
    for DATABASE in "${DATABASES[@]}"
    do
        echo "Backing up MySql database: $DATABASE"
        mysqldump --defaults-extra-file=/"$HOME"/"$MySqlDetailsFile" --quick  "$DATABASE" | bzip2 > "$DATABASE".sql.bz2
    done
    mv "$HOME"/"$MySqlTempFolderName"/*.bz2 "$HOME"/"$MySqlFinalFolderPath/"
}


SyncWwwToGridFiles()
{
    echo "Syncronising www folders to grid files"
    rsync -azP --delete $FolderForRsyncToBackup "$HOME"/$FolderForRsynToBackupTo
}

GridFilesBackup()
{
    echo "starting backup of grid files"
    source "$HOME"/"$ResticEnviroment"
    restic backup "$HOME"/"$FolderToBackup"
    echo "File backup complete, starting repository prune"
    restic forget --keep-last "$NumberOfBackupsToKeep"
}


BackupMySql
SyncWwwToGridFiles
GridFilesBackup

Save the script with a sensible file name like “BackupAll.sh” and make it executable using chmod

chmod 775 BackupAll.sh

Ubuntu Opensim Server Backup – Crontab

The final stage in this Ubuntu Opensim Server Backup tutorial is using crontab to launch the script at regular intervals. Use the command below to open the user settings for crontab.

crontab -e

A file looking similar to the contents blow will be displayed.

# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command

Add a single line to the bottom and remember to adjust your user name. Additionally, change the file name if needed. The example shown will do a backup at 6am every day. Further details on crontab are in Ubuntu’s documentation.

0      06     *       *       * /home/**YourUserName**/BackupAll.sh