Ubuntu Opensim Server Backup

• 

• 

• 

• 

• 

• 

Ubuntu Opensim Server Backup. An article describing one way to fully backup Opensimulator using Ubuntu and other opensource software. It follows on directly from Opensimulator Grid Backup & Restores; additionally, it is part of a more extensive Opensim Server Setup Using Ubuntu & Windows.

Data Stores – Ubuntu Opensim Server Backup

• Primary Backup – FTP folder on an OVH host.
• Secondary Backup – Backblaze Bucket
• Third Backup – Google Drive
• Fourth Backup – Local compuater

Summery Of Intent

• Mount the primary backup
• Create credential file for Backblaze
• Mount the Backblaze bucket
• Create credential file for Restic
• Mount Google Drive
• Create credential file for the SQL dump
• Initiate Restic repositories on the Primary and Secondary backup
• Dump Database Files to the ~/OpenSim/Data/MySql
• Syncronise /var/www to ~/OpenSim/Data/www
• Make the backup run automatically

Mounting the FTP Folder – Ubuntu Opensim Server Backup

First, create a mount point for the backup. This can be anywhere the user account can access. However, readers intending to use the sample scripts later should create

mkdir ~/ResticBackup

It is possible to mount any FTP folder using curl. The example below is generic, and the second is an actual example with substituted hypothetical values.

curlftpfs Login:password@server MountPoint

curlftpfs ip-155-4394-44.zz:djdHS573"37££l@ftpback-rbx2-144.ovh.net ~/ResticBackup

Further down the page, a Bash shell script that mounts all critical drives uses this. Details of finding the login information for OVH backup is in Buying Server Hosting For Opensimulator.

Mounting Google Drive – Ubuntu Opensim Server Backup

Mount Google drive using google-drive-ocamlfuse. There are full instructions available; since it is a server with no GUI, either use the instructions for Headerless Usage And Authorization or Service Accounts. Both methods require visiting Google Cloud Console to set up credentials. In addition, Google offers a helpful tutorial video for Creating and Using Service Accounts. Finally, take a look at the Team Drives instructions. If using the service account route, do not forget to use chmod 600 on the json credentials file(stops any other user on the system from accessing the file).

One area where their instructions could be more apparent is in the “label”. For example, Google-Drive-Ocamlfuse creates a new hidden folder in the users home directory called “.gdfuse.” The label then becomes the name of a subfolder inside the “.gdfuse” directory.

The team drives Id’s are found by opening the drive and copying the last part of the URL from a browser.

mkdir ~/GoogleDrives
mkdir ~/GoogleDrives/OpenSim
mkdir ~/GoogleDrives/OpenSim/Grid

From this point on, the drive can be mounted with

google-drive-ocamlfuse -label label MountPoint

google-drive-ocamlfuse -label OpenSimGridDrive ~/GoogleDrives/OpenSim/Grid/

A BASH shell script that mounts all drives uses this further down the page.

Create a credentials file for SQL dump

This will work for MySql and MariaDB; it will not work with PostgreSQL, added later.

This section refers back to Creating Databases For Opensimulator, as it requires the user name and password set for the databases.

Create the file with

nano ~/.mySqlLogin.cnf

[client]
user = Opensimulator
password = !*3Nhdgyle*$%

save and exit, then change the permissions so no other user on the system can access the file.

chmod 600 ~/.mySqlLogin.cnf

Create a credentials file for the Restic Repositories

Backblase uses an S3 style API. In Buying Remote Storage From Backblaze, it describes creating buckets and keys. The information from there is necessary for this step.

Both the Restic repositories will use the same credentials file. Create a new file in the users home folder called “.restic-env”. Using nano, this is done with

nano ~/.restic-env

Next, enter values like the example below. Like previous examples, the first is totally generic, and the second uses precise hypothetical values.

export RESTIC_PASSWORD="Password"  
export B2_ACCOUNT_ID="Key Id"
export B2_ACCOUNT_KEY="Key"
export RESTIC_REPOSITORY="BucketName"

export RESTIC_PASSWORD="Mhd73"$£hs5"  
export B2_ACCOUNT_ID="000SH29ssa3290000000050"
export B2_ACCOUNT_KEY="K000aCBEQja4+Qjaqwt7B5S4bdVlQog"
export RESTIC_REPOSITORY="b2:ServerFiles-192-168-1-254"

Save and exit with

CTRL+O
CTRK+X

Use chmod to prevent any other user on the system from accessing the file

chmod 600 ~/.restic-env

Mount Backblaze And Initiate the Restic Repository

Firstly create a credentials file for the mount. It uses the same credentials as the restic-env but needs layout out differently.

Firstly create a new file in the user home folder called “.passwd-s3fs”. Each time Resitic is called, it will use this file. Using nano

nano .passwd-s3fs

Add the key and id and application key as shown below. The first is a generic example, and the second is an actual example with hypothetical substitute examples.

Key Id:Application Key

000SH29ssa3290000000050:K000aCBEQja4+Qjaqwt7B5S4bdVlQog

Finally, make sure no other users on the system can access this file by changing the permissions. It is possible to mount the bucket using this credentials file.

chmod 600 ~/.passwd-s3fs

Next, create the mount point its self, in this example, its

mkdir ~/BackBlazeDrive

Mounting the Backblaze Bucket

Mount the Backblaze repository with this command; again, there is a generic and precise example. Further down the page, this is used in a script.

s3fs BucketName PathToRepository -o passwd_file=Path/FileName -o url=EndPoint -o use_path_request_style

s3fs OarsMount ~/BackBlazeDrive -o passwd_file=$HOME/.passwd-s3fs -o url=https://s3.us-west-000.backblazeb2.com -o use_path_request_style

Initiate The Backblaze repository

cd ~/
source = ResticCredentialsFile
restic -r b2:BucketName init

source ~/.restic-env
restic -r b2:ServerFiles-192-168-1-254 init

Mount The FTP Drive from the OVH

This section uses the login details sent by OVH when the FTP backup is activated. Details are available in Buying Server Hosting For Opensimulator. Like other times there are two examples below, one generic and one precise. Also, like previous times later in the article, a script will run this.

curlftpfs Login:Password@Server

curlftpfs ns1025137.ip-192-168-1.eu:9stLZRAs5@ftpback-rbx5-1346.ovh.net ~/ResticBackup

Initiate the Restic Repository on the Mounted FTP folder

cd ~/
source = ResticCredentialsFile
restic -r b2:BucketName init

source ~/.restic-env
restic -r b2:~/ResticBackup init

Create a BASH shell script to mount folders and create a SystemD service to run them at startup.

Create a new shell script

nano ~/OpenSim/Scripts/MountDrives.sh

#!/bin/bash
s3fs ServerFiles-192-168-1-254 init $HOME/BackBlazeDrive -o passwd_file=$HOME/.passwd-s3fs -o url=https://s3.us-west-000.backblazeb2.com -o use_path_request_style

curlftpfs ns3085157.ip-145-239-4.eu:2mtKZcwRUs@ftpback-rbx2-144.ovh.net $HOME/ResticBackup

google-drive-ocamlfuse -label OpenSimGridDrive $HOME/GoogleDrives/OpenSim/Grid

Repeat the process to create an unmount script

#!/bin/bash
fusermount -u $HOME/BackBlazeDrive
fusermount -u $HOME/GoogleDrives/OpenSim/Grid/
umount $HOME/ResticBackup

Next, make both of these files executable

chmod +x ~/OpenSim/Scripts/MountDrives.sh
chmod +x ~/OpenSim/Scripts/UnMountDrives.sh

Save and exit, then repeat a similar process to create a new systemD service file. However, this one must be completed with an editor using sudo privileges.

sudo nano /etc/systemd/system/MountDrives.service

  GNU nano 4.8                                                                   opensimulator.service                                                                              
[Unit]
Description= Mounting Drives
Documentation=none
After=syslog.target network.target ufw.service mysql.service cron.service
[Service]
Type=simple
User=opensim
Group=opensim
WorkingDirectory=/home/opensim
ExecStart=/home/opensim/Scripts/MountDrives.sh
ExecStop=/home/opensim/Scripts/UnMountDrives.sh
RemainAfterExit=yes
KillMode=none
Environment=USER=opensim  HOME=/home/opensim
[Install]
WantedBy=multi-user.target

Save and exit, then enable the service.

sudo systemctl enable MountDrives.service

if the drives are not already mounted, start them with

sudo systemctl start MountDrives.service

Create a BASH shell script to perform a Backup All operation

Near the top of the script is an array of databases to backup. This will need changing on a per-server basis and the other values for the remote backup location, IP address etc.

Create a Cron task to perform the backup at regular intervals

#!/usr/bin/env bash
# shellcheck source=/home/sara/.restic-env

DATABASES=( Simulator00 Simulator01 Simulator02 )
IPADDRESS="192.168.1.254"
RemoteBackupLocation="b2:ServerFiles-192-168-1-254"
LocalBackupLocation="$HOME/ResticBackup"
SourceLocation="$HOME/OpenSim"

BackupMySql()
{
    cd $HOME/OpenSim/Data/MySql || return
    for DATABASE in "${DATABASES[@]}"
    do
        echo "Backing up MySql database: $DATABASE"
        mysqldump --defaults-extra-file=$HOME/.mySqlLogin.cnf --quick $DATABASE | bzip2 > $DATABASE.sql.bz2
    done
}

SyncWwwToGridFiles()
{
    echo "Syncronising www folders to grid files"
    rsync -azP --delete /var/www/ $HOME/OpenSim/Data/www/
}

SyncMapTiles()
{
    echo "syncing map tiles"
    rsync -azP --delete $HOME/OpenSim/Data/Maptiles/ $HOME/GoogleDrives/OpenSim/Grid/GridBackups/Maptiles/$IPADDRESS/
}

ResticBackupRemote()
{
    echo "Starting Restic Remote Backup"
    source $HOME/.restic-env
    echo "ensuring no locks stop this backup"
    restic -r $RemoteBackupLocation unlock
    echo "starting restic backup of grid files"
    restic -r $RemoteBackupLocation backup $SourceLocation
    echo "File backup complete, starting repository prune"
    restic -r $RemoteBackupLocation forget --keep-last 14 --prune
}

ResticBackupLocal()
{
    echo "Starting Local Remote Backup"
    echo "Starting Restic Remote Backup"
    source $HOME/.restic-env
    echo "ensuring no locks stop this backup"
    restic -r $LocalBackupLocation unlock
    echo "starting restic backup of grid files"
    restic -r $LocalBackupLocation backup $SourceLocation
    echo "File backup complete, starting repository prune"
    restic -r $LocalBackupLocation forget --keep-last 14 --prune
}

EnsureGdriveMounted()
{
    echo "checking Gdrives mounted"
    cd $HOME/GoogleDrives/OpenSim/Grid/OarAccess || RemountGdrive
}

RemountGdrive()
{
    echo "remounting Gdrives"
    cd $HOME/OpenSim/Scripts/
    ./UnMountGoogleDrives.sh
    ./MountGoogleDrives.sh
    return
}

EnsureFTPdriveMounted()
{
    echo "checking FTP drives mounted"
    cd $HOME/Restic/locks || RemountFtpDrive
}

RemountFtpDrive()
{
    echo "remounting FTP Drive"
}

SyncMySqlToBackupDrive()
{
    echo "Syncronising MySql Backup To Backup Drive"
    rsync -azP --delete $HOME/OpenSim/Data/MySql/ $HOME/GoogleDrives/OpenSim/Grid/MySqlBackup/$IPADDRESS/
}

SyncOarsToBackupDrive()
{
    echo "Syncronising Oars To Backup Drive"
    rsync -azP --delete $HOME/OpenSim/Data/Oars/ $HOME/GoogleDrives/OpenSim/Grid/OarAccess/$IPADDRESS/
}


BackupMySql
SyncWwwToGridFiles
SyncMapTiles
EnsureFTPdriveMounted
ResticBackupLocal
ResticBackupRemote
EnsureGdriveMounted
SyncMySqlToBackupDrive
SyncOarsToBackupDrive

The final task in this section is making a cron job to run the backup at regular intervals. If this is the first time running crontab, it will offer a choice of editors. Anyone unfamiliar with the options should probably choose nano.

crontab -e

add the following line to run the backup once a day at 06:00 (by the servers clock).

#Backup all every dat at 6am
0      06       *       *       *      /home/sara/BackupAll.sh

Related Articles – Ubuntu Opensim Server Backup

• Installing Ubuntu Server Guide
• Secure A Ubuntu Server
• Extensible & Maintainable Opensimulator Config