Ubuntu Opensim Server Backup.
Ubuntu Opensim Server Backup – Opensimulator logo Ubuntu Opensim Server Backup – Ubuntu logo Rsync logo restic logo Google Drive logo Contabo Logo
This article looks at Ubuntu Opensim Server Backup. It covers how to do to so adequately while being affordable.
Previous and Related Articles
The guide is written to accompany Grid backups and oar files on the Fire And Ice Grid blog. Additionally, it follows on from Setup Ubuntu 20.04 for Opensimulator, Opensim with Robust multiple services on Ubuntu, Automated Opensim Startup and Shutdown and Securing An Opensim Ubuntu Server. Most of the article is suitable to use on any small scale server setup. However, automatic OAR generation is specific to opensimulator.
Ubuntu Opensim Server Backup Principles
The 3-2-1 backup rule is a standard which many adhere to. Firstly it states to keep three coppies of all data. Secondly that the data should be stored in two locations. Finally that one of those locatoins needs to be off site. Importantly, one of these backups must be available quickly.
The 3-2-1 rule is a little dated, however, the principle still applies. Firstly, keep several copies of all data. Then secondly store each backup in a different datacentre. Finally, ensure that the data centres used are not operated by the same company.
The primary server should only have access to the first backup store. Then the VPS server needs SSH access to the primary server to push files quickly if required. Both the VPS and the office server duplicate the first backup independently. Similarly, it is essential to ensure that neither the VPS nor the primary server has access to the office location.
- Keep a minimum of 3 Copies of all data
- One copy of the data must be available quickly
- Disconnected servers for backup replication
- Use Multiple data centres
- More than one company should host the data centres
- Encrypt data before transmission
- Use incremental backups and subsequently reduce time and bandwidth
Opensim Specific Backup Information
No single solution is adequate to restore an opensimulator grid from a catastrophic event. Regions usually restore correctly from OAR files, but exceptions do exist. Inventory archives can rebuild user inventories, but grid owners can not use them. To restore users friends lists, groups and other details; the Robust database needs saving and restoring. In most cases, the Asset folder is also required (if FSAssets is used). Some scripts can cause objects to break after OAR restoration. As a result, a copy of the simulator folders needs saving along with the simulators database. This also means keeping each simulator in its own folder, and with its own database is a good idea
Opensim Inventory Archive
Inventory archives are less useful than they could be. To make an inventory archive, the user’s password has to be available. Individuals running their own simulators will find this helpful, but grid owners are unable to use this feature. Full details are on the opensimulator Inventory Archives wiki page.
Opensim Region Backup – Automatic OAR
Full details of the specification for are on the Auto Backup page of the opensimulator wiki. To use auto-backup add a few lines to the OpenSim.ini configuration file. The path needs to set the first backup location or duplicated there afterwards. Exceptions exist with individual scripts however so these can not be the whole solution.
[AutoBackupModule] AutoBackupModuleEnabled = true ; enabled AutoBackup = true ; daily = 1440 minutes, 2880 = 2 days, 3 backups per week, restarts at 7 days AutoBackupInterval = 2880 ; Always run AutoBackupBusyCheck = true ;Skip the backup if more than this many avatars are present AutoBackupThreshold = 1 ; we do want assets AutoBackupSkipAssets = false ; save 2 weeks data (7 files per region over 14 days) AutoBackupKeepFilesForDays = 14 ; AutoBackupNaming = Time ; save into a bin/Autobackup folder AutoBackupDir = "/**folder path for the OAR files**"
The auto-backup directory can be inside your mounted G-Suite drive.
Opensim Server Backup – Keeping The Cost Down
There are many storage providers. Their prices vary dramatically along with their specification. The example in this article actually saves data to the secondary store (G-Suite) first. The VPS server then uses Rsync to clone the G-Suite drive to the VPS. Another machine in the office clones the secondary store again.
The VPS can push data to the central server at speed if required. The VPS needs SSH keypair access to the primary server. However, the main server must not have access to the VPS. Equally, neither the main server nor the VPS can access the office backup. Working this way protects two backups from main server breaches.
Use a VPS (virtual private server) for the primary Backup.
VPS servers are often cheaper for small servers than cloud storage such as Google Cloud, Microsoft Azure and Amazon S3. A VPS server can also act independently of the primary server. Thus it provides storage and the disconnected replication of backups. Contabo offer competitively priced VPS.
Google G-Suite (Google Drive) – Secondary backup store.
Googles G-Suite drive space is very low-cost cloud storage. Each user costs around £7 per month and gets 1 TB of drive space. In contrast, a G-Suite account with more than 4 users gets unlimited space. However, there are restrictions on the total daily bandwidth and speed to consider. It is too slow for a primary backup store.
Office / Home high capacity storage drive
Keep one copy of the data in your own office. This copy of the backup data will is the last resort backup. To add an even higher level of security make this a drive which can be unplugged when not in use.
Backup Software
- google-drive-ocamlfuse – mount google drive to a local folder
- SystemD (used to start ocamlfuse at startup)
- Restic
- MySql dump/plump
- Percona Xtra Backup
- Rsync
- SSH key pairs
- Bash Scripts
- Crontab – to automate the bash scripts
Mounting a G-Suite Google Drive – Google Drive Ocamlfuse
Google Drive Ocamlfuse is a command-line client for mounting google drives on Linux. Ocamlfuse can use team drive space, multiple accounts and headerless installs. To help automate the backups, it is a good idea to save the automatic OAR backups to a folder in google drive.
Start ocamlfuse when the server boots – use systemd
Create a systemd service file using Nano or another text editor. Next, Change the name at the end of the line if you wish to name the service differently.
sudo nano /etc/systemd/system/gdrive-opensim
Copy-paste the content below. All places where a double asterisk (** **) surrounds something, replace the content with system-specific information.
[Unit] Description= Ocamlfuse startup and keep alive After=syslog.target network.target [Service] Type=simple User=**YourUserName** Group=**YourUserGroup** WorkingDirectory=/home/**YourUserName** ExecStart=/home/**YourUserName**/google-drive-ocamlfuse -label **DriveLable** /**PathToYourMountPoint**/ ExecStop=/home/**YourUserName**/fusermount -u /**PathToYourMountPoint**/ RemainAfterExit=yes KillMode=none Restart=always Environment=USER=**YourUserName** HOME=/home/**YourUserName** [Install] WantedBy=multi-user.target
Save and exit the file
CTRL + O
ENTER
CONTROL + X
Finally enable and start the new service
sudo systemctl enable gdrive-opensim.service
sudo systemctl start gdrive-opensim.service
Ubuntu Opensim Server Backup – Restic folder backup
Restic is a command-line backup system using high-level encryption. It also supports environmental variables so everything can be automated, and almost any storage space is compatible. Data encryption happens on the local machine before transmission. Each snapshot restic takes only saves the extra data not stored by the previous jobs.
Install and setup Restic
To install restic on Ubuntu use the command below, for other operating systems please see the main restic installation page.
apt-get install restic
Decide on a file structure. Eg.
GoogleDrives/
└── Opensim
└── GridBackups
├── MySqlBackup
├── Server1
├── Server2
└── Server3
Create a Restic environment file and add the information in the example below.
nano .restic-opensim-environment
Add in the following, changing the path, user name and password to match the system being used.
export RESTIC_PASSWORD="**MyPassword**" export RESTIC_REPOSITORY="/home/**YourUserName**/GoogleDrives/Opensim/GridBackups/Server1"
Make sure no other users can read this file
chmod go-rwx .restic-opensim-environment
Initiate the restic repository with these commands
cd ~/
source .restic-opensim-environment
restic init --repo /home/**YourUserName**/GoogleDrives/Opensim/GridBackups/Server1
Wait for confirmation.
Backup using Restic
To do a backup execute the following commands. Rember to change **YourUser** to your user name and **Opensim** to the location of the opensim directory.
cd ~/
source .restic-opensim-environment
restic backup /home/**YourUser**/**Opensim**
Over time the size of the restic backup folder will grow without stopping. Keep the data volume down by combining “forget” and “prune. To keep the last two backups use the command below. Alternatively, adjust it to suit your requirements.
restic forget --keep-last 2 --prune
Adding this command to a bash script allows automation of the backup.
Ubuntu Opensim Server Backup – MySql Databases
MySql dump is compatible with all versions of MySql and MariaDB. Users of MySQL V5 or higher should consider using MySql Pump instead. Firstly create a new file using nano or another similar text editor. Secondly, paste in the structure below. Thirdly adjust the unique details (the ones with a ** at the start and end). Finally, save and exit. MySql dump locks the database during a backup. It is worth considering Percona Xtra backup discussed below for more extensive Robust databases. While the database is locked, users will not be able to log in.
[client] user = **DatabaseUserName** password = **DatabaseUserPassword**
Ensure only your user can read the file
chmod go-rwx .MySql-opensim.cnf
Create a database backup using the command below.
mysqldump --defaults-extra-file=/home/**YourUserName**/.MySql-opensim.cnf --quick -u **DatabaseUserName** **DatabaseName** | bzip2 > **DatabaseName**.sql.bz2"
Backing up a large database file directly to Google drive can cause a timeout issue. To avoid this issue, back up the database to a local drive first, then move the file to Google drive.”
cd ~/
mkdir MySqlBackupTemp
Ubuntu Opensim Server Backup – MySql Databases – Percona
Percona Xtra backup is an alternative to using MySql dump/plump. It comes with the advantage that during backup the databases are not locked. Additionally, the backup process is substantially quicker than using MySql Dump. However, there is a cost when restoring. Percona makes binary backups of the MySql data folder. In order to restore a database into this folder, the MySql service must be stopped. How problematic this is, depends on various factors. It is worth considering when choosing the backup method.
Ubuntu Opensim Server Backup – Rsync
Rsync is a utility for efficiently transferring and synchronising files. Files can be local or on a different server using SSH. To install rsync, use the following command in terminal.
sudo apt-get install rsync
Rsync – Duplicating The First Backup
On each server you wish to duplicate the first back on, setup Google Drive Ocamlfuse and make a folder to synchronise the backup to.
cd ~/
mkdir GridBackupClone
rsync -azP --delete ~/GoogleDrives/Opensim/GridBackups ~/GridBackupClone
Rsync – Synchronize other folders
This example shows rsync to synchronise the contents of the default www folders with a folder called WwwBackup inside the Opensim folder. Additionally, note that “–delete” is used, consequently, if files are removed from the source, they are also removed from the destination.
rsync -azP --delete /var/www ~/Opensim/WwwBackup
Bash Script to automate backing up all database and folders
The penultimate stage is to create a bash script to launch the entire process. Starting with the MySql databases and then moving on to synchronising other folders into the one being backed up, and creation of a snapshot. Finally, when everything else is finished, a prune will be carried out.
#!/usr/bin/env bash #!/usr/bin/env bash ####################### #Start User Variables# MySqlTempFolderName="MySqlTempBackup" MySqlFinalFolderPath="GoogleDrives/Opensim/GridBackups/MySqlBackup/" MySqlDetailsFile=".MySql-opensim.cnf" ResticEnviroment=".restic-opensim-environment" FolderToBackup="Opensim" NumberOfBackupsToKeep="4" FolderForRsyncToBackup="/var/www" FolderForRsynToBackupTo="OpenSim/WwwBackup" #Databases is a list of all Mysql Databases to be backed up. DATABASES=( Robust Simulator1 Simulator2 Simulator3 Simulator4 ) #End User Variables ##################### BackupMySql() { cd "$HOME"/"$MySqlTempFolderName" || return for DATABASE in "${DATABASES[@]}" do echo "Backing up MySql database: $DATABASE" mysqldump --defaults-extra-file=/"$HOME"/"$MySqlDetailsFile" --quick "$DATABASE" | bzip2 > "$DATABASE".sql.bz2 done mv "$HOME"/"$MySqlTempFolderName"/*.bz2 "$HOME"/"$MySqlFinalFolderPath/" } SyncWwwToGridFiles() { echo "Syncronising www folders to grid files" rsync -azP --delete $FolderForRsyncToBackup "$HOME"/$FolderForRsynToBackupTo } GridFilesBackup() { echo "starting backup of grid files" source "$HOME"/"$ResticEnviroment" restic backup "$HOME"/"$FolderToBackup" echo "File backup complete, starting repository prune" restic forget --keep-last "$NumberOfBackupsToKeep" } BackupMySql SyncWwwToGridFiles GridFilesBackup
Save the script with a sensible file name like “BackupAll.sh” and make it executable using chmod
chmod 775 BackupAll.sh
Ubuntu Opensim Server Backup – Crontab
The final stage in this Ubuntu Opensim Server Backup tutorial is using crontab to launch the script at regular intervals. Use the command below to open the user settings for crontab.
crontab -e
A file looking similar to the contents blow will be displayed.
# Edit this file to introduce tasks to be run by cron. # # Each task to run has to be defined through a single line # indicating with different fields when the task will be run # and what command to run for the task # # To define the time you can provide concrete values for # minute (m), hour (h), day of month (dom), month (mon), # and day of week (dow) or use '*' in these fields (for 'any'). # # Notice that tasks will be started based on the cron's system # daemon's notion of time and timezones. # # Output of the crontab jobs (including errors) is sent through # email to the user the crontab file belongs to (unless redirected). # # For example, you can run a backup of all your user accounts # at 5 a.m every week with: # 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/ # # For more information see the manual pages of crontab(5) and cron(8) # # m h dom mon dow command
Add a single line to the bottom and remember to adjust your user name. Additionally, change the file name if needed. The example shown will do a backup at 6am every day. Further details on crontab are in Ubuntu’s documentation.
0 06 * * * /home/**YourUserName**/BackupAll.sh
Pingback: Grid backups and oar files - Fire And Ice Grid Blog
Pingback: Automated Opensim Startup and Shutdown - Sara Payne's BlogSara Payne's Blog
Pingback: Setup Ubuntu 20.04 for Opensimulator - Sara Payne's BlogSara Payne's Blog
Pingback: Opensim with multiple Robust services on Ubuntu - Sara Payne's BlogSara Payne's Blog
Pingback: Securing An Opensim Ubuntu Server - Sara Payne's BlogSara Payne's Blog
Pingback: Fire And Ice Grid Upgrade & Study possibilities. - Sara Payne's BlogSara Payne's Blog
Pingback: Fire and Ice Grid opens - A dream realised. - Sara Payne's BlogSara Payne's Blog