Automated Opensim Startup and Shutdown

• 

• 

• 

Introduction to Automated Opensim Startup and Shutdown

Automated Opensim startup and shutdown tutorial. This is an alternate approach to the Opensimulator wiki pages. This tutorial aims to give greater flexibility and improve ease of use. Written for Ubuntu but suitable for many Linux distributions. Additionally, the system described here is in use on the Fire and Ice opensim grid. Fire And Ice Blog details

Full Article Here

An article that covers setting up Opensimulator from Start to Finish is available here

Previous Related Articles

• Setup Ubuntu 20.04 for Opensimulator
• Opensim with multiple Robust service instances

Automated Opensim Startup and Shutdown Key Goals

• In-World residents will get advanced warning of a restart.
• Minimise downtime when a server restarts.
• Clean shutdown even when it is unexpected
• Possible to use every simulator individually.
• Avoid bottlenecks during oar backups.

To achieve this the following will happen:

• Systemd will launch opensimulator automatically when the server boots.
• Each simulator to be accessed through a tmux window.
• Residents get message warnings every min for ten mins before shutdown.
• Unexpected shutdown to be clean and still give residents minimal warning.
• Simulators will restart with a delay between each one after a server restart (reduces bottleneck during oar backup)

user@user-MS-7522:~/src$ tmux ls
Robust: 6 windows (created Fri May 29 17:07:18 2020)
SimulatorsMain: 6 windows (created Fri May 29 19:56:48 2020)
SimulatorsTesting: 7 windows (created Fri May 29 17:11:16 2020)
user@user-MS-7522:~/src$ 

• 

• 

• 

Anyone unfamiliar with tmux is encouraged to look at the tmux chat sheet. I may consider doing a tmux focused blog later if there is interest.

Automated Opensim Startup and Shutdown – Systemd

Create the bash shell scripts ready for systemd

Initially create two bash shell scripts, to keep the demonstration simple they will be placed in the users home folder.

• Opensim_Start.sh
• Opensim_Stop.sh

The shell scripts need to have the bash shebang at the top and need to be made executable. The process is the same for each one, only the first is shown. Start by creating the script in the home folder.

cd ~/
nano OpenSim_Start.sh

Add the shebang to the top of the script

#!/usr/bin/env bash

Now save and exit with

CTRL + O Enter
CTRL + X

Repeat the process for all three scripts, then make them executable using:

chmod 775 OpenSim_Start.sh
chmod 775 OpenSim_Stop.sh

Adding the new scripts to systemd, so they start when the server boots up

Create a new service file in the systemd directory. This may be in a different location on different distributions, but the same process will work with all distributions which use systemd.

sudo nano /etc/systemd/system/opensim.service

Now add the contents. **UserName** needs changing to your user name with no asterisk.

[Unit]
Description= OpenSimulator Inside Tmux
Documentation=man:tmux(1)
After=syslog.target network.target ufw.service mysql.service cron.service 

[Service]
Type=simple
User=**UserName**
Group=**UserName**
WorkingDirectory=/home/**UserName** 
ExecStart=/home/**UserName** /OpenSim_Start.sh
ExecStop=/home/**UserName** /OpenSim_Stop_.sh fast
RemainAfterExit=yes
KillMode=none
Environment=USER=**UserName**  HOME=/home/**UserName** 

[Install]
WantedBy=multi-user.target

Save and exit with

CTRL + O Enter
CTRL + X

Finally enable the service with

sudo systemctl enable opensim.service

Now when the server boots up it will run OpenSim_Start.sh and when it closes down it will run OpenSim_Stop.sh passing “fast” into the script. The next section covers tmux and more shell scripting.

Automated Opensim Startup and Shutdown – Description and explanation

Usually, a server runs Robust or Simulators but not both. Doing both together is convenient as a demonstration.

3 tmux sessions will be created :

• Robust
• SimulatorsMain
• SimulatorsTesting

Each of these sessions contains multiple windows. Each window has an instance of Robust or a Simulator. Two simulator sessions exist to demonstrate grouping simulators.

Folder Structure

└── Opensimulator
   ├── Robust
   └── Simulators
       ├── Main
       │   ├── Instance1
       │   ├── Instance2
       │   ├── Instance3
       │   ├── Instance4
       │   └── Instance5
       └── Testing
           ├── 0821
           ├── 0900
           ├── 0901
           ├── 0910
           ├── 0911
           └── 0920

Inside each simulator folder, the file and folder structure matches a standard opensimulator download. Additionally .ini files are demonstrated in Opensim with multiple Robust services are in Opensimulator/Robust/bin. Conversely, The folder structure can change if the scripts are modified to match.

Automated Opensim Startup and Shutdown- Startup Scripts

Main Opensim Startup Script (OpenSim_Start.sh)

#!/usr/bin/env bash
# This script assumes the other scripts it calls are in the same folder if they are not please adjust accordingly
#############################
###Start of user variables###
SIMULATORGROUPS=( SimulatorsMain SimulatorsTesting ) #list of all simulator names to start up
MINSCOUNT=15 #number of mins to wait after simulators start is initiated before restarts begin
###End of user variables###
############################
#only edit below this line if you are sure of what you are doing
WINDOW="Terminal"
TAIL="_Start.sh"

#launch the Robust startup script
#no robust on this server comment out
#./Robust_Start.sh
#sleep 5s #give robust time to start (Should take no noticeable time).

#start simulators. 
#do this for every simulator group listed above
for SIMULATORGROUP in "${SIMULATORGROUPS[@]}"
do
    echo "checking for session - $SIMULATOR"
    #if the session doesn't already exist make it
    SESSIONEXISTS=$(tmux list-sessions | grep "$SIMULATORGROUP")
    if [ "$SESSIONEXISTS" = "" ]
    then
        echo "$SIMULATORGROUP not found, starting session "
        #Create new Session and rename it
        tmux new-session -d -s "$SIMULATORGROUP"
        tmux rename-window -t 0 "$WINDOW"
    else
        echo "$SIMULATORGROUP already started, skipping"
        echo "checking for $WINDOW window"
        #if a terminal window doesn't already exist make one
        WINDOWEXISTS=$(tmux list-windows -t "$WINDOW" | grep "$SIMULATORGROUP")    
        if [ "$WINDOWEXISTS" = "" ]
        then
            echo "Starting window $WINDOW"
            tmux new-window -t "$SIMULATORGROUP" -n "$WINDOW"
        fi  
    fi
    #send the start script command to the terminal window for this session
    tmux send-keys -t "$SIMULATORGROUP:$WINDOW" "bash $SIMULATORGROUP$TAIL" C-m
done
while [ $MINSCOUNT -ge 1  ]
do
      echo "$MINSCOUNT minutes until simulator restarts begin"
    sleep 1m
    (("MINSCOUNT--"))
done
for SIMULATORGROUP in "${SIMULATORGROUPS[@]}"
do
    RESTARTTAIL="_Restart.sh"
    #unlike the initial startup, the restarts should be done sequentially to avoid bottlenecks during oar backup.
    #so the scripts should be called directly, 
    #equates to ./SimulatorMain_Restart.sh if the simulator is called Main
    ./"$SIMULATORGROUP""$RESTARTTAIL"
done

Robust Startup Script

#!/usr/bin/env bash
COMMANDPATH="$HOME/Opensimulator/Robust/bin/"
COMMAND="mono --server" 
FILE="Robust.exe"
INIFILEPRE="-inifile=Robust.HG."
INIFILETAIL=".ini"
SESSION="Robust"

if [ "$1" != "" ]
then
    Services=( "$1" )
else
    #######################################
    ###Start Of User Changeable Variables##
    Services=( Asset GridUser Inventory Main Map ) #list of all simulators in this group
    ###End Of User Changeable Variables###
    ######################################
fi

echo "checking for session - $SESSION"
SESSIONEXISTS=$(tmux list-sessions | grep $SESSION)
if [ "$SESSIONEXISTS" = "" ]
then
    echo "Starting session $SESSION"
    #Create new Session and rename it
    tmux new-session -d -s $SESSION
    tmux rename-window -t 0 'Terminal'
else
    echo "$SESSION already started, skipping creation"
fi
#loops through all instances of robust to start launching each on its its own window
for Service in "${Services[@]}"
do
    WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$Service")
    if [ "$WINDOWEXISTS" = "" ]
    then
        echo "creating window $Service in $SESSION"
        tmux new-window -t "$SESSION" -n "$Service"
    else
        echo "window $Service already exits in $SESSION, skipping creation"
    fi
    echo "starting $Service"
    tmux send-keys -t "$SESSION:$Service" 'ulimit -s 1048576' C-m
    tmux send-keys -t "$SESSION:$Service" 'export TERM=xterm' C-m
    tmux send-keys -t "$SESSION:$Service" "cd $COMMANDPATH" C-m
    tmux send-keys -t "$SESSION:$Service" "$COMMAND $FILE $INIFILEPRE$Service$INIFILETAIL" C-m    
done

Simulator Group Startup Script

Create startup scripts to match your simulator groups. e.g

• SimulatorMain_Sart.sh
• SimulatorTesting_Start.sh

Edit the values at the top for each simulator group startup script.

#!/usr/bin/env bash
# second parameter passed into the script
if [ "$1" != "" ]
then
    SIMULATORS=( "$1" )
else
    #######################################
    ###Start Of User Changeable Variables##
    SIMULATORS=( Simulator1 Simulator2 Simulator3 Simulator4) #list of all simulators in this group
    ###End Of User Changeable Variables###
    ######################################
fi

#######################
###Start User Variables
SIMULATORGROUP="Main" #name of the simulator gorup
SIMULATORSPATH="$HOME/Opensim/Simulators/$SIMULATORGROUP" #edit this only if your main Opensim folder is not a subfolder of your users home folder
COMMAND="mono --server"  #if you're using a desktop pc instead of a server amend this to --desktop
###End User Variables
#####################
#Only edit below this line if you are sure about what you are doing
FILE="OpenSim.exe" #name of the file mono runs
SESSION="Simulators$SIMULATORGROUP" 
#check the session exits and start it, if it doesnt
echo "checking for session - $SESSION"
SESSIONEXISTS=$(tmux list-sessions | grep $SESSION)
if [ "$SESSIONEXISTS" = "" ]
then
    echo "Starting session $SESSION"
    #Create new Session and rename it
    tmux new-session -d -s $SESSION
    tmux rename-window -t 0 'Terminal'
else
    echo "$SESSION alredy started, skipping"
    echo "checking for terminal window"
    #if a window for this simulator doesn't exist make it
    WINDOW="Terminal"
    WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$WINDOW")
    if [ "$WINDOWEXISTS" = "" ]
    then
        echo "creating window $WINDOW in $SESSION"
        tmux new-window -t "$SESSION" -n "$SIMULATOR"
    else
        echo "window $WINDOW already exits in $SESSION, skipping"
    fi
fi
for SIMULATOR in "${SIMULATORS[@]}"
do
    echo "checking for window $SIMULATOR in $SESSION"
    WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$SIMULATOR")
    if [ "$WINDOWEXISTS" = "" ]
    then
        echo "creating window $SIMULATOR in $SESSION"
        tmux new-window -t "$SESSION" -n "$SIMULATOR"
    else
        echo "window $SIMULATOR in $SESSION already exits"
    fi
    echo "starting $SIMULATOR"
    SIMULATORPATH="$SIMULATORSPATH/$SIMULATOR/bin"
    tmux send-keys -t "$SESSION:$SIMULATOR" 'ulimit -s 1048576' C-m
    tmux send-keys -t "$SESSION:$SIMULATOR" 'export TERM=xterm' C-m
    tmux send-keys -t "$SESSION:$SIMULATOR" "cd $SIMULATORPATH" C-m
    tmux send-keys -t "$SESSION:$SIMULATOR" "$COMMAND $FILE" C-m
done 

Simulator Group Restart Script

Create one copy of this script for each simulator group. Editing the variables at the top to match. Eg.

• SimulatorsMain_Restart.sh
• SimulatorsTesting_Restart.sh

#!/usr/bin/env bash
SIMULATORGROUP="Main"
SIMULATORS=( Simulator1 Simulator2 Simulator3 Simulator4 )
count_down_warning()
{
    MINSCOUNT=30
    while [ $MINSCOUNT -ge 1  ]
    do
        echo "$MINSCOUNT minutes until next simulator restart begins"
        sleep 1m
        #sleep 1s
        ((MINSCOUNT--))
    done
}
SESSION="Simulators$SIMULATORGROUP"
STOP="_Stop.sh"
START="_Start.sh"
##Stop each simulator in turn, each simulator has its own built in timer warnings for residents
for SIMULATOR in "${SIMULATORS[@]}"
do
    count_down_warning
    echo "Starting shutdown of $SIMULATOR"
    bash "$SESSION$STOP" slow "$SIMULATOR"
    echo "Begin startup of $SIMULATOR"
    ./$SESSION$START $SIMULATOR
done

Automated Opensim Startup and Shutdown – Shutdown Scripts

Main Opensim Shutdown Script (OpenSim_Stop.sh)

#!/usr/bin/env bash
#this script assumes the scripts it executes are in the same folder as this script.
#if your setup is different this script will require modification.
##########################
###Start Of User Variables
#Simulator Sessions robust is dealt with later
SESSIONS=( SimulatorsMain SimulatorsTesting )
###End Of User Variables###
###########################
PRE="./"
TAIL="_Stop.sh fast"
WINDOW="Terminal"
#For every session listed above, send the execute the simulator group shutdown script inside the terminal window of that session.
#This allows all sessions to be shutdown at the same time.
for SESSION in "${SESSIONS[@]}"
do
        ##loop through the sessions, checking they exist, 
        SESSIONEXISTS=$(tmux list-sessions | grep "$SESSION")
        if [ "$SESSIONEXISTS" != "" ]
        then
                #if they do then send the shutdown keys to the terminal window for that session
                tmux send-keys -t "$SESSION"":""$WINDOW" "$PRE""$SESSION""$TAIL"
        fi
done
#stop robust now
./Robust_Stop.sh
#wait for all mono process to close before exit
while pgrep -x mono >/dev/null;
do
        sleep 1s
        echo "shutting down, please wait"
done
echo "shut down compete" 

Simulator Group Shutdown

Create one of these scripts for each simulator group. Edit the variables at the top to match. E.g.

• SimulatorsMain_Stop.sh
• SimulatorsTesting_Stop.sh

#!/usr/bin/env bash
#script parameters $1 = fast/other,
#script parameters $2 = empty / name of a specific simulator
#launch using ./SimulatorMain_Start.sh fast/slow simulatorName/empty
SPEED="${1,,}" # first parameter passed into the script, this also converts it to lower case to avoid typing errors.
# second parameter passed into the script
if [ "$2" != "" ]
then
    WINDOWS=( "$2" )
else
    #######################################
    ###Start Of User Changeable Variables##
    WINDOWS=( Simulator1 Simulator2 Simulator3 Simulator4 )
    ###End Of User Changeable Variables###
    ######################################
fi
########################################
###Start Of User Changeable Variables###
#The name of this simulator group - change this to match your requirements
SESSIONNAME="Main"
WARNINGSNUMBER=20 # The number of warnings provided, under a slow stop this is mins, under fast shutdown this is seconds.
###End Of User Changeable Variables###
######################################
#Do not edit below this line unless you are sure what you are doing.
SESSION="Simulators""$SESSIONNAME" 

TIMEUNITS="m" #used to set the timer to minutes
TIMEUNITTEXT="minutes"

#if fast is passed into the script, set the time to seconds instead of minutes
if [ "$SPEED" = "fast" ]
then
    TIMEUNITS="s" #sets the time to seconds
    TIMEUNITTEXT="seconds"
fi
echo "Checking for the session - $SESSION"
SESSIONEXISTS=$(tmux list-sessions | grep $SESSION)
#only do any of this is the session actually exists, other wise skip and send warning
if [ "$SESSIONEXISTS" != "" ] 
then        
    #every min send a warning to residents about shut down and to operator in terminal
    echo "Session $SESSION found, starting shutdown"
    while [ $WARNINGSNUMBER -ge 1  ]
    do
        #for every Tmux window (which is really a Simulator, send an inworld message to all avis)
        for WINDOW in "${WINDOWS[@]}"
        do
            #check if the window exits, it it does send the warning
            WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$WINDOW")
            if [ "$WINDOWEXISTS" != "" ]
            then
                echo "sending $WARNINGSNUMBER $TIMEUNITTEXT warning to $WINDOW"
                tmux send-keys -t "$SESSION:$WINDOW" "change region root" C-m
                tmux send-keys -t "$SESSION:$WINDOW" "alert Region Restart in $WARNINGSNUMBER $TIMEUNITTEXT" C-m
            fi    
        done
        sleep 1"$TIMEUNITS"
        (("WARNINGSNUMBER--"))
    done
    
    #call close inside each Tmux Window
    for WINDOW in "${WINDOWS[@]}"
    do
        tmux send-keys -t "$SESSION:$WINDOW" 'quit' C-m
        echo "Called close on $WINDOW"
    done
    
    #Set time to 1 min if the shutdown type is fast and 5 mins if it is a normal shutdown
    if [ "$SPEED" = "fast" ]
    then
        WARNINGSNUMBER=1 
        #The shutdown routine of the actual server limits each section to 90 seconds. 
        #20 seconds are used up in customer warnings during fast shut down. This leaves one for the simulator to exit before closing robust
    else
        WARNINGSNUMBER=5
    fi
    #Allow adequate time for each window to finish closing, send terminal notifications at regular intervals
    while [ $WARNINGSNUMBER -ge 1  ]
    do
        echo "final shut down in $WARNINGSNUMBER minutes"
        sleep 1m
        (("WARNINGSNUMBER--"))
    done
      
    for WINDOW in "${WINDOWS[@]}"
    do
        #check if the window exits, it it does tell the simulator to quit
        WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$WINDOW")
        if [ "$WINDOWEXISTS" != "" ]
        then
            tmux kill-window -t "$SESSION":"$WINDOW"
            echo "Called close on $WINDOW"
        fi   
    done      
else
   echo "session not found skipping"
fi

Robust Shutdown Script

One of these scripts is required, the suggested name is

• Robust_Stop.sh

#!/usr/bin/env bash
SESSION="Robust"
if [ "$1" != "" ]
then
    Services=( "$1" )
else
    #######################################
    ###Start Of User Changeable Variables##
    Services=( Asset GridUser Inventory Main Map ) #list of all simulators in this group
    ###End Of User Changeable Variables###
    ######################################
fi

echo "checking for session - $SESSION"
SESSIONEXISTS=$(tmux list-sessions | grep $SESSION)
#only do any of this is the session actually exists, other wise skip and send warning
if [ "$SESSIONEXISTS" != "" ] 
then            
    #call close inside each Tmux Window
    for Service in "${Services[@]}"
    do
        #check if the window exits, it it does send shutdown
        WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$Service")
            if [ "$WINDOWEXISTS" != "" ]
            then
                tmux send-keys -t "$SESSION:$Service" 'quit' C-m
                echo "Called close on $Service"
            fi
    done
    
    #Allow adequate time for each window to finish closing, send terminal notifications at regular intervals
    echo "final shut down in 5s."
    sleep 5s
    
    for Service in "${Services[@]}"
    do
        #check if the window exits, it it does tell the simulator to quit
        WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$Service")
        if [ "$WINDOWEXISTS" != "" ]
        then
            tmux kill-window -t "$SESSION":"$Service"
            echo "Called close on $Service"
        fi   
    done        
else
   echo "session not found skipping"
fi

Keep the processes alive

For reasons I do not yet fully understand I never managed to get systemD to restart a crashed mono process which runs inside tmux. For this reason, make another bash script for each of your simulator groups. Consequently, inside each script, you will need to change the simulator group name and the names of each simulator in the group. Finally, after its made, add it to the cron jobs as described below.

Simulators Keep Alive Script

#!/bin/bash

SimulatorGroup="Main"
Simulators=( Simulator1 Simulator2 Simulator3 Simulator4 )

check_pid_file_exists()
{
    #$1=file $2=Simulator
    echo "checking for $1"
    if [ -f "$1" ]; then
        echo "$1 exists "
        read_pid_file $1 $2
    else 
        REASON="because_no_pid_file"
        restart_simulator $2 $REASON
    fi
}

read_pid_file()
{
    #$1=file $2=Simulator
    PID=$(<$1)
    echo "process number = $PID"
    check_if_process_running $PID $2
}

check_if_process_running()
{
    #$1=pidNumber  $2=Simulator
    echo "checking to see if $1 is running"
    PROCESS=$1  
    REASON="because_process_not_running"
    pgrep mono | grep $1 >/dev/null && echo "Process $PROCESS found" || restart_simulator $2 $REASON
}

restart_simulator()
{
    # $1=Simulator $3=Reason
    echo "restarting $1 $2"
    Pre="Simulators"
    Stop="_Stop.sh"
    Start="_Start.sh"
    Switch="fast"
    CommandStop="$Pre$SimulatorGroup$Stop $Switch $1"
    CommandStart="$Pre$SimulatorGroup$Start $1" 
    ./$CommandStop
    ./$CommandStart
}

for Simulator in "${Simulators[@]}"
do
    FILEPATH="/tmp/"
    FILENAME="Simulator"$Simulator".pid"
    FILE=$FILEPATH$FILENAME
    check_pid_file_exists $FILE $Simulator    
done 

Robust Keep Alive Script

#!/bin/bash

Services=( Asset GridUser Inventory Main Map )

check_pid_file_exists()
{
    #$1=file $2=Simulator
    echo "checking for $1"
    if [ -f "$1" ]; then
        echo "$1 exists "
        read_pid_file $1 $2
    else 
        REASON="because_no_pid_file"
        restart_service $2 $REASON
    fi
}

read_pid_file()
{
    #$1=file $2=Simulator
    PID=$(<$1)
    echo "process number = $PID"
    check_if_process_running $PID $2
}

check_if_process_running()
{
    #$1=pidNumber  $2=Service
    echo "checking to see if $1 is running"
    PROCESS=$1  
    REASON="because_process_not_running"
    pgrep mono | grep $1 >/dev/null && echo "Process $PROCESS found" || restart_service $2 $REASON
}

restart_service()
{
    # $1=Service $3=Reason
    echo "restarting $1 $2"
    Pre="Robust"
    Stop="_Stop.sh"
    Start="_Start.sh"
    CommandStop="$Pre$Stop $1"
    CommandStart="$Pre$Start $1" 
    #echo $CommandStop
    #echo $CommandStart
    ./$CommandStop
    ./$CommandStart
}

for Service in "${Services[@]}"
do
    FILEPATH="/tmp/"
    FILENAME="Robust"$Service".pid"
    FILE=$FILEPATH$FILENAME
    check_pid_file_exists $FILE $Service   
done 

Add these script to the local cron jobs

In terminal

crontab -e

add the following line, adjusting the number to the number of mins wanted between each check

*/6 * * * * /home/opensim/SimulatorsMain_KeepAlive.sh
*/5 * * * * /home/opensim/SimulatorsMain_KeepAlive.sh

The timings on these are set to 6 mins and 5 mins. These can be adjusted, but it is a good idea to make this restart time slightly longer than the final shutdown time during a planned restart, in these examples that is set to 5 minutes.

Final Touches – Make sure your scripts are executable.

Use chmod in the same way as above when the first bash shell scripts were created to do this.

chmod 775 fileName.sh
#or do them all with 
chmod 775 *.sh

Related Posts – Automated Opensim Startup and Shutdown

• Set up Ubuntu 20.04 for Opensimulator
• Opensim with multiple Robust services on Ubuntu
• Backup Ubuntu opensim servers
• Securing An Opensim Ubuntu Server