Opensimulator Logo - Opensimulator Useful Control Scripts
Opensimulator Logo

Opensimulator Useful Control Scripts is a collection of helpful BASH/BAT scripts for starting, stopping and running crash recovery. It is part of a larger whole on setting up and maintaining Opensimlator Grids.

Full Guide

TMUX – Versions

Start Everything – Opensimulator Useful Control Scripts

This is the script run when the system starts. Near the top of the script is an array of Simulator groups. Usually, this only contains a single simulator group; however, potentially, it can have many more. There is then an option to start Robust, which is commented out if there is no Robust instance on the server. Similarly, if there are no Simulator instances, leave the array of Simulators empty. The script then launches every simulator in the Simulator groups and Robust using their own scripts.

~/OpenSim/Scripts/OpenSim_Start.sh
#!/usr/bin/env bash
cd $HOME/OpenSim/Scripts/
./MountGoogleDrives.sh
# This script assumes the other scripts it calls are in the same folder if they are not please adjust accordingly
#############################
###Start of user variables###
SIMULATORGROUPS=( SimulatorsMain ) #list of all simulator names to start up
MINSCOUNT=30 #number of mins to wait after simulators start is initiated before restarts begin
###End of user variables###
############################
#only edit below this line if you are sure of what you are doing
WINDOW="Terminal"
TAIL="_Start.sh"

#launch the Robust startup script
#no robust on this server comment out
#./Robust_Start.sh
#sleep 5s #give robust time to start (Should take no noticeable time).

#start simulators. 
#do this for every simulator group listed above
for SIMULATORGROUP in "${SIMULATORGROUPS[@]}"
do
    echo "checking for session - $SIMULATOR"
    #if the session doesn't already exist make it
    SESSIONEXISTS=$(tmux list-sessions | grep "$SIMULATORGROUP")
    if [ "$SESSIONEXISTS" = "" ]
    then
        echo "Starting session $SIMULATORGROUP"
        #Create new Session and rename it
        tmux new-session -d -s "$SIMULATORGROUP"
        tmux rename-window -t 0 "$WINDOW"
    else
        echo "$SIMULATORGROUP already started, skipping"
        echo "checking for $WINDOW window"
        #if a terminal window doesn't already exist make one
        WINDOWEXISTS=$(tmux list-windows -t "$WINDOW" | grep "$SIMULATORGROUP")    
        if [ "$WINDOWEXISTS" = "" ]
        then
            echo "Starting window $WINDOW"
            tmux new-window -t "$SIMULATORGROUP" -n "$WINDOW"
        fi  
    fi
    #send the start script command to the terminal window for this session
    tmux send-keys -t "$SIMULATORGROUP:$WINDOW" "./$SIMULATORGROUP$TAIL" C-m
done
while [ $MINSCOUNT -ge 1  ]
do
      echo "$MINSCOUNT minutes until simulator restarts begin"
    sleep 1m
    (("MINSCOUNT--"))
done

for SIMULATORGROUP in "${SIMULATORGROUPS[@]}"
do
    RESTARTTAIL="_Restart.sh"
    #unlike the initial startup, the restarts should be done sequentially to avoid bottlenecks during oar backup.
    #so the scripts should be called directly, 
    #equates to ./SimulatorMain_Restart.sh if the simulator is called Main
    ./"$SIMULATORGROUP""$RESTARTTAIL"
done

Starting Simulators – Opensimulator Useful Control Scripts

This script is run by the start all script; however, it can also be run independently from the command line. Near the top of the script is an array of Simulators it can start. When run with no parameters, it will start all simulators in the array. Alternatively, if a Simulator name is added to the parameters when the script is run, it will launch that single simulator. Each simulator is started inside its own tmux window.

~/OpenSim/Scripts/SimulatorsMain_Start.sh
#!/usr/bin/env bash
#if a name is passed in only restart the individual sim

cd $HOME/OpenSim/Scripts

if [ "$1" != "" ]
then
    Simulators=( "$1" )
else
#######################################
###Start Of User Changeable Variables##
Simulators=( Simulator00 Simulator01 Simulator02 ) #list of all simulators in this group
###End Of User Changeable Variables###
######################################
fi

#######################
###Start User Variables
SimulatorGroup="Main" #name of the simulator gorup
OpenSimPath="$HOME/OpenSim/Bin" #edit this only if your main Opensim folder is not a subfolder of your users home folder
Command="mono --server"  #if you're using a desktop pc instead of a server amend this to --desktop
IniFilePre="-inifile=$HOME/OpenSim/Settings/Simulators"
LogConfigPre="-logconfig="
LogConfigFile="$HOME/OpenSim/Settings/OpenSimLogs.config"
LogPath="$HOME/OpenSim/Logs"
LogConfig="$LogConfigPre$LogConfigFile"
Console="-console=local"

###End User Variables
#####################
#Only edit below this line if you are sure about what you are doing
File="OpenSim.exe" #name of the file mono runs
Session="Simulators$SimulatorGroup" 
#check the session exits and start it, if it doesnt
echo "checking for session - $Session"
SessionExists=$(tmux list-sessions | grep $Session)
if [ "$SessionExists" = "" ]
then
    echo "Starting session $Session"
    #Create new Session and rename it
    tmux new-session -d -s $Session
    tmux rename-window -t 0 'Terminal'
else
    echo "$Session alredy started, skipping"
    echo "checking for terminal window"
    #if a window for this simulator doesn't exist make it
    Window="Terminal"
    WindowsExists=$(tmux list-windows -t "$Session" | grep "$Window")
    if [ "$WindowsExists" = "" ]
    then
        echo "creating window $Window in $Session"
        tmux new-window -t "$Session" -n "$Simulators"
    else
        echo "window $Window already exits in $Session, skipping"
    fi
fi
for Simulator in "${Simulators[@]}"
do
    echo "checking for window $Simulator in $Session"
    WindowsExists=$(tmux list-windows -t "$Session" | grep "$Simulator")
    if [ "$WindowsExists" = "" ]
    then
        echo "creating window $Simulator in $Session"
        tmux new-window -t "$Session" -n "$Simulator"
    else
        echo "window $Simulator in $Session already exits"
    fi
    echo "starting $Simulator"
    SimulatorPath="$OpenSimPath"
    IniFile="$IniFilePre/$Simulator/OpenSim.ini"
    Execute="$Command $File $IniFile $Console $LogConfig"
    LogFile="$LogPath/$Simulator.log"
    tmux send-keys -t "$Session:$Simulator" "export LogFile=$LogFile" C-m
    tmux send-keys -t "$Session:$Simulator" 'ulimit -s 1048576' C-m
    tmux send-keys -t "$Session:$Simulator" 'export TERM=xterm' C-m
    tmux send-keys -t "$Session:$Simulator" "cd $SimulatorPath" C-m
    tmux send-keys -t "$Session:$Simulator" "$Execute" C-m
done

Stopping Simulators – Opensimulator Useful Control Scripts

This script is run by the script that stops everything. It can also be run independently from the terminal. Two parameters can be passed at startup. The first controls the speed of the shutdown process. There are three options available “slow”, “medium” and “fast”. The second parameter is a simulator name. If no simulator name is passed, it will stop all simulators in the Array of Simulators near the top of the script. If no parameters are passed, it will shut down all Simulators slowly. Additionally, while shutting down a simulator, this script sends messages in the world counting down to the shut-off point.

Bash shell script

~/OpenSim/Scripts/SimulatorsMain_Stop.sh
#!/usr/bin/env bash
cd $HOME/OpenSim/Scripts
#script parameters $1 = fast/other,
#script parameters $2 = empty / name of a specific simulator
#launch using ./SimulatorMain_Start.sh fast/slow simulatorName/empty
SPEED="${1,,}" # first parameter passed into the script, this also converts it to lower case to avoid typing errors.
# second parameter passed into the script
if [ "$2" != "" ]
then
    WINDOWS=( "$2" )
else
    #######################################
    ###Start Of User Changeable Variables##
    WINDOWS=( Simulator00 Simulator01 Simulator02 )
    ###End Of User Changeable Variables###
    ######################################
fi
########################################
###Start Of User Changeable Variables###
#The name of this simulator group - change this to match your requirements
SESSIONNAME="Main"
WARNINGSNUMBER=20 # The number of warnings provided, under a slow stop this is mins, under fast shutdown this is seconds.
###End Of User Changeable Variables###
######################################
#Do not edit below this line unless you are sure what you are doing.
SESSION="Simulators""$SESSIONNAME" 

TIMEUNITS="m" #used to set the timer to minutes
TIMEUNITTEXT="minutes"

#if fast is passed into the script, set the time to seconds instead of minutes
if [ "$SPEED" = "fast" ]
then
    TIMEUNITS="s" #sets the time to seconds
    TIMEUNITTEXT="seconds"
fi

if [ "$SPEED" = "medium" ]
then
    WARNINGSNUMBER=120
    TIMEUNITS="s"
    TIMEUNITTEXT="seconds"
fi

echo "Checking for the session - $SESSION"
SESSIONEXISTS=$(tmux list-sessions | grep $SESSION)
#only do any of this is the session actually exists, other wise skip and send warning
if [ "$SESSIONEXISTS" != "" ] 
then        
    #every min send a warning to residents about shut down and to operator in terminal
    echo "Session $SESSION found, starting shutdown"
    while [ $WARNINGSNUMBER -ge 1  ]
    do
        #for every Tmux window (which is really a Simulator, send an inworld message to all avis)
        for WINDOW in "${WINDOWS[@]}"
        do
            #check if the window exits, it it does send the warning
            WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$WINDOW")
            if [ "$WINDOWEXISTS" != "" ]
            then
                echo "sending $WARNINGSNUMBER $TIMEUNITTEXT warning to $WINDOW"
                tmux send-keys -t "$SESSION:$WINDOW" "change region root" C-m
                tmux send-keys -t "$SESSION:$WINDOW" "alert Region Restart in $WARNINGSNUMBER $TIMEUNITTEXT" C-m
            fi    
        done
        sleep 1"$TIMEUNITS"
        (("WARNINGSNUMBER--"))
    done
    
    #call close inside each Tmux Window
    for WINDOW in "${WINDOWS[@]}"
    do
        tmux send-keys -t "$SESSION:$WINDOW" 'quit' C-m
        echo "Called close on $WINDOW"
    done
    
    #Set time to 1 min if the shutdown type is fast and 5 mins if it is a normal shutdown
    if [ "$SPEED" = "fast" ]
    then
        WARNINGSNUMBER=1 
        #The shutdown routine of the actual server limits each section to 90 seconds. 
        #20 seconds are used up in customer warnings during fast shut down. This leaves one for the simulator to exit before closing robust
    else
        WARNINGSNUMBER=5
    fi
    #Allow adequate time for each window to finish closing, send terminal notifications at regular intervals
    while [ $WARNINGSNUMBER -ge 1  ]
    do
        echo "final shut down in $WARNINGSNUMBER minutes"
        sleep 1m
        (("WARNINGSNUMBER--"))
    done
      
    for WINDOW in "${WINDOWS[@]}"
    do
        #check if the window exits, it it does tell the simulator to quit
        WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$WINDOW")
        if [ "$WINDOWEXISTS" != "" ]
        then
            tmux kill-window -t "$SESSION":"$WINDOW"
            echo "Called close on $WINDOW"
        fi   
    done      
else
   echo "session not found skipping"
fi

Restarting Simulators – Opensimulator Useful Control Scripts

This script is run by OpenSim_Start.sh after everything is up and running. Each simulator in the demonstration system runs OAR back of regions periodically. Unfortunately, while very useful, generating OAR files is resource-intensive. Consequently, it is not a good idea to run them at the exact moment. This script restarts all the simulators in the array at the top of the script one by one. Additionally, it adds a delay between each restart. Finally, the script takes parameters for speed.

SimulatorsMain_Restart.sh
#!/usr/bin/env bash
SIMULATORGROUP="Main"

cd $HOME/OpenSim/Scripts/

if [ $1 != "" ]
then
    SPEED="${1,,}" # first parameter passed into the script, this also converts it to lower case to avoid typing errors.
else
    SPEED="slow"
fi


if [ "$2" != "" ]
then
    SIMULATORS=( "$2" )
else
    #######################################
    ###Start Of User Changeable Variables##
    SIMULATORS=( Simulator00 Simulator01 Simulator02 )
    ###End Of User Changeable Variables###
    ######################################
fi

if [ $SPEED == fast ]
then
    MINSCOUNT=0
else
    MINSCOUNT=5
fi

count_down_warning()
{
    
    while [ $MINSCOUNT -ge 1  ]
    do
        echo "$MINSCOUNT minutes until next simulator restart begins"
        sleep 1m
        #sleep 1s
        ((MINSCOUNT--))
    done
}
SESSION="Simulators$SIMULATORGROUP"
STOP="_Stop.sh"
START="_Start.sh"
##Stop each simulator in turn, each simulator has its own built in timer warnings for residents
for SIMULATOR in "${SIMULATORS[@]}"
do
    count_down_warning
    echo "Starting shutdown of $SIMULATOR"
    bash "$SESSION$STOP" $SPEED "$SIMULATOR"
    echo "Begin startup of $SIMULATOR"
    ./$SESSION$START $SIMULATOR
done

Simulator Crash Recovery

This script is run periodically by crontab. It cycles through each of the simulators. Firstly it checks to make sure the process is running. Then, assuming the process is running, it checks for responsiveness. Finally, if either of them fails, it shuts the simulator down and restarts it.

~/OpenSim/Scripts/SimulatorsMain_KeepAlive.sh
#!/bin/bash
cd $HOME/OpenSim/Scripts

SimulatorGroup="Main"
Simulators=( Simulator00 Simulator01 Simulator02 )
Port=9000

check_pid_file_exists()
{
    #$1=file $2=Simulator $3=Port
    echo "checking for $1"
    if [ -f "$1" ]; then
        echo "$1 exists "
        read_pid_file $1 $2 $3
    else 
        REASON="because_no_pid_file"
        #echo "Debug: Restart goes here due to no pid file"
        restart_simulator $2 $REASON
    fi
}

read_pid_file()
{
    #$1=file $2=Simulator $3=Port
    PID=$(<$1)
    echo "process number = $PID"
    check_if_process_running $PID $2 $3
}

check_if_process_running()
{
    #$1=pidNumber  $2=Simulator $3=Port
    echo "checking to see if $1 is running"
    PROCESS=$1  
    REASON="because_process_not_running"
    #pgrep mono | grep $1 >/dev/null && echo "Process $PROCESS found" || restart_simulator $2 $REASON
    pgrep mono | grep $1 >/dev/null && check_if_process_frozen $2 $3 || restart_simulator $2 $REASON
}

check_if_process_frozen()
{
    # $1=Simulator $2=Port
    echo "checking to see if $1 on port $2 is frozen"
    REASON="because_process_frozen"
    #curl localhost:9001/simstatus/OK "echo process is ok" || restart_simulator $2 $REASON
    Check="OK"
    Curl="/usr/bin/curl -s"
    Address="localhost:$2/simstatus/"
    Args="-w '%{response_code}'"
    Status=$(timeout 10s $Curl $Address)
    #Status=$(timeout 10s $Curl "9001")
    if [ $Status = "OK" ];
        then
        echo "Simulator: $1 is: $Status"    
    else
        echo "Simulator: $1 is: Frozen, begin resart"
        restart_simulator $1 $Reason
    fi
}

restart_simulator()
{
    # $1=Simulator $3=Reason
    echo "restarting $1 $2"
    Pre="Simulators"
    Stop="_Stop.sh"
    Start="_Start.sh"
    Switch="medium"
    CommandStop="$Pre$SimulatorGroup$Stop $Switch $1"
    CommandStart="$Pre$SimulatorGroup$Start $1" 
    ./$CommandStop
    ./$CommandStart
}

for Simulator in "${Simulators[@]}"
do
    FILEPATH="/tmp/"
    FILENAME="$Simulator.pid"
    FILE=$FILEPATH$FILENAME
    check_pid_file_exists $FILE $Simulator $Port
    Port=$(($Port+10))
done

Starting Robust

This script is run by the overall startup script to start the Robust services. Each Instance name is in the array of services near the top of the script. Every instance starts inside its own tmux window.

~/OpenSim/Scripts/Robust_Start.sh
#!/usr/bin/env bash
CommandPath="$HOME/OpenSim/Bin/"
Command="mono --server" 
File="Robust.exe"
IniFilePre="-inifile=$HOME/OpenSim/Settings/Robust/Robust.HG."
IniFileTail=".ini"
#RESTCONSOLE="-console rest"
LogConfigPre="-logconfig="
LogConfigFile="$HOME/OpenSim/Settings/OpenSimLogs.config"
LogPath="$HOME/OpenSim/Logs"
LogConfig="$LogConfigPre$LogConfigFile"
Console="-console=local"

SESSION="Robust"

if [ "$1" != "" ]
then
    Services=( "$1" )
else
    #######################################
    ###Start Of User Changeable Variables##
    Services=( Asset GridUser Main Map ) #list of all simulators in this group
    ###End Of User Changeable Variables###
    ######################################
fi

echo "checking for session - $SESSION"
SESSIONEXISTS=$(tmux list-sessions | grep $SESSION)
if [ "$SESSIONEXISTS" = "" ]
then
    echo "Starting session $SESSION"
    #Create new Session and rename it
    tmux new-session -d -s $SESSION
    tmux rename-window -t 0 'Terminal'
else
    echo "$SESSION already started, skipping creation"
fi
#loops through all instances of robust to start launching each on its its own window
for Service in "${Services[@]}"
do
    WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$Service")
    if [ "$WINDOWEXISTS" = "" ]
    then
        echo "creating window $Service in $SESSION"
        tmux new-window -t "$SESSION" -n "$Service"
    else
        echo "window $Service already exits in $SESSION, skipping creation"
    fi
    echo "starting $Service"
    LogFile="$LogPath/Robust$Service.log"
    Execute="$Command $File $IniFilePre$Service$IniFileTail $Console $LogConfig"
    echo "Execute Command: $Execute"
    tmux send-keys -t "$Session:$Simulator" "export LogFile=$LogFile" C-m
    tmux send-keys -t "$SESSION:$Service" 'ulimit -s 1048576' C-m
    tmux send-keys -t "$SESSION:$Service" 'export TERM=xterm' C-m
    tmux send-keys -t "$SESSION:$Service" "cd $CommandPath" C-m
    #tmux send-keys -t "$SESSION:$Service" "$Execute $RESTCONSOLE" C-m    
    tmux send-keys -t "$SESSION:$Service" "$Execute" C-m    
done

Stopping Robust

This script is run when the system is shutting down, cleaning stopping each robust instance in the array at the top.

~/OpenSim/Scripts/Robust_Stop.sh
#!/usr/bin/env bash
SESSION="Robust"
if [ "$1" != "" ]
then
    Services=( "$1" )
else
    #######################################
    ###Start Of User Changeable Variables##
    Services=( Asset GridUser Main Map ) #list of all simulators in this group
    ###End Of User Changeable Variables###
    ######################################
fi

echo "checking for session - $SESSION"
SESSIONEXISTS=$(tmux list-sessions | grep $SESSION)
#only do any of this is the session actually exists, other wise skip and send warning
if [ "$SESSIONEXISTS" != "" ] 
then            
    #call close inside each Tmux Window
    for Service in "${Services[@]}"
    do
        #check if the window exits, it it does send shutdown
        WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$Service")
            if [ "$WINDOWEXISTS" != "" ]
            then
                tmux send-keys -t "$SESSION:$Service" 'quit' C-m
                echo "Called close on $Service"
            fi
    done
    
    #Allow adequate time for each window to finish closing, send terminal notifications at regular intervals
    echo "final shut down in 5s."
    sleep 5s
    
    for Service in "${Services[@]}"
    do
        #check if the window exits, it it does tell the simulator to quit
        WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$Service")
        if [ "$WINDOWEXISTS" != "" ]
        then
            tmux kill-window -t "$SESSION":"$Service"
            echo "Called close on $Service"
        fi   
    done        
else
   echo "session not found skipping"
fi

Restarting Robust

This script is run by crontab once a day to restart all robust instances.

~/OpenSim/Scripts/Robust_Restart.sh
#!/usr/bin/env bash
./Robust_Stop.sh
./Robust_Start.sh

Robust Crash Recovery

#!/bin/bash

Services=( Asset GridUser Inventory Main Map )

check_pid_file_exists()
{
    #$1=file $2=Simulator
    echo "checking for $1"
    if [ -f "$1" ]; then
        echo "$1 exists "
        read_pid_file $1 $2
    else 
        REASON="because_no_pid_file"
        restart_service $2 $REASON
    fi
}

read_pid_file()
{
    #$1=file $2=Simulator
    PID=$(<$1)
    echo "process number = $PID"
    check_if_process_running $PID $2
}

check_if_process_running()
{
    #$1=pidNumber  $2=Service
    echo "checking to see if $1 is running"
    PROCESS=$1  
    REASON="because_process_not_running"
    pgrep mono | grep $1 >/dev/null && echo "Process $PROCESS found" || restart_service $2 $REASON
}

restart_service()
{
    # $1=Service $3=Reason
    echo "restarting $1 $2"
    Pre="Robust"
    Stop="_Stop.sh"
    Start="_Start.sh"
    CommandStop="$Pre$Stop $1"
    CommandStart="$Pre$Start $1" 
    #echo $CommandStop
    #echo $CommandStart
    ./$CommandStop
    ./$CommandStart
}

for Service in "${Services[@]}"
do
    FILEPATH="/tmp/"
    FILENAME="Robust"$Service".pid"
    FILE=$FILEPATH$FILENAME
    check_pid_file_exists $FILE $Service   
done 

Stop Everything – Opensimulator Useful Control Scripts

This script is run by SystemD when the server is shutting down. It stops all the simulators and Robust cleanly before a server shuts down.

~/OpenSim/Scripts/OpenSim_Stop.sh
#!/usr/bin/env bash
#this script assumes the scripts it executes are in the same folder as this script.
#if your setup is different this script will require modification.
cd $HOME/OpenSim/Scripts/
##########################
###Start Of User Variables
#Simulator Sessions robust is dealt with later
SESSIONS=( Simulators )
###End Of User Variables###
###########################
PRE="./"
TAIL="_Stop.sh fast"
WINDOW="Terminal"
#For every session listed above, send the execute the simulator group shutdown script inside the terminal window of that session.
#This allows all sessions to be shutdown at the same time.
for SESSION in "${SESSIONS[@]}"
do
        ##loop through the sessions, checking they exist, 
        SESSIONEXISTS=$(tmux list-sessions | grep "$SESSION")
        if [ "$SESSIONEXISTS" != "" ]
        then
                #if they do then send the shutdown keys to the terminal window for that session
                tmux send-keys -t "$SESSION"":""$WINDOW" "$PRE""$SESSION""$TAIL"
        fi
done
#stop robust now
./Robust_Stop.sh
#wait for all mono process to close before exit
while pgrep -x mono >/dev/null;
do
        sleep 1s
        echo "shutting down, please wait"
done
echo "shut down compete" 

Start Opensim with the server – SystemD Service file

[Unit]
Description= OpenSimulator Inside Tmux
Documentation=man:tmux(1)
After=syslog.target network.target ufw.service mysql.service cron.service
[Service]
Type=simple
User=opensim
Group=opensim
WorkingDirectory=/home/opensim/OpenSim/Scripts
ExecStart=/home/opensim/OpenSim/Scripts/OpenSim_Start.sh
ExecStop=/home/opensim/OpenSim/Scripts/OpenSim_Stop_.sh fast
RemainAfterExit=yes
KillMode=none
Environment=USER=opensim  HOME=/home/opensim
[Install]
WantedBy=multi-user.target

Save the above script using root permissions to

/etc/systemd/system/opensimulator.service

Finally, start the service and enable it, so it runs when the server starts.

sudo systemctl start opensimulator.service
sudo systemctl enable opensimulator.service

Screen Versions

Coming soon

Windows Versions

Coming soon.

Related Articles

This article has 1 comments

  1. Pingback: Opensim Server Setup Using Ubuntu & Windows. - Sara Payne's BlogSara Payne's Blog

Leave a Comment

Your email address will not be published. Required fields are marked *