Opensimulator Useful Control Scripts is a collection of helpful BASH/BAT scripts for starting, stopping and running crash recovery. It is part of a larger whole on setting up and maintaining Opensimlator Grids.
TMUX – Versions
Start Everything – Opensimulator Useful Control Scripts
This is the script run when the system starts. Near the top of the script is an array of Simulator groups. Usually, this only contains a single simulator group; however, potentially, it can have many more. There is then an option to start Robust, which is commented out if there is no Robust instance on the server. Similarly, if there are no Simulator instances, leave the array of Simulators empty. The script then launches every simulator in the Simulator groups and Robust using their own scripts.
~/OpenSim/Scripts/OpenSim_Start.sh
#!/usr/bin/env bash cd $HOME/OpenSim/Scripts/ ./MountGoogleDrives.sh # This script assumes the other scripts it calls are in the same folder if they are not please adjust accordingly ############################# ###Start of user variables### SIMULATORGROUPS=( SimulatorsMain ) #list of all simulator names to start up MINSCOUNT=30 #number of mins to wait after simulators start is initiated before restarts begin ###End of user variables### ############################ #only edit below this line if you are sure of what you are doing WINDOW="Terminal" TAIL="_Start.sh" #launch the Robust startup script #no robust on this server comment out #./Robust_Start.sh #sleep 5s #give robust time to start (Should take no noticeable time). #start simulators. #do this for every simulator group listed above for SIMULATORGROUP in "${SIMULATORGROUPS[@]}" do echo "checking for session - $SIMULATOR" #if the session doesn't already exist make it SESSIONEXISTS=$(tmux list-sessions | grep "$SIMULATORGROUP") if [ "$SESSIONEXISTS" = "" ] then echo "Starting session $SIMULATORGROUP" #Create new Session and rename it tmux new-session -d -s "$SIMULATORGROUP" tmux rename-window -t 0 "$WINDOW" else echo "$SIMULATORGROUP already started, skipping" echo "checking for $WINDOW window" #if a terminal window doesn't already exist make one WINDOWEXISTS=$(tmux list-windows -t "$WINDOW" | grep "$SIMULATORGROUP") if [ "$WINDOWEXISTS" = "" ] then echo "Starting window $WINDOW" tmux new-window -t "$SIMULATORGROUP" -n "$WINDOW" fi fi #send the start script command to the terminal window for this session tmux send-keys -t "$SIMULATORGROUP:$WINDOW" "./$SIMULATORGROUP$TAIL" C-m done while [ $MINSCOUNT -ge 1 ] do echo "$MINSCOUNT minutes until simulator restarts begin" sleep 1m (("MINSCOUNT--")) done for SIMULATORGROUP in "${SIMULATORGROUPS[@]}" do RESTARTTAIL="_Restart.sh" #unlike the initial startup, the restarts should be done sequentially to avoid bottlenecks during oar backup. #so the scripts should be called directly, #equates to ./SimulatorMain_Restart.sh if the simulator is called Main ./"$SIMULATORGROUP""$RESTARTTAIL" done
Starting Simulators – Opensimulator Useful Control Scripts
This script is run by the start all script; however, it can also be run independently from the command line. Near the top of the script is an array of Simulators it can start. When run with no parameters, it will start all simulators in the array. Alternatively, if a Simulator name is added to the parameters when the script is run, it will launch that single simulator. Each simulator is started inside its own tmux window.
~/OpenSim/Scripts/SimulatorsMain_Start.sh
#!/usr/bin/env bash #if a name is passed in only restart the individual sim cd $HOME/OpenSim/Scripts if [ "$1" != "" ] then Simulators=( "$1" ) else ####################################### ###Start Of User Changeable Variables## Simulators=( Simulator00 Simulator01 Simulator02 ) #list of all simulators in this group ###End Of User Changeable Variables### ###################################### fi ####################### ###Start User Variables SimulatorGroup="Main" #name of the simulator gorup OpenSimPath="$HOME/OpenSim/Bin" #edit this only if your main Opensim folder is not a subfolder of your users home folder Command="mono --server" #if you're using a desktop pc instead of a server amend this to --desktop IniFilePre="-inifile=$HOME/OpenSim/Settings/Simulators" LogConfigPre="-logconfig=" LogConfigFile="$HOME/OpenSim/Settings/OpenSimLogs.config" LogPath="$HOME/OpenSim/Logs" LogConfig="$LogConfigPre$LogConfigFile" Console="-console=local" ###End User Variables ##################### #Only edit below this line if you are sure about what you are doing File="OpenSim.exe" #name of the file mono runs Session="Simulators$SimulatorGroup" #check the session exits and start it, if it doesnt echo "checking for session - $Session" SessionExists=$(tmux list-sessions | grep $Session) if [ "$SessionExists" = "" ] then echo "Starting session $Session" #Create new Session and rename it tmux new-session -d -s $Session tmux rename-window -t 0 'Terminal' else echo "$Session alredy started, skipping" echo "checking for terminal window" #if a window for this simulator doesn't exist make it Window="Terminal" WindowsExists=$(tmux list-windows -t "$Session" | grep "$Window") if [ "$WindowsExists" = "" ] then echo "creating window $Window in $Session" tmux new-window -t "$Session" -n "$Simulators" else echo "window $Window already exits in $Session, skipping" fi fi for Simulator in "${Simulators[@]}" do echo "checking for window $Simulator in $Session" WindowsExists=$(tmux list-windows -t "$Session" | grep "$Simulator") if [ "$WindowsExists" = "" ] then echo "creating window $Simulator in $Session" tmux new-window -t "$Session" -n "$Simulator" else echo "window $Simulator in $Session already exits" fi echo "starting $Simulator" SimulatorPath="$OpenSimPath" IniFile="$IniFilePre/$Simulator/OpenSim.ini" Execute="$Command $File $IniFile $Console $LogConfig" LogFile="$LogPath/$Simulator.log" tmux send-keys -t "$Session:$Simulator" "export LogFile=$LogFile" C-m tmux send-keys -t "$Session:$Simulator" 'ulimit -s 1048576' C-m tmux send-keys -t "$Session:$Simulator" 'export TERM=xterm' C-m tmux send-keys -t "$Session:$Simulator" "cd $SimulatorPath" C-m tmux send-keys -t "$Session:$Simulator" "$Execute" C-m done
Stopping Simulators – Opensimulator Useful Control Scripts
This script is run by the script that stops everything. It can also be run independently from the terminal. Two parameters can be passed at startup. The first controls the speed of the shutdown process. There are three options available “slow”, “medium” and “fast”. The second parameter is a simulator name. If no simulator name is passed, it will stop all simulators in the Array of Simulators near the top of the script. If no parameters are passed, it will shut down all Simulators slowly. Additionally, while shutting down a simulator, this script sends messages in the world counting down to the shut-off point.
Bash shell script
~/OpenSim/Scripts/SimulatorsMain_Stop.sh
#!/usr/bin/env bash cd $HOME/OpenSim/Scripts #script parameters $1 = fast/other, #script parameters $2 = empty / name of a specific simulator #launch using ./SimulatorMain_Start.sh fast/slow simulatorName/empty SPEED="${1,,}" # first parameter passed into the script, this also converts it to lower case to avoid typing errors. # second parameter passed into the script if [ "$2" != "" ] then WINDOWS=( "$2" ) else ####################################### ###Start Of User Changeable Variables## WINDOWS=( Simulator00 Simulator01 Simulator02 ) ###End Of User Changeable Variables### ###################################### fi ######################################## ###Start Of User Changeable Variables### #The name of this simulator group - change this to match your requirements SESSIONNAME="Main" WARNINGSNUMBER=20 # The number of warnings provided, under a slow stop this is mins, under fast shutdown this is seconds. ###End Of User Changeable Variables### ###################################### #Do not edit below this line unless you are sure what you are doing. SESSION="Simulators""$SESSIONNAME" TIMEUNITS="m" #used to set the timer to minutes TIMEUNITTEXT="minutes" #if fast is passed into the script, set the time to seconds instead of minutes if [ "$SPEED" = "fast" ] then TIMEUNITS="s" #sets the time to seconds TIMEUNITTEXT="seconds" fi if [ "$SPEED" = "medium" ] then WARNINGSNUMBER=120 TIMEUNITS="s" TIMEUNITTEXT="seconds" fi echo "Checking for the session - $SESSION" SESSIONEXISTS=$(tmux list-sessions | grep $SESSION) #only do any of this is the session actually exists, other wise skip and send warning if [ "$SESSIONEXISTS" != "" ] then #every min send a warning to residents about shut down and to operator in terminal echo "Session $SESSION found, starting shutdown" while [ $WARNINGSNUMBER -ge 1 ] do #for every Tmux window (which is really a Simulator, send an inworld message to all avis) for WINDOW in "${WINDOWS[@]}" do #check if the window exits, it it does send the warning WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$WINDOW") if [ "$WINDOWEXISTS" != "" ] then echo "sending $WARNINGSNUMBER $TIMEUNITTEXT warning to $WINDOW" tmux send-keys -t "$SESSION:$WINDOW" "change region root" C-m tmux send-keys -t "$SESSION:$WINDOW" "alert Region Restart in $WARNINGSNUMBER $TIMEUNITTEXT" C-m fi done sleep 1"$TIMEUNITS" (("WARNINGSNUMBER--")) done #call close inside each Tmux Window for WINDOW in "${WINDOWS[@]}" do tmux send-keys -t "$SESSION:$WINDOW" 'quit' C-m echo "Called close on $WINDOW" done #Set time to 1 min if the shutdown type is fast and 5 mins if it is a normal shutdown if [ "$SPEED" = "fast" ] then WARNINGSNUMBER=1 #The shutdown routine of the actual server limits each section to 90 seconds. #20 seconds are used up in customer warnings during fast shut down. This leaves one for the simulator to exit before closing robust else WARNINGSNUMBER=5 fi #Allow adequate time for each window to finish closing, send terminal notifications at regular intervals while [ $WARNINGSNUMBER -ge 1 ] do echo "final shut down in $WARNINGSNUMBER minutes" sleep 1m (("WARNINGSNUMBER--")) done for WINDOW in "${WINDOWS[@]}" do #check if the window exits, it it does tell the simulator to quit WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$WINDOW") if [ "$WINDOWEXISTS" != "" ] then tmux kill-window -t "$SESSION":"$WINDOW" echo "Called close on $WINDOW" fi done else echo "session not found skipping" fi
Restarting Simulators – Opensimulator Useful Control Scripts
This script is run by OpenSim_Start.sh after everything is up and running. Each simulator in the demonstration system runs OAR back of regions periodically. Unfortunately, while very useful, generating OAR files is resource-intensive. Consequently, it is not a good idea to run them at the exact moment. This script restarts all the simulators in the array at the top of the script one by one. Additionally, it adds a delay between each restart. Finally, the script takes parameters for speed.
SimulatorsMain_Restart.sh
#!/usr/bin/env bash SIMULATORGROUP="Main" cd $HOME/OpenSim/Scripts/ if [ $1 != "" ] then SPEED="${1,,}" # first parameter passed into the script, this also converts it to lower case to avoid typing errors. else SPEED="slow" fi if [ "$2" != "" ] then SIMULATORS=( "$2" ) else ####################################### ###Start Of User Changeable Variables## SIMULATORS=( Simulator00 Simulator01 Simulator02 ) ###End Of User Changeable Variables### ###################################### fi if [ $SPEED == fast ] then MINSCOUNT=0 else MINSCOUNT=5 fi count_down_warning() { while [ $MINSCOUNT -ge 1 ] do echo "$MINSCOUNT minutes until next simulator restart begins" sleep 1m #sleep 1s ((MINSCOUNT--)) done } SESSION="Simulators$SIMULATORGROUP" STOP="_Stop.sh" START="_Start.sh" ##Stop each simulator in turn, each simulator has its own built in timer warnings for residents for SIMULATOR in "${SIMULATORS[@]}" do count_down_warning echo "Starting shutdown of $SIMULATOR" bash "$SESSION$STOP" $SPEED "$SIMULATOR" echo "Begin startup of $SIMULATOR" ./$SESSION$START $SIMULATOR done
Simulator Crash Recovery
This script is run periodically by crontab. It cycles through each of the simulators. Firstly it checks to make sure the process is running. Then, assuming the process is running, it checks for responsiveness. Finally, if either of them fails, it shuts the simulator down and restarts it.
~/OpenSim/Scripts/SimulatorsMain_KeepAlive.sh
#!/bin/bash cd $HOME/OpenSim/Scripts SimulatorGroup="Main" Simulators=( Simulator00 Simulator01 Simulator02 ) Port=9000 check_pid_file_exists() { #$1=file $2=Simulator $3=Port echo "checking for $1" if [ -f "$1" ]; then echo "$1 exists " read_pid_file $1 $2 $3 else REASON="because_no_pid_file" #echo "Debug: Restart goes here due to no pid file" restart_simulator $2 $REASON fi } read_pid_file() { #$1=file $2=Simulator $3=Port PID=$(<$1) echo "process number = $PID" check_if_process_running $PID $2 $3 } check_if_process_running() { #$1=pidNumber $2=Simulator $3=Port echo "checking to see if $1 is running" PROCESS=$1 REASON="because_process_not_running" #pgrep mono | grep $1 >/dev/null && echo "Process $PROCESS found" || restart_simulator $2 $REASON pgrep mono | grep $1 >/dev/null && check_if_process_frozen $2 $3 || restart_simulator $2 $REASON } check_if_process_frozen() { # $1=Simulator $2=Port echo "checking to see if $1 on port $2 is frozen" REASON="because_process_frozen" #curl localhost:9001/simstatus/OK "echo process is ok" || restart_simulator $2 $REASON Check="OK" Curl="/usr/bin/curl -s" Address="localhost:$2/simstatus/" Args="-w '%{response_code}'" Status=$(timeout 10s $Curl $Address) #Status=$(timeout 10s $Curl "9001") if [ $Status = "OK" ]; then echo "Simulator: $1 is: $Status" else echo "Simulator: $1 is: Frozen, begin resart" restart_simulator $1 $Reason fi } restart_simulator() { # $1=Simulator $3=Reason echo "restarting $1 $2" Pre="Simulators" Stop="_Stop.sh" Start="_Start.sh" Switch="medium" CommandStop="$Pre$SimulatorGroup$Stop $Switch $1" CommandStart="$Pre$SimulatorGroup$Start $1" ./$CommandStop ./$CommandStart } for Simulator in "${Simulators[@]}" do FILEPATH="/tmp/" FILENAME="$Simulator.pid" FILE=$FILEPATH$FILENAME check_pid_file_exists $FILE $Simulator $Port Port=$(($Port+10)) done
Starting Robust
This script is run by the overall startup script to start the Robust services. Each Instance name is in the array of services near the top of the script. Every instance starts inside its own tmux window.
~/OpenSim/Scripts/Robust_Start.sh
#!/usr/bin/env bash CommandPath="$HOME/OpenSim/Bin/" Command="mono --server" File="Robust.exe" IniFilePre="-inifile=$HOME/OpenSim/Settings/Robust/Robust.HG." IniFileTail=".ini" #RESTCONSOLE="-console rest" LogConfigPre="-logconfig=" LogConfigFile="$HOME/OpenSim/Settings/OpenSimLogs.config" LogPath="$HOME/OpenSim/Logs" LogConfig="$LogConfigPre$LogConfigFile" Console="-console=local" SESSION="Robust" if [ "$1" != "" ] then Services=( "$1" ) else ####################################### ###Start Of User Changeable Variables## Services=( Asset GridUser Main Map ) #list of all simulators in this group ###End Of User Changeable Variables### ###################################### fi echo "checking for session - $SESSION" SESSIONEXISTS=$(tmux list-sessions | grep $SESSION) if [ "$SESSIONEXISTS" = "" ] then echo "Starting session $SESSION" #Create new Session and rename it tmux new-session -d -s $SESSION tmux rename-window -t 0 'Terminal' else echo "$SESSION already started, skipping creation" fi #loops through all instances of robust to start launching each on its its own window for Service in "${Services[@]}" do WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$Service") if [ "$WINDOWEXISTS" = "" ] then echo "creating window $Service in $SESSION" tmux new-window -t "$SESSION" -n "$Service" else echo "window $Service already exits in $SESSION, skipping creation" fi echo "starting $Service" LogFile="$LogPath/Robust$Service.log" Execute="$Command $File $IniFilePre$Service$IniFileTail $Console $LogConfig" echo "Execute Command: $Execute" tmux send-keys -t "$Session:$Simulator" "export LogFile=$LogFile" C-m tmux send-keys -t "$SESSION:$Service" 'ulimit -s 1048576' C-m tmux send-keys -t "$SESSION:$Service" 'export TERM=xterm' C-m tmux send-keys -t "$SESSION:$Service" "cd $CommandPath" C-m #tmux send-keys -t "$SESSION:$Service" "$Execute $RESTCONSOLE" C-m tmux send-keys -t "$SESSION:$Service" "$Execute" C-m done
Stopping Robust
This script is run when the system is shutting down, cleaning stopping each robust instance in the array at the top.
~/OpenSim/Scripts/Robust_Stop.sh
#!/usr/bin/env bash SESSION="Robust" if [ "$1" != "" ] then Services=( "$1" ) else ####################################### ###Start Of User Changeable Variables## Services=( Asset GridUser Main Map ) #list of all simulators in this group ###End Of User Changeable Variables### ###################################### fi echo "checking for session - $SESSION" SESSIONEXISTS=$(tmux list-sessions | grep $SESSION) #only do any of this is the session actually exists, other wise skip and send warning if [ "$SESSIONEXISTS" != "" ] then #call close inside each Tmux Window for Service in "${Services[@]}" do #check if the window exits, it it does send shutdown WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$Service") if [ "$WINDOWEXISTS" != "" ] then tmux send-keys -t "$SESSION:$Service" 'quit' C-m echo "Called close on $Service" fi done #Allow adequate time for each window to finish closing, send terminal notifications at regular intervals echo "final shut down in 5s." sleep 5s for Service in "${Services[@]}" do #check if the window exits, it it does tell the simulator to quit WINDOWEXISTS=$(tmux list-windows -t "$SESSION" | grep "$Service") if [ "$WINDOWEXISTS" != "" ] then tmux kill-window -t "$SESSION":"$Service" echo "Called close on $Service" fi done else echo "session not found skipping" fi
Restarting Robust
This script is run by crontab once a day to restart all robust instances.
~/OpenSim/Scripts/Robust_Restart.sh
#!/usr/bin/env bash ./Robust_Stop.sh ./Robust_Start.sh
Robust Crash Recovery
#!/bin/bash Services=( Asset GridUser Inventory Main Map ) check_pid_file_exists() { #$1=file $2=Simulator echo "checking for $1" if [ -f "$1" ]; then echo "$1 exists " read_pid_file $1 $2 else REASON="because_no_pid_file" restart_service $2 $REASON fi } read_pid_file() { #$1=file $2=Simulator PID=$(<$1) echo "process number = $PID" check_if_process_running $PID $2 } check_if_process_running() { #$1=pidNumber $2=Service echo "checking to see if $1 is running" PROCESS=$1 REASON="because_process_not_running" pgrep mono | grep $1 >/dev/null && echo "Process $PROCESS found" || restart_service $2 $REASON } restart_service() { # $1=Service $3=Reason echo "restarting $1 $2" Pre="Robust" Stop="_Stop.sh" Start="_Start.sh" CommandStop="$Pre$Stop $1" CommandStart="$Pre$Start $1" #echo $CommandStop #echo $CommandStart ./$CommandStop ./$CommandStart } for Service in "${Services[@]}" do FILEPATH="/tmp/" FILENAME="Robust"$Service".pid" FILE=$FILEPATH$FILENAME check_pid_file_exists $FILE $Service done
Stop Everything – Opensimulator Useful Control Scripts
This script is run by SystemD when the server is shutting down. It stops all the simulators and Robust cleanly before a server shuts down.
~/OpenSim/Scripts/OpenSim_Stop.sh
#!/usr/bin/env bash #this script assumes the scripts it executes are in the same folder as this script. #if your setup is different this script will require modification. cd $HOME/OpenSim/Scripts/ ########################## ###Start Of User Variables #Simulator Sessions robust is dealt with later SESSIONS=( Simulators ) ###End Of User Variables### ########################### PRE="./" TAIL="_Stop.sh fast" WINDOW="Terminal" #For every session listed above, send the execute the simulator group shutdown script inside the terminal window of that session. #This allows all sessions to be shutdown at the same time. for SESSION in "${SESSIONS[@]}" do ##loop through the sessions, checking they exist, SESSIONEXISTS=$(tmux list-sessions | grep "$SESSION") if [ "$SESSIONEXISTS" != "" ] then #if they do then send the shutdown keys to the terminal window for that session tmux send-keys -t "$SESSION"":""$WINDOW" "$PRE""$SESSION""$TAIL" fi done #stop robust now ./Robust_Stop.sh #wait for all mono process to close before exit while pgrep -x mono >/dev/null; do sleep 1s echo "shutting down, please wait" done echo "shut down compete"
Start Opensim with the server – SystemD Service file
[Unit] Description= OpenSimulator Inside Tmux Documentation=man:tmux(1) After=syslog.target network.target ufw.service mysql.service cron.service [Service] Type=simple User=opensim Group=opensim WorkingDirectory=/home/opensim/OpenSim/Scripts ExecStart=/home/opensim/OpenSim/Scripts/OpenSim_Start.sh ExecStop=/home/opensim/OpenSim/Scripts/OpenSim_Stop_.sh fast RemainAfterExit=yes KillMode=none Environment=USER=opensim HOME=/home/opensim [Install] WantedBy=multi-user.target
Save the above script using root permissions to
/etc/systemd/system/opensimulator.service
Finally, start the service and enable it, so it runs when the server starts.
sudo systemctl start opensimulator.service sudo systemctl enable opensimulator.service
Screen Versions
Coming soon
Windows Versions
Coming soon.
Hello Sara,
I have just built a grid using your instructions, and I must say they are very fine, love the video, screamed at the screen… Nooo it is 5007 not 007 etc. All is good except for one issue. OpenSim seems to insist that the Region.ini file come from Bin/Regions (./Regions). I tried every which way to convince it otherwise, even trying. .xml version, nope it was stubborn. So in the end I added a couple of line to the Startup Scrip and OpenSim is now happy. The work around is :-
rm $HOME/OpenSim/Bin/Regions/Regions.ini
cp -a $HOME/OpenSim/Settings/Simulators/$Simulator/Regions/Regions.ini $HOME/OpenSim/Bin/Regions/Regions.ini
Run just after echo “starting $Simulator”
So I bung the Sim copy into the standard location before the sim starts.
Is there a better way?
Cheers From Western Australia Grid.