Creating an LXC Containerized Selenium Grid

2015-12-08T20:30:00Z

This guide shows you how to build one or more self-contained Selenium grids using:

The approach used here has been inspired by Dimitri Baeli's Selenium in the Sky with LXC article on the comparethemarket.com's tech blog. In this installation, the intent is providing developers fast feedback from a suite of Selenium Browser/API tests incorporated into a Continuous Integration/Deployment process. Thus we are provisioning Selenium grids for use in a private network. If you intend hosting a similar solution on cloud based infrastructure, you will need to take the necessary steps to secure communications and harden the operating systems.

I have opted to run multiple self-contained Selenium Grid's. Each Grid runs on one physical server, with host operating system running a Selenium hub. Each server will then run multiple Selenium node instances, each of which will be running in their own LXC container.

Throughout this guide, "sghost0" is used to refer to the Selenium grid host server we're building. You maybe building multiple Selenium grid host servers, in which case they are referred to as "sghost1", "sghost2" etc.

This guide assumes you have a cleanly installed Ubuntu Server 14.04.3 host with OpenSSH installed. You will need root privileges either directly or via sudo. It's also assumed that you have a working DNS integrated DHCP service.

Setup the LXC / selenium hub host (sghost0)

Set-up Bridged Networking

Install bridge-utils

$ sudo apt-get install -y bridge-utils

Edit the /etc/network/interfaces configuration file. We will use a dynamic IP for now.

$ sudo vi /etc/network/interfaces

Comment out the current primary network interface; change

auto eth0
iface eth0 inet dhcp

to

#auto eth0
#iface eth0 inet dhcp

And then add:

auto br0
iface br0 inet dhcp
    bridge_ports eth0
    bridge_fd 0
    bridge_maxwait 0

Implement the change

$ sudo ifdown eth0 ; sudo ifup br0

Install and configure GlusterFS

This is an optional step - if you are setting up a single Selenium grid host, then you can skip this section. If you are planning to run two, three or four Selenium Grids, then using a GlusterFS volume for synchronising the Selenium log and HTTP Archive files can be beneficial. This configuration has not been tested with more than four Selenium Grids - I suspect that increase disk and network I/O could start to become a concern with more grids.

You might be asking why not run one "grid" comprising a hub on one host and nodes in LXC containers on all three/four hosts? I've found that a "distributed grid" results in tests that take much longer to run because they get delayed by the cross-network communcations between the hub and node Selenium instances. It's better to ensure all the communication between the hub and the node remains on the host. The Linux kernel should ensure that network traffic between the host and the bridged LXC containers does not move onto the network. Our development team have produced a wrapper for the Selenium Web Driver used to run the tests which means they can be run against multiple grids in parellel.

Install $ sudo apt-get install glusterfs-common glusterfs-server glusterfs-client attr

Choose one Selenium grid host to be your primary GlusterFS node (e.g. "sghost0"). On that host run the following once for each other host:

$ sudo gluster peer probe <hostname>.yourdomain.yourtld

For example:

$ sudo gluster peer probe sghost1.yourdomain.yourtld
$ sudo gluster peer probe sghost2.yourdomain.yourtld

On each host, create the folders needed for Gluster fs

$ sudo mkdir /media/gluster-sel-har
$ sudo mkdir -p /var/log/selenium/har
$ sudo chmod -R 777 /media/gluster-sel-har
$ sudo chmod -R 777 /var/log/selenium

Back to the primary GlusterFS, "sghost0". Create the GlusterFS distributed volume and set the necessary properties. Replace "replica 3" with the appropriate value for actual number of Selenium grid hosts you are planning to use. $ sudo gluster volume create SelHarVol replica 3 transport tcp sghost0.yourdomain.yourtld:/media/gluster-sel-har sghost1.yourdomain.yourtld:/media/gluster-sel-har sghost2.yourdomain.yourtld::/media/gluster-sel-har sghost0.yourdomain.yourtld:/media/gluster-sel-har force

$ sudo gluster volume set SelHarVol server.allow-insecure on
$ sudo gluster volume set SelHarVol auth.allow "*"  

$ sudo gluster volume start SelHarVol

#add to fstab on each host
$ localhost:/SelHarVol    /var/log/selenium/har   glusterfs  defaults,nobootwait,_netdev,fetch-attempts=10 0 2  
$ sudo mount -a

# Verify gluster volumes are running with bricks on each host
$ sudo gluster volume info all

Volume Name: SelHarVol
Type: Replicate
Volume ID: 3553fcf7-cf6f-49ee-8c15-e7e02a9309b7
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: sghost0.yourdomain.yourtld:/media/gluster-sel-har
Brick2: sghost1.yourdomain.yourtld:/media/gluster-sel-har
Brick3: sghost2.yourdomain.yourtld:/media/gluster-sel-har
Options Reconfigured:
server.allow-insecure: on
auth.allow: *

Install and Configure samba

Install samba $ sudo apt-get install -y samba

Edit the configuration $ sudo vi /etc/samba/smb.conf

Replace the contents of the file with the following: [global] workgroup = YOURDOMAIN.YOURTLD # or "WORKGROUP" if you don't have a domain server string = %h server wins support = no dns proxy = no log file = /var/log/samba/log.%m max log size = 1000 panic action = /usr/share/samba/panic-action %d server role = standalone server encrypt passwords = true passdb backend = tdbsam unix password sync = no map to guest = bad user

[selenium]
    path = /var/log/selenium
    browseable = yes
    writeable = no
    guest ok = yes

[firebug]
    path = /var/log/selenium/har
    browseable = yes
    writeable = yes
    guest ok = yes

Restart samba

$ sudo service smbd restart
$ sudo service nmbd restart

Install Java

Install Oracle/SUN Java. In my experience, Selenium does not play too well with openJDK

$ sudo apt-get install software-properties-common # Only required for Ubuntu Minimal installations
$ sudo apt-add-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install -y oracle-java7-installer

In the last step you will need to confirm you are happy to accept the end user license agreement.

Install and configure Selenium server on the host

If you skipped installing and configuring GlusterFS, then create a folder for the selenium log files and make it world writeable.

$ sudo mkdir -p /var/log/selenium/har
$ sudo chmod -R 777 /var/log/selenium

Create a folder for the selenium-server jar file

$ sudo mkdir /opt/selenium-server
$ cd /opt/selenium-server

Download Selenium server

$ sudo wget http://selenium-release.storage.googleapis.com/2.48/selenium-server-standalone-2.48.2.jar

Create a shell script to start a Selenium node instance within the LXC containers

$ sudo vi start-selenium.sh

Add the following lines

#!/bin/bash
portNumber=$((5555 + $(echo -n $(hostname | grep -o -P '\d+$'))))
logFile=/var/log/selenium/node_$(hostname)_$(date +%Y%m%d).log
echo "$(date +%Y%m%d)T$(date +%H%M%S) : [selenium-server] Launching selenium-server-standalone on $(hostname)}" >> $logFile
java -jar /opt/selenium-server/selenium-server-standalone-2.48.2.jar -role node \
        -hub http://$(hostname | grep -P -o '^\w*').yourdomain.yourtld:4444/grid/register -port $portNumber \
        -maxSession 4 -registerCycle 5000 -timeout 150000 -browserTimeout 120000 \
        -browser browserName=firefox,version=38.4.0,maxInstances=4,platform=LINUX &>> $logFile

Save and exit. Then make the script executable.

$ sudo chmod 755 start-selenium.sh

Later on we will mount the /opt/selenium and /var/log/selenium folders within the LXC containers. This affords a couple of benefits:

Create a selenium user, specifying the group and user id

$ sudo adduser selenium -uid 1234 -gid 1234 # (password and now for...)
Adding user `selenium' ...
Adding new group `selenium' (1234) ...
Adding new user `selenium' (1234) with group `selenium' ...
Creating home directory `/home/selenium' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
Sorry, passwords do not match
passwd: Authentication token manipulation error
passwd: password unchanged
Try again? [y/N] y
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for selenium
Enter the new value, or press ENTER for the default
        Full Name []:
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
Is the information correct? [Y/n] y

Create a SystemV init script to start the selenium grid. Note the "GID" and "UID" variables set to '1234'.

$ sudo vi /etc/init.d/selenium-hub

Insert the following text:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          selenium-hub
# Required-Start:    $network $syslog
# Required-Stop:     $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts Selenium Server hub instance
# Description:       Run's a hub instance of Selenium Server in a 
#                    Selenium Grid
### END INIT INFO

# Author: .BN <biscuitninja@someDomain.someTld>

PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Selenium Server - Hub"
NAME=selenium-hub
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
JARFILE=/opt/selenium-server/selenium-server-standalone-2.48.2.jar
LOGFILE=/var/log/selenium/hub_$(hostname)_$(date +%Y%m%d).log
SELENIUM_ARGS="-role hub -log $LOGFILE -nodePolling 5000 -nodeStatusCheckTimeout 5000 -downPollingLimit 3 -unregisterIfStillDownAfter 30000 -browserTimeout 120000"
DAEMON=/usr/bin/java
DAEMON_ARGS="-jar $JARFILE $SELENIUM_ARGS"

# Use the selenium UID/GID
UID=1234
GID=1234

# Exit if the package is not installed
[ -s "$JARFILE" ] || (echo "${JARFILE} not found" && exit 0)
[ -x "$DAEMON" ] || (echo "${DAEMON} not installed" && exit 0)

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.2-14) to ensure that this file is present
# and status_of_proc is working.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
    # Return
    #   0 if daemon has been started
    #   1 if daemon was already running
    #   2 if daemon could not be started
    start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
        || return 1
    start-stop-daemon --start --quiet --pidfile $PIDFILE --chuid $UID:$GID --background --exec $DAEMON -- \
        $DAEMON_ARGS \
        || return 2
    # Add code here, if necessary, that waits for the process to be ready
    # to handle requests from services started subsequently which depend
    # on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
    # Return
    #   0 if daemon has been stopped
    #   1 if daemon was already stopped
    #   2 if daemon could not be stopped
    #   other if a failure occurred
    start-stop-daemon --stop --quiet --retry=KILL/5 --pidfile $PIDFILE --name $NAME
    RETVAL="$?"
    [ "$RETVAL" = 2 ] && return 2
    # Wait for children to finish too if this is a daemon that forks
    # and if the daemon is only ever run from this initscript.
    # If the above conditions are not satisfied then add some other code
    # that waits for the process to drop all resources that could be
    # needed by services started subsequently.  A last resort is to
    # sleep for some time.
    start-stop-daemon --stop --quiet --oknodo --retry=KILL/5 --exec $DAEMON
    [ "$?" = 2 ] && return 2
    # Many daemons don't delete their pidfiles when they exit.
    rm -f $PIDFILE
    return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
    #
    # If the daemon can reload its configuration without
    # restarting (for example, when it is sent a SIGHUP),
    # then implement that here.
    #
    start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
    return 0
}

case "$1" in
  start)
    [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
    do_start
    case "$?" in
        0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
        2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  stop)
    [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
    do_stop
    case "$?" in
        0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
        2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  status)
    status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
    ;;
  #reload|force-reload)
    #
    # If do_reload() is not implemented then leave this commented out
    # and leave 'force-reload' as an alias for 'restart'.
    #
    #log_daemon_msg "Reloading $DESC" "$NAME"
    #do_reload
    #log_end_msg $?
    #;;
  restart|force-reload)
    #
    # If the "reload" option is implemented then remove the
    # 'force-reload' alias
    #
    log_daemon_msg "Restarting $DESC" "$NAME"
    do_stop
    case "$?" in
      0|1)
        do_start
        case "$?" in
            0) log_end_msg 0 ;;
            1) log_end_msg 1 ;; # Old process is still running
            *) log_end_msg 1 ;; # Failed to start
        esac
        ;;
      *)
        # Failed to stop
        log_end_msg 1
        ;;
    esac
    ;;
  *)
    #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
    echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2
    exit 3
    ;;
esac

:

Make the init script executable:

$ sudo chmod 755 /etc/init.d/selenium-hub

Add the new init script to the default run levels

$ sudo update-rc.d selenium-hub defaults

Start the service

$ sudo service selenium-hub start

Check the service is running

$ps -ef | grep selenium
selenium  1457     1  0 17:07 ?        00:00:01 /usr/bin/java -jar /opt/selenium-server/selenium-server-standalone-2.48.2.jar -role hub -log /var/log/selenium/hub_sghost0_20151110.log -nodePolling 5000 -nodeStatusCheckTimeout 5000 -downPollingLimit 3 -unregisterIfStillDownAfter 30000 -browserTimeout 120000

Install LXC on the Host

Install:

$ sudo apt-get install -y lxc 

If you have configured apt not to install recommends, also install the recommended packages: $ sudo apt-get install -y uidmap lxc-templates debootstrap

Create and Initialize a Container

Create a container named "sel-node-template", running Ubuntu Trusy x64

$ sudo lxc-create -t ubuntu -n sel-node-template -- --release trusty --arch amd64

List containers $ sudo lxc-ls sel-node-template

Switch the container onto bridged networking (as opposed to NAT)

$sudo vi /var/lib/lxc/sel-node-template/config

Change the line

lxc.network.link = (lxcnet0|lxcbr0)

To

lxc.network.link = br0

Whilst we are here configure the folders from the host which we want to be directly accessible to the container@

lxc.mount.entry = /opt/selenium-server opt/selenium-server none defaults,bind,optional,create=dir 0 0
lxc.mount.entry = /var/log/selenium var/log/selenium none defaults,bind,optional,create=dir 0 0

Save changes and close the file. Start the container

$ sudo lxc-start -n sel-node-template -d

Connect to container

$ sudo lxc-attach -n sel-node-template

Setup LXC / Selenium Grid Node Container

Install X (Openbox)

Update

root@sel-node-template$ apt-get update ; apt-get upgrade -y

Apt tweaks

root@sel-node-template$ vi /etc/apt/apt.conf.d/99tweaks

Insert the following text into the file

APT
{
        Install-Recommends "false";
        Install-Suggests "false";
}

Save changes and close the file.

Install openbox, vnc4server, etc...

root@sel-node-template$ apt-get install -y openbox xinit xorg vnc4server tint2 haveged

Install SUN Java (not openJDK)

root@sel-node-template$ apt-get install software-properties-common
root@sel-node-template$ apt-add-repository ppa:webupd8team/java
root@sel-node-template$ apt-get update
root@sel-node-template$ apt-get install -y oracle-java7-installer

Configure Openbox

Make xinitrc executable root@sel-node-template$ chmod 755 /etc/X11/xinit/xinitrc

Add a selenium user root@sel-node-template$ adduser selenium --uid 1234 --gid 1234 # (password and now for...) root@sel-node-template$ su - selenium

Tweak the Openbox menu
selenium@sel-node-template$ mkdir -p ~/.config/openbox selenium@sel-node-template$ cp /etc/xdg/openbox/menu.xml ~/.config/openbox/. selenium@sel-node-template$ vi ~/.config/openbox/menu.xml

Insert the following lines after "Web Browser"

    <item label="Selenium Server">
        <action name="Execute"><execute>/opt/selenium-server/start-selenium.sh</execute></action>
    </item>

Delete the following lines...

<item label="ObConf">
  <action name="Execute"><execute>obconf</execute></action>
</item>
<item label="Restart">
  <action name="Restart" />
</item>
<separator />
<item label="Exit">
  <action name="Exit" />
</item>

...but not

<item label="Reconfigure">
  <action name="Reconfigure" />
</item>

Save changes and close the file.

Configure Openbox start up applications

selenium@sel-node-template$ vi ~/.config/openbox/autostart 

Add the following lines, save, close

# Run tint2 to provide a taskbar within the desktop environment
/usr/bin/tint2 &

# Start a Selenium grid instance
/opt/selenium-server/start-selenium.sh &

Make the autostart script executable

selenium@sel-node-template$ chmod 764 ~/.config/openbox/autostart

Create a desktop folder

selenium@sel-node-template$ mkdir ~/Desktop

Configure vnc4server

Run vnc4server and specify a password (e.g. "123456") - we're not going to use any security as the container will only be accessible from a local network but vnc4server still requires a password on first use. Additionally, ignore the warning that "xauth: file /home/selenium/.Xauthority does not exist" does not exist. This is for information only as vnc4server automatically creates the file.

selenium@sel-node-template$ vnc4server  

Kill vnc4server.

selenium@sel-node-template$ vnc4server -kill :1 #Kill off the instance of vnc4server we just started
selenium@sel-node-template$ vi ~/.vnc/xstartup

Uncomment lines 4 & 5. Save changes. Quit

Remove the log file that vnc4server has just generated.

selenium@sel-node-template$ rm .vnc/*\:1.log 

Close the selenium user's shell selenium@sel-node-template$ exit

Install Firefox ESR

Lets first grab the dependencies Firefox needs: root@sel-node-template$ apt-get install libasound2 libasound2-data libgtk2.0-0 libgtk2.0-common

root@sel-node-template$ cd /opt
root@sel-node-template$ wget https://ftp.mozilla.org/pub/firefox/releases/38.4.0esr/linux-x86_64/en-GB/firefox-38.4.0esr.tar.bz2
root@sel-node-template$ tar -xjf firefox*.tar.bz2
root@sel-node-template$ rm firefox*.tar.bz2
root@sel-node-template$ mv firefox firefox38_4_0/.
root@sel-node-template$ ln -s /opt/firefox38_4_0/firefox /usr/bin/firefox
root@sel-node-template$ ln -s /usr/bin/firefox /usr/bin/x-www-browser

Add an init script to start xvncserver

Initially, grab the user id and group id of the selenium user:

root@sel-node-template$ grep selenium /etc/passwd
selenium:x:1001:1001:,,,:/home/selenium:/bin/bash

Create the SystemV init script:

root@sel-node-template$ vi /etc/init.d/vnc4server

Paste the following, noting the UID and GID that match the selenium user.

#! /bin/sh
### BEGIN INIT INFO
# Provides:          vnc4server
# Required-Start:    selenium-hub havaged
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts selenium-grid node within XVNC instance
# Description:       Starts an vnc4server VNC-XWindow instance, which
#                    in turn starts a selenium-grid node
### END INIT INFO

# Author: .BN <biscuitninja@someDomain.someTld>
DISPLAY="1"
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="selenium-grid node/XVNC instance"
NAME="vnc4server-${DISPLAY}"
DAEMON=/usr/bin/vnc4server
DAEMON_ARGS="-geometry 1600x900 -SecurityTypes None :${DISPLAY}"
DAEMON_KILL_ARGS="-kill :${DISPLAY}"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

# Use the selenium UID/GID
UID=1234
GID=1234

# Set value for home directory used by vnc4server
export HOME=/home/selenium

# Exit if the package is not installed
[ -s "$JARFILE" ] || (echo "${JARFILE} not found" && exit 0)
[ -x "$DAEMON" ] || (echo "${DAEMON} not installed" && exit 0)

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.2-14) to ensure that this file is present
# and status_of_proc is working.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
                || return 1
        start-stop-daemon --start --quiet --pidfile $PIDFILE --chuid $UID:$GID --exec $DAEMON -- \
                $DAEMON_ARGS \
                || return 2
        # Add code here, if necessary, that waits for the process to be ready
        # to handle requests from services started subsequently which depend
        # on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{

        $DAEMON $DAEMON_KILL_ARGS
        RETVAL="$?"
        [ "$RETVAL" !=  0 ] && return 2
        rm -f $PIDFILE
        return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
        #
        # If the daemon can reload its configuration without
        # restarting (for example, when it is sent a SIGHUP),
        # then implement that here.
        #
        start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
        return 0
}

case "$1" in
start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
        do_start
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        do_stop
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
status)
        status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
        ;;
#reload|force-reload)
        #
        # If do_reload() is not implemented then leave this commented out
        # and leave 'force-reload' as an alias for 'restart'.
        #
        #log_daemon_msg "Reloading $DESC" "$NAME"
        #do_reload
        #log_end_msg $?
        #;;
restart|force-reload)
        #
        # If the "reload" option is implemented then remove the
        # 'force-reload' alias
        #
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_stop
        case "$?" in
        0|1)
                do_start
                case "$?" in
                        0) log_end_msg 0 ;;
                        1) log_end_msg 1 ;; # Old process is still running
                        *) log_end_msg 1 ;; # Failed to start
                esac
                ;;
        *)
                # Failed to stop
                log_end_msg 1
                ;;
        esac
        ;;
*)
        #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2
exit 3
        ;;
esac

:

Make the new init script executable

root@sel-node-template$ chmod 755 /etc/init.d/vnc4server

Add the new init script to the default run levels

root@sel-node-template$ update-rc.d vnc4server defaults

Free some space:

Exit and stop the container

root@sel-node-template$ exit
$ sudo lxc-stop -n sel-node-template

Or just shut it down:

root@sel-node-template$ shutdown -P now

Automate the Cloning and Start-up of the Containers

First, lets create a new location for storing our initial template:

$ sudo mkdir /var/lib/lxc-base

Then lets create a compressed archive in which to store the template container

$ sudo -i
$ cd /var/lib/lxc
$ tar --numeric-owner -czvf /var/lib/lxc-base/sel-node-template.tar.gz sel-node-template
$ exit  # sudo session

Create a SystemV init script to automatically start the Selenium grid containers on boot

$ sudo vi /etc/init.d/selenium-containers

Insert the following text into the file, then save and close it:

#! /bin/bash
### BEGIN INIT INFO
# Provides:          selenium-containers
# Required-Start:    $local_fs $syslog $network
# Required-Stop:     $local_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Selenium Containers
# Description:       Clones and starts LXC selenium grid node
#            containers.
### END INIT INFO

# Author: .BN <biscuitninja@someDomain.someTld>
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="selenium-containers"
NAME="lxc-start"
DAEMON="/usr/bin/${lxc-start}"
DAEMON_ARGS="-d"
SCRIPTNAME=/etc/init.d/$NAME

# Exit if the package is not installed
/usr/bin/dpkg-query -W lxc &>/dev/null || exit 0

# Selenium/LXC specific variables
# Name of template container:
SEL_LXC_TEMPLATE_CONTAINER="sel-node-template"
# Path to template archive file:
SEL_LXC_TEMPLATE_ARCHIVE="/var/lib/lxc-base/sel-node-template.tar.gz"
# Number of clone containers to instantiate:
SEL_LXC_NUMBER_CLONES=2

# Exit if template archive not available
[ -f $SEL_LXC_TEMPLATE_ARCHIVE ] || exit 0

. /lib/lsb/init-functions

do_pre_start()
{
    log_daemon_msg "Executing ${DESC} do_pre_start" "$NAME"
    if [ ! -d "/var/lib/lxc/sel-node-template" ]; then
        log_daemon_msg "Template not initialised" "$NAME"
        log_action_begin_msg "Extracting template"
            /bin/tar -xzf $SEL_LXC_TEMPLATE_ARCHIVE -C /var/lib/lxc
        log_action_end_msg $?
    fi

    for (( i=1; i<=$SEL_LXC_NUMBER_CLONES; i++)) ; do
        if [ -d /var/lib/lxc/$(hostname)-$i ]; then
            log_daemon_msg "LXC Selenium container $(hostname)-${i} still loiters around from prior run" "$NAME"
            log_action_begin_msg "Cleaning up"
            rm -rf /var/lib/lxc/$(hostname)-$i
            log_action_end_msg $?
        fi
        log_action_begin_msg "Cloning LXC Selenium container $(hostname)-${i}"
            /usr/bin/lxc-clone -o sel-node-template -n "$(hostname)-${i}" >/dev/null
        log_action_end_msg $?
    done
}

do_start()
{
    # Return
    #   0 if daemon has been started
    #   1 if daemon was already running
    #   2 if daemon could not be started
    /usr/bin/lxc-info -n "$(hostname)-${1}" 2>/dev/null | /bin/grep "RUNNING" &>/dev/null && return 1
    log_action_begin_msg "Starting LXC Selenium container $(hostname)-${i}"
    /usr/bin/lxc-start -n "$(hostname)-${1}" -d || return 2
    log_action_end_msg $?
}

do_start_daemons()
{
    do_pre_start

    OVERALL_RESULT=0
    for((i=1; i<=$SEL_LXC_NUMBER_CLONES; i++)) ; do
        do_start $i
        RESULT=$?
        [ $RESULT -gt $OVERALL_RESULT ] && OVERALL_RESULT=$RESULT
        [ $RESULT -eq 2 ] && log_failure_msg "Starting $DESC (${NAME} $(hostname)-${i}) failed"
        [ $RESULT -eq 1 ] && log_warning_msg "$DESC (${NAME} $(hostname)-${i}) already running ?"
    done
    return $OVERALL_RESULT
}

do_stop()
{
    # Return
    #   0 if daemon has been stopped
    #   1 if daemon was already stopped
    #   2 if daemon could not be stopped
    /usr/bin/lxc-info -n "$(hostname)-${1}" 2>/dev/null | /bin/grep "RUNNING" &>/dev/null || return 1
    log_action_begin_msg "Stopping LXC Selenium container $(hostname)-${i}"
    /usr/bin/lxc-stop -n "$(hostname)-${1}" || return 2
    log_action_end_msg $?
}

do_post_stop()
{
    for((i=1; i<=$SEL_LXC_NUMBER_CLONES; i++)) ; do
        log_action_begin_msg "Destroying LXC Selenium container $(hostname)-${i}"
            /usr/bin/lxc-destroy -n "$(hostname)-${i}"
        log_action_end_msg $?
    done
    log_action_begin_msg "Destroying LXC Selenium container ${SEL_LXC_TEMPLATE_CONTAINER}"
        /usr/bin/lxc-destroy -n $SEL_LXC_TEMPLATE_CONTAINER
    log_action_end_msg $?
}

do_stop_daemons()
{
    OVERALL_RESULT=0
    for((i=1; i<=$SEL_LXC_NUMBER_CLONES; i++)) ; do
        do_stop $i
        RESULT=$?
        [ $RESULT -gt $OVERALL_RESULT ] && OVERALL_RESULT=$RESULT
        [ $RESULT -eq 2 ] && log_failure_msg "Stopping $DESC (${NAME} $(hostname)-${i}) failed"
        [ $RESULT -eq 1 ] && log_warning_msg "$DESC (${NAME} $(hostname)-${i} already stopped ?"
    done
    if [ $OVERALL_RESULT -eq 0 ] ; then
        do_post_stop
    fi
    return $OVERALL_RESULT
}

get_status()
{
    /usr/bin/lxc-info -n "$(hostname)-${1}" 2>/dev/null | /bin/grep "RUNNING" &>/dev/null
    STATUS="$?"
    if [ "$STATUS" = 0 ]; then
        log_success_msg "$DESC (${NAME} $(hostname)-${i}) is running"
        return 0
    else
        log_failure_msg "$DESC (${NAME} $(hostname)-${i}) is not running"
        return $status
    fi
}

get_daemon_statuses()
{
    OVERALL_RESULT=0
    for (( i=1; i<=$SEL_LXC_NUMBER_CLONES; i++)) ; do
        get_status $i
        RESULT=$?
        [ $RESULT -gt $OVERALL_RESULT ] && OVERALL_RESULT=$RESULT
    done
    return $OVERALL_RESULT
}

case "$1" in
  start)
    [ "$VERBOSE" != no ] && log_daemon_msg "Starting ${DESC} (${NAME} $(hostname)-*)" "${NAME}"
    do_start_daemons
    case "$?" in
        0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
        2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  stop)
    [ "$VERBOSE" != no ] && log_daemon_msg "Stopping ${DESC} (${NAME} $(hostname)-*)" "${NAME}"
    do_stop_daemons
    case "$?" in
        0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
        2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  status)
    get_daemon_statuses && exit 0 || exit $?
    ##status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
    ;;
  restart|force-reload)
    #
    # If the "reload" option is implemented then remove the
    # 'force-reload' alias
    #
    log_daemon_msg "Restarting ${DESC} (${NAME} $(hostname)-*)" "${NAME}"
    do_stop_daemons
    case "$?" in
      0|1)
        do_start_daemons
        case "$?" in
            0) log_end_msg 0 ;;
            1) log_end_msg 1 ;; # Old process is still running
            *) log_end_msg 1 ;; # Failed to start
        esac
        ;;
      *)
        # Failed to stop
        log_end_msg 1
        ;;
    esac
    ;;
  *)
    echo "Usage: ${SCRIPTNAME} {start|stop|status|restart|force-reload}" >&2
    exit 3
    ;;
esac

:

Make the new init script executable:

$ sudo chmod 750 /etc/init.d/selenium-containers

Add the script to the default rc run levels:

$ sudo update-rc.d selenium-containers defaults

And that's it. Restart your Selenium grid server, visit http://sghost0:4444/grid/console and you should see your grid up and running, something like this:

1

Further Considerations

TmpFS

I had intended to clone the containers to TmpFS (RAM disk) backed storage in order to reduce the time it takes to start the FireFox browser at the start of a test fixture. I couldn't get this working on Ubuntu Server 14.04.3 with LXC 1.0.8 but it did work with Ubuntu Server 15.10 and LXC 1.1.4

Snapshot based LXC Clones

A further avenue for exploration, if continuing to rely on disk backed LXC containers is using a snapshot clones instead of copy clones. This means switching backing store to aufs, btrfs or LVM.