Background

I’m investigating a strange problem we are seeing in one of our environments with Apache HTTP Server. In some circumstances an HTTP 400 status code is returned in response to a TLS Client Hello.

I want to see whether the HTTPD2 socache_shmcb module has a role to play in this unexpected behaviour. I suspect the problem will be more likely to occur with more clients, and less likely to occur with fewer clients.

To test the hypothesis, I want to easily change the number of clients. I have been using Locust for these type of load tests. Locust unfortunately does not support binding the http client to an IP, or at least if it does, having checked the docs and looked at the source code, I wasn’t able to see how. Thus I chose to hack around this problem using Docker. I began by building a Locust Docker image, which is covered in this post.

I hope to, as I build out this solution, create multiple Locust containers with one Docker image, binding each container to its own IP address. It sounds like it should be possible.

Posts in this Series

Part 1 - Creating and Building The Docker Image

Prerequisites

Before we start, we can install Debian (version 10, Buster) on a box and then install both Docker and Docker Compose:

sudo apt install -y docker.io docker-compose

Once you have Docker installed, you can check that the service is up and running with:

systemctl status docker.service

The output should show that the service is enabled and active (running).

You can of course opt to use your own preferred Linux distribution, or even a non-Linux based operating system. I have written this series of posts based on my experience of Debian. Things invariably change over time. Irrespective of your chosen OS, your mileage may vary!

It is worth noting, before we start work, that I have opted to keep everything contained in one directory:

mkdir ~/loadtest

Creating a Docker Image with a Docker File

Creating a Docker image means writing a dockerfile. Here is my minimum viable example:

FROM debian

COPY locust.sh /

ENV DEBIAN_FRONTEND=noninteractive
RUN  apt-get update \
    && apt-get install -y apt-utils \
    && apt-get install -y python3-pip python3-setuptools python3-wheel python3-dev build-essential \
    && pip3 install locust \
    && chmod 555 /locust.sh

RUN  mkdir /locust
EXPOSE 8089 5557
WORKDIR /locust
ENTRYPOINT ["/locust.sh"]

Save this file as ~/loadtest/dockerfile.

The FROM keyword initialises the build and specifies a base Docker image.

COPY adds the shell script we need to run Locust to the root of the container image.

ENV specifieds an environment variable which apt-get seems to need running non-interactively.

RUN is self explanatory. In the first RUN instruction we update repositories, install some packages including pip and then using pip install Locust.

WORKDIR sets the working directory for commands that follow.

EXPOSE specifies the ports we want the container to expose, default protocol is TCP.

Finally, ENTRYPOINT specifies an executable to run, in this case a shell script.

Shell Script

When we spin up a Docker container from the image we have built, we need it to do something. That is where the locust.sh shell script comes in. For the first stab at the locust.sh shell script, I worked up something like this:

#!/bin/bash
set -e

locustMode=${locustMode:-standalone}
locustMasterBindPort=${locustMasterBindPort:-5557}
locustFile=${locustFile:-locust.py}

[ -z ${targetHost+x} ] && (echo 'variable targetHost not set' ; exit 1)
[ $(echo ${locustMode} | tr 'a-z' 'A-Z') == "WORKER" ] && [ -z ${locustMaster+x} ] && \
    (echo 'variable locustMaster must be set if locustMode=="WORKER"'; exit 1)

locustOptions="-f ${locustFile} --host=${targetHost} $locustOptions"

[ $(echo ${locustMode} | tr 'a-z' 'A-Z') == "MASTER" ] && locustOptions="--master --master-bind-port=${locustMasterBindPort} $locustOptions"

[ $(echo ${locustMode} | tr 'a-z' 'A-Z') == "WORKER" ] && locustOptions="--worker --master-host=${locustMaster} --master-port=${locustMasterBindPort} $locustOptions"

cd /locust
locust ${locustOptions}

Save this file as ~/loadtest/locust.sh.

For some flexibility, this script means we can run Locust as master, worker or standalone instance. For most of the environment variables used, we have set sensible default values. The only value not set is $targetUrl, which is the URL of the target you want to load test, e.g. https://mywebsite.sometld/index.html.

Build

At this juncture, we are ready to test the build. From within the ~/loadtest directory, run:

sudo docker build -t debian:locust .

After waiting a while, the image is successfully created.

Successfully built 99af74077500

You can list Docker images with:

docker images

When you do this, your results should show two images, one identified as debian:latest and another identified as debian:locust. If you have made multiple attempts to build the debian:locust images you will see other images listed that don’t have a repository, tag. You can remove these with:

docker rmi <IMAGE_ID>

Where image id, which is a 12 character hex string, is taken from the output of docker images.

In part 2 I will examine how we can use the Docker image we have just created.