I'm creating an open source sync program called ClearSkies, with the help of some others. While unit testing can be done like usual, integration testing involves some additional trickery. This post contains what has worked well for me so far.

Iterate Quickly

As a preamble, the first thing that I feel I need to be able to do with any test is to be able to iterate quicky. I have a tool to do this, called auto-go, which looks in the current directory for a file called .auto-go, and then searches the parent directory and all ancestors until it finds it. The magical part is how it is set up in vim. I make it so that if I press ",r", it saves my file and then runs auto-go, even in insert mode.

There are other methods for running tests automatically, such as with Guard, but I prefer this one because it lets me adjust my area of focus quickly (by changing the contents of .auto-go).

Integration Test with bash

To do an integration test, we need to run two copies of the program simultaneously. When I wrote a very similar program during 2007 I'd launch each copy by hand and then connect them together, which was tedious and slowed my progress. Basically, two copies of the program need to be launched and then the progress of the sync can be checked by looking directly at the directory contents:

rm -rf /tmp/1 /tmp/2

./clearskies start --data-dir=/tmp/1
./clearskies start --data-dir=/tmp/2

mkdir /tmp/1/stuff
echo some file contents > /tmp/1/stuff/test-file
code=`./clearskies --data-dir=/tmp/1 share /tmp/1/stuff read-write`
./clearskies --data-dir=/tmp/2 add /tmp/2/stuff $code

echo "Waiting for test-file to sync"
while ! test -e /tmp/2/stuff/test-file
do
  sleep 0.1
done

./clearskies stop --data-dir=/tmp/1
./clearskies stop --data-dir=/tmp/2

Of course, it's nice to see the logging output of both programs, so they should be launched in the Foreground:

./clearskies start --no-fork --data-dir=/tmp/1 &
./clearskies start --no-fork --data-dir=/tmp/2 &

Gnome-terminal

It's then hard to distinguish between the output of both programs, so I tried launching each in its own gnome-terminal:

gnome-terminal -e './clearskies start --no-fork --data-dir=/tmp/1'
gnome-terminal -e './clearskies start --no-fork --data-dir=/tmp/2'

Tmux

I used the gnome-terminal version for quite a while, but I found it was annoying that the new windows would steal my focus, stopping me from interrupting them.

So, I started them in tmux instead:

if [ -z "$TMUX" ]
then
  exec tmux new-session $0
fi

tmux split-window -v './clearskies start --no-fork --data-dir=/tmp/1'
tmux split-window -v './clearskies start --no-fork --data-dir=/tmp/2'
tmux select-layout even-vertical
tmux select-pane -t 0

This works fantastically well.

Combined Logs

One problem with this approach is that if there are bugs involving the interaction of both programs, it's hard to figure out the timeline of events. I solved this by having each program write to a log file with a timestamp at the beginning of the line, including the test's output, and then combined them into a single log file, sorted by the time.

Other details

I had to capture ctrl-c in my main script so that I could shut down the other programs, and I also had the extra windows sleep after launching so that I could see any exceptions.

See the actual integration script for more details.