Distributed testing, part 1

Introduction

We are working on a series of improvements to our platform. These consists of

• Improving support for parallel test runs
• Better job distribution
• Support for multi-agent tests
• Test handoff

Improving parallelism stems from the need to shorten execution times. In end-to-end tests, execution times are often very long. In a tool where test creation is fast (such as Boozang), long execution times dictate the upper limit of test coverage.

To shorten execution times, it’s also essential to have fair job distribution. Parallelism does little if one worker gets all the work. 

As we have several workers doing work in parallel, this opens up for multi-agent tests. Imagine where you have two active agents acting in a work-flow, like a two-sided marketplace. Usually, we solve this by writing a linear test that switches between the roles and often requires several logins and logouts. In a multi-agent test, you highlight which parts of the test are done by which agents and run the test with at least two active sessions simultaneously. 

In the new Boozang test architecture, we also support test handoff. In Boozang, being able to see the test executing in the browser greatly speeds up test creation. In some cases, the user wants to know the test outcome without seeing the test running in the browser window. Now the user can handoff the test to someone else and continue working un-impeded. 

Borrowing from Agile

To be able to do this well, we have used the analogy of Agile teams. In a team, different team members can have different skillsets. It’s ideal if the team volunteers for the tasks within their skillset. It’s also good for the team to work as an independent unit and handle assigned tasks without any centralized unit meddling. The team will report back to management when they have completed their assigned tasks and are ready to take on new ones.

Much in the same way, servers can have different areas of specialization. Some might live in a specific part of the network, giving them access to systems that others don’t. Some might run as users with a particular role, such as “admin” or “unauthorized.”

New terminology

To model this out on the application level, we introduce the following terminology.

• Worker: An Boozang client instance that can do work. In Boozang, this is an instance created using the Docker container bz-docker-xvfb (or bz-docker-playwright).
• Group: A worker belongs to a group, like a team member belongs to a team. In Boozang, a group is a set of client instances that are ready to accept work. 
• Scope / Tag: Classification of a group. In our group analogy, the scope of a development team is “development.” In Boozang, the scope defines a type of worker. 

Now with this terminology at hand, we can understand the new Boozang view.  

Practical uses: Parallel runs

There are many practical uses for this functionality, but let’s start with the most important: setting up a test suite to run its tests parallel. If there is no specific need to group your workers, you don’t even have to define a group. 

1. Go to Settings -> Integration -> Generate My account Cooperation Server Link
2. Enter a unique server number (1, 2, 3 …)
3. Enter password

The generated URL will look something like:

"http://ai.boozang.com/extension?token=xxx&env=0&number=2#5bea26946c43587a5950f410/0.0.1"

Now you have an executable URL for that worker. Note that the generated URL contains the server number, so it can be automatically assigned by Jenkins if needed. Use the boozang-runner Docker container (or boozang-playwright) to start clients using the URL, ensuring that each client has a unique server number. 

docker run --rm -v "$(pwd):/var/boozang/" styrman/boozang-runner "http://staging-be.boozang.com/extension?token=xxx&env=4&number=2#5e3f275e64f84941a326d4d8/ci44"
docker run --rm -v "$(pwd):/var/boozang/" styrman/boozang-runner "http://staging-be.boozang.com/extension?token=xxx&env=4&number=3#5e3f275e64f84941a326d4d8/ci44"

...
docker run --rm -v "$(pwd):/var/boozang/" styrman/boozang-runner "http://staging-be.boozang.com/extension?token=xxx&env=4&number=1#5e3f275e64f84941a326d4d8/ci44/m2/t1/run"

In the above test, we have created two “helper” workers first; then, by adding the test suite’s module and test suite in one of the clients (in this case, the master: number 1), the test is started.

"http://ai.boozang.com/extension?token=xxx&env=0&number=3#5bea26946c43587a5950f410/0.0.1/m5/t2/run"

The test clients will split up the test suite tests on a “first-come” – first-serve” basis and automatically stop upon completion.

Practical uses: Test handoff

To handoff tests in Boozang we created a new play mode in Boozang called “Remote play.” To test remote play, generate the URL as the previous example and use the Docker boozang-runner to start up the worker.

docker run --rm -v "$(pwd):/var/boozang/" styrman/boozang-runner "http://staging-be.boozang.com/extension?token=xxx&env=4&number=2#5e3f275e64f84941a326d4d8/ci44"
docker run --rm -v "$(pwd):/var/boozang/" styrman/boozang-runner "http://staging-be.boozang.com/extension?token=xxx&env=4&number=3#5e3f275e64f84941a326d4d8/ci44"

The workers will now idly wait for something to do. 

1. Open the IDE
2. Go to a test or test suite you want to run
3. Use the Play dropdown to Click remote play
4. The workers now show up as idling
5. Select “Free combination of services”
6. Click “Go”

You will now see the test being assigned to the first available worker. If it’s a test suite with several tests, the tests will be served up to the workers as in a “first-come”-“first-serve” basis.

As soon as the tests are finished, you should see the workers going back to “idle” status, available to take on more work.

Util functions

To create logic inside the tests that depend on this new code, we have added a set of util functions.

$util.getCoopKey
$util.getCoopScope
$util.getCoopGroup

It’s now possible to be able to do conditional logic inside the tests based on these parameters. For instance, to make sure all workers are using different user accounts, you can keep username and password in a matrix data file userAccounts defined as

_key	username	password
1	tom	secret1
2	anna	secret2

and simply retrieve the username and password using Javascript

let key = $util.getCoopKey;
$test.username  = userAccounts[key].username;
$test.password  = userAccounts[key].password;

Jenkins examples

You can find workable Jenkins examples in the Forum here. They are all based on bash and Docker and should work (almost) equally well for any CI server.

Read more

There is another blog post dealing with more complex parallel tests, where we introduce runner groups and scope. This can be found here.