Write better software tests with random test execution in mind

Improve the quality of your software tests with random test execution

Tests, whether unit tests or end-to-end tests, should not depend on other tests. That means that whether you run one test for itself or the whole test suite, the test should always work. One feature which can help to accomplish that goal is random test execution.

If enabled, the tests will not executed in the order they are written but semi-randomly. This feature is supposed to help us writing better tests. This way, we might also detect bugs that we would not notice if the tests always run in the same order.

The popular JavaScript testing library Jasmine has set random execution of specs to true by default since version 3 (you can use it in older versions, too). Other testing libraries like JUnit already do this as well. Although you can disable this feature in some libraries, you should not do that as it helps you to write better tests.

In the following, I will explain how to write tests which work independent of the execution order. The principles apply for both unit tests and end-to-end tests. For examples, I will use Jasmine but the concepts apply for other testing libraries in other programming languages as well.

How to write tests that always work independently?

• Use beforeEach and beforeAll for setting up all you need for your tests: e.g. go to a specific route and create some data for your tests. Keeping your setup at a consistent place will make it easier to clean up afterwards.
• Use afterEach and afterAll to clean up: e.g. delete things you created in the tests. This ensures that tests which will be executed after will not be affected.
• Do not assume an order how tests will be executed. For example, do not count on testA running before testB. Tests should be isolated and work for themselves independent of other tests.
• Use a random seed. In Jasmine, you can provide it as a runtime argument. If a test fails when executing a test suite, then you can use the seed to rerun the tests in the exact order it ran last time. Then, you can find out why that one test fails given the execution order.
• Use continuous integration (CI). This is not necessarily a requirement but it will be super useful and there is no reason not to use CI in software development projects. The CI server can run all tests on every commit (preferably with a random seed) and catch errors in your tests early.
• Keep your tests at a reasonable size. Tests should be easy to read and easy to write. If you have too much logic or nested tests (in case of Jasmine: you can put a describe block inside of another describe block) then the tests and the control flow will be harder to understand.
• Include a retry mechanism. I recommend this for end-to-end tests which can sometimes be flaky due to multiple reasons (e.g. network conditions). For example, there is a solution for the Protractor testing library to rerun flaky tests. This way, you can identify flaky tests easier and you can also reduce false positives.

Running tests with random test execution and a random seed using Jasmine.

Conclusion

Writing and understanding tests should be fairly easy for developers in order for them to be effective. Random test execution helps developers to build more reliable and consistent tests. What are your experiences e when it comes to random test execution? Let me know in the comments.