How to write tests in Python - Recipes and Patterns

Even though tests should contain as little amount of logic as possible, that doesn't mean that tests cannot be DRY. This section will detail how I write tests in order to be abstract enough to easily add additional tests while maintaining readable/understandable tests. One very important thing to remember is that tests, like code, should be easy to read and understand; there should be a very good balance between understandability and maintenance. I would argue that understandability is better than maintenance in the case of tests. I've heard it said that 10% of a programmer's time is spent writing code, the other 90% is spent reading code. Since so much time is spent reading, it better be easy to read.

Configuration-style subclassing

I like to use Python's class inheritance as a way to write abstract tests without introducing custom logic. One way to do this is to write a base class that contains all of the abstract tests and then use subclassing to provide information for the tests. With this pattern, you write the tests once, but run them for each configuration (or subclass).

While at Garmin, I worked on writing tests for the Build System that we use. We support multiple toolsets: Microsoft's Visual C, ARM C, ADI Blackfin, and GCC. For the most part, we always wanted to perform the same kind of test, but for each toolset. So, I employed the configuration subclassing pattern to solve this problem. It looked something like this:

from testlib.testcase import BaseTestCase


class TestMsvc8Toolset(BaseTestCase):
    toolset = 'msvc'
    toolset_version = '8.0'
    executable_extension = '.exe'

    def setUp(self):
        super(TestMsvcToolset, self).setUp()
        # write a hello world program
        self.write(
            'src/hello.c',
            """
            #include <stdio.h>

            int main(int argc, char **argv){
                printf("%s\n", "Hello, World!");
                return 0;
            }
            """
        )

    def test_single_file_compilation(self):
        # run the build system with the toolset and toolset version
        # specified. This pulls from the class-level attributes
        # to provide the correct values to run.
        self.run_build_system([self.toolset, self.version])
        # make sure that 'hello.c' has been compiled and linked
        # into the correct 'hello' executables.
        self.assert_file_created('hello{}'.format(self.executable_extension))


class TestMsvc12Toolset(TestMsvc8Toolset):
    version = '12.0'


class TestArm12Toolset(TestMsvc8Toolset):
    toolset = 'arm'
    version = '1.2'
    executable_extension = '.axf'


class TestArm505Toolset(TestArm12Toolset):
    # note that this subclasses the other ARM test
    # which is why only the version needs to be specified
    version = '5.05'


class TestAdiBlackFin(TestMsvc8Toolset):
    toolset = 'adi-bf'
    version = '8.2'
    executable_extension = '.dxe'


class TestGcc481Toolset(TestMsvc8Toolset):
    toolset = 'gcc'
    version = '4.8.1'
    executable_extension = ''
    # this should be an acceptable use of logic
    if os.name == 'nt':
        executable_extension = '.exe'

An Object Oriented expert may frown upon this pattern as having an ARM toolset test case inherit from an MSVC test case doesn't really make any sense, but it has more than saved me time in maintenance because it does an excellent job at keeping the tests DRY. As it stands, when running test discovery on the above, 6 tests will be run. Adding a new test (maybe checking to see if library creation works correctly) will suddenly add a new test for all toolsets. There would then be a total of 12 tests (2 tests for 6 toolsets), but only one place for the definition of the test!

Depending on the type of tests that need to be written, problems may arise by subclassing in this particular way and so it might make sense to create a test mixin instead. The above pattern could be rewritten like so:

from testlib.testcase import BaseTestCase


# note the inheritance from object
class TestsMixin(object):
    toolset = ''
    toolset_version = ''
    executable_extension = ''

    def setUp(self):
        super(TestMsvcToolset, self).setUp()
        # write a hello world program
        self.write(
            'src/hello.c',
            """
            #include <stdio.h>

            int main(int argc, char **argv){
                printf("%s\n", "Hello, World!");
                return 0;
            }
            """
        )

    def test_single_file_compilation(self):
        # run the build system with the toolset and toolset version
        # specified. This pulls from the class-level attributes
        # to provide the correct values to run.
        self.run_build_system([self.toolset, self.version])
        # make sure that 'hello.c' has been compiled and linked
        # into the correct 'hello' executables.
        self.assert_file_created('hello{}'.format(self.executable_extension))


class BaseToolsetTestCase(BaseTestCase):
    pass


class TestMsvc8Toolset(BaseToolsetTestCase, TestsMixin):
    toolset = 'msvc'
    toolset_version = '8.0'
    executable_extension = '.exe'

class TestArm12Toolset(BaseToolsetTestCase, TestsMixin):
    toolset = 'arm'
    version = '1.2'
    executable_extension = '.axf'

This sort of inheritance model is more correct by OO standards.

Configurable method

Another way to keep tests dry is to create a run-and-assert method. This kind of method is in charge of running the application or function-under-test and assert that some sort of event has occurred. The method will then allow parameters (either required-positional or optional-keyword parameters) to configure how the run is performed or how to assert that an event occurred.

The above class-based configuration method could be rewritten using this style to illustrate the point.

from testlib.testcase import BaseTestCase


class TestToolsets(BaseTestCase):

    # the setup function will be exactly the same
    def setUp(self):
        super(TestMsvcToolset, self).setUp()
        # write a hello world program
        self.write(
            'src/hello.c',
            """
            #include <stdio.h>

            int main(int argc, char **argv){
                printf("%s\n", "Hello, World!");
                return 0;
            }
            """
        )

    def run_and_assert(self, toolset, version, extension):
        self.run_build_system([toolset, version])
        # running the build system should have produced
        # the correct executable
        self.assert_file_created('hello{}'.format(extension))

    def test_msvc_8_0_toolset(self):
        self.run_and_assert('msvc', '8.0', '.exe')

    def test_msvc_12_0_toolset(self):
        self.run_and_assert('msvc', '12.0', '.exe')

    def test_arm_1_2_toolset(self):
        self.run_and_assert('arm', '1.2', '.axf')

    def test_arm_5_05_toolset(self):
        self.run_and_assert('arm', '5.05', '.axf')

    def test_adi_bf_8_2_toolset(self):
        self.run_and_assert('adi-bf', '8.2', '.dxe')

    def test_gcc_4_8_1_toolset(self):
        extension = ''
        if os.name == 'nt':
            extension = '.exe'
        self.run_and_assert('gcc', '4.8.1', '.exe')

Performing the toolset tests this way would be terrible, but hopefully it illustrates how to use a method as a way to configure and run tests.

Why would this be bad for the Toolset tests?

Since there are many tests required to test the output of a toolset, it would require essentially copying and pasting the same configuration data over and over (i.e. toolset name, version, etc.).

Another possible scenario is that the base toolset test case may have a test that doesn't apply for a particular toolset. Using the method configuration pattern, you would simply omit the test. To a reader, it might appear that the test case had simply been forgotten and so, they might try to recreate the test for that particular toolset. Using the configuration by subclass pattern, you can simply override the test, either by completely rewriting the test, or providing a pass for the body. In either case, a docstring can and should be provided explaining why this test/toolset combination is special.

Previous: Writing Tests


Comments