How to write tests in Python - Project Structure

In school, we were never taught how to write tests. Like my beginning employment at Garmin, most testing was just running a program to see that the output was as expected. The life span of the average program in college is roughly a week, so there was no need to write tests. Writing tests was a completely foreign idea to me. So, before writing the testing framework for my project at work, I did a lot of research on how to write tests. Sadly, most of the articles and examples that I found only talked about the most basic of things. Most examples only contained a single module and a single test file that tested the module. Nothing ever touched on how tests should be structured in a real project.

The project structure that I tend to use should work well with any of the testing frameworks out there. I use a combination of Python's unittest for testing helper functions and nose2 for test discovery and running. unittest will work just fine for test discovery and running, but nose2 allows you to supply plugins for additional functionality. One example is the built-in nose2 coverage plugin that automatically collects coverage for all of your Python files.

Project and Testing structure

A typical Python library or framework will contain all modules inside of a package. In order to provide a working example, assume we have a library named example. The example library might have a project structure like this

example
|- subpackage1
|  |- __init__.py
|  |- module1.py
|- subpackage2
|  |- __init__.py
|  |- module2.py
|- ...
|- __init__.py

While I've seen it done, I personally don't care for placing test modules adjacent to the files that they test. For example, the tests for module1.py would be named test_module1.py and live in the subpackage1 directory. I think it's cleaner to keep the tests separate from library.

Thus, I favor the following development project structure

example
|- example
|  |- subpackage1
|  |- subpackage2
|  |- ...
|- tests
|  |- subpackage1
|  |  |- module1
|  |  |  |- __init__.py
|  |  |  |- test_some_common_functionality_in_module1.py
|  |  |- module2
|  |  |  |- __init__.py
|  |  |  |- test_some_common_functionality_in_module2.py
|  |  |- __init__.py
|  |  |- test_any_common_functionality_for_subpackage1.py
|  |- testlib
|  |  |- __init__.py
|  |  |- testcase.py
|  |  |- environment.py
|  |  |- renderables.py

There is a tests directory that is adjacent to the library under test, example. Both are contained in a directory of the same name as the library (just a personal convention, there's no reason something like exampleproject can't be used).

I normally have a testlib package that lives in the tests directory. The testlib is a mini library of classes or functions that are useful for performing tests. For example, both environment.py and renderables.py might contain mixin classes to add functionality to the BaseTestCase. environment.py might be in charge of setting up and tearing down temporary projects and renderables.py might be in charge of rendering temporary files.

Note that the tests directory structure mirrors the directory structure of the library under test except for one important difference: the modules module1.py and module2.py are directories on the tests side. I like to do this so that I don't have a single test_module1.py file that contains all the tests for that module. Depending on the module, it has the potential to contain a lot of tests. Instead, I like to split the tests for the module into separate test modules grouping by common functionality. Sometimes that common functionality might be just testing a single function in the module. Assume that module1.py has a function, foo(). A test module under the module1 directory might be test_foo.py doing nothing but testing the example.subpackage1.module1.foo() function.

Also note that each test module has a name that starts with the same prefix test_. This has been done for test discovery purposes. For this same reason, testlib and testlib.testcase do not have the underscore in their names. Each directory contains an __init__.py. When using test discovery by a test runner such as py.test, nose2, or even the built-in unittest, there is usually a way to specify a pattern for what module should be considered a test module.

TestCase inheritance hierarchy

I would highly recommend the built-in testing framework unittest. You can read more about that framework on Python's documentation. unittest provides a class, TestCase, that provides helper methods for assertions, setup and teardown methods, class setup and teardown methods, etc. And don't let the name fool you. It works well for system tests as well. Although I am suggesting unittest, the same principles should apply to any class-based testing framework.

I have developed the practice of creating a BaseTestCase class that all tests inherit from. This base class will go into testlib/testcase.py.

# testlib.testcase.py

from unittest import TestCase


class BaseTestCase(TestCase):
    """
    All test cases should inherit from this class as any common
    functionality that is added here will then be available to all
    subclasses. This facilitates the ability to update in one spot
    and allow all tests to get the update for easy maintenance.
    """

The BaseTestCase class will extend unittest.TestCase by providing additional helper methods. What kind of methods that will be written are dependent upon what is needed in order to test. When I first started out writing my testing framework, my BaseTestCase class literally inherited from unittest.TestCase and just had a pass as the class's body. As time went on, I started to abstract tests that were written and provided helper methods on the BaseTestCase class. Thus, all tests would have access to these new helper methods.

Another thing that I like to do is create a base class for each level in the tests hierarchy. For example, in tests/subpackage1/__init__.py I will create a class named BaseSubpackage1TestCase that inherits from testlib.testcase.BaseTestCase.

# tests/subpackage1/__init__.py

from testlib.testcase import BaseTestCase


class BaseSubpackage1TestCase(BaseTestCase):
    pass

Each base class in the hierarchy adds another level of granularity and control for the tests that inherit from it. The BaseSubpackage1TestCase class might provide helper methods that only make sense to subpackage1 modules, classes, and functions. The majority of the time, I will create these classes with a pass in the body and the majority of the time, that's how they will remain. But not always. Eventually, I will need to perform some action for all tests under a base class and, because I provided a base class at each level in the hierarchy, I will only have to make an update in one location.

Real life example

We had a module that interacted with git. In order to test that module, there needed to be an actual git repo on disk. So, to augment the existing environment creation and teardown, the BaseGitTestCase class created a git repo and a text file, then committed that file on setup. The git repo was destroyed on teardown from the environment handling. Creating a git repo only made sense in the context of testing the git module; therfore, this functionality only existed on the BaseGitTestCase class.

Finally, the actual test case class will inherit from its closest base test case. Continuing with the foo() testing example, the following test case would be constructed

# tests/subpackage1/module1/test_foo.py

# note that this is importing BaseModule1TestCase from
# the tests/subpackage1/module1/__init__.py file
from . import BaseModule1TestCase

from example.subpackage1 import module1

class FooTestCase(BaseModule1TestCase):
    def test_that_foo_does_not_crash_when_called(self):
        module1.foo()

In the end, the following test case hierarchy will be formed. Each class listed inherits from the previous.

BaseTestCase               - testlib/testcase.py
\_ BaseSubpackage1TestCase - tests/subpackage1/__init__.py
   \_ BaseModule1TestCase  - tests/subpackage1/module1/__init__.py
      \_ FooTestCase       - tests/subpackage1/module1/test_foo.py

Previous: Introduction

Next: Writing Tests


Comments