In school, we were never taught how to write tests. Like my beginning employment at Garmin, most testing was just running a program to see that the output was as expected. The life span of the average program in college is roughly a week, so there was no need to write tests. Writing tests was a completely foreign idea to me. So, before writing the testing framework for my project at work, I did a lot of research on how to write tests. Sadly, most of the articles and examples that I found only talked about the most basic of things. Most examples only contained a single module and a single test file that tested the module. Nothing ever touched on how tests should be structured in a real project.
The project structure that I tend to use should work well with any of the
testing frameworks out there. I use a combination of Python's
for testing helper functions and
nose2 for test discovery and running.
unittest will work just fine for test discovery and running, but nose2
allows you to supply plugins for additional functionality. One example is the
built-in nose2 coverage plugin that automatically collects coverage for all
of your Python files.
Project and Testing structure
A typical Python library or framework will contain all modules inside of a
package. In order to provide a working example, assume we have a library named
example library might have a project structure like this
example |- subpackage1 | |- __init__.py | |- module1.py |- subpackage2 | |- __init__.py | |- module2.py |- ... |- __init__.py
While I've seen it done, I personally don't care for placing test modules
adjacent to the files that they test. For example, the tests for
would be named
test_module1.py and live in the
subpackage1 directory. I
think it's cleaner to keep the tests separate from library.
Thus, I favor the following development project structure
example |- example | |- subpackage1 | |- subpackage2 | |- ... |- tests | |- subpackage1 | | |- module1 | | | |- __init__.py | | | |- test_some_common_functionality_in_module1.py | | |- module2 | | | |- __init__.py | | | |- test_some_common_functionality_in_module2.py | | |- __init__.py | | |- test_any_common_functionality_for_subpackage1.py | |- testlib | | |- __init__.py | | |- testcase.py | | |- environment.py | | |- renderables.py
There is a
tests directory that is adjacent to the library under test,
example. Both are contained in a directory of the same name as the library
(just a personal convention, there's no reason something like
can't be used).
I normally have a
testlib package that lives in the
tests directory. The
testlib is a mini library of classes or functions that are useful for
performing tests. For example, both
might contain mixin classes to add functionality to the
environment.py might be in charge of setting up and tearing down temporary
renderables.py might be in charge of rendering temporary files.
Note that the
tests directory structure mirrors the directory structure of
the library under test except for one important difference: the modules
module2.py are directories on the tests side. I like to do
this so that I don't have a single
test_module1.py file that contains all
the tests for that module. Depending on the module, it has the potential to
contain a lot of tests. Instead, I like to split the tests for the module into
separate test modules grouping by common functionality. Sometimes that common
functionality might be just testing a single function in the module. Assume that
module1.py has a function,
foo(). A test module under the
directory might be
test_foo.py doing nothing but testing the
Also note that each test module has a name that starts with the same prefix
test_. This has been done for test discovery purposes. For this same reason,
testlib.testcase do not have the underscore in their names. Each
directory contains an
__init__.py. When using test discovery by a test runner
such as py.test,
nose2, or even the
built-in unittest, there is usually a way to specify a pattern for what module
should be considered a test module.
TestCase inheritance hierarchy
I would highly recommend the built-in testing framework
unittest. You can read
more about that framework on Python's documentation.
unittest provides a class,
TestCase, that provides helper methods for
assertions, setup and teardown methods, class setup and teardown methods, etc.
And don't let the name fool you. It works well for system tests as well.
Although I am suggesting
unittest, the same principles should apply to
any class-based testing framework.
I have developed the practice of creating a
BaseTestCase class that all tests
inherit from. This base class will go into
# testlib.testcase.py from unittest import TestCase class BaseTestCase(TestCase): """ All test cases should inherit from this class as any common functionality that is added here will then be available to all subclasses. This facilitates the ability to update in one spot and allow all tests to get the update for easy maintenance. """
BaseTestCase class will extend
unittest.TestCase by providing additional
helper methods. What kind of methods that will be written are dependent upon
what is needed in order to test. When I first started
out writing my testing framework, my
BaseTestCase class literally inherited
unittest.TestCase and just had a
pass as the class's body. As time went
on, I started to abstract tests that were written and provided helper methods
BaseTestCase class. Thus, all tests would have access to these new
Another thing that I like to do is create a base class for each level in the
tests hierarchy. For example, in
tests/subpackage1/__init__.py I will create
a class named
BaseSubpackage1TestCase that inherits from
# tests/subpackage1/__init__.py from testlib.testcase import BaseTestCase class BaseSubpackage1TestCase(BaseTestCase): pass
Each base class in the hierarchy adds another level of granularity and control
for the tests that inherit from it. The
BaseSubpackage1TestCase class might
provide helper methods that only make sense to
subpackage1 modules, classes,
and functions. The majority of the time, I will create these classes with a
pass in the body and the majority of the time, that's how they will remain.
But not always. Eventually, I will need to perform some action for all tests
under a base class and, because I provided a base class at each level in the
hierarchy, I will only have to make an update in one location.
Real life example
We had a module that interacted with git. In order to test that module, there needed to be an actual git repo on disk. So, to augment the existing environment creation and teardown, the
BaseGitTestCaseclass created a git repo and a text file, then committed that file on setup. The git repo was destroyed on teardown from the environment handling. Creating a git repo only made sense in the context of testing the git module; therfore, this functionality only existed on the
Finally, the actual test case class will inherit from its closest base test
case. Continuing with the
foo() testing example, the following test case would
# tests/subpackage1/module1/test_foo.py # note that this is importing BaseModule1TestCase from # the tests/subpackage1/module1/__init__.py file from . import BaseModule1TestCase from example.subpackage1 import module1 class FooTestCase(BaseModule1TestCase): def test_that_foo_does_not_crash_when_called(self): module1.foo()
In the end, the following test case hierarchy will be formed. Each class listed inherits from the previous.
BaseTestCase - testlib/testcase.py \_ BaseSubpackage1TestCase - tests/subpackage1/__init__.py \_ BaseModule1TestCase - tests/subpackage1/module1/__init__.py \_ FooTestCase - tests/subpackage1/module1/test_foo.py
Next: Writing Tests