Usage#

Installation#

To use Colorful Test, first install it using pip:

$ pip install colorful-test

Basic Example#

Colorful Test provides a handful of useful tools to help you construct and run tests. Here’s a basic example of some of those tools in action:

from colorful_test import TestCase, show_message
from solution import add, mul, div

class TestSolution(TestCase):

    @show_message(
        success='Your add method works as expected',
        fail='''
        Your add method doesn't work as expected. Hints:

        Expected: %s
        Received: %f
        '''
    )
    def test_basic_addition_2(self):
        self.assert_equal(add(1, 1), 2)
        self.assert_equal(add(1, 2), 3)

    @show_message(
        success='Your mul method works as expected',
        fail='''
        Your mul method doesn't work as expected. Hints:

        Expected: %s
        Received: %f
        '''
    )
    def test_basic_multiplication_0(self):
        self.assert_equal(mul(1, 1), 1)
        self.assert_equal(mul(1, 2), 2)

    @show_message(
        success='Your div method works as expected',
        fail='Your div method should raise a ZeroDivisionError if the second argument is 0'
    )
    def test_basic_division_error_1(self):
        self.assert_raises(ZeroDivisionError, div, 3, 0)

if __name__ == '__main__':
    TestSolution.run_and_output_results()

A TestCase is created by inheriting TestCase. The test runner looks for methods that start with test_ and considers them as tests. These tests are then ordered based on the number appended to their names.

In each test, assert_equal or assert_raises can be used—these work similarly to assertEqual and assertRaises from the unittest framework. assert_equal checks if the first and second arguments are equivalent, while assert_raises verifies whether the specified callable raises an error with the given arguments. However, you don’t have to use these methods—you can simply use assert, and the test runner will still accumulate all test results and generate a test report.

One thing to note is that test_basic_multiplication_0 will be executed first, followed by test_basic_division_error_1, and so on. This happens because of how the tests are ordered. Appending a number to test method names is not mandatory—if omitted, test methods will be ordered alphabetically.

Command Line Interface#

The tests are run in the command-line interface. While the framework doesn’t yet offer as many commands or as much flexibility as other Python frameworks, you can still run tests easily.

To execute a test file, run one of the following commands:

$ python <test_file>

$ python -m colorful-test <test_file>/<directory>

If you specify a directory, Colorful Test will find all Python files in that directory that start with test_ and execute them.

Organizing your code#

The basic building blocks of unit testing are test cases—single scenarios that must be set up and checked for correctness. Colorful Test’s TestCase provides functionalities to create test cases and test methods, making it easy to test different scenarios.

You don’t always have to use show_message, as seen in the Basic Example, when writing test methods. Here’s another basic example of a test case:

from colorful_test import TestCase
from person import person
from account import Account

class TestBankAccount(TestCase):

    def test_account_deposit(self):
        josh = Person('Josh', 'Gilder')
        account = Account(josh)
        account.open()

        account.deposit(500)
        self.assert_equal(account.get_balance(), 500)

if __name__ == '__main__':
    TestBankAccount.run_and_output_results()

That is a simple test case that checks if money can be deposited into an account. It uses one of our assert_* methods to achieve this.

Note that we have to call the run_and_output_results method to run our test case. Without calling the runner, nothing will happen. But if we do call it, we’ll get a report on how our unit tests ran—even without using show_message.

This is how the output will look:

Ran 1 tests in 0.000078 ms
1 passed
0 failed
0 skipped
0 error(s)

✓ Grade: 100.0%

Grade is just another metric that specifies the success rate of our tests. It can be useful in educational environments.

Tests can be numerous, and their setup can be repetitive. That’s where set_up comes in—it’s a method that is called before each test runs, providing a setup that will be used by the test methods.

Here’s an example:

from colorful_test import TestCase
from person import person
from account import Account

class TestBankAccount(TestCase):

    def set_up(self):
        self.josh = Person('Josh', 'Gilder')
        self.account = Account(josh)
        self.account.open()

    def test_account_deposit(self):
        self.account.deposit(500)
        self.assert_equal(account.get_balance(), 500)

if __name__ == '__main__':
    TestBankAccount.run_and_output_results()

If the set_up() method raises an exception while the test is running, the framework will consider the test to have encountered an error, and the test method will not be executed.

Similarly, we can provide a tear_down() method to clean up after the test method has run:

from colorful_test import TestCase
from person import person
from account import Account

class TestBankAccount(TestCase):

    def set_up(self):
        self.josh = Person('Josh', 'Gilder')
        self.account = Account(josh)
        self.account.open()

    def tear_down(self):
        self.account.close()

By default, tests will fail fast, meaning the run will stop immediately after encountering the first error or failure. You can change this by setting fail_fast to False in run_and_output_results.

Here’s an example:

if __name__ == '__main__':
    TestBankAccount.run_and_output_results(fail_fast=False)

Using run_and_output_results will run your test case and display the results in the console. However, you can also use the run method, which won’t print anything to the console but will return a TestResults object, allowing you to customize how you handle errors and different scenarios.

Example using the run method:

if __name__ == '__main__':
    TestBankAccount.run(fail_fast=False)

Skipping tests#

Colorful Test supports skipping test methods. Skipping a test is as simple as raising a SkipTest exception—the test runner will recognize this as a skipped test.

Here’s an example of how you can skip tests:

import colorful_test

class ExampleTestCase(colorful_test.TestCase):

    @colorful_test.skip(reason='showing how skip works')
    def test_one(self):
        # Test will be skipped
        pass

    @colorful_test.skip_if(not feature_enabled("dark_mode"), reason="Dark mode is not enabled")
    def test_two(self):
        # Test will be skipped only if dark_mode is not enabled
        pass

    @colorful_test.skip_unless(sys.platform == "win32", reason="Only supports Windows")
    def test_three(self):
        # Test will be skipped unless platform is Windows
        pass

The following decorators and exception implement test skipping:

colorful_test.skip_test.skip_test(reason)#: Calling this during a test method or set_up() skips the current test.

colorful_test.skip_test.skip_test_if(reason)#: Calling this during a test method or set_up() skips the current test if the condition evaluates to True.

colorful_test.skip_test.skip_test_unless(reason)#: Calling this during a test method or set_up() skips the current test unless the condition evaluates to True.

Classes and functions#

TestCase#

class colorful_test.testcase.TestCase#

Bases: object

Base class that implements the interface needed by the runner to allow it do drive the tests, and methods that the test code can use to check for and report various kinds of failures.

class TestResult#

Bases: object

Class that stores the results of the tests ran by run(). It provides functionalities to display output to the screen.

add_test(status: str, order_num: int, name: str, errors: Exception | None = None) → None#: Adds a test and its results.

get_total_tests_ran() → int#: Returns the total number of tests ran.

classmethod add_cleanup(function: Callable[[Any], None], *args: Any, **kwargs: Any) → None#

Add a function to be called after tear_down() to clean up resources used during the test. Functions will be called in reverse order to the order they are added (LIFO). They are called with any arguments and keyword arguments passed into add_cleanup when they are added.

If set_up() fails meaning that tear_down() is not called, then any cleanup functions will still be called.

assert_almost_equal(first: Any, second: Any, places: int = 7) → None#: Test that first and second are approximately equal by computing the difference, rounding to the given number of decimal places (default 7), and comparing to zero.

assert_count_equal(first: Any, second: Any) → None#: Test that sequence first contains the same elements as second, regardless of their order. Duplicates are not ignored.

assert_dict_equal(first: Any, second: Any) → None#: Test that two dictionaries are equal.

assert_does_not_raises(exception: BaseException, callable: Callable[[Any], Any], *args: Any, **kwargs: Any) → None#: Test that an exception (specific) is not raised when callable is called with any positional or keyword arguments. The test passes if exception is not raised, is an error if another exception is raised, or fails if exception is raised.

assert_equal(first: Any, second: Any) → None#: Test that first and second are equal. If the values does not compare, the test will fail.

assert_false(expr: Any) → None#: Test that expr is False.

assert_greater(first: Any, second: Any) → None#: Test that first is > than the second. If not, the test will fail.

assert_greater_equal(first: Any, second: Any) → None#: Test that first is >= than the second. If not, the test will fail.

assert_in(first: Any, second: Any) → None#: Test that first is in second.

assert_is(first: Any, second: Any) → None#: Test that first and second evaluate to the same object.

assert_is_instance(obj: object, cls: type) → None#: Test that obj is an instance of cls.

assert_is_none(expr: Any) → None#: Test that expr is None.

assert_is_not(first: Any, second: Any) → None#: Test that first and second does not evaluate to the same object.

assert_is_not_none(expr: Any) → None#: Test that expr is not None.

assert_less(first: Any, second: Any) → None#: Test that first is < than the second. If not, the test will fail.

assert_less_equal(first: Any, second: Any) → None#: Test that first is <= than the second. If not, the test will fail.

assert_list_equal(first: Any, second: Any) → Any#: Test that two lists are equal.

assert_not_almost_equal(first: Any, second: Any, places: int = 7) → None#: Test that first and second are not approximately equal by computing the difference, rounding to the given number of decimal places (default 7), and comparing to zero.

assert_not_equal(first: Any, second: Any) → None#: Test that first and second are not equal. If the values do compare equal, the test will fail

assert_not_in(first: Any, second: Any) → None#: Test that first is not in second.

assert_not_is_instance(obj: object, cls: type) → None#: Test that obj is not an instance of cls.

assert_not_regex(text: str, regex: Pattern[str]) → None#: Test that a regex search does not math the text.

assert_raises(exception: BaseException, callable: Callable[[Any], Any], *args: Any, **kwargs: Any) → None#: Test that an exception (specific) is raised when callable is called with any positional or keyword arguments. The test passes if exception is raised, is an error if another exception is raised, or fails if no exception is raised.

assert_raises_regex(exception: Exception, regex: str, callable: Callable[[Any], Any], *args: Any, **kwargs: Any) → None#: Like assert_raises() but also tests that regex matches on the string representation of the raised exception.

assert_regex(text: str, regex: Pattern[str]) → None#: Test that a regex search matches the text.

assert_sequence_equal(first: Any, second: Any, seq_type: Any | None = None) → None#: Test that two sequences are equal. If a seq_type is supplied, both first and second must be instances of seq_type or a failure will be raised.

assert_set_equal(first: Any, second: Any) → None#

Test that two sets are equal.

Fails if either of first or second does not have a set.difference() method.

assert_true(expr: Any) → None#: Test that expr is True.

assert_tuple_equal(first: Any, second: Any) → None#: Test that two tuples are equal.

classmethod do_cleanups()#: The method is called unconditionally after tear_down(), or after set_up() if set_up() raises an exception.

final_message() → str | None#: This method is called after tests in an individual class run. The returned message is printed as the last message if all tests passed. The default implementation returns None

classmethod run(fail_fast: bool = True) → TestResult#: Run the tests, collecting the results into TestResult object. The result object is returned to run()’s caller.

classmethod run_and_output_results(fail_fast: bool = True, show_grade: bool = True) → TestResult#

set_up() → None#: This method is called immediately before calling a test method; other than AssertionError or SkipTest, any exception raised by this method will be considered an error rather than a test failure. The default implementation does nothing.

classmethod set_up_class() → None#: This method is called before tests in an individual class run. The default implementation does nothing.

tear_down() → None#: This method is called immediately after calling a test method; other than AssertionError or SkipTest, any exception raised by this method will be considered an error rather than a test failure. This method will be executed only if set_up succeeds. The default implemantation does nothing.

classmethod tear_down_class() → None#: This method is called after tests in an individual class run. The default implementation does nothing.

Message#

colorful_test.message.show_message(fail: str | None = None, success: str | None = None) → Callable[[Callable[[], None]], Callable[[Tuple[Any], Dict[str, Any]], None]]#

Decorator used to print a failure message if a test fails or a success message if a test passes.

If one or the other is specified, show_message will only print for that case. For example, if you don’t want to print anything when a test passes, you can omit the success message.