Cove Mountain Software

Embedded Software, Firmware and the Internet of Things

Unit Testing of Active Objects and State Machines

on by 2 Comments

Spotted on Twitter and inspiring this post:

Reviewing the board, we find various state machine related questions:

  • “Still not clear how to test state machines. If I have a home grown switch-style state machine, how would I test it? Using a Mock?”
  • “How does TDD work with higher level designs? Such as State Machines, Operating Systems?”

Testing a State Machine

First: Let me emphasize:

Testing a state machine is fundamentally no different than testing any other class or module.

Matthew Eshleman

When writing unit tests for a state machine, the tests should drive behavior using the module’s public interface while confirming the module’s reactions and outputs. Internally, the module may be using an advanced event driven hierarchical state machine or simply tracking state with internal variables and switch-statements. As with any code-under-test we must be aware of static or member variables retaining state within the code. Then, guided by our Test Driven Development (TDD) process, we add backdoors or new public methods to force the module to a known startup or initialized state before each test. However, because backdoor methods may expose internal design details, I do not recommend them except for initializing to the initial state.

Our goal in writing the tests should be to avoid as much internal knowledge of the module as possible. We do not directly test a state machine, rather we test a module that is internally driven by a state machine.

Let us review a concrete example. The module to be developed will control an external physical hardware lock. This lock may either be “locked” or “unlocked.” Additionally, the external lock supports a self-test function. When completed, the lock’s self-test behavior always returns the lock to its “locked” state. A driver was provided with the following external “C” APIs:

HwLockCtrl “C” driver API

The above HwLockCtrl API represents a “mock’able” interface. More on that later.

Given the limitations in the driver and associated hardware, the team decides to create an event driven active object with an internal state machine to guarantee certain desired behavior. The team required that the module guarantee a locked state upon startup. Additionally, the team required a history-behavior to return the lock to its previous state after completing any requested self-test. The team also decided that tracking lock-state across power loss conditions will be deferred to a higher level application module. The following state chart diagram represents the desired behavior for this active object (aka Service).

Hw Lock Ctrl Service State Chart
The HwLockCtrl Service Internal State Chart Design

At this point we have two critical ingredients necessary to start our TDD process:

  1. An initial set of requirements for the module to be developed.
  2. A software interface for the hardware to be controlled.

The Flat State Machine Class

To avoid any external dependencies, this demo project created its own flat event driven state machine abstract base class. This class was created following a TDD process, but admittedly, I already knew what I wanted from past experience. Please review the code here. Key points:

  • The base class provides basic flat state machine behavior. However, it is abstract and therefore can not be tested directly.
  • In the unit test project for this code we create a concrete “test” state machine. Internally this test state machine records events and state behavior using the CppUTest mocking feature.
  • The unit tests then drive the test state machine and confirm the expected ENTER, EXIT, and event handling behavior. Therefore, these tests confirm the behavior provided by the base FlatStateMachine abstract class.
  • Since this is “just a state machine”, there are no threading concerns.
  • Each test restarts with a new object. Helper methods were created as the TDD process moved forward to “drive” the test state machine to the desired state for further testing.
    • i.e. we do not attempt to “force” the state machine to a particular state to enable additional testing on that state.
    • This avoids backdoors for this particular class.

The Active Object Base Class

Following the example set by Samek and his QActive class, this project then created a simple active object class derived from our flat state machine class. In other words, our active object class “is a” flat state machine. In most firmware or embedded software projects, this class would have dependencies to a particular operating system. We ignore this constraint for this demonstration and instead use C++11 std::thread facilities. Guided by past experience, I knew exactly what was needed in this class to facilitate PC hosted unit testing. Review the code here, noting the following:

  • The “Start()” method provides for a unit testing option where the internal thread context is not created. Instead the internal state machine is immediately initialized in-line.
  • The “ProcessOneEvent()” method is public. In a typical active object class design, this method may not exist or would be private. This method becomes our backdoor to unit testing an active object without creating threads. A key point: this method processes exactly one event and returns true if an event was processed. The method returns false if the internal queue is empty.

To emphasize: This class anticipates the desire to unit test active objects without testing threading behavior. In other words, these backdoors allow our unit tests to focus entirely on the module’s state machine logic.

If the firmware project uses legacy or third-party active object designs, similar “backdoors” will be required to enable PC hosted unit testing. FWIW, I have modified several designs to enable unit testing. If your project could use similar help, please consider our related services.

The Active Object

Now we come to the target code: the “HwLockCtrlService” active object. With the project’s flat state machine class and associated active object class, we now have the foundations necessary to deliver this service.

Mocking the driver

We know the service will access the assigned hardware driver. We also know that we want our unit tests to confirm both the external outputs of the service as well as the critical details of how the service controls the hardware. A mocked driver will enable unit testing observations of the hardware, which will not exist when we execute on our host PC. The demo project uses CppUTest’s mocking facilities. See the demo source code here for our example, noting:

  • The public “C” API for the “HwLockCtrl” driver is clean of unnecessary external dependencies. This makes creating mocks or fakes easier.
  • Just because we are writing unit tests or mock code does not excuse us from software engineering best practices. Adhere to the DRY principle while writing tests and mocks.
  • Workaround for a defect in version 3.8 of CppUTest with respect to “bool” handling.

Testing the Service

With the associated demo project available for review, we note the following TDD steps taken during this development:

  1. Create an empty test project. Build, link, and execute with no real tests. Confirm expected testing framework output.
  2. Add a single empty test. I generally start with a trivial “create and init does not crash” style test.
  3. Create headers and source files for the module-under-test. Once a basic empty class is available, add the files to the test project, link, and execute. If the existing test(s) pass, then move to the first desired behavior. Add a test for the behavior, implement the code, and then fix/improve until all tests are passing.
  4. Each test focuses (ideally) on a single behavior.
  5. As we add tests, we refactor as needed. This includes refactoring the tests themselves to create appropriate helper functions. Even in our unit testing code we should adhere to the DRY principle and software engineering best practices.
  6. The unit tests provide their own callback functions to the active object for certain event notifications from the object under test. These callbacks use the mocking framework to register their activity.
  7. Key point: Since the active object presents an entirely asynchronous public interface, the unit tests must “give CPU time” to the active object to process its internal event queue. See the code for details, but this is where the unit tests use the “backdoor” provided by our base active object class. See the helper method “GiveProcessingTime()” excerpt below:
The GiveProcessingTime() method

Another note on the above “GiveProcessingTime()” method: the method repeats the call to the active object’s ProcessOneEvent() method until the internal queue is empty. This is important, as the target active object may post additional events “to self” while processing the event that kicked-off the test.

At no point in our unit tests are we directly testing the underlying state machine in our target class. Rather, we are testing behaviors accessible through the active object’s public interface and confirming the unit’s impact on the underlying hardware.

A Demo Application

Finally after completing all the TDD driven modules noted above, we created a trivial PC demo application to exercise our Hw Lock Ctrl Service. Doing so confirms the desired behavior with real threads. See the code for details. This demo app is also an example of creating a simple “integration test” application separate from our primary application (firmware or otherwise.)

Summary

Unit testing state machines is as simple as not unit testing state machines. Rather, unit test a software module which internally uses a state machine, making use of the module’s public interface while mocking the dependencies of the module-under-test. Take special care to avoid threading in unit tests and before long you too will be executing hundreds of PC hosted unit tests.

Please comment or reach out if any questions on this topic!

References and Inspiration

The following are useful related references or sources of inspiration for the materials, code, or design behind this post.

2 comments

  1. Pingback: What is an Active Object? - Cove Mountain Software
  2. Pingback: Unit Testing of QP/C (qpc) using CppUTest - Cove Mountain Software

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: