Stopping Bugs: Seven Layers of Defense

Bugs. Defects. Errors. Glitches. Regardless of the term we use, software development teams must work diligently to prevent bugs from invading our code and disturbing our customer’s experience. But how? Many of the best development processes involve layers of defense against bugs. This post explores seven key defensive layers.

Layer 1: Requirements and Specifications

Use Cases. Acceptance criteria. Requirements. Specifications. These are but a few of the terms and forms of documentation that a development team may require to successfully deliver the desired product, service, or application. Poorly conceived or poorly communicated requirements and specifications give bugs their first opportunity to worm their way into our product. Many formal or safety related processes require teams to create appropriate levels of requirements and specification documentation before development proceeds. Mistakes and lack of attention to this layer, if caught late in the product development cycle, are typically the most expensive defects to repair.

Layer 2: Architecture and Design

The overall architecture and design of our system, hardware, or software delivers our next defensive layer. This post is focused on software, so what should we be concerned with here? Will the firmware use a traditional super loop, raw threads, or a flexible publish/subscribe event driven system? How will the software architecture handle communications channels? How will the software abstract interfaces to hardware? Will the user interface code follow a pattern such a MVC? What frameworks will be deployed? How will the software leverage open source projects, if any? Does the architecture play to the team’s development strengths? How does the architecture and design accommodate future changes in requirements? There are many questions to ask and consider to properly construct this defensive layer.

Layer 3: The Compiler

With our other layers well underway and available to guide the software team, developers will likely be ready to write initial code, drivers, or exploratory spikes of functionality. This is the time to ensure the foundations of code quality. Ratchet up the compiler warnings. Turn them into errors. Consider starting the project with at least the gcc equivalent of “-Wall -Wextra -Werror”.

With the compiler preventing as many common coding errors as possible, now is the time to ensure the team is aware of another compile time defense: static asserts. I love a good static assert.

When constructing this defensive layer a team should also consider static analysis tools, such as Lint, clang, CppCheck or related. Integrate the selected tool into a continuous integration system to help squash common coding bugs as soon as possible.

Layer 4: Unit Tests

The code is now building warning free with various forms of analysis alerting us to common coding errors. Now we need to ensure the code is executing our algorithms, business logic, and hardware operations correctly and as expected. Unit tests, preferably following a Test Driven Design process, represent our next defensive layer. To learn more about TDD and unit testing, see my recent presentation here, or just purchase the master’s book. I have also summarized my TDD story and experience here. Don’t forget to add automatic execution of the project’s unit tests to the same continuous integration system assisting with layer three.

Layer 5: Asserts

Layers one through four all help prevent bugs before our code is even executing on the target. The use of run-time asserts represent our first layer of defense where the code is executing and performing its desired function.

Firmware projects are often created with ‘Debug’ and ‘Production’ versions of the code, where various compile options trigger debug outputs, behavior, or asserts. In some project’s the asserts are removed in the ‘Production’ or ‘Release’ builds. I would advise otherwise and would recommend treating these valuable asserts as another strategy and defensive layer in our fight against releasing bugs into the wild. From NASA we receive the wisdom:

“Test what you fly, fly what you test.”

NASA

In other words, keep your software’s debug build as close to the production version as possible. Every deviation becomes another opportunity for bugs to sneak into the project’s release.

I had more to say on this topic, but instead found an external detailed post at memfault expressing my thoughts. To learn more on best practices in using asserts in our firmware and embedded software, check it out.

Layer 6: QA Automated Tests

In a typical consumer electronics embedded software project, this layer of defense is rarely deployed. However, I have seen the benefits of automating the black box testing of products, and depending upon the team’s instincts on the matter, this defensive layer should be considered. Why? Automated testing improves release velocity, confidence in the release, and helps remove human error from our testing results. Automated tests may require some “white box” support from the development team, so we should give some consideration to this defensive layer during our requirements and specifications development stage. If automated tests are desired, be sure to embed a QA presence into the development team’s release cycles to maximize the benefit of this defensive layer.

Layer 7: QA Manual Testing

A well run Quality Assurance (QA) department creates our final defensive layer against bugs. The QA team is likely also in charge of Layer Six, although this may be handled by the development and engineering team. Many QA teams are driven by manual testing, test scripts, and freestyle testing. To ensure the effectiveness of a QA department, the engineering teams should solicit software requirements from QA, especially where it might help the QA team observe the status of the device or extract logs when functionality fails to pass a test. QA should be involved in sprints or other scoped development activities. Ultimately, the QA team has the last and most critical vote on whether a software or product release is ready for manufacturing or customer updates.

Layer Oops: The Customer

Oops. Our defensive layers have failed. A customer is now complaining to technical support or submitting a negative review on Amazon. Despite our best efforts pesky bugs may still manage to crawl into our code. Yet the more we invest time and energy into our defensive layers the less likely it is that this will happen. If a bug does manage to make it to our customers, a well run team will then investigate how the error occurred, perform a root cause analysis, and then re-enforce the appropriate defensive layer.

One last point. The effort required to build the project’s defensive layers will now serve a new purpose: the bug will be easier to fix, easier to prevent moving forward, and far less likely to introduce regressions when repaired.

Did I miss a layer of defense against bugs?

Which layer will you help improve in your team’s current project?

Need help or advice on matters like these? Please see our services.