The Matthew Test: 15 Steps to Better Embedded Software (and Firmware)

Many years ago, I helped drive the selection of a shiny new case and bug tracking system for our embedded software team at Toshiba. During that process I discovered Joel Spolsky, founder of Fog Creek Software. We ultimately selected FogBugz and thus began my exposure to Joel’s world-class software development advice, including his famous “The Joel Test” post.

In “The Joel Test“, he lays out a set of questions to help a team assess their software development practices. Inspired by Joel’s legendary post, I hereby introduce:

“The Matthew Test: 15 Steps to Better Embedded Software (and Firmware).”

The Questions

Tally a point for each “yes” answer. Add up the points to rank the organization.

1. Do you use source control?
2. Can you build, deploy to the target, and execute the code in one step?
3. Can you create release images in one step?
4. Do you use continuous integration?
5. Do you track development tasks and bugs using a modern system?
6. Does the product have reasonably complete specifications?
7. Do the hardware and software teams regularly collaborate?
8. Do engineers have quiet working conditions?
9. Does the team perform design and architecture tasks before coding?
10. Is Test Driven Development encouraged?
11. Do you use the best tools money can buy?
12. Are future coworkers vetted appropriately?
13. Is the schedule realistic and updated?
14. Does the software undergo peer review regularly?
15. Is there a dedicated quality assurance team?

Ranking

• Excellent: 12-15 points
  • This embedded software organization is in the top-tier, consistently delivering quality software. Prepare continuous improvement plans targeting the full 15 points.
• Good: 8-11 points
  • Your team is on track. Keep it up and pick the next step to improve upon.
• Lucky: 4-7 points
  • This team may be successful in the short term but probably runs on a great deal of luck, sweat, and perseverance.
• Ouch: 0-3 points
  • This organization needs serious short- and long-term guidance on improving their software development practices.

Description and Details

2. Can you build, deploy to the target, and execute the code in one step?

The lifeblood of a firmware engineer is the process of building a firmware image, deploying the image, and executing on the development target hardware. These steps might be executed hundreds of times a day. Therefore, the process must be reliable and efficient. Embedded software engineering teams should regularly invest time into improving this process with the ideal end-goal of reducing the steps to a single mouse-click or makefile command. Modern IDEs such as Eclipse, CLion, and Qt Creator all include the necessary functionality to enable this goal.

3. Can you create release images in one step?

Despite the similarity and overlap with the previous question, I have broken this topic out due to the critical nature of this process in our industry. Getting this process wrong may result in a delay at the factory or an embarrassing and costly product recall. Or worse, it may cause loss of life and property.

Modern embedded systems often consist of families of products, each with a unique configuration of options, secrets, bootloaders, application executables, and other supporting data. Packaging these artifacts to ensure a coherent and consistent release to the factory should be automated such that a release to production is created with a single step. Documented release checklists are the bare minimum requirement here, but they are still prone to human error. Automate the process to avoid human error.

4. Do you use continuous integration?

A fully configured continuous integration (CI) server monitors your project’s source code repository for the latest revisions, performing various build or analysis functions as desired by the team. The server may automatically build the software project while confirming an error free build. If the build fails, the server automatically sends an email to the person responsible for the failing commit. These automated features alert the engineers and thereby help ensure rapid repair efforts.

The server may be configured to automatically run any unit tests, perform static analysis of the code, or whatever tasks are required by the team. A fully configured continuous integration server can help measure overall software health. For example, the pass/fail history of builds may become a proxy measurement of software maintenance concerns. Too many random build failures? Maybe the software needs refactoring. I find it particularly valuable to have the CI server fully integrated with the team’s ongoing source code review and/or pull request system.

Examples of continuous integration systems include: Jenkins, Bamboo, TeamCity, and many others.

5. Do you track development tasks and bugs using a modern system?

A single source of truth. Within a year or two of our team’s transition to FogBugz, our little server in Nashville was being accessed globally by Toshiba project managers, executives, and engineers everywhere. It quickly became the single source of truth for our assigned projects. The system dynamically generated graphs demonstrating the rate of change in bugs or tasks being closed, invaluable information as projects approached critical mass production deadlines. Each engineer knew exactly where to go to determine which task or bug to tackle next. Eventually, we even managed all specifications as unique cases in FogBugz, allowing anyone interested in the spec the ability to subscribe to the case and receive an automated email when it was updated.

By the time our business unit started to shut down, our “little” server was all grown up with many hundreds of users, multiple dozens of projects, and approaching hundreds of thousands of cases, tasks, bugs, and inquiries. It was a gold mine of meta information on our history, progress, and state. It was invaluable.

If your team is still managing tasks or bugs with emails, shared spreadsheets, or other informal methods, then stop, do not pass go, and begin your own selection process.

My recommendation: anything other than emails and spreadsheets. I would recommend FogBugz, but it has new owners with whom I have no direct experience. These days everyone seems to use Jira, a system I almost like.

6. Does the product have reasonably complete specifications?

Reliability requirements. Self-tests. Factory requirements. Warranty rework and service requirements. Real-time processing requirements. Security requirements. Battery management requirements. Hardware timing requirements. White box QA testing requirements. Certification requirements. Connectivity requirements. Oh, and by-the-way, the end-customer probably has a few requirements too.

As we can see, the creation of modern electronic products involves many specifications and requirements. Does the company maintain a written product specification, or does the organization informally communicate the latest requirements to the engineering team? Without written specifications the team will inevitably lose requirements or overlook an important use case. Additionally, if a key project manager or engineer departs for another opportunity, critical specification knowledge may be lost. Write it down and agree to it. Yes, in the age of Agile-derived processes this may seem a bit old-school, but it works and may very well save the company a great deal of internal confusion and rework costs.

7. Do the hardware and software teams regularly collaborate?

In some product development organizations, there may be a formal (or informal) wall between the hardware and software engineering teams. As Ronald Reagan once said, “Tear Down this wall!”

Bringing the hardware and software teams together especially during the foundational early design stages is critical to later productivity, usability, and debug-ability of the product in question. The hardware team should include the software team in any milestone schematic reviews. For example, the firmware team may request a spare GPIO line be easily accessible and dedicated to software usage. Additionally, the software team may note that the JTAG or SWD port is inaccessible when the product is assembled. Such requests and tradeoffs may not be considered without the full team’s input.

My recommendations:

• Include software engineers in schematic reviews.
• Schedule a recurring meeting between key members of the hardware and software engineering teams.

8. Do engineers have quiet working conditions?

I am a very focused and productive engineer. To some extent this is part of my gifts and talents. However, today I am officially revealing my secret sauce: I have a dedicated office with a door, and I frequently keep the door shut. My thoughts are not interrupted by the latest office announcements nor by an office mate boisterously scheduling an appointment. Aggravating this common work-place challenge, many firmware or embedded software engineers are co-located within a factory or production environment where interruptions and noise may be commonplace. There have been many studies on this topic, and it seems that software development is particularly prone to reduced productivity due to interruptions. Quoting Joel:

“With programmers, it’s especially hard. Productivity depends on being able to juggle a lot of little details in short term memory all at once. Any kind of interruption can cause these details to come crashing down.”

Joel Spolsky: “8. Do programmers have quiet working conditions?“

Solving problems with interruptions and noise in your environment may require a concerted effort and cultural shift, but it may result in surprising gains in quality and time to market. My recommendation: do not ignore this critical issue.

9. Does the team perform design and architecture tasks before coding?

If creating a confusing mess of spaghetti software is the goal, then by all means start writing code with no upfront design. However, if you would like to create maintainable, useful, and reliable software, I would recommend a bit of up-front-design.

This is a difficult activity to balance out. In my career I have seen rather terrible spaghetti code driving commercially successful products. I have also observed an architect or engineer stuck in analysis paralysis, resulting in delays and later panic development efforts. A balanced up-front design effort is required, proportional to the business scope and safety impact of the product in question. Will this code be used for derivative products for many years to come or is it a one-off project? Would the business benefit from creating an internal library of re-usable software? Is there a standard software pattern we should be using for this module? Questions like these should be considered during the design and architecture of embedded software.

If working on a smaller or more informal project, my favorite tool for this activity is the whiteboard, a cell phone camera, and a wiki. As the project scope, scale, or seriousness increases, then so do my tools. I’ve been using Visio since my first professional engineering job, but admittedly it is not licensed as fairly by Microsoft as it once was. There are many alternatives from heavy and strict UML tools to simple shared diagraming web services.

My recommendation: a team wiki with a section on architecture and design for each project with a breakdown of each formally defined software module. Just remember to adhere to the DRY principle and avoid documenting details best left to the code itself, and if your team could use experienced guidance on this topic, please review our related services.

A popular and related post: “Questions to Ask: Software Architecture, Design, And Coding.”

10. Is Test Driven Development encouraged?

Upon departing Toshiba (I miss that team!) I first sampled Test-Driven Development (TDD) in the context of a C#/.Net based project. I immediately loved it. Sometime later I transitioned into software consulting and contracting focusing on my favorite world: embedded software and firmware. As I was making this transition, I absorbed James Grenning’s excellent book “Test Driven Development for Embedded C.” My first firmware contracting job was substantially driven by a TDD process inspired by Grenning’s book, and I was forever sold. Unit testing and TDD processes remain uncommon in the firmware and embedded software domains, yet I believe TDD is an important indicator of a professional and modern software engineering team. Go for it!

My recommendation: hire Grenning to train your team on TDD for embedded. If Grenning isn’t available, then perhaps consider this option.

11. Do you use the best tools money can buy?

Penny wise and pound foolish, many organizations fall prey to this mental trap. I am frequently surprised by managers overlooking the high-costs associated with engineering personnel while quibbling over the price of a build-server, an oscilloscope, or a signal generator. Engineering time is expensive and time to market is critical. Do not short-change the equipment needed by the development team. Thankfully, I had a boss that fully understood this issue during my tenure at Toshiba. See my related post: “It makes engineers more productive.”

12. Are future coworkers vetted appropriately?

The equivalent question in Joel’s post was worded: “Do new candidates write code during their interview?” A great question, but I believe the root concern involves the broader question of how we interview and sort through job candidates. After all, we may end up working for years or even decades with the resume, oops sorry, the person sitting across the table.

I love writing code, but I must admit: I don’t particularly enjoy writing yet-another-linked-list-class while sitting in a conference room using a strange laptop with some odd editor in front of random people I just met five minutes ago. Sorry Joel, just isn’t fun.

What is fun and useful? Talk with the candidate. Ask them about their best moments and their hardest moments. Dig in. Ask them about the device driver they recently implemented. Ask them if the datasheet for the device was accurate or not. See if they have a sense of humor! While doing all that, make sure they can handle criticism in a professional manner. Then dig in a bit further to make sure they are willing to learn and apply that knowledge to your company’s engineering challenges. Because ultimately that is the coworker we all need.

13. Is the schedule realistic and updated?

“Death marches”. The very phrase evokes fear in the hearts of software engineers everywhere. I have been on a death march project and was not sent to Cancun afterwards unlike Joel’s Microsoft Word experience. There is only one good thing about such experiences. Afterwards you swear:

NEVER AGAIN!

All software engineers and developers everywhere after a death march experience.

There are many possible root causes to such a terrible experience, but one that typically ranks high is an unrealistic development schedule. Was the schedule created by pre-selecting the end-date and working backwards? This is typical in the consumer electronics world. A particular trade show, Christmas, or Best Buy’s warehousing deadlines may drive the business and with it millions of dollars in revenue or worse: contractual penalties. Did the hardware development team create the schedule with no input from the firmware team? Sadly, this too is typical. A realistic and thoughtful schedule should keep a project on track as well as feed into the original business plan.

I personally learned a great deal about estimating, project management, architecture, design, and software development best practices as a result of my death march experience. There are many excellent references, but I would recommend starting with almost anything written by Steve McConnell. “Code Complete” and “Rapid Development” were my software bibles for many years, and it appears I have missed a new book from Steve: “More Effective Agile: A Roadmap for Software Leaders.” I am adding that to my cart right now.

14. Does the software undergo peer review regularly?

As engineering professionals, we generate many different artifacts from design documents to schematics and source code. The electrical engineering team we partner with understands the benefits of peer review: schematic reviews are commonplace. The benefits are understood in software engineering as well, although many firmware teams continue to skip this valuable practice. In some situations, a firmware team may consist of only a single solo engineer. Such teams may benefit from an external code review service.

If the software team already uses a modern revision control process, then you are only a few steps away from benefiting from ongoing peer code reviews. For example, if the project is using git, then with minimal effort the team could (and should) embrace a source code review process centered around pull-requests.

To learn more about this critical best practice, I’ll refer you to one of my embedded heroes, Jack Ganssle, and his writeup on software code inspections.

15. Is there a dedicated quality assurance (QA) team?

Automated black box testing. Manual human-driven test scripts. Pre-certification quality checks. Long term reliability testing. Interference tests. Brown-out testing. Edge case testing. Performance tests. Usability tests. Acceptance tests. Smoke testing.

Those are but a few of the tasks a dedicated quality assurance (QA) department may tackle. As an engineer delivering software to a QA team and receiving feedback and bugs from a QA team, it is difficult for me to imagine an organization efficiently and confidently releasing a product without a QA team. And yet it happens. Many companies skirt by without a formal QA team, relying on their engineers, managers, executives, and even the sales team to informally test and find deficiencies in the product. It is certainly possible to successfully launch a product without a QA team, yet just as unit testing increases confidence when delivering software, a solid QA team will help ensure a no-surprises product launch with minimal customer complaints and warranty claims.

Conclusion

I sincerely hope this post is useful. It was a labor of love and a salute to the many people that influenced my career, including the hard-working engineers and executives at Toshiba, Joel Spolsky, Jack Ganssle, Steve McConnell, and others. Please let me know if any of this information was useful and especially let me know if your organization applies any of the advice here. I would love to hear from you!

Does your organization need help on one or more of these items? If so, please check out our related services.