In honor of the many influences marking my engineering career, I authored the post “The Matthew Test: 15 Steps to Better Embedded Software (and Firmware).” In the past year this post has simmered, brewed, aged, and hit its prime. With invites to a podcast and other events related to ‘The Matthew Test,’ I was inspired to launch a non-scientific internet survey to glean insights into where teams are in their continuous improvement programs. The results are encouraging.
The Basics
The survey first presented a handful of basic demographic questions. Results for the 65 responders are presented below.
Company Size
The survey garnered a nice distribution of responses from people working at the four company size survey options.
Team Size
Small teams dominate the embedded software world and the survey results reflected this reality. This matched my expectations and also aligned with the last embedded.com survey I happen to read.
Team Type
Most people responding belonged to software focused teams, which was perfect for this survey.
Analysis
Overall Average Score
Without further ado, the average Matthew Test score of the 2022 survey was:
9 (i.e. Good)
Great job!
Now, how about the average score broken down by background demographics. First up, Company Size.
| Company Size | Avg Score | Entries |
|---|---|---|
| Mega Company (999+) | 10.2 | 13 |
| Large Company (100-999) | 8.5 | 16 |
| Medium Company (25 to 99) | 8.8 | 16 |
| Small Company (1-25) | 8.9 | 20 |
Perhaps not too surprising, the Mega Companies take the lead. However, Large Companies appear to be missing out, coming in with the lowest average score. I wonder why?
Next up: Team Size.
| Team Size | Avg Score | Entries |
|---|---|---|
| Very Large (50+) | 9.8 | 4 |
| Large (10-50) | 9.2 | 6 |
| Medium (5-10) | 9.4 | 17 |
| Small (1-4) | 8.8 | 38 |
Again, not too surprising. The Very Large teams take the lead, however, most of the entries were from Small teams.
Next up, Team Focus.
| Team Focus | Avg Score | Entries |
|---|---|---|
| Embedded SW | 9.4 | 53 |
| EE/HW | 7.1 | 8 |
| Generic Software | 8.5 | 4 |
Hard to make any particular conclusions given the entry counts. However, it is nice to see that the bulk of entries were from embedded software focused teams.
Results for Each Matthew Test Question
| Matthew Test Question | Yes | No |
|---|---|---|
| Item 1: Does your team use source control? | 62 (95.4%) | 3 (4.6%) |
| Item 2: Can team members build, deploy to the target, and execute the code in one step? | 28 (43.1%) | 37 (56.9%) |
| Item 3: Can the team create a release image in one step? | 34 (52.3%) | 31 (47.7%) |
| Item 4: Does the project use continuous integration? | 31 (47.7%) | 34 (52.3%) |
| Item 5: Does the team track development tasks and bugs using a modern system? | 49 (75.4%) | 16 (24.6%) |
| Item 6: Does the product have reasonably complete specifications? | 43 (66.2%) | 22 (33.8%) |
| Item 7: Do the hardware and software teams regularly collaborate? | 52 (80.0%) | 13 (20.0%) |
| Item 8: Do engineers have quiet working conditions? | 49 (75.4%) | 16 (25.6%) |
| Item 9: Does the team perform design and architecture tasks before coding? | 41 (63.1%) | 24 (36.9%) |
| Item 10: Is Test Driven Design/Development (TDD) encouraged? | 30 (46.2%) | 35 (53.8%) |
| Item 11: Do you use the best tools money can buy? | 24 (36.9%) | 41 (63.1%) |
| Item 12: Are future coworkers vetted appropriately? | 37 (56.9%) | 28 (43.1%) |
| Item 13: Is the schedule realistic and updated? | 37 (56.9%) | 28 (43.1%) |
| Item 14: Does the software undergo peer review regularly? | 45 (69.2%) | 20 (30.8%) |
| Item 15: Is there a dedicated quality assurance team? | 25 (38.5%) | 40 (61.5%) |
The Top Five
| Matthew Test Question | Yes | No |
|---|---|---|
| Item 1: Does your team use source control? | 62 (95.4%) | 3 (4.6%) |
| Item 7: Do the hardware and software teams regularly collaborate? | 52 (80.0%) | 13 (20.0%) |
| Item 5: Does the team track development tasks and bugs using a modern system? | 49 (75.4%) | 16 (24.6%) |
| Item 8: Do engineers have quiet working conditions? | 49 (75.4%) | 16 (24.6%) |
| Item 14: Does the software undergo peer review regularly? | 45 (69.2%) | 20 (30.8%) |
The Bottom Five
| Matthew Test Question | Yes | No |
|---|---|---|
| Item 11: Do you use the best tools money can buy? | 24 (36.9%) | 41 (63.1%) |
| Item 15: Is there a dedicated quality assurance team? | 25 (38.5%) | 40 (61.5%) |
| Item 2: Can team members build, deploy to the target, and execute the code in one step? | 28 (43.1%) | 37 (56.9%) |
| Item 10: Is Test Driven Design/Development (TDD) encouraged? | 30 (46.2%) | 35 (53.8%) |
| Item 4: Does the project use continuous integration? | 31 (47.7%) | 34 (52.3%) |
Of the worst performing items, we could probably remove Item 11 from consideration. Let’s face it, that is a rather subjective question.
However, “Item 2: Can team members build, deploy to the target, and execute the code in one step?” is generally low hanging fruit. If your team is answering “No” to that question, then take a look at the following resources to help jump start your journey to answering “Yes” to this question:
- Qt Creator: https://doc.qt.io/qtcreator/creator-developing-baremetal.html (I have used this IDE and approach on multiple firmware projects)
- CLion: https://blog.jetbrains.com/clion/2016/06/clion-for-embedded-development/ (On my TODO list, as I love JetBrains IDEs)
- Visual Studio Code: https://mcuoneclipse.com/2021/05/01/visual-studio-code-for-c-c-with-arm-cortex-m-part-1/
- Eclipse IDE: I refuse to provide a link. Skip it and use Qt Creator instead! 🙂
Of the remaining bottom five performing items, “Item 15: Is there a dedicated quality assurance team?” requires management, budgets, and executive buy in. It is probably the toughest sell. Ultimately some product or feature will launch, thousands of customers will make their purchase, then suddenly hundreds of negative reviews and returns pour in. Sadly, it seems to always take a major failure to force positive change. When your company takes the leap, consider a world class consultant like Rex Black to help guide your foundational efforts for maximum quality assurance effect.
Conclusion
Thank you very much to the many people that responded to this survey. I hope this helps us all gain a new perspective on the embedded software world and where we are collectively hitting our stride and also where we need the most help. I would like to extend a special thanks to James Grenning for promoting the survey, resulting in at least 20% of the entries.
What were you surprised by in the survey results? — Please comment or let us know via Twitter or LinkedIn.
Photo by Daniel Lerman on Unsplash.
Via LinkedIn, was asked to breakdown CI Usage (Item 4) by Team Size. Here ya go!
TeamSize,Yes,No
Very Large,2, 2 (i.e. 50/50)
Large,3,3 (again, 50/50)
Medium,11,6 (Medium size teams almost 2 to 1 using CI)
Small,15,23 (Small teams frequently not using CI)