The Matthew Test: 2022 Survey Results

In honor of the many influences marking my engineering career, I authored the post “The Matthew Test: 15 Steps to Better Embedded Software (and Firmware).” In the past year this post has simmered, brewed, aged, and hit its prime. With invites to a podcast and other events related to ‘The Matthew Test,’ I was inspired to launch a non-scientific internet survey to glean insights into where teams are in their continuous improvement programs. The results are encouraging.

The Basics

The survey first presented a handful of basic demographic questions. Results for the 65 responders are presented below.

Company Size

The survey garnered a nice distribution of responses from people working at the four company size survey options.

Team Size

Small teams dominate the embedded software world and the survey results reflected this reality. This matched my expectations and also aligned with the last embedded.com survey I happen to read.

Team Type

Most people responding belonged to software focused teams, which was perfect for this survey.

Analysis

Overall Average Score

Without further ado, the average Matthew Test score of the 2022 survey was:

9 (i.e. Good)

Great job!

Now, how about the average score broken down by background demographics. First up, Company Size.

Company SizeAvg ScoreEntries
Mega Company (999+)10.213
Large Company (100-999)8.516
Medium Company (25 to 99)8.816
Small Company (1-25)8.920
Average Scores versus Company Size

Perhaps not too surprising, the Mega Companies take the lead. However, Large Companies appear to be missing out, coming in with the lowest average score. I wonder why?

Next up: Team Size.

Team SizeAvg ScoreEntries
Very Large (50+)9.84
Large (10-50)9.26
Medium (5-10)9.417
Small (1-4)8.838
Average Score versus Team Size

Again, not too surprising. The Very Large teams take the lead, however, most of the entries were from Small teams.

Next up, Team Focus.

Team FocusAvg ScoreEntries
Embedded SW9.453
EE/HW7.18
Generic Software8.54
Average Score versus Team Focus

Hard to make any particular conclusions given the entry counts. However, it is nice to see that the bulk of entries were from embedded software focused teams.

Results for Each Matthew Test Question

Matthew Test QuestionYesNo
Item 1: Does your team use source control?62 (95.4%)3 (4.6%)
Item 2: Can team members build, deploy to the target, and execute the code in one step?28 (43.1%)37 (56.9%)
Item 3: Can the team create a release image in one step?34 (52.3%)31 (47.7%)
Item 4: Does the project use continuous integration?31 (47.7%)34 (52.3%)
Item 5: Does the team track development tasks and bugs using a modern system?49 (75.4%)16 (24.6%)
Item 6: Does the product have reasonably complete specifications?43 (66.2%)22 (33.8%)
Item 7: Do the hardware and software teams regularly collaborate?52 (80.0%)13 (20.0%)
Item 8: Do engineers have quiet working conditions?49 (75.4%)16 (25.6%)
Item 9: Does the team perform design and architecture tasks before coding?41 (63.1%)24 (36.9%)
Item 10: Is Test Driven Design/Development (TDD) encouraged?30 (46.2%)35 (53.8%)
Item 11: Do you use the best tools money can buy?24 (36.9%)41 (63.1%)
Item 12: Are future coworkers vetted appropriately?37 (56.9%)28 (43.1%)
Item 13: Is the schedule realistic and updated?37 (56.9%)28 (43.1%)
Item 14: Does the software undergo peer review regularly?45 (69.2%)20 (30.8%)
Item 15: Is there a dedicated quality assurance team?25 (38.5%)40 (61.5%)
The Results

The Top Five

Matthew Test QuestionYesNo
Item 1: Does your team use source control?62 (95.4%)3 (4.6%)
Item 7: Do the hardware and software teams regularly collaborate?52 (80.0%)13 (20.0%)
Item 5: Does the team track development tasks and bugs using a modern system?49 (75.4%)16 (24.6%)
Item 8: Do engineers have quiet working conditions?49 (75.4%)16 (24.6%)
Item 14: Does the software undergo peer review regularly?45 (69.2%)20 (30.8%)
The Top Five

The Bottom Five

Matthew Test QuestionYesNo
Item 11: Do you use the best tools money can buy?24 (36.9%)41 (63.1%)
Item 15: Is there a dedicated quality assurance team?25 (38.5%)40 (61.5%)
Item 2: Can team members build, deploy to the target, and execute the code in one step?28 (43.1%)37 (56.9%)
Item 10: Is Test Driven Design/Development (TDD) encouraged?30 (46.2%)35 (53.8%)
Item 4: Does the project use continuous integration?31 (47.7%)34 (52.3%)
The Bottom Five

Of the worst performing items, we could probably remove Item 11 from consideration. Let’s face it, that is a rather subjective question.

However, “Item 2: Can team members build, deploy to the target, and execute the code in one step?” is generally low hanging fruit. If your team is answering “No” to that question, then take a look at the following resources to help jump start your journey to answering “Yes” to this question:

Of the remaining bottom five performing items, “Item 15: Is there a dedicated quality assurance team?” requires management, budgets, and executive buy in. It is probably the toughest sell. Ultimately some product or feature will launch, thousands of customers will make their purchase, then suddenly hundreds of negative reviews and returns pour in. Sadly, it seems to always take a major failure to force positive change. When your company takes the leap, consider a world class consultant like Rex Black to help guide your foundational efforts for maximum quality assurance effect.

Conclusion

Thank you very much to the many people that responded to this survey. I hope this helps us all gain a new perspective on the embedded software world and where we are collectively hitting our stride and also where we need the most help. I would like to extend a special thanks to James Grenning for promoting the survey, resulting in at least 20% of the entries.

What were you surprised by in the survey results? — Please comment or let us know via Twitter or LinkedIn.

Photo by Daniel Lerman on Unsplash.

1 comment

  1. Via LinkedIn, was asked to breakdown CI Usage (Item 4) by Team Size. Here ya go!

    TeamSize,Yes,No
    Very Large,2, 2 (i.e. 50/50)
    Large,3,3 (again, 50/50)
    Medium,11,6 (Medium size teams almost 2 to 1 using CI)
    Small,15,23 (Small teams frequently not using CI)

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: