Test Automation Should Start with a Person in the Lab – Not a Script in the Repo
You’re automating your Hardware-in-the-Loop (HiL) testing, right?
Automation holds the promise of efficiency, scalability, and consistency, particularly automated testing in embedded systems development. You’re gonna want to automate. But when’s the right time?
Rushing to automate without the right approach or at the right time often leads to wasted costs, unnecessary delays, and inflexible systems.
Let’s explore the pitfalls of premature automation, how to identify the right moment, and the tools that help ensure automation succeeds.
Why Automating Too Soon Can Hurt Your Progress
“Automation applied to an inefficient operation will magnify the inefficiency.” – Bill Gates
While automation offers undeniable benefits, jumping in too early can derail your development process. Automation magnifies the efficiency, or inefficiency, of your workflows. Automation exacerbates existing problems if you automate too soon.
Complexity of Hardware Integration
Embedded systems tightly couple with specific hardware components, which makes testing inherently complex. Often, teams don’t receive production hardware until later in development.
When triaging test failures, they originate from one of three areas: 1) a “real” bug, 2) a test (content) bug, or 3) an infrastructure bug. The infrastructure bugs are the killers. They waste your time needlessly and destroy trust in the work your doing.
Even with simulation or emulation, teams that automate too early frequently overlook the subtle realities of real hardware integration and end up debugging too many infrastructure failures.
Wasted Resources
Building and maintaining automation systems requires investment in time, tools, and expertise. If your testing processes, hardware, and software continue to evolve, you may find yourself constantly reworking automation scripts, or even frameworks, which wastes both time and budget.
Loss of Flexibility
Automating immature processes locks you into workflows that may not be optimized. When changes inevitably occur, you’ll face the burden of revising your automation frameworks to keep up.
Slowing Down Development
Ironically, automating too early will hinder progress. Instead of improving workflows, your team may spend valuable time troubleshooting automation tools rather than developing and testing the product itself. Troubleshooting automation is inevitable, but it’s slows down progress and wastes time if you’re troubleshooting automation that eventually must be replaced.
When Is the Right Time to Automate?
“[Automate] comes last. The big mistake was that I began by trying to automate every step.” – Elon Musk, in his 5-step process
Automated testing should be introduced as an accelerator, not as an early development step. To achieve this, you must first establish a stable, repeatable, and optimized manual testing process. Here are three ways to know its the right time to automate.
When You’ve Perfected a Process with People
“Yes, excessive automation at Tesla was a mistake. To be precise, my mistake. Humans are underrated.” – Elon Musk, response to a Wall Street Journal interviewer
Your people perfect the process better than any tool can. They are your most valuable resource. Let them run the workflow, refine it, and uncover what matters. Once you’ve done so, then you can dive in with automation.
When You Have Something to Automate
Maybe that sounds simplistic, but it’s also truth. Processes that require frequent repetition, such as unit tests and hardware-in-the-loop (HIL) regression tests, make the best candidates for automation. If you must perform a task the same way dozens or hundreds of times, automation saves time and ensures consistency.
When You’re Ready to Ramp and Scale
As your team grows, ramps up production, or supports multiple hardware variants, complexity increases fast. To keep pace, you can rely on automation. Your team can handle manual regression tests and nightly builds for a small batch, but at scale, those repetitive, time-sensitive tasks will consume valuable engineering time and delay delivery. Automation absorbs that burden and keeps your momentum. You may be able to deliver a hundred of your products with your current process, but can you deliver a million?
People First: Automate the Process, Not the Problem
“Automation is not the enemy of jobs. It frees up human beings to do higher-value work.” – Andy Stern, SEIU President
Effective automation starts with people. The strength of your testing process lies in human insight. How your team identifies critical test scenarios, organizes workflows, and executes tests. Automation should amplify this work, not replace it.
The best automation enhances each person’s contribution to the team and the product.
- Automate testing with the tools, equipment, and procedures that your people already use.
- Automate in place.
- Optimize before you automate.
Best Practices for Successful Automated Testing
When you’re ready to automate, use these strategies to ensure your efforts deliver meaningful results.
The Right Tool: The Person in the Lab
We talked about it above. Start with the person and automate what they do. That means, in addition, that the framework tool that you use (whether you build it yourself, or use an OTS tool like Fuze) must allow you to automate what they do. It must take the position and perspective of the person in the lab.
Start Small and Build From There
Automate low-complexity tasks first, such as unit tests, HiL smoke/basic tests, and nightly regression runs. Building momentum with early wins helps build trust in the process and a solid foundation for more complexity over time.
Embrace Continuous Integration (CI)
Integrate automation into your CI pipeline. Every pull request and merge to main should trigger automated builds and tests, providing immediate feedback to developers, the QA team, and management.
Build a Hierarchy of Coverage vs Runtime
Move from basic CI on PR’s and main merges into nightly, weekly, and release testing. Since you have a finate and inelastic DUT farm, design what to test when based on a hierarchy of coverage balanced against how long it takes to run the testing.
Aim for 10-15 minutes of testing during PR’s. Up that to 30 for a main merge. Aim for a couple hours each night. On the weekends, build in soak and repetitions for many hours at a time.
Automate as many tests as possible for release testing. If you have more than 30 days between releases, think about building a once-a-month release test operation.
Automate Results Publication
Make test results effortlessly visible to your team, or they’re useless. Design your test automation framework to automatically publish results and ensure they’re easy to find and understand in whatever communication tool/workspace that your company uses. Use pass/fail indicators, graphs, and simple visuals to help developers and stakeholders consume the information quickly and confidently.
Mindset Change
As you move toward automation and through the process, your team’s mindset moves from “I must execute testing” to “I must develop tests.” Encourage this because it’s an idicator that you’re on the right path.
Conclusion
Test automation only delivers value when it scales something that already works. For engineering leaders, that means building clarity and repeatability into the process before scaling it. For developers and testers, it means shaping the workflow by hand before encoding it into a framework.
Start with the person in the lab. Let the process mature. Then automate. Because if a process doesn’t work manually, it won’t work at scale—no matter how sophisticated your tooling.
