Hardware-in-the-loop is hard! Sometimes.

What is HIL?

The concept of hardware-in-the-loop (HIL) is not new in the embedded space. The basic idea is to automate the functional tests of your device under development to verify if your implementation does what it’s supposed to do.

Although every embedded project has a phase where the development work is checked and verified in the hardware, it’s not always the case that this battery of tests is performed automatically, systematically, and against a set of requirements.

Moreover, in some cases, the hardware tests are performed in the latter stages of the project. If an error in implementation, a bug, or a failed requirement is found in this stage, the development workload can skyrocket and blow up your timeline.

Why is HIL important?

Unit tests and platform simulations are effective tools to verify that your code is correct. But running tests on hardware to confirm that it’s operational is the best way to fully check the behavior of the system. It is therefore desirable to test early and often!

A key downside to testing your code on hardware is that the initial setup may be time-consuming, and could increase the budget required for your project. This is why it is common to only see full system testing done in the final phase of development of embedded projects.

With an up-front commitment to setting up HIL at the onset of a project, it’s possible to quickly and regularly perform ongoing tests during the development process. Automating a particular test may take some time to set up, but it can then be performed without the intervention of a developer, making it cheaper, faster, and more reliable in the long run. Developers can test device behavior at an early stage in the project, and can systematically check each component as functionality is added.

No single definition of “hardware-in-the-loop”

You may now have a basic understanding of what HIL enables and are wondering why developers don’t just do HIL all the time. HIL doesn’t (usually) just come in a box with a simple implementation. Each project has unique caveats, and understanding testing needs is critical to deciding whether an HIL implementation is worth the additional effort. Automatic tests may be critical to the success of one project, but negligible for another.

What does a real HIL setup look like?

A HIL system may resemble the following: The Device Under Test (DUT) is the device or system whose correct functionality we are testing. The HIL component will be the main component checking the proper functionality of DUT. This means at least being able to load an image into the DUT and retrieving its status. Ideally, we would prefer to read all the registers and the memory of the DUT. Even better we can modify them at will – although that can be difficult, depending on the chipset or the platform used.

Usually, embedded projects have some communication, either WiFi, Bluetooth, Ethernet, SPI, I2C, or CAN, etc. In those cases, HIL should communicate with the DUT, transmitting and receiving messages when required.

The same thing happens with GPIOs; the HIL should be able to toggle and detect toggles if the DUT uses them.

In some cases, the HIL can’t accommodate all the testing needs: not fast enough GPIO toggles, a particular communication interface missing, etc.

In those cases, it should act as a controller for other devices that will perform the actual interaction with the DUT.

In the same way, we may want to measure things that the HIL can’t, like speedy changes in the signal, the power consumption of the DUT, or its temperature. External devices can do this at the HIL’s command.

And the HIL should have some way to retrieve all the data and possibly evaluate if the tests have passed or not. Finally, it should report that.

If you have ever worked on an embedded project, you are likely picturing how HIL would look in an actual project of yours. It may be more complex than you may think. HIL can simulate environmental controls such as higher temperatures, increased humidity, different pressures, or even atmospheric environments. You can perform cyberattacks to assess security, test the system over time to see how it evolves, measure its spectrum radiation, or apply mechanical vibrations. If there’s a requirement to test against, HIL can help.

HIL is HARD! Sometimes.

A basic HIL setup can get you started fairly quickly, but the complexity can increase exponentially as more elements are included in the test set.

For example, a system composed of 3 Finite State Machines, each with five states, will have 125 total states. If there are also 5 peripheral pins to be tested along with a CAN interface with 10 possible messages, the number of combinations to test just some of the desirable paths (assuming that all messages are correctly processed and the queues are not full) is at least 7250. Assuming, optimistically, that each test takes 2 minutes to run and report, and that the tester has no inefficiencies, and works eight hours per day…12.5 working days will be required to pass all of the tests. This slows development and increases costs.

Fortunately, in the development stages of the project, not all the tests need to be passed. This puts the developer in the difficult position of having to decide which tests are part of the release pipeline and which tests are for development. Estimating this is a crucial step, as it may drastically change the testing time. Depending on relevant industry standards, developers often need to present these tests to regulatory agencies. Moreover, depending on the test-set and the hardware required, the tests could be parallelized in multiple HIL setups, and reduce the testing time.

Compromises in the Product Lifecycle

As you can see, doing HIL by the book can be costly and complex. Companies often choose to compromise and decide which subset of tests will be the ones they pursue. A method of testing one component of the system at a time and ensuring that it works may be the best balance between the complexity of the HIL test-set and DUT verification. But a developer should consider that this does not verify the DUT thoroughly. In some cases (e.g. safety-related applications), that’s just not enough. For the rest, this means that although we may have a high level of certainty that the system works, we need to constantly monitor the system after deployment. The ability to generate, send and analyze status reports of deployed devices may help you detect possible problems fast, add more specific tests to your use cases, and iterate in the development of your product.