Arm is no stranger to DevOps or Continuous Integration (CI). They have several pipeline scripts available for Travis CI and outline how they use Jenkins to build and test Mbed OS. But so far, there has not been a demonstration of how to use a simulator in pipelines. We partnered with Arm to show how to use Fixed Virtual Platforms (FVPs) to test your code in a CI pipeline. We are using EmbedOps, a CI tool compatible with GitLab that is designed for embedded developers. We will share how we’ve seen onboard testing, real, and simulated; how we have helped embedded teams and how we accomplished it using Arm Ecosystem FVPs for the Cortex-M55; and we share some of the improvements we would like to see to make this tool work even better for FVPs.
What simulated hardware testing does for your team
In the development of embedded systems, real-device testing is mandatory before releasing to capture bugs that show up only in real hardware. To have consistent functional verification, manual testing is done by developers when new features are implemented. This can be time-consuming and error-prone. Sometimes, the developer could be debugging a problem with the firmware that could actually be a hardware issue. By using simulated hardware, we can reduce the chance of seeing bugs on the real hardware, and have a reliable way to eliminate firmware as a root cause when they do occur.
Complete development on a tight schedule
Development time can be reduced even further by integrating the hardware simulation into CI pipelines. With CI integration, the firmware can be tested with the simulated hardware with every push to the version control platform.
Start development in advance
With simulated hardware, development can be started before hardware is available. It normally takes time for the development boards or custom boards to be delivered to the developers. Instead of waiting for the boards, developers can start implementing and testing new features by using simulated hardware.
Lower the cost of prototyping
With the advancement of technology, electronics hardware is getting less expensive. But that is not the case when it comes to product development, where lots of expensive prototypes are usually required. Instead of every developer having real hardware, each of them can have a copy of the simulated hardware on their computers. Let us take the recent pandemic as an example. Developers have started to work remotely from home at unprecedented rates. With each developer having the simulated hardware, development can get started immediately. And while final validation is performed with real hardware before releasing, testing can happen earlier in the process, avoiding the costs of spinning up new hardware revisions.
How we built a CI pipeline for simulation
The CI pipeline of the Arm Cortex-M55 hello world example has two simple jobs: a “build” job that builds the source code and generates the AXF files, and a “simulate” job that runs the simulation. The AXF file is an object file generated by the Arm compiler, which contains both object code and debug information. For this demonstration the source is not actually compiled. Instead, a simple script fakes the build and copies a pre-built AXF file to the next stage.
With a Docker container, the simulation can be run in an isolated environment. While a developer has to build the docker container manually, on the EmbedOps platform the same developer can modify the Dockerfile included in this example project to build your own FVP image. The Dockerfile downloads the FVP from the Arm developer website, installs the requirements for running the simulation, and has the simulator ready to be used in the CI pipeline. To create a custom Docker image for a different FVP, update the URL and file name with the desired FVP.
FVPs include a UI and the UI is opened automatically when simulation starts. When working locally with the simulators, this visual representation is valuable. However, to incorporate simulation in a CI pipeline, the UI is more of an impediment than an asset. Having a version of the simulator that is designed to work on the command line smooths this process and increases the value of using the FVPs in a CI system. Automated reports on how the model is running save the developer time manually checking the model and provide a recorded history of how the model performed as code was developed. For CI purposes, we created a config-ci.txt based on the config.txt file you would use locally with the UI and made the following modifications.
Set the visualization parameter to 1 to disable the UI window.
Add a file location for an output file.
In the “simulate” job, a shell script is executed to run the simulation. To run the simulation, the AXF file and the config-ci.txt file are passed into the model executable and the simulation is limited to run for one minute as shown in the following example.
While the simulation is running, a telnet session is created to capture the serial print statement output by the program. To validate that the AXF file is actually running on the simulator, the script checks if the expected output string exists in the uartOutputFile. The job passes if the output matches known-good output.
A screenshot example of the FVP simulator GUI.
00000000 ff fd 03 ff fb 03 ff fd 00 ff fb 00 ff fb 01 0d |................| 00000010 0a 3d 20 42 6c 69 6e 6b 79 20 69 73 20 72 75 6e |.= Blinky is run| 00000020 6e 69 6e 67 20 3d 0d 0a 4c 45 44 20 74 69 6d 65 |ning =..LED time| 00000030 72 20 69 6e 74 65 72 76 61 6c 20 69 73 20 73 65 |r interval is se| 00000040 74 20 74 6f 20 31 73 0d 0a |t to 1s..|
Where we are hoping to see simulators go
Official Docker images
We created an FVP model Docker image for this example, so we know it is not impossible nor difficult. But maintaining these images and supporting security checks and updates is often put on the back burner when teams are busy with all the work of getting a new product shipped. With official FVP model Docker images, the time for developers to get started using the Arm simulators both locally and on a CI pipeline would be markedly shortened. And it would allow for teams to work together without worrying about differences in their development environments affecting results. In the meantime, the EmbedOps container registry hosts a Corstone-300-FVP-M55 container that is regularly scanned and updated.
Dojo Five brings modern tools, techniques, and best-practices from the web and mobile development environments, paired with leading-edge innovations in firmware to our customers to help them build successful products and successful clients. We have talented engineers on hand ready to help you with all aspects of your EmbedOps journey. Bring your interesting problems that need solving – we are always happy to help out. You can reach out at any time on LinkedIn or through email.