The Case for EmbedOps (Embedded DevOps)

From even before the first computer was working, the technology industry has been looking for the best way to write code. Today, as code “eats the world”, the types of things developers need to do is not limited to coding. There are managing releases and versions, managing development environments, testing, integrating, and so on. In the web, desktop, and mobile industries, tools have been created to automate and streamline the additional work as well as make good code easier to write. But in embedded systems, where the languages are long-lived, the early standards still stand, and the history runs deep, we have been left out of many of those tool advances. At Dojo Five, we are designing a modern embedded-first suite of tools and practices modeled after the best of the advancements seen in web and mobile. We call it EmbedOps

What is EmbedOps?

An embedded-first implementation of DevOps. We take the best practices from web and mobile and apply them to embedded development. We also share everything we’ve learned about the great tools that already exist in the embedded world and make them easy to find, use, and benefit from.

Back up, what is DevOps?

It is a method of organizing a company’s development and IT operations functions into a single unit so that the entire development cycle is visible to everyone. It aims to shorten the feedback loop from how the code you write performs in the real world. The goal of DevOps is to improve code quality, shorten the time it takes to fix bugs and outages, and reduce the amount of unproductive work for the dev team.

Why is Dojo Five working on EmbedOps?

We’ve talked before about how embedded development seems stuck doing things the way they’ve always been done. We have been improving our own processes and tools while working on our clients’ products. We’ve done our best to ease our clients’ pain points and leave them not only with good firmware but also tools and resources gleaned from state of the art DevOps practices to make their own work better and easier. After implementing the same changes for many clients, we realized that the need for DevOps designed to work for embedded systems was something many more companies could use. EmbedOps was the result.

Even if your team has already started your DevOps journey, EmbedOps is still for you. Maintaining CI/CD systems, monitoring them for security, and adding new automated tools is done by dedicated teams at large companies for a reason. It’s a skill set your developers may not have experience with, and takes a lot of time they could be using to work on your product. So how do you convince your team, your boss, or your company to make the switch? Let’s chat about the benefits that may just sway them.

Why your team needs EmbedOps

The Internet is coming for your Things

Recent studies have shown that DevOps will be a $15B market by 2023. But we don’t make websites, you might say! We can’t push new changes every three minutes, you might argue! That’s true, but you’ve heard that the world is more connected than ever, and it’s just as true for embedded devices. Many devices are connected to the internet or at least their owners’ computers, and can be updated after they’re deployed, sometimes with no intervention from the end-user. These “Over the Air” updates are becoming more critical every year as more devices are connected, and therefore vulnerable to the same kinds of security and privacy risks traditionally seen in the web and mobile world. Since you can now send updates to devices without all of the gates you’re used to seeing before manufacturing, knowing that your code has been thoroughly tested and checked for security and safety before you send it is critical. Continuous Integration and Continuous Delivery are now standard practice for the web world for this very reason.

Bugs: The faster they’re found, the faster they’re fixed

James Grenning has been touting Test Driven Development for embedded devices for years. As he says in his book, “In principle, there is no difference between a dependency on a hardware device and one on a database.” And yet, they’re seen as more difficult on an embedded system. A CI system can help testing be a value rather than a painful part of your everyday coding process. Finding a bug seconds after you write it is easier to fix than one you find weeks after it’s written, and far less stressful than finding one after your code is deployed. EmbedOps helps reduce the amount of effort it takes to set up unit tests. Our container registry is uniquely designed for cross compilation and embedded system development at every stage. In addition, there are multiple templates for common chips and frameworks that have unit tests and the build system ready to run them pre-configured.

Show your work

If you’re creating medical devices, writing firmware for planes or cars, or working in other safety-critical industries, you’ll have to document not only your code but your development processes thoroughly before you’re allowed to sell your product. Think ISO standards like 26262, 62304, 13485, and others. With EmbedOps you can track your peer reviews, connect issues or merge requests with requirements, enforce review gateways, and interact with your quality management system without the developer overhead.

Modern tools make familiar ones more powerful

There are a plethora of tools out there for embedded development. But until EmbedOps, it was hard to combine them into a process to make them useful enough to use daily. With flash and RAM metrics you can see the code size not only on your own work but on every branch. You won’t have any surprises when merging and can take action before you have to scramble to fit everyone’s code in the device. And with EmbedOps’ Docker registry, you’ll know your code is building in a standard and repeatable environment. Each image is designed with embedded development at the forefront, with cross-compilation and board support packages baked in. You can also see the routine security scans for the registry and be assured that your code is building in a well maintained secure environment.

Build vs Buy

If you’re already convinced of the benefits of CI or are already working on modernizing your development processes, you may be wondering if you should just build out CI yourself. We’ve found it takes between 2 and 4 weeks for an FTE employee to set up a CI system if they have no other responsibilities, and about 15% of an engineer’s time to maintain it for each project once it’s deployed.

If you have enough resources to dedicate an engineer, and possibly a team full time to maintaining CI, building in-house may be the right choice for your company. On the other end of the spectrum, if your company is running a lean engineering department or your developers are already feeling overwhelmed, there are plenty of companies out there creating simplified CI interfaces at bargain-basement monthly subscription prices. But in exchange, you and your team still have to do quite a bit of setup internally to configure these web-focused tools for your very non-web projects. In addition, you’ll share their support team, which likely does not understand embedded development, with every other customer they have. But if you’re just trying to make initial steps into modern dev practices or developing a guerrilla CI system under management’s radar to improve your team’s life, these options are attractive.

But if you have a small team and need to have rock-solid processes in place yesterday, EmbedOps engineers are here to support you with dedicated embedded-first services, training, and maintenance. We are especially excited about helping teams building medical devices and safety-critical firmware!

As of our last count, Wikipedia lists 58 frameworks for unit testing on C and 81 for C++, but according to the most recent survey by the Barr Group, only 28% of teams use Test-Driven Development. This tells a story that there are plenty of resources available, too many maybe. And the same story is repeated at every step of the development process. At EmbedOps, we’ll save you from the paradox of choice and show you which services are best for you and your team, configure them for your unique needs, and keep them up and running so your team can focus on developing great firmware.

Trust your process, trust your firmware

DojoFive brings modern tools, techniques, and best-practices from the web and mobile development environments, paired with leading-edge innovations in firmware to our customers to help them build successful products and successful clients. We have talented engineers on hand ready to help you with all aspects of your EmbedOps journey. Bring your interesting problems that need solving – we are always happy to help out. You can reach out at any time on LinkedIn or through email.