One of the most frustrating things during any type of development (be that firmware or software), is to grab a project’s source code and have it not build. No doubt everyone who has done some form of programming has encountered this at least once in their life.
Maybe you’ve pulled a repository from Github, built it according to the readme file, and it DOESN’T BUILD. Surely you’ve missed something….right? You look at the bug reports for the repository and see that no one else had issues. Have you discovered a new bug?
In another scenario, let’s say you’ve just made a change to some code at your job. On your local machine you’ve built it successfully and all the pre-checks passed. You put out a pull request/review and go about your day. Later you look back and see uh oh….it’s failed to build on your company’s pipeline. But….it built and ran fine on your PC, what happened?
The answers to the above scenarios is what we’ll dive into here. In reality there are probably 100 different reasons it works on one machine and not yours, but we’ll try to generalize the five most common ones here:
- You are not building your source code with the same compiler version. There are a few ways this can cause issues:
- You are using an older compiler version than the other machine and a bug in this compiler version was fixed by the newer one.
- You are using a newer compiler version than the other machine and your newer compiler has introduced a bug.
- Both a) and b) above can also cause issues when the different versions of the compiler modify how they handle warnings. This can lead to one compiler spitting out more warnings than the other.
- The items in a), b), and c) would all be considered “loud” failures. They are obvious failures where the build either flat out failed or had new warnings that weren’t present on the other machine.
A compiler mismatch can cause more serious “quiet” failures which are much harder to diagnose. These “quiet” failures occur when the build looks exactly the same on both machines, but the compiler differences on one machine cause erroneous assembly instructions to be placed in the final build output. This failure is only caught during run operation, in test, or worst case when the product has already been released. This is usually one of the main arguments in favor of fixing the “It builds on my machine but not yours!” problem.
- The program was compiled for x86 and you’re on ARM or ARM64. Flat out mismatches in target architecture will cause a program not to run.
- Using a different version of libraries (for example you’re using an embedded filesystem and you don’t have the correct version on your local machine to compile against).
- Some undocumented library or build tool is missing on your computer. A library required to build the code is not contained in the repository and not documented, but expected to be on your machine to build properly.
- Operating system versions are different. (i.e. Ubuntu 23 vs. 24 or Windows 10 vs 11)
How do we Correct the Problem?
Fortunately, the issue of mis-matched builds has been solved already. The best solution for this is to use some sort of Dev Container and then ensure that the same Container is also used for your dedicated build machine or automated cloud build. This type of setup will make sure all developers are using the same build tools and that the build on local machines matches the one on the dedicated build machine/cloud build.
About Dev Containers
Dev Containers (or just Containers) allow ONE unified build system that all developers use. One example of a Containerized system is Docker. The idea behind this is that a Container encompasses specific versions of all tools, compilers, and programs used to build and/or run your code. Containers are preferable to Virtual Machines because they are small, fast, and portable. A Container can run on Windows, Mac, and Linux and can be scripted to run automatically on build servers.
A Container allows your source code to live locally on your PC. When the Container runs it will treat your source code directory as a shared folder and be able to build it.
This means you can develop locally and use the Container to build -or- you can use the Container to both develop and build. (VSCode is great for facilitating this) The simplest option will be doing development and builds inside the container, but there may be use cases where it makes sense to develop outside the container and then use the container separately just to build.
To read more about Dev Containers and how Visual Studio Code can be used with them: https://code.visualstudio.com/docs/devcontainers/containers
