Ask an embedded engineer what they do while a Yocto image builds and you get a version of the same answer. They switch to something else and forget to come back, or they can’t concentrate because the thing might finish any minute. As one told us, the wait is hard to put to any effective use.
On the image we benchmark against, the first build takes an hour and 43 minutes. The ones after it take 8 to 12 minutes of billed compute and cost under 25 cents. Same repo, same recipes, same pipeline.
Waiting on CI costs more than waiting on your own machine. The image you flash comes out of the pipeline, not off your laptop, so until the pipeline finishes you have nothing to put on the board: no bring-up, no hardware test. When a build dies at 80%, the board stays idle for the two hours it takes to start over.
The numbers behind the wait, from teams we’ve talked to this year: two to two and a half hours per CI build, and six to seven for a clean rebuild.
Your CI deletes the cache every job
Pushing sstate into your CI provider’s cache rarely works, because these trees run past 25 GB and keep growing, well past most cache tiers. Your cache has to outlive the runner, and so does your build.
Move the build off the runner
In the workflow the CLI emits, the GitHub-hosted runner checks out the repo, installs a pinned yocto-cloud binary, exchanges its OIDC token for short-lived AWS credentials, and runs one command. The build happens on an EC2 spot instance, streaming into the job log.
The instance talks to your bucket directly. The runner holds credentials and collects the result.
You can run that on a standard GitHub-hosted runner and stop paying for the always-on build host. Every team we asked was running one, and none believed it was cost-optimal. One had grown theirs twice after out-of-memory failures and kept paying for its storage while it sat stopped. Another rents expensive high-RAM compute to keep a 50-minute cold build tolerable.
The repository holds no AWS keys. The runner mints an OIDC token and trades it for a role that a CloudFormation stack the CLI ships created, and that role’s trust policy names one branch. The credentials last eight hours, because Yocto builds routinely run two to four and the IAM default of one hour expires mid-build.
One cache, both paths
Three cache prefixes in one S3 bucket per project: sstate, downloads, and a hash-equivalence database. Every build pulls before bitbake starts and pushes when it’s done, and engineers run the same CLI against the same bucket: a change validated locally warms the cache the pipeline uses, and last night’s pipeline warms what they pull this morning.
We added that third prefix because of a complaint we heard more than once: caching helps until you touch a header included everywhere. Hash equivalence handles that case. Bitbake asks a hashserv daemon whether a changed input produced a different output, and remaps the hash so downstream tasks still hit sstate. Run that daemon on an ephemeral instance and its database dies with it, so every run invents its own remappings, and sstate uploaded by one run is unreachable to the next. We were pushing gigabytes of cache and hitting under half of it.
We now persist that one sqlite file, which fixed the hit rate and created a sharper failure: a torn database in S3 poisons every later build. So that prefix skips mid-build checkpoints and uploads only once bitbake has exited and released the file, behind a PRAGMA integrity_check. A lost run’s equivalence entries regenerate on the next build. A corrupt file would not.
Five measured builds
The image is a scarthgap ARM64 build: 7,874 tasks, 3,832 sstate objects, on a 64-vCPU spot instance. Billed time runs from instance launch to termination, and pulling artifacts down to the runner happens after that. The last column is spot compute alone. The build summary further down adds EBS and artifact egress, so a run total reads higher than the figure here.
| What changed | sstate objects missed | bitbake | billed | spot compute |
| Nothing | 4 of 3,832 | 3m 30s | 8m 36s | $0.16 |
| One leaf app recipe | 11 | 3m 36s | 8m 04s | $0.15 |
| One kernel config option | 68 | ~7m | 10m 28s | $0.19 |
| Upstream layer bump, 29 recipes | 101 | ~8m | 11m 49s | $0.25 |
| Everything, empty bucket | 3,832 | ~1h 42m | 1h 43m 39s | $1.87 |
The warm bar is the same work, minus what the cache already holds.
Change one leaf recipe and you pay six seconds more than changing nothing, because do_rootfs runs either way and dominates both. The narrow, deep kernel change invalidated fewer objects than the wide, shallow layer bump, so blast radius and depth are separate axes.
One thing of note if you benchmark this: bitbake’s sstate match percentage reads 99% for “changed one leaf recipe” and 99% for “changed nothing.” The percentage you’d put on a slide can’t tell those apart, and the missed-objects column is the one that discriminates. We ran two rows twice and landed within 1.4% on bitbake time.
Every row ran with SSTATE_MIRRORS empty. The project’s config had pointed it at a public CDN mirror, and on the cold row that mirror served 1,111 of the 3,832 objects while making the build 15 minutes slower and 35% slower per task, because bitbake keeps re-checking a remote for the objects it won’t find there. Turning it off was worth more than what the mirror served. The tool never sets SSTATE_MIRRORS in either direction, so if your config names a public mirror, that cost stays yours until you remove it.
Thoughtful architecture against upstream mirror downtime
The most useful thing anyone told us was that roughly one build in three or four failed because an upstream mirror was down, and with no state persisted, each cost another one to two hours from zero.
We proved their point by accident. A regional package mirror started returning 503s mid-benchmark and killed four of nine instance launches, about a dollar of compute billed for nothing. The measurements were the easy part of that week.
The fix has to cover the boring failures too. The build script starts two watchers before bitbake and reaps them on exit: one checkpoints sstate and downloads to S3 every five minutes; the other polls the instance metadata endpoint, which returns nothing until AWS schedules a spot interruption and then returns JSON, at which point the watcher forces one last sync and stops.
That makes the build resumable. Whatever kills it, you lose at most five minutes of work, and a re-triggered job picks up warm. The downloads cache covers the same problem from the other end: sources fetched once come from your bucket instead of whichever mirror is down today. Resuming is safe because the store is append-only, and a tripwire in the code enforces that. Even –fresh deletes nothing remotely.
The build runs detached over SSM rather than SSH, so cancelling a job stops the stream and leaves the build running. SSM kept working through the mirror outage too: one command into a stuck instance found the failing mirror, which console output had not.
The logs outlive the instance
The instance is disposable, so anything you need to diagnose a failure leaves it first. Logs and artifacts upload to S3 before shutdown, and the build command brings the logs back to yocto-cloud-out/logs/ when it finishes, whether the build passed or failed. Reattaching after a dropped stream does the same download. So a recipe error waits on your disk instead of being trapped on a machine that no longer exists, and in CI the workflow uploads that same directory as a job artifact on every run.
Regulated systems put the build host in scope
An image is only as trustworthy as the machine that produced it, and the build host is the part of a pipeline least likely to have been reviewed. A long-lived box accumulates packages, state, and credentials nobody can account for a year later.
The design above covers most of that. Each build gets a fresh instance that terminates itself, its credentials expire with the run, and every resource lives in your own account with no static keys and no inbound ports. Two things matter more here than they did earlier. Nothing but your own roles can write to the cache, and the CI session carries the run id, so the AWS calls a build makes trace back to the job that made them.
The CLI holds itself to the same standard. The workflow pins a version and verifies the binary against checksums published from a different project than the one serving it, then refuses to run if they disagree.
None of this makes a Yocto build compliant with anything. Compliance is a property of your system and your processes. What it does is make the build ordinary to explain, which is most of the work when someone asks.
A pipeline that can spend money
CI that launches cloud instances needs guardrails a test suite doesn’t, and most live in the emitted workflow: path filters so a README edit doesn’t start a billable build, a concurrency group so two pushes don’t mean two spot instances, a shutdown timer armed at boot, and a tagged monthly budget. Every run prints a cost estimate. The tail of a cold one:
Wall time: 2h 4m 38s
Artifacts downloaded: 150
Billed: 1h 58m 1s on c9gd.16xlarge (spot $1.25/hr · us-east-1c)
Run cost (this build): ~$3.01
compute (spot) $2.46
EBS (gp3 root) $0.01
egress (5.9 GB) $0.53 (max; $0 if puller is in us-east-1)
Persistent storage (monthly): ~$0.60
sstate 8.4 GB (11596 objects)
downloads 17.5 GB (833 objects)
persistent 3.6 MB (3 objects)
total 26.0 GB
Two knobs trim the rest:
–local-globs, or [yocto].local_artifact_globs, whitelists which images land on the runner. It filters the download only, so S3 keeps the full archive. On the benchmark runs that pull ran from 5% to 62% of wall time.
A cold instance type sizes the machine to the job. The CLI probes the sstate prefix before launch and takes the big box only when the cache is empty, because a cold build uses the cores and a warm one waits on S3.
One command, one approval
Caching stays inconsistent across an org because setting it up is a side project: Terraform, an AMI, a cache layout, spot-interruption handling. yocto-cloud bootstrap does it once. It fills in what it can find for itself, the project name and repo and branch from your directory and origin remote, and the region from your AWS profile. Then it shows what it will create in your account and waits for a typed yes.
Each stage checks the world rather than a saved checkpoint, so a re-run picks up at the first incomplete one. One stage needs rights a project profile lacks, so it prints the command an admin runs and stops there, and that stop is a resume point. It also reads your spot vCPU allowance before creating anything: if one build of your instance type won’t fit, you get the exact increase request instead of a MaxSpotInstanceCountExceeded at launch.
The next change
Warm builds still sync the whole cache before starting. Of eight minutes billed, bitbake takes three and a half minutes and the cache sync two and a half, so boot and sync now cost more than the build does. The sync re-lists and compares every object at both ends of every run, so its cost tracks the size of the cache rather than the size of your change.
The fix is to stop syncing and start fetching: point SSTATE_MIRRORS and PREMIRRORS at your own bucket over the in-region VPC endpoint, so a build pulls the objects it wants and ignores the rest. That is a different proposition from a public mirror on the far side of the internet, where every miss buys a round trip. In-region, a miss costs almost nothing.
GitHub Actions and AWS are what ship today, and build hosts are ARM64 Graviton. None of those three is fixed. Adding a CI system, a cloud, or x86-64 hosts does not mean rebuilding the tool, and we take them on in the order teams ask. Tell us which one you need.
The most common objection we hear is fair. We already have a build host that works. What does it charge you per build, counting the box that idles between runs and the work it rebuilds every time?
If you want that number for your own image, the CLI is a single binary: request documentation and install instructions and point it at a kas project. yocto-cloud bootstrap provisions the AWS side and writes the workflow.
yocto-cloud is the Yocto corner of EmbedOps, our suite for embedded teams [sign up for the free product right here]: templates for build environments that run the same on a laptop and in CI, hardware-in-the-loop testing, memory and test analytics.
