I've been using Nix for "embedded" systems for few years now and it works fine. I don't quite understand why "embedded" has historically meant learning a completely separate tool. I've been building for x86 and ARM servers for 5+ years now, so why should targeting an ARM board be any different?
Like the article points out, the software stacks people use for embedded devices are the same as people use non-embedded use; Rust, Go, NodeJS, and sure still C++. The only real difference with embedded devices is non-OS components like the bootloader (u-boot, EDK2) and customizing the device tree. (And of course firmware flashing). Fundamentally those are all just packages that I can describe in Nix. I don't need a separate tool just because the board is small.
IMO the embedded space, especially in the US, is already pretty Niche. Most companies either just ship the vendors BSP example (Ubuntu/Debian/Yocto) and pay very little attention to the detail or re-useablity. Once you vendor declares the BSP EOL you are stuck unless you re-implement it yourself.
Using Nix (or Guix) has the massive advantage of a large and active community that isn't fractured like the Yoctoo/Buildroot community. (By fractured I mean there may by many, many people using those tools at $DAYJOB but due to vendor BSP customization they likely share much less with the upstream community maintained sources).
> I don't quite understand why "embedded" has historically meant learning a completely separate tool. I've been building for x86 and ARM servers for 5+ years now, so why should targeting an ARM board be any different?
Because embedded usually means dealing with complete functionality provided by the soc, bare minimum for the init system to get quick boot times.
If you just want to boot the CPU and run some IoT app, device tree + kernel + uboot should be enough.
But for anything with a GUI, You'd probably need the vendor provided binary blobs for GPU accelerated UI, proper gstreamer packages with all the patches to make it work with the binary blobs... properly packaged Qt etc ... Not to mention read only rootfs and A/B partitions for upgrades to deal with power loss headaches.
I haven't used Nix for any of these but most of these things are available ready to use with the vendor provided buildroot or yocto setups. And it would've been just more work to get this all working with yet another build system/package manager
However, reading through the docs, the author clearly knows about Nix/Nixpks and has factored some of its properties into the design: https://docs.yoebuild.org/nix.html
Actually, quite an honest and in-depth review with many genuine good points about the state of Nix. If you genuinely need sub 1GB image sizes, you would have to reimplement many of the Nixpkgs conventions like glibc/systemd.
I’d still probably just implement this in Nix/Nixpkgs so I could leverage the rest of the ecosystem but that’s not trivial at all.
I wouldn’t even say Nix as a language is better than Starlark. The author is certainly well informed in the choices made.
Nitpick: binfmt.emulatedSystems is not true cross compilation. It sets up a nixos to enable emulated native compilation. After painful tinkering, i prefer this way too. When i recall correctly, for cross compilation you'd have to use nixpkgs flags.
At some point it's a matter of perspective. If the emulated system is fast, it feels the same as "true" cross compilation. In the end it's a bunch of bits and bytes which produces some other bits and bytes when we poke it. If it goes fast, it's good. If it goes slow, it's painful.
I share some of the same observations that seem to have motivated this project, but my solution was to just use Alpine on the same architecture as the target (possibly via qemu) and then export a subset of it to a filesystem image. I implemented it as a perl module collection with no dependencies other than core perl modules, and then run the export from within the image itself. Among other things, it lets me use strace to find the runtime library dependencies of the things that need to be in the image.
Modern SBCs are just normal computers and not "embedded" in the traditional sense. You can generally just use Debian, and spend time on the actual project, instead of wrestling with the system
They aren't normal computers in the same way x86 is. They usually need proprietary drivers in custom boot processes, forked kernels with patches, etc. I'm yet to see a single SBC I can just download the iso from the debian website and get going. You always need some custom build.
ARM boards need a custom kernel and bootloader. These aren't things managed by the distro. The userspace IS managed by the distro and IS standard. Using yocto to manage userspace maybe made sense 15 years ago, but it has long since become far more trouble than its worth. Debian supports most architectures out of the box, has good cross-building infrastructure, many thousands of ready-to-use packages, and is non-weird.
Debian handles userspace fine but there is a lot of image customization needed a lot of the time. I have experience with developing images for a beaglebone black, where I forked the bespoke image builder (to create user accounts, mount points, set up udev rules, device tree overlays, ssh keys, etc) and that was a pain to maintain. I suppose I could have create debian packages for some of those things, but I would still have needed to set up repositories. Now I'm using an SoM with Yocto and while the learning curve is quite a bit steeper, it is much easier to customize for our experiment's needs. Also, AI assistants are really good at spitting out yocto recipes (that usually don't work immediately, but gets you 90% of the way there).
You're missing the point that when you're building the product, and not using SBC just for fun, the image you are building, and yes even the userspace, is highly customized. Running a full-blown Linux distro is a non-requirement, and often incorrect.
For more mainstream SBCs, totally agree. However a lot of the niche industry boards (especially new ones) only support Yocto out of the gate. Debian might be "supported" by the SoM/CPU vendor but manually reapplying kernel patches and praying it boots is a PITA.
I agree. In the past, I have successfully used Debian and Alpine for embedded. Never needed to compile OS kernels or standard DLLs, other people already did and published in these package repositories.
I'm surprised to see no mention of Guix. It solves all of these issues, and already has a good story both for cross and native builds on a variety of architectures.
Adding new targets is deceptively easy, just copy an existing template and substitute your values.
What do you mean by image? In Nix you instantiate a NixOS system and specify the architecture. There are a number of tools to build a disk image from a NixOS system.
IMO the nixos-rebuild scripts are unnecessary and just obscures what’s being built.
build-vm is the same as building the configs “config.system.build.vm” attribute.
nixos-rebuild is fine for getting started but you end up needing to use the attribute path to do anything more complicated like CI caching or inspecting the drvPath.
I haven't used it, but there's also Avocado OS [1]. Myself, I use Yocto for our embedded build. It's not perfect but for our application, which involves building a bunch of software that uses both Swift and Flutter for an embedded audio product running on a RPi CM4, it works just fine. Shout out to Mender for A/B updates.
Bitbake (Yocto's build system) started originally as a fork of portage (gentoo package manager), but they have diverged a ton. Very similar concepts though.
Architecture emulation with qemu-userspace is 5-10x slower than running a native build.
The only 'proper' no-friction cross-compilation system I know is Debian/Ubuntu, because it allows to install foreign architecture libraries into your native system, and has all the wrappers for cross-compilation in dpkg-buildpackage.
This way you don't have to maintain cross-compiler or toolchain/sysroot, you just install whatever dependencies you need with the regular `apt install` of another architecture.
I'd like to have more friendly tooling around that instead of a slow compilation of architecture emulation.
Joke: The Linux community will continue to overlap and reinvent parts of itself until there is a universal Linux kernel that can be fully adapted and customized to fit onto almost any hardware..
This sounds like fun, but I'm not sure how changing build systems fixes the vendor problem. They will just ship their proprietary blobs built on top of this new build system instead of the old one, and will not change other behavior. Buying from vendors who upstream or minimally open source their work might help.
I'm a little doubtful on the post's assertion that yocto is only needed for "deeply embedded regulated products." For products that have to follow the https://en.wikipedia.org/wiki/Cyber_Resilience_Act coming up, they will almost certainly need:
And I'll echo what some other folks have said here: if you don't need features like that, bundle your custom bootloaders and kernel with a binary distribution like Debian and call it a day.
The problems building with Linux and GNU environments exist because they were terribly designed with assumptions like
- You're building on the same native system as GNU and Linux packages, you install them globally in the same places that servers and desktops use
- Your C, C++ compiler and entire toolchain and other binary utilities with the kernel is a one single unit that you can only change one part at a time
- You use the same up to date headers with glibc, gcc and Linux kernel
- You're building software in the same universe of all the other packages, especially gcc libraries (libgcc_s, libstdc++), glibc (especially bad since ld-linux.so is part of it)
- The build system only uses standard paths
The reason Yocto is so complicated is that developing in a Linux environment actually sucks when you're not writing web-oriented or server / VM software. Yocto fixes it. It introduces a good set of abstractions that work around terrible design decisions that were made in overall Linux ecosystem. There are a lot because the OS design is fundamentally broken, especially with C-based toolchains which is 99.999% of the ecosystem. Current C toolchains including MSVC strongly ties OS with the C's internal types and bad decisions of 70s.
As always all articles whose title asks a question are answered with NO, 99% time. By taking away the cross-compiling abilities and the workarounds doesn't fix the brokenness of Linux and overall FOSS ecosystem.
If you're looking for how a better embedded environment looks like, look at Rust toolchains. For Linux take a look at musl-libc based ones (you 100% need a systemd distro to get away from nss complexities that musl introduces). Or even better take a look at relibc. There are barely any assumptions about the target filesystem and tooling in Rust toolchains, unlike C/C++/Make toolchains. There is redox OS but it is still in slow development and they stuck with Make, which I think was a bad decision. Android uses its own build tooling but cannot run away from C/C++ tooling unless Google revives Fuschia.
My experience is that Yocto is a bit like CMake: people hate the mess they make with it and blame the system, even though they never bothered learning to use it.
When someone starts with "why cross-compiling the system if it's bearable to compile it directly on the device?", it really sounds like they are just not comfortable with cross-compilation. To me it would be similar to say "why would you need a compiled language, with this weird compilation step that nobody understands, when you can just write in an interpreted language?". Except that developers wouldn't generally proudly admit that they don't know how to use a compiler.
A valid reason I see to require a new embedded linux build system would be if the existing ones (mostly buildroot and Yocto) were unable to build a modern OS (e.g. by not being able to integrate some security features). But I don't think that's the case. This article sounds more like very verbose marketing for a system claiming that it can do similarly without having to learn as much (which I honestly doubt).
To me it's often the same pattern:
* Meson is nice, but it doesn't remove the need to use CMake, and it's not fundamentally simpler or more powerful (it's more comfortable at most).
* Jujutsu is nice, but it doesn't remove the need to learn git, and it's not fundamentally simpler or more powerful.
* Yoebuild sounds like even if it got traction like Meson and Jujutsu, it wouldn't remove the need to interact with Yocto for embedded linux engineers.
As an embedded linux engineer, if anyway I have to interact with Yocto and therefore learn Yocto, why would I learn another one that isn't fundamentally better? Usually people who go for those "simpler" (at least on the surface) tools do it because they don't want to learn Yocto. And for an embedded linux engineer, isn't it a red flag to not want to learn the most famous embedded linux build system?
It reminds me of Balena/Resin, which seems like it's targetting people who are not embedded linux engineers, don't want to become embedded linux engineers, but still want to distribute an embedded linux. To me that's generally a red flag.
> My experience is that Yocto is a bit like CMake: people hate the mess they make with it and blame the system, even though they never bothered learning to use it.
I agree this is true a lot of the time, speaking as someone who has made many Yocto messes and later regretted. But you also can't discount all of the messes made by silicon OEMs that you are forced to deal with to use their SoCs.
E.g. I recently did a project with Yocto on a Zynq US+. Look at the meta-xilinx layerset (or, God forbid, petalinux) and tell me they haven't made a complete mess of it.
The complexity and lack of upstreaming in kernel and userspace for so many of these SoCs, coupled with non standard boot process and devicetrees means you will be forced to use these terrible OEM layers. I don't think a new build system can fix this at all, it's a cultural/commitment that needs to change from the OEMs.
At the end of the day, of you are doing any complex embedded Linux work as a small team, you must use Yocto because it's what the OEM supports. I don't have time or resource to debug DFX not working because some userspace tool was built with slightly different flags than expected by the flaky ass xilinx upstream.
> But you also can't discount all of the messes made by silicon OEMs that you are forced to deal with to use their SoCs.
Yes, totally! It's a bit sad because we don't usually have the luxury to select the hardware based on the quality of the BSP. First because they all seem to be terrible, and second because it is somehow "hidden" cost. One selects the hardware based on its price and capabilities, and the software part is left as an exercise.
But as you say it's not a problem of Yocto, it's a problem of the OEMs being in a position to make a mess because it doesn't have an impact on them. A bit like Apple is terrible for developers (in my experience) because anyway developers don't have a choice.
CMake at least used to be like this: a black plastic bag of LEGOs, you empty it on the floor, there are some instructions there, but some of the images don't look like any of the pieces you see on the floor. Also there are some random rusty bolts, lint, and pieces from an Erector Set. There's an empty tape roll, someone wrote "buy more, ask forum" with a sharpie on it.
You can make great things with that, but it was easy to be "holding it wrong".
I still recommend CMake at work, it's the least offensive solution in most situations.
I get the feeling, it is a... "different" language. But I never understood the criticism about the documentation. At least in the last decade, I have always found the CMake documentation pretty good. Everything is documented.
Probably CMake is not a good fit for very, very big projects. But the vast majority of projects are not that big.
Usually what I see with CMake is self-inflicted pain, often where the devs start adding custom functions and mix it with python and stuff. If you keep it simple, it's very clear and does the job with surprisingly few keywords, I find. And I have setup tens of projects with CMake.
Whilst I find yocto perfectly fine. I also see that it has its limitations. I think there is a need to make yocto more nimble rather than designing a whole new build system.
yoebuild does seem interesting at least as a proof-of-concept.
I've used buildroot extensively, and an occasional yockto.
My impression in recent years is that these image cross build environments are just not as frequently needed as they were back in the day of their invention.
My most recent embedded linux environments were just embedded archlinux.
No need to cross build an image, just install and run the minimized linux environment right on the target.
Of course, a big part of the need for these cross-tools is that it seems most modern embedded linux developers are running windoze on their development workstations 8-/
Are you proposing compiling on the target? For a vast number of embedded systems that is not only impossible (insufficient disk and memory) but also incredibly slow.
At the end of the day you need some cross compilation just for board bring up.
If you're playing with some platform for which this has already been done, then sure, but that's not really the "normal" way of doing embedded.
Embedded just means ARM 99% of the time and it's cheaper and easier to use native ARM servers (AWS has them cheap) than to make 100% of software cross compile. Some parts of the firmware might need to be cross compiled but those projects are designed to cross compile.
Your target build environment often differs significantly from your server environment, so you end up needing a different toolchain and all of the problems that come with cross-compilation anyway. The toolchain and ABI settings that produce small, battery or instruction cache efficient are usually not you want on servers.
My toolchain for both is managed by the Nix derivation wrappers. It’s trivial to set custom C/C++/Rust/Go/NodeJS flags. As long as I have one flexible build tool, I can customize it for both server and embedded.
I'm not familiar with the problems that would cause "target build environment often differs significantly from your server environment". If you have an ARM server, then you should be able to download and run Alpine docker images, and then fetch most system utilities from the Alpine package repo and then compile a kernel and a few other binaries on top of that, then export an image.
The main reason to not run Debian is that Debian usually makes lots of compile-time decisions for you, and chooses to maximize the enabled options of the software. Alpine makes the opposite choices for you, creating very minimal feature sets.
It's been working well for me, but my use cases is a lot more like a custom in-house DDWRT firmware kind of a thing. If you've experienced additional constraints that make this pattern unworkable, I'd be interested to hear about them.
Run on server, chroot into target environment, configure compiler flags for eventual target, done. It's really just an afternoon of setup and then the problem is solved.
I’ve been experimenting with what a next-generation embedded Linux build system might look like: native builds on the target architecture, modern language package managers as first-class citizens, and AI as a primary interface to the system.
Instead of cross-compiling with a large meta-layer stack, the tool builds kernel, rootfs, and applications together using one engine, with a CLI, TUI, and AI assistant talking to the same core. All you need is the tool, Docker, and Git — no global SDKs or hidden state.
It’s pre-1.0 and rough around the edges; I’m sharing it early to get feedback from people who live in Yocto/OpenEmbedded, Buildroot, Nix, etc. I’d love to hear where this breaks on your boards, what workflows feel wrong, and whether the “native builds + AI-aware build graph” direction seems promising.
It was true in 2005, but still? As I described in another post here, a modern strategy is to get a beefy server than can run the same ABI, then start a docker container and assemble a system from Alpine's package repo, then compile a kernel and a few in-house things, then extract a subset of that into an image. No cross-compile, and most of the useful software is in pre-built binary packages with the compile-time options you would have selected anyway.
Even if you don't have a beefy server of the same architecture, you can probably run it in qemu instead of docker to the same effect. And even if qemu is slow, you can run a build of the kernel and your in-house stuff in parallel on 64 cores and not really be affected by the qemu slowdown.
What if this hegemony in ISAs is partially because of the high costs of porting software and toolchains?
Maybe the next generation of embedded devices will have ISAs that are, say, extremely optimized for real time industrial controllers, but terrible at compiling C++.
It seems a little silly to give up on cross-compilation just because _today_ everything runs on ARM.
Especially because cross-compilation is not a particularly hard problem, the problem is with decades of tooling assuming the build and target environments are the same.
Well, I'd say cross-compiling is at least a little problematic... if nothing else, it is a hurdle to running automated tests. And cross-compiling still needs to exist for the people building the package repositories. I'm just sort of advocating that if you have limited amounts of time but want to put together a system image to run in an embedded or embedded-adjacent manner, the least pain / most agile solution is by building on top of a popular binary distro and then using native compilation on appropriate hardware inside a chroot or container that resembles the target.
"Ampere processors natively support both 32 & 64 bit Android applications and require no binary translation for maximum instance density"
...
"Run up to 120+ 3D cloud game instances per socket"
...
"DRAM: 384GB - 512GB"
I've not personally tested this, but it very much appears that just like x86, you can use a 64-bit ARM kernel that has 32-bit support enabled to run a 32-bit userspace, likely compatible with docker, or even just in a chroot. Then with 120 cores, "make -j 120"
Yocto and Buildroot will compile, from scratch, an entire gcc crosstool chain with standard library suite and headers to build fast and then deploy to your target. This exists.
Armbian and Qemu worked well for me when I needed to compile packages for an orange pi without enough RAM to actually run cargo build. Built the image on an emulator with more RAM then the target system, dropped the customized image on the SD card and booted right into the entry script.
(For the unaware, OP above is the article author.)
As a long time user of Bitbake and Yocto, and watcher of Yoe Linux... I kind of have to respectfully disagree with the approach.
One look at the Python or Node packaging ecosystem, and you can see how difficult trying to integrate those in a 'sane' way in embedded, with repeatability and security in mind, nevertheless wrangling something like C/C++ dependencies and native packages in a qemu-cross (or binfmt emulation) environment.
I feel like Bitbake and the wider Yocto ecosystem has essentially 'solved' cross compiling. Sure there are the incredibly complex codebases like Chromium that require lots and lots of study and patience, or esoteric compilers, etc. but for most applications I feel that especially AI tooling can write up a good basic recipe for integration with Yocto.
The unfortunate thing about AI tooling is Kernighan's law (where debugging is twice as hard as writing it the first time.) Especially in Embedded Linux, where 99.99% of the product code is written by others, trying to figure out where the AI didn't quite get it right or missed something can break things unexpectedly and impossibly for the unaware developer to fix, even breaking at runtime. So the build environment has to be simple and predictable. I think Yocto/Bitbake already strikes a good balance here, with other nice things like offline builds, repeatable builds, sstate caching across machines, PRbuild servers... things useful for big teams, so consider those as well.
Although, if someone does a 'uv'-like rewrite of Bitbake into Rust with the same feature set but faster, I am all for it...
Maybe, having Yoe or some other centralized distro make specific opinionated choices and then provide an open sstate feed, working inside Bitbake, could get a lot of the speed and customizability gains back without a big rewrite effort.
I wish you luck! The pain points you identified are definitely real and solving them would be valuable.
The workflow for user space can definitely improve some of this pain but I feel like a large portion of any embedded Linux development effort still ends up in the weeds for boot related items (secure boot, proper updates, nuanced kernel patches, bootloaders, device trees, and supporting machine variants, etc). Solving those to make them easy is a hard problem for sure.
RAM is expensive now. Reasonably priced seems to be counter to that fact.
The whole article feels like a promo-tour.
> The catch is that nobody handed us the tools to keep the place running.
Software such as homebrew exists. Why is the article ignoring this? Not only does it feel like promo, it now feels like propaganda. Homebrew is also just one example; LFS/BLFS also exists for embedded linux. Perhaps the quality is less than for desktop computer systems but it exists.
> Edge devices have started behaving like cloud systems. They run increasingly complex stacks, are frequently updated, and are managed remotely over their entire life.
Companies want to become dependent on others? Because that is what cloud is all about. Top-down control. A mafia for later blackmail rack-up-the-price strategies.
> That puts the cross-compile burden squarely on embedded developers, and maintaining recipes for thousands of packages has been a steady drain on the Yocto community
Seems untrue. If you have a compiler that works, you can compile. Thousands of packages? I am tracking 3888 right now. That covers really most software by far, including python pypi addons. If a single person can do that, a team has an even easier time. The whole article seems to narrate a story that is just made up.
> Building everything from source in Yocto means long builds, heavy memory use, and powerful workstations.
No it does not. I do so every day. You need a reasonably fast computer and RAM, which is now driven up by the existing hardware mafia that must be put in jail, but it is easily possible for solo persons too. Now imagine a team.
The whole article is really really crap. Absolute garbage. The claims it makes are for the most part simply not correct. It narrates a story as if all linux developers are incompetent. I highly doubt that. It also never mentions any existing software here that would change the rhetoric such as homebrew, LFS/BLFS and so forth. It feels as if that article is just engineered to tell a "we thus need xyz". That's not good writing. I mean just look at this:
"We can leverage these change agents and rethink the world of connected products."
What does that even mean? Did AI write this? Because humans are usually smarter when it comes to opinion pieces. "connected products" - everything is now connected?
Oi, more AI slop. After dealing with things that have external package management and yocto, I understand the pain. But integrating that into yocto is very doable and is a far better path forward.
And "I've been experiment with" seems to just mean "I've been vibe coding"
I've been using Nix for "embedded" systems for few years now and it works fine. I don't quite understand why "embedded" has historically meant learning a completely separate tool. I've been building for x86 and ARM servers for 5+ years now, so why should targeting an ARM board be any different?
Like the article points out, the software stacks people use for embedded devices are the same as people use non-embedded use; Rust, Go, NodeJS, and sure still C++. The only real difference with embedded devices is non-OS components like the bootloader (u-boot, EDK2) and customizing the device tree. (And of course firmware flashing). Fundamentally those are all just packages that I can describe in Nix. I don't need a separate tool just because the board is small.
IMO the embedded space, especially in the US, is already pretty Niche. Most companies either just ship the vendors BSP example (Ubuntu/Debian/Yocto) and pay very little attention to the detail or re-useablity. Once you vendor declares the BSP EOL you are stuck unless you re-implement it yourself.
Using Nix (or Guix) has the massive advantage of a large and active community that isn't fractured like the Yoctoo/Buildroot community. (By fractured I mean there may by many, many people using those tools at $DAYJOB but due to vendor BSP customization they likely share much less with the upstream community maintained sources).
> I don't quite understand why "embedded" has historically meant learning a completely separate tool. I've been building for x86 and ARM servers for 5+ years now, so why should targeting an ARM board be any different?
Because embedded usually means dealing with complete functionality provided by the soc, bare minimum for the init system to get quick boot times.
If you just want to boot the CPU and run some IoT app, device tree + kernel + uboot should be enough.
But for anything with a GUI, You'd probably need the vendor provided binary blobs for GPU accelerated UI, proper gstreamer packages with all the patches to make it work with the binary blobs... properly packaged Qt etc ... Not to mention read only rootfs and A/B partitions for upgrades to deal with power loss headaches.
I haven't used Nix for any of these but most of these things are available ready to use with the vendor provided buildroot or yocto setups. And it would've been just more work to get this all working with yet another build system/package manager
I was thinking the same.
However, reading through the docs, the author clearly knows about Nix/Nixpks and has factored some of its properties into the design: https://docs.yoebuild.org/nix.html
Actually, quite an honest and in-depth review with many genuine good points about the state of Nix. If you genuinely need sub 1GB image sizes, you would have to reimplement many of the Nixpkgs conventions like glibc/systemd.
I’d still probably just implement this in Nix/Nixpkgs so I could leverage the rest of the ecosystem but that’s not trivial at all.
I wouldn’t even say Nix as a language is better than Starlark. The author is certainly well informed in the choices made.
I'm not very familiar with nix, how well does it do with cross compiling? is anyone actually using it for yocto sorts of domains?
You can generally remotely build for any supported architecture[1]. The build process will be painfully slow though if you have cache misses.
[1] https://search.nixos.org/options?channel=26.05&query=boot.bi...
Nitpick: binfmt.emulatedSystems is not true cross compilation. It sets up a nixos to enable emulated native compilation. After painful tinkering, i prefer this way too. When i recall correctly, for cross compilation you'd have to use nixpkgs flags.
At some point it's a matter of perspective. If the emulated system is fast, it feels the same as "true" cross compilation. In the end it's a bunch of bits and bytes which produces some other bits and bytes when we poke it. If it goes fast, it's good. If it goes slow, it's painful.
Nixos can cross build os images for Le Potato with device tree stuff: https://hub.libre.computer/t/nixos-for-aml-s905x-cc-le-potat...
We do Nix-based cross-compiling for non-embedded software (ARM vs x64). I'd say it works mostly transparently.
[dead]
Hey, you completely forgot about embedded devices that dont run linux. It's not all linux ya know
> I don't quite understand why "embedded" has historically meant learning a completely separate tool.
Because historically, and I would argue should still, refer to very limited hardware, in terms of memory, processing power, and energy.
If you're sticking the Linux kernel into something you've already past the scale this should require a separate build system to accomplish.
I guess this was just so true everyone found it uninteresting like saying water is wet eh?
I share some of the same observations that seem to have motivated this project, but my solution was to just use Alpine on the same architecture as the target (possibly via qemu) and then export a subset of it to a filesystem image. I implemented it as a perl module collection with no dependencies other than core perl modules, and then run the export from within the image itself. Among other things, it lets me use strace to find the runtime library dependencies of the things that need to be in the image.
https://metacpan.org/pod/Sys::Export
Modern SBCs are just normal computers and not "embedded" in the traditional sense. You can generally just use Debian, and spend time on the actual project, instead of wrestling with the system
They aren't normal computers in the same way x86 is. They usually need proprietary drivers in custom boot processes, forked kernels with patches, etc. I'm yet to see a single SBC I can just download the iso from the debian website and get going. You always need some custom build.
ARM boards need a custom kernel and bootloader. These aren't things managed by the distro. The userspace IS managed by the distro and IS standard. Using yocto to manage userspace maybe made sense 15 years ago, but it has long since become far more trouble than its worth. Debian supports most architectures out of the box, has good cross-building infrastructure, many thousands of ready-to-use packages, and is non-weird.
Debian handles userspace fine but there is a lot of image customization needed a lot of the time. I have experience with developing images for a beaglebone black, where I forked the bespoke image builder (to create user accounts, mount points, set up udev rules, device tree overlays, ssh keys, etc) and that was a pain to maintain. I suppose I could have create debian packages for some of those things, but I would still have needed to set up repositories. Now I'm using an SoM with Yocto and while the learning curve is quite a bit steeper, it is much easier to customize for our experiment's needs. Also, AI assistants are really good at spitting out yocto recipes (that usually don't work immediately, but gets you 90% of the way there).
You're missing the point that when you're building the product, and not using SBC just for fun, the image you are building, and yes even the userspace, is highly customized. Running a full-blown Linux distro is a non-requirement, and often incorrect.
For more mainstream SBCs, totally agree. However a lot of the niche industry boards (especially new ones) only support Yocto out of the gate. Debian might be "supported" by the SoM/CPU vendor but manually reapplying kernel patches and praying it boots is a PITA.
I agree. In the past, I have successfully used Debian and Alpine for embedded. Never needed to compile OS kernels or standard DLLs, other people already did and published in these package repositories.
Except you may have just a 512 M industrial sd card to run on. Or even less.
I'm surprised to see no mention of Guix. It solves all of these issues, and already has a good story both for cross and native builds on a variety of architectures.
Adding new targets is deceptively easy, just copy an existing template and substitute your values.
https://codeberg.org/guix/guix/src/branch/master/gnu/system/...
https://codeberg.org/guix/guix/src/branch/master/gnu/bootloa...
Does guix have any advantage here over nix?
I didn't know Nix had support for embedded systems. Where are the images defined?
What do you mean by image? In Nix you instantiate a NixOS system and specify the architecture. There are a number of tools to build a disk image from a NixOS system.
$ my-arm-system = nixpkgs.lib.nixosSystem { system = "aarch64-linux"; modules = [ ./configuration.nix ]; };
$ :b arm-system.config.system.build.images.iso
PSA:
nixos-rebuild build-vm (--flake DIR#HOSTNAME / configuration.nix)
./result/bin/SCRIPT
to build and launch a nixos vm with qemu.
IMO the nixos-rebuild scripts are unnecessary and just obscures what’s being built.
build-vm is the same as building the configs “config.system.build.vm” attribute.
nixos-rebuild is fine for getting started but you end up needing to use the attribute path to do anything more complicated like CI caching or inspecting the drvPath.
Lisp
I haven't used it, but there's also Avocado OS [1]. Myself, I use Yocto for our embedded build. It's not perfect but for our application, which involves building a bunch of software that uses both Swift and Flutter for an embedded audio product running on a RPi CM4, it works just fine. Shout out to Mender for A/B updates.
[1] https://www.peridio.com/avocado-os
Gentoo is literally designed for this but people keep coming up with other non-solutions. A Gentoo stage 3 with a custom kernel can run on a potato.
Doesn't Yocto come from Gentoo to some extent?
Bitbake (Yocto's build system) started originally as a fork of portage (gentoo package manager), but they have diverged a ton. Very similar concepts though.
Wow, no cross-compilation as a feature?!
Architecture emulation with qemu-userspace is 5-10x slower than running a native build.
The only 'proper' no-friction cross-compilation system I know is Debian/Ubuntu, because it allows to install foreign architecture libraries into your native system, and has all the wrappers for cross-compilation in dpkg-buildpackage.
This way you don't have to maintain cross-compiler or toolchain/sysroot, you just install whatever dependencies you need with the regular `apt install` of another architecture.
I'd like to have more friendly tooling around that instead of a slow compilation of architecture emulation.
Joke: The Linux community will continue to overlap and reinvent parts of itself until there is a universal Linux kernel that can be fully adapted and customized to fit onto almost any hardware..
I heard it’s going to be called emacs.
This sounds like fun, but I'm not sure how changing build systems fixes the vendor problem. They will just ship their proprietary blobs built on top of this new build system instead of the old one, and will not change other behavior. Buying from vendors who upstream or minimally open source their work might help.
Yes, skipping builds from source can be faster, and, I don't know that you need to throw away yocto to do that https://rootcommit.com/pub/conferences/2024/elce/yocto-binar...
From the statement about "Caches builds so no piece of software is built twice." I'm guessing that the author has not enabled sstate caching https://docs.yoctoproject.org/dev/overview-manual/concepts.h...
I'm a little doubtful on the post's assertion that yocto is only needed for "deeply embedded regulated products." For products that have to follow the https://en.wikipedia.org/wiki/Cyber_Resilience_Act coming up, they will almost certainly need:
* SBOM management https://docs.yoctoproject.org/next/dev-manual/sbom.html
* CVE (or its successor) tracking https://docs.yoctoproject.org/dev/security-manual/vulnerabil...
* License management https://docs.yoctoproject.org/next/dev-manual/licenses.html
And I'll echo what some other folks have said here: if you don't need features like that, bundle your custom bootloaders and kernel with a binary distribution like Debian and call it a day.
Yes, and if they have done the hard work to deliver features in the kernel that provide a moat, that blob is the hardware.
Reversing and reimplementing the closed bits are what I want my Buildroot pipelines to do.
That’s my use case though; targeting COTS devices with (mainline) Linux support that don’t do all they could.
I have done it the ugly way (in production) and I want elegance in my old age.
The problems building with Linux and GNU environments exist because they were terribly designed with assumptions like
- You're building on the same native system as GNU and Linux packages, you install them globally in the same places that servers and desktops use
- Your C, C++ compiler and entire toolchain and other binary utilities with the kernel is a one single unit that you can only change one part at a time
- You use the same up to date headers with glibc, gcc and Linux kernel
- You're building software in the same universe of all the other packages, especially gcc libraries (libgcc_s, libstdc++), glibc (especially bad since ld-linux.so is part of it)
- The build system only uses standard paths
The reason Yocto is so complicated is that developing in a Linux environment actually sucks when you're not writing web-oriented or server / VM software. Yocto fixes it. It introduces a good set of abstractions that work around terrible design decisions that were made in overall Linux ecosystem. There are a lot because the OS design is fundamentally broken, especially with C-based toolchains which is 99.999% of the ecosystem. Current C toolchains including MSVC strongly ties OS with the C's internal types and bad decisions of 70s.
As always all articles whose title asks a question are answered with NO, 99% time. By taking away the cross-compiling abilities and the workarounds doesn't fix the brokenness of Linux and overall FOSS ecosystem.
If you're looking for how a better embedded environment looks like, look at Rust toolchains. For Linux take a look at musl-libc based ones (you 100% need a systemd distro to get away from nss complexities that musl introduces). Or even better take a look at relibc. There are barely any assumptions about the target filesystem and tooling in Rust toolchains, unlike C/C++/Make toolchains. There is redox OS but it is still in slow development and they stuck with Make, which I think was a bad decision. Android uses its own build tooling but cannot run away from C/C++ tooling unless Google revives Fuschia.
I just use Debian these days
My experience is that Yocto is a bit like CMake: people hate the mess they make with it and blame the system, even though they never bothered learning to use it.
When someone starts with "why cross-compiling the system if it's bearable to compile it directly on the device?", it really sounds like they are just not comfortable with cross-compilation. To me it would be similar to say "why would you need a compiled language, with this weird compilation step that nobody understands, when you can just write in an interpreted language?". Except that developers wouldn't generally proudly admit that they don't know how to use a compiler.
A valid reason I see to require a new embedded linux build system would be if the existing ones (mostly buildroot and Yocto) were unable to build a modern OS (e.g. by not being able to integrate some security features). But I don't think that's the case. This article sounds more like very verbose marketing for a system claiming that it can do similarly without having to learn as much (which I honestly doubt).
To me it's often the same pattern:
* Meson is nice, but it doesn't remove the need to use CMake, and it's not fundamentally simpler or more powerful (it's more comfortable at most).
* Jujutsu is nice, but it doesn't remove the need to learn git, and it's not fundamentally simpler or more powerful.
* Yoebuild sounds like even if it got traction like Meson and Jujutsu, it wouldn't remove the need to interact with Yocto for embedded linux engineers.
As an embedded linux engineer, if anyway I have to interact with Yocto and therefore learn Yocto, why would I learn another one that isn't fundamentally better? Usually people who go for those "simpler" (at least on the surface) tools do it because they don't want to learn Yocto. And for an embedded linux engineer, isn't it a red flag to not want to learn the most famous embedded linux build system?
It reminds me of Balena/Resin, which seems like it's targetting people who are not embedded linux engineers, don't want to become embedded linux engineers, but still want to distribute an embedded linux. To me that's generally a red flag.
> My experience is that Yocto is a bit like CMake: people hate the mess they make with it and blame the system, even though they never bothered learning to use it.
I agree this is true a lot of the time, speaking as someone who has made many Yocto messes and later regretted. But you also can't discount all of the messes made by silicon OEMs that you are forced to deal with to use their SoCs.
E.g. I recently did a project with Yocto on a Zynq US+. Look at the meta-xilinx layerset (or, God forbid, petalinux) and tell me they haven't made a complete mess of it.
The complexity and lack of upstreaming in kernel and userspace for so many of these SoCs, coupled with non standard boot process and devicetrees means you will be forced to use these terrible OEM layers. I don't think a new build system can fix this at all, it's a cultural/commitment that needs to change from the OEMs.
At the end of the day, of you are doing any complex embedded Linux work as a small team, you must use Yocto because it's what the OEM supports. I don't have time or resource to debug DFX not working because some userspace tool was built with slightly different flags than expected by the flaky ass xilinx upstream.
> But you also can't discount all of the messes made by silicon OEMs that you are forced to deal with to use their SoCs.
Yes, totally! It's a bit sad because we don't usually have the luxury to select the hardware based on the quality of the BSP. First because they all seem to be terrible, and second because it is somehow "hidden" cost. One selects the hardware based on its price and capabilities, and the software part is left as an exercise.
But as you say it's not a problem of Yocto, it's a problem of the OEMs being in a position to make a mess because it doesn't have an impact on them. A bit like Apple is terrible for developers (in my experience) because anyway developers don't have a choice.
CMake at least used to be like this: a black plastic bag of LEGOs, you empty it on the floor, there are some instructions there, but some of the images don't look like any of the pieces you see on the floor. Also there are some random rusty bolts, lint, and pieces from an Erector Set. There's an empty tape roll, someone wrote "buy more, ask forum" with a sharpie on it.
You can make great things with that, but it was easy to be "holding it wrong".
I still recommend CMake at work, it's the least offensive solution in most situations.
I get the feeling, it is a... "different" language. But I never understood the criticism about the documentation. At least in the last decade, I have always found the CMake documentation pretty good. Everything is documented.
Probably CMake is not a good fit for very, very big projects. But the vast majority of projects are not that big.
Usually what I see with CMake is self-inflicted pain, often where the devs start adding custom functions and mix it with python and stuff. If you keep it simple, it's very clear and does the job with surprisingly few keywords, I find. And I have setup tens of projects with CMake.
Whilst I find yocto perfectly fine. I also see that it has its limitations. I think there is a need to make yocto more nimble rather than designing a whole new build system.
yoebuild does seem interesting at least as a proof-of-concept.
I've used buildroot extensively, and an occasional yockto.
My impression in recent years is that these image cross build environments are just not as frequently needed as they were back in the day of their invention.
My most recent embedded linux environments were just embedded archlinux.
No need to cross build an image, just install and run the minimized linux environment right on the target.
Of course, a big part of the need for these cross-tools is that it seems most modern embedded linux developers are running windoze on their development workstations 8-/
Are you proposing compiling on the target? For a vast number of embedded systems that is not only impossible (insufficient disk and memory) but also incredibly slow.
At the end of the day you need some cross compilation just for board bring up.
If you're playing with some platform for which this has already been done, then sure, but that's not really the "normal" way of doing embedded.
Embedded just means ARM 99% of the time and it's cheaper and easier to use native ARM servers (AWS has them cheap) than to make 100% of software cross compile. Some parts of the firmware might need to be cross compiled but those projects are designed to cross compile.
Your target build environment often differs significantly from your server environment, so you end up needing a different toolchain and all of the problems that come with cross-compilation anyway. The toolchain and ABI settings that produce small, battery or instruction cache efficient are usually not you want on servers.
My toolchain for both is managed by the Nix derivation wrappers. It’s trivial to set custom C/C++/Rust/Go/NodeJS flags. As long as I have one flexible build tool, I can customize it for both server and embedded.
I'm not familiar with the problems that would cause "target build environment often differs significantly from your server environment". If you have an ARM server, then you should be able to download and run Alpine docker images, and then fetch most system utilities from the Alpine package repo and then compile a kernel and a few other binaries on top of that, then export an image.
The main reason to not run Debian is that Debian usually makes lots of compile-time decisions for you, and chooses to maximize the enabled options of the software. Alpine makes the opposite choices for you, creating very minimal feature sets.
It's been working well for me, but my use cases is a lot more like a custom in-house DDWRT firmware kind of a thing. If you've experienced additional constraints that make this pattern unworkable, I'd be interested to hear about them.
Run on server, chroot into target environment, configure compiler flags for eventual target, done. It's really just an afternoon of setup and then the problem is solved.
Why not Bazel?
I’ve been experimenting with what a next-generation embedded Linux build system might look like: native builds on the target architecture, modern language package managers as first-class citizens, and AI as a primary interface to the system.
Instead of cross-compiling with a large meta-layer stack, the tool builds kernel, rootfs, and applications together using one engine, with a CLI, TUI, and AI assistant talking to the same core. All you need is the tool, Docker, and Git — no global SDKs or hidden state.
It’s pre-1.0 and rough around the edges; I’m sharing it early to get feedback from people who live in Yocto/OpenEmbedded, Buildroot, Nix, etc. I’d love to hear where this breaks on your boards, what workflows feel wrong, and whether the “native builds + AI-aware build graph” direction seems promising.
>native builds
This is the complete opposite way, actually.
We need cross-compiling that is just as effortless as native compilation.
You should be able to build complex software on a powerful computer and perform costly optimization, then run it on a low-powered device.
It was true in 2005, but still? As I described in another post here, a modern strategy is to get a beefy server than can run the same ABI, then start a docker container and assemble a system from Alpine's package repo, then compile a kernel and a few in-house things, then extract a subset of that into an image. No cross-compile, and most of the useful software is in pre-built binary packages with the compile-time options you would have selected anyway.
Even if you don't have a beefy server of the same architecture, you can probably run it in qemu instead of docker to the same effect. And even if qemu is slow, you can run a build of the kernel and your in-house stuff in parallel on 64 cores and not really be affected by the qemu slowdown.
I'm interested to hear counterexamples, though.
I don’t have counter examples, but hear me out.
What if this hegemony in ISAs is partially because of the high costs of porting software and toolchains?
Maybe the next generation of embedded devices will have ISAs that are, say, extremely optimized for real time industrial controllers, but terrible at compiling C++.
It seems a little silly to give up on cross-compilation just because _today_ everything runs on ARM.
Especially because cross-compilation is not a particularly hard problem, the problem is with decades of tooling assuming the build and target environments are the same.
Well, I'd say cross-compiling is at least a little problematic... if nothing else, it is a hurdle to running automated tests. And cross-compiling still needs to exist for the people building the package repositories. I'm just sort of advocating that if you have limited amounts of time but want to put together a system image to run in an embedded or embedded-adjacent manner, the least pain / most agile solution is by building on top of a popular binary distro and then using native compilation on appropriate hardware inside a chroot or container that resembles the target.
There are no beefy RISC-V servers. There are no beefy 32-bit ARM servers.
Beowulf time?
I had been maintaining a fork of the U of KY BDR with SBCs and Arm devices in mind, might need to revive it for RISC-V.
> There are no beefy 32-bit ARM servers.
"Ampere processors natively support both 32 & 64 bit Android applications and require no binary translation for maximum instance density" ... "Run up to 120+ 3D cloud game instances per socket" ... "DRAM: 384GB - 512GB"
https://amperecomputing.com/solutions/arm-native
I've not personally tested this, but it very much appears that just like x86, you can use a 64-bit ARM kernel that has 32-bit support enabled to run a 32-bit userspace, likely compatible with docker, or even just in a chroot. Then with 120 cores, "make -j 120"
> It was true in 2005, but still? As I described in another post here, a modern strategy is to get a beefy server than can run the same ABI
I’m already sitting at an expensive computer, if everything else were equal I’d rather cross compile than buy more hardware.
I’m open to reasons that’s presently too hard of a workflow to enable so I’m (literally) paying for a penalty, but I don’t see how it’s a better idea.
Yocto and Buildroot will compile, from scratch, an entire gcc crosstool chain with standard library suite and headers to build fast and then deploy to your target. This exists.
I wouldn’t call either of those effortless, though. I’m thinking of something similar to Zig’s DX for cross-compilation.
zig makes cross-compiling easy for a single project - but not everything is zig (or supported by zig cc)
the various embedded build systems make cross-compiling easy at scale, which means you often don't need to think about it when adding a new package
Armbian and Qemu worked well for me when I needed to compile packages for an orange pi without enough RAM to actually run cargo build. Built the image on an emulator with more RAM then the target system, dropped the customized image on the SD card and booted right into the entry script.
(For the unaware, OP above is the article author.)
As a long time user of Bitbake and Yocto, and watcher of Yoe Linux... I kind of have to respectfully disagree with the approach.
One look at the Python or Node packaging ecosystem, and you can see how difficult trying to integrate those in a 'sane' way in embedded, with repeatability and security in mind, nevertheless wrangling something like C/C++ dependencies and native packages in a qemu-cross (or binfmt emulation) environment.
I feel like Bitbake and the wider Yocto ecosystem has essentially 'solved' cross compiling. Sure there are the incredibly complex codebases like Chromium that require lots and lots of study and patience, or esoteric compilers, etc. but for most applications I feel that especially AI tooling can write up a good basic recipe for integration with Yocto.
The unfortunate thing about AI tooling is Kernighan's law (where debugging is twice as hard as writing it the first time.) Especially in Embedded Linux, where 99.99% of the product code is written by others, trying to figure out where the AI didn't quite get it right or missed something can break things unexpectedly and impossibly for the unaware developer to fix, even breaking at runtime. So the build environment has to be simple and predictable. I think Yocto/Bitbake already strikes a good balance here, with other nice things like offline builds, repeatable builds, sstate caching across machines, PRbuild servers... things useful for big teams, so consider those as well.
Although, if someone does a 'uv'-like rewrite of Bitbake into Rust with the same feature set but faster, I am all for it...
Maybe, having Yoe or some other centralized distro make specific opinionated choices and then provide an open sstate feed, working inside Bitbake, could get a lot of the speed and customizability gains back without a big rewrite effort.
I wish you luck! The pain points you identified are definitely real and solving them would be valuable.
The workflow for user space can definitely improve some of this pain but I feel like a large portion of any embedded Linux development effort still ends up in the weeds for boot related items (secure boot, proper updates, nuanced kernel patches, bootloaders, device trees, and supporting machine variants, etc). Solving those to make them easy is a hard problem for sure.
Native builds on target can be very slow?
[dead]
> Fast, reasonably priced prototyping (PCB assembly, mechanical 3D printing, etc.).
RAM is expensive now. Reasonably priced seems to be counter to that fact.
The whole article feels like a promo-tour.
> The catch is that nobody handed us the tools to keep the place running.
Software such as homebrew exists. Why is the article ignoring this? Not only does it feel like promo, it now feels like propaganda. Homebrew is also just one example; LFS/BLFS also exists for embedded linux. Perhaps the quality is less than for desktop computer systems but it exists.
> Edge devices have started behaving like cloud systems. They run increasingly complex stacks, are frequently updated, and are managed remotely over their entire life.
Companies want to become dependent on others? Because that is what cloud is all about. Top-down control. A mafia for later blackmail rack-up-the-price strategies.
> That puts the cross-compile burden squarely on embedded developers, and maintaining recipes for thousands of packages has been a steady drain on the Yocto community
Seems untrue. If you have a compiler that works, you can compile. Thousands of packages? I am tracking 3888 right now. That covers really most software by far, including python pypi addons. If a single person can do that, a team has an even easier time. The whole article seems to narrate a story that is just made up.
> Building everything from source in Yocto means long builds, heavy memory use, and powerful workstations.
No it does not. I do so every day. You need a reasonably fast computer and RAM, which is now driven up by the existing hardware mafia that must be put in jail, but it is easily possible for solo persons too. Now imagine a team.
The whole article is really really crap. Absolute garbage. The claims it makes are for the most part simply not correct. It narrates a story as if all linux developers are incompetent. I highly doubt that. It also never mentions any existing software here that would change the rhetoric such as homebrew, LFS/BLFS and so forth. It feels as if that article is just engineered to tell a "we thus need xyz". That's not good writing. I mean just look at this:
"We can leverage these change agents and rethink the world of connected products."
What does that even mean? Did AI write this? Because humans are usually smarter when it comes to opinion pieces. "connected products" - everything is now connected?
Oi, more AI slop. After dealing with things that have external package management and yocto, I understand the pain. But integrating that into yocto is very doable and is a far better path forward.
And "I've been experiment with" seems to just mean "I've been vibe coding"