diff options
Diffstat (limited to 'static/src/_posts/2021-09-22-building-appimages-with-nix.md')
| -rw-r--r-- | static/src/_posts/2021-09-22-building-appimages-with-nix.md | 266 |
1 files changed, 0 insertions, 266 deletions
diff --git a/static/src/_posts/2021-09-22-building-appimages-with-nix.md b/static/src/_posts/2021-09-22-building-appimages-with-nix.md deleted file mode 100644 index 612e601..0000000 --- a/static/src/_posts/2021-09-22-building-appimages-with-nix.md +++ /dev/null @@ -1,266 +0,0 @@ ---- -title: >- - Building AppImages with Nix -description: >- - With some process trees thrown in there for fun. -series: nebula -tags: tech ---- - -It's been a bit since I've written an update on the cryptic nebula project, -almost 5 months (since [this post][lastnix], which wasn't officially part of the -blog series but whatever). Since then it's switched names to "cryptic-net", and -that we would likely use [MinIO](https://min.io/) as our network storage -service, but neither of those is the most interesting update. - -The project had been stalled because of a lack of a build system which could -fulfill the following requirements: - -* Network configuration (static IP, VPN certificates) of individual hosts is - baked into the binary they run. - -* Binaries are completely static; no external dependencies need to exist on the - host in order to run them. - -* Each binary runs a composition of multiple sub-services, each being a separate - sub-process, and all of them having been configured to work together (with - some possible glue code on our side) to provide the features we want. - -* The builder itself should be deterministic; no matter where it runs it should - produce the same binary given the same input parameters. - -Lacking such a build system we're not able to distribute cryptic-net in a way -which "just works"; it would require some kind of configuration, or some kind of -runtime environment to be set up, both of which would be a pain for users. And -lacking a definite build system makes it difficult to move forward on any other -aspect of a project, as it's not clear what may need to be redone in the future -when the build system is decided upon. - -## Why not nix-bundle? - -My usage of [nix-bundle][nix-bundle] in a [previous post][lastnix] was an -attempt at fulfilling these requirements. Nix in general does very well in -fulfilling all but the second requirement, and nix-bundle was supposed to -fulfill even that by packaging a nix derivation into a static binary. - -And all of this it did! Except that the mechanism of nix-bundle is a bit odd. -The process of a nix-bundle'd binary jails itself within a chroot, which it then -uses to fake the `/nix/store` path which nix built binaries expect to exist. - -This might work in a lot of cases, but it did not work in ours. For one, [nebula -can't create its network interface when run from inside -nix-bundle's chroot][nix-bundle-issue]. For another, being run in a chroot means -there's going to be strange restrictions on what our binary is able to do and -not. - -## AppImage - -What we really needed was an [AppImage][appimage]. AppImages are static binaries -which can bundle complex applications, even those which don't expect to be -bundled into single binaries. In this way the end result is the same as -nix-bundle, but the mechanism AppImage uses is different and places far fewer -restrictions on what we can and can't do with our program. - -## Building Sub-Services Statically with Nix - -It's probably possible to use nix to generate an AppImage which has the -`/nix/store` built into it, similar to what nix-bundle does, and therefore not -worry about whether the binaries it's bundling are static or not. But if your -services are written in sane languages it's not that difficult to build them -statically and dodge the issue. - -For example, here is how you build a go binary statically: - -``` -{ - buildGoModule, - fetchFromGitHub, -}: - buildGoModule rec { - pname = "nebula"; - version = "1.4.0"; - - src = fetchFromGitHub { - owner = "slackhq"; - repo = pname; - rev = "v${version}"; - sha256 = "lu2/rSB9cFD7VUiK+niuqCX9CI2x+k4Pi+U5yksETSU="; - }; - - vendorSha256 = "p1inJ9+NAb2d81cn+y+ofhxFz9ObUiLgj+9cACa6Jqg="; - - doCheck = false; - - subPackages = [ "cmd/nebula" "cmd/nebula-cert" ]; - - CGO_ENABLED=0; - tags = [ "netgo" ]; - ldflags = [ - "-X main.Build=${version}" - "-w" - "-extldflags=-static" - ]; - }; -``` - -And here's how to statically build a C binary: - -``` -{ - stdenv, - glibcStatic, # e.g. pkgs.glibc.static -}: - stdenv.mkDerivation rec { - pname = "dnsmasq"; - version = "2.85"; - - src = builtins.fetchurl { - url = "https://www.thekelleys.org.uk/dnsmasq/${pname}-${version}.tar.xz"; - sha256 = "sha256-rZjTgD32h+W5OAgPPSXGKP5ByHh1LQP7xhmXh/7jEvo="; - }; - - nativeBuildInputs = [ glibcStatic ]; - - makeFlags = [ - "LDFLAGS=-static" - "DESTDIR=" - "BINDIR=$(out)/bin" - "MANDIR=$(out)/man" - "LOCALEDIR=$(out)/share/locale" - ]; - }; -``` - -The derivations created by either of these expressions can be plugged right into -the `pkgs.buildEnv` used to create the AppDir (see AppDir section below). - -## Process Manager - -An important piece of the puzzle for getting cryptic-net into an AppImage was a -process manager. We need something which can run multiple service processes -simultaneously, restart processes which exit unexpectedly, gracefully handle -shutting down all those processes, and coalesce the logs of all processes into a -single stream. - -There are quite a few process managers out there which could fit the bill, but -finding any which could be statically compiled ended up not being an easy task. -In the end I decided to see how long it would take me to implement such a -program in go, and hope it would be less time than it would take to get -`circus`, a python program, bundled into the AppImage. - -2 hours later, [pmux][pmux] was born! Check it out. It's a go program so -building it looks pretty similar to the nebula builder above, so I won't repeat -it. However I will show the configuration we're using for it within the -AppImage, to show how it ties all the processes together: - -```yaml -processes: - - name: nebula - cmd: bin/nebula - args: - - "-config" - - etc/nebula/nebula.yml - - - name: dnsmasq - cmd: bin/dnsmasq - args: - - "-d" - - "-C" - - ${dnsmasq}/etc/dnsmasq/dnsmasq.conf -``` - -## AppDir -> AppImage - -Generating an AppImage requires an AppDir. An AppDir is a directory which -contains all files required by a program, rooted to the AppDir. For example, if -the program expects a file to be at `/etc/some/conf`, then that file should be -places in the AppDir at `<AppDir-path>/etc/some/conf`. - -[These docs](https://docs.appimage.org/packaging-guide/manual.html#ref-manual) -were very helpful for me in figuring out how to construct the AppDir. I then -used the `pkgs.buildEnv` utility to create an AppDir derivation containing -everything cryptic-net needs to run: - -``` - appDir = pkgs.buildEnv { - name = "cryptic-net-AppDir"; - paths = [ - - # real directory containing non-built files, e.g. the pmux config - ./AppDir - - # static binary derivations shown previously - nebula - dnsmasq - pmux - ]; - }; -``` - -Once the AppDir is built one needs to use `appimagetool` to turn it into an -AppImage. There is an `appimagetool` build in the standard nixpkgs, but -unfortunately it doesn't seem to actually work... - -Luckily nix-bundle is working on AppImage support, and includes a custom build -of `appimagetool` which does work! - -``` -{ - fetchFromGitHub, - callPackage, -}: let - src = fetchFromGitHub { - owner = "matthewbauer"; - repo = "nix-bundle"; - rev = "223f4ffc4179aa318c34dc873a08cb00090db829"; - sha256 = "0pqpx9vnjk9h24h9qlv4la76lh5ykljch6g487b26r1r2s9zg7kh"; - }; -in - callPackage "${src}/appimagetool.nix" {} -``` - -Using `callPackage` on this expression will give you a functional `appimagetool` -derivation. From there's it's a simple matter of writing a derivation which -generates the AppImage from a created AppDir: - -``` -{ - appDir, - appimagetool, -}: - pkgs.stdenv.mkDerivation { - name = "cryptic-net-AppImage"; - - src = appDir; - buildInputs = [ appimagetool ]; - ARCH = "x86_64"; # required by appimagetool - - builder = builtins.toFile "build.sh" '' - source $stdenv/setup - cp -rL "$src" buildAppDir - chmod +w buildAppDir -R - mkdir $out - - appimagetool cryptic-net "$out/cryptic-net-bin" - ''; - } -``` - -Running that derivation deterministically spits out a binary at -`result/cryptic-net-bin` which can be executed and run immediately, on any -system using the `x86_46` CPU architecture. - -## Fin - -I'm extremely hyped to now have the ability to generate binaries for cryptic-net -that people can _just run_, without them worrying about which sub-services that -binary is running under-the-hood. From a usability perspective it's way nicer -than having to tell people to "install docker" or "install nix", and from a dev -perspective we have a really solid foundation on which to build a quite complex -application. - -[lastnix]: {% post_url 2021-04-22-composing-processes-into-a-static-binary-with-nix %} -[nix-bundle]: https://github.com/matthewbauer/nix-bundle -[nix-bundle-issue]: https://github.com/matthewbauer/nix-bundle/issues/78 -[appimage]: https://appimage.org/ -[pmux]: https://github.com/cryptic-io/pmux |