summaryrefslogtreecommitdiff
path: root/static/src/_posts/2021-09-22-building-appimages-with-nix.md
diff options
context:
space:
mode:
Diffstat (limited to 'static/src/_posts/2021-09-22-building-appimages-with-nix.md')
-rw-r--r--static/src/_posts/2021-09-22-building-appimages-with-nix.md267
1 files changed, 267 insertions, 0 deletions
diff --git a/static/src/_posts/2021-09-22-building-appimages-with-nix.md b/static/src/_posts/2021-09-22-building-appimages-with-nix.md
new file mode 100644
index 0000000..956f912
--- /dev/null
+++ b/static/src/_posts/2021-09-22-building-appimages-with-nix.md
@@ -0,0 +1,267 @@
+---
+title: >-
+ Building AppImages with Nix
+description: >-
+ With some process trees thrown in there for fun.
+series: nebula
+tags: tech
+---
+
+It's been a bit since I've written an update on the cryptic nebula project,
+almost 5 months (since [this post][lastnix], which wasn't officially part of the
+blog series but whatever). Since then it's switched names to "cryptic-net", and
+that we would likely use [MinIO](https://min.io/) as our network storage
+service, but neither of those is the most interesting update.
+
+The project had been stalled because of a lack of a build system which could
+fulfill the following requirements:
+
+* Network configuration (static IP, VPN certificates) of individual hosts is
+ baked into the binary they run.
+
+* Binaries are completely static; no external dependencies need to exist on the
+ host in order to run them.
+
+* Each binary runs a composition of multiple sub-services, each being a separate
+ sub-process, and all of them having been configured to work together (with
+ some possible glue code on our side) to provide the features we want.
+
+* The builder itself should be deterministic; no matter where it runs it should
+ produce the same binary given the same input parameters.
+
+Lacking such a build system we're not able to distribute the program in a way
+which "just works"; it would require some kind of configuration, or some kind of
+runtime environment to be set up, both of which would be a pain for users. And
+lacking a definite build system makes it difficult to move forward on any other
+aspect of a project, as it's not clear what may need to be redone in the future
+when the build system is decided upon.
+
+## Why not nix-bundle?
+
+My usage of [nix-bundle][nix-bundle] in a [previous post][lastnix] was an
+attempt at fulfilling these requirements. Nix in general does very well in
+fulfilling all but the second requirement, and nix-bundle was supposed to
+fulfill even that by packaging a nix derivation into a static binary.
+
+And all of this it did! Except that the mechanism of nix-bundle is a bit odd.
+The process of a nix-bundle'd binary jails itself within a chroot, which it then
+uses to fake the `/nix/store` path which nix built binaries expect to exist.
+
+This might work in a lot of cases, but it did not work in ours. For one, [nebula
+can't create its network interface when run from inside
+nix-bundle's chroot][nix-bundle-issue]. For another, being run in a chroot means
+there's going to be strange restrictions on what our binary is able to do and
+not.
+
+## AppImage
+
+What we really needed was an [AppImage][appimage]. AppImages are static binaries
+which can bundle complex applications, even those which don't expect to be
+bundled into single binaries. In this way the end result is the same as
+nix-bundle, but the mechanism AppImage uses is different and places far fewer
+restrictions on what we can and can't do with our program.
+
+## Building Sub-Services Statically with Nix
+
+It's probably possible to use nix to generate an AppImage which has the
+`/nix/store` built into it, similar to what nix-bundle does, and therefore not
+worry about whether the binaries it's bundling are static or not. But if your
+services are written in sane languages it's not that difficult to build them
+statically and dodge the issue.
+
+For example, here is how you build a go binary statically:
+
+```
+{
+ buildGoModule,
+ fetchFromGitHub,
+}:
+ buildGoModule rec {
+ pname = "nebula";
+ version = "1.4.0";
+
+ src = fetchFromGitHub {
+ owner = "slackhq";
+ repo = pname;
+ rev = "v${version}";
+ sha256 = "lu2/rSB9cFD7VUiK+niuqCX9CI2x+k4Pi+U5yksETSU=";
+ };
+
+ vendorSha256 = "p1inJ9+NAb2d81cn+y+ofhxFz9ObUiLgj+9cACa6Jqg=";
+
+ doCheck = false;
+
+ subPackages = [ "cmd/nebula" "cmd/nebula-cert" ];
+
+ CGO_ENABLED=0;
+ tags = [ "netgo" ];
+ ldflags = [
+ "-X main.Build=${version}"
+ "-w"
+ "-extldflags=-static"
+ ];
+ };
+```
+
+And here's how to statically build a C binary:
+
+```
+{
+ stdenv,
+ glibcStatic, # e.g. pkgs.glibc.static
+}:
+ stdenv.mkDerivation rec {
+ pname = "dnsmasq";
+ version = "2.85";
+
+ src = builtins.fetchurl {
+ url = "https://www.thekelleys.org.uk/dnsmasq/${pname}-${version}.tar.xz";
+ sha256 = "sha256-rZjTgD32h+W5OAgPPSXGKP5ByHh1LQP7xhmXh/7jEvo=";
+ };
+
+ nativeBuildInputs = [ glibcStatic ];
+
+ makeFlags = [
+ "LDFLAGS=-static"
+ "DESTDIR="
+ "BINDIR=$(out)/bin"
+ "MANDIR=$(out)/man"
+ "LOCALEDIR=$(out)/share/locale"
+ ];
+ };
+```
+
+The derivations created by either of these expressions can be plugged right into
+the `pkgs.buildEnv` used to create the AppDir (see AppDir section below).
+
+## Process Manager
+
+An important piece of the puzzle for getting our nebula project into an AppImage
+was a process manager. We need something which can run multiple service
+processes simultaneously, restart processes which exit unexpectedly, gracefully
+handle shutting down all those processes, and coalesce the logs of all processes
+into a single stream.
+
+There are quite a few process managers out there which could fit the bill, but
+finding any which could be statically compiled ended up not being an easy task.
+In the end I decided to see how long it would take me to implement such a
+program in go, and hope it would be less time than it would take to get
+`circus`, a python program, bundled into the AppImage.
+
+2 hours later, [pmux][pmux] was born! Check it out. It's a go program so
+building it looks pretty similar to the nebula builder above, so I won't repeat
+it. However I will show the configuration we're using for it within the
+AppImage, to show how it ties all the processes together:
+
+```yaml
+processes:
+ - name: nebula
+ cmd: bin/nebula
+ args:
+ - "-config"
+ - etc/nebula/nebula.yml
+
+ - name: dnsmasq
+ cmd: bin/dnsmasq
+ args:
+ - "-d"
+ - "-C"
+ - ${dnsmasq}/etc/dnsmasq/dnsmasq.conf
+```
+
+## AppDir -> AppImage
+
+Generating an AppImage requires an AppDir. An AppDir is a directory which
+contains all files required by a program, rooted to the AppDir. For example, if
+the program expects a file to be at `/etc/some/conf`, then that file should be
+places in the AppDir at `<AppDir-path>/etc/some/conf`.
+
+[These docs](https://docs.appimage.org/packaging-guide/manual.html#ref-manual)
+were very helpful for me in figuring out how to construct the AppDir. I then
+used the `pkgs.buildEnv` utility to create an AppDir derivation containing
+everything the nebula project needs to run:
+
+```
+ appDir = pkgs.buildEnv {
+ name = "my-AppDir";
+ paths = [
+
+ # real directory containing non-built files, e.g. the pmux config
+ ./AppDir
+
+ # static binary derivations shown previously
+ nebula
+ dnsmasq
+ pmux
+ ];
+ };
+```
+
+Once the AppDir is built one needs to use `appimagetool` to turn it into an
+AppImage. There is an `appimagetool` build in the standard nixpkgs, but
+unfortunately it doesn't seem to actually work...
+
+Luckily nix-bundle is working on AppImage support, and includes a custom build
+of `appimagetool` which does work!
+
+```
+{
+ fetchFromGitHub,
+ callPackage,
+}: let
+ src = fetchFromGitHub {
+ owner = "matthewbauer";
+ repo = "nix-bundle";
+ rev = "223f4ffc4179aa318c34dc873a08cb00090db829";
+ sha256 = "0pqpx9vnjk9h24h9qlv4la76lh5ykljch6g487b26r1r2s9zg7kh";
+ };
+in
+ callPackage "${src}/appimagetool.nix" {}
+```
+
+Using `callPackage` on this expression will give you a functional `appimagetool`
+derivation. From there's it's a simple matter of writing a derivation which
+generates the AppImage from a created AppDir:
+
+```
+{
+ appDir,
+ appimagetool,
+}:
+ pkgs.stdenv.mkDerivation {
+ name = "program-name-AppImage";
+
+ src = appDir;
+ buildInputs = [ appimagetool ];
+ ARCH = "x86_64"; # required by appimagetool
+
+ builder = builtins.toFile "build.sh" ''
+ source $stdenv/setup
+ cp -rL "$src" buildAppDir
+ chmod +w buildAppDir -R
+ mkdir $out
+
+ # program-name needs to match the desktop file in the AppDir
+ appimagetool program-name "$out/program-name-bin"
+ '';
+ }
+```
+
+Running that derivation deterministically spits out a binary at
+`result/program-name-bin` which can be executed and run immediately, on any
+system using the same CPU architecture.
+
+## Fin
+
+I'm extremely hyped to now have the ability to generate binaries for our nebula
+project that people can _just run_, without them worrying about which
+sub-services that binary is running under-the-hood. From a usability perspective
+it's way nicer than having to tell people to "install docker" or "install nix",
+and from a dev perspective we have a really solid foundation on which to build a
+quite complex application.
+
+[lastnix]: {% post_url 2021-04-22-composing-processes-into-a-static-binary-with-nix %}
+[nix-bundle]: https://github.com/matthewbauer/nix-bundle
+[nix-bundle-issue]: https://github.com/matthewbauer/nix-bundle/issues/78
+[appimage]: https://appimage.org/
+[pmux]: https://github.com/cryptic-io/pmux