Nix and buck2: from enemies to lovers with snowydeer
Nix does dependencies and distribution well, but has a controlling personality: it wants to build everything in the build graph.
Buck2 delivers fast, user-friendly, and scalable project builds, but has an equally controlling personality and a lacking public dependency ecosystem.
What if their build graphs touched ... and they were both girls?
In this talk, I will demonstrate how we go from Nix to buck2 to Nix then deploy with containers:
* Using Nix for dependencies in buck2
* Using buck2 in the project build: remote caching, fast builds, ~zero evaluation time
* Importing store paths to Lix from buck2 output while correctly handling dependencies
* Extending these techniques to build Docker images defined entirely in buck2, using nixpkgs `dockerTools`
This talk focuses on buck2, but the techniques used apply to any powerful non-Nix build system.
Hermetic build systems are hard to interoperate with, but there are cheat codes: by resolving their trust issues, you can combine their respective benefits.
For the last year, my team and I have been working on a Buck2 migration for one of the largest Haskell codebases in industry to reduce build times from dozens of minutes to seconds and improve user experience.
We use Nix to orchestrate hundreds of Haskell dependencies and hundreds more dependencies across the Rust, TypeScript, Python and C ecosystems.
On top of that, we are running a much newer Haskell compiler than most of the industry due to active involvement in GHC development, which means a *heavily* patched nixpkgs Haskell dependency set to be able to incorporate these improvements.
By building a Nix-Buck2-Nix sandwich, we were able to focus on rewriting our product build in buck2 while keeping both our Nix-based dependencies and our deployment pipelines to both NixOS and AWS ECS completely as-is.
The novel contributions of this talk are:
* Creating full-featured store paths *entirely* outside of Nix with Snowydeer:
* Reference scanning outside of Nix using ripgrep
* Using a new Lix CLI feature: output-addressed paths with references
* Referencing those store paths from Nix language for further processing
* Using Nix as a build step rather than a build driver to create an ergonomic Docker image builder with Snowydeer Container
This talk builds on previous work, such as:
- ["Integrating Nix and Buck2 for fun and profit"](https://www.youtube.com/watch?v=pDkFk7iggIE) by Claudio Bley
- ["Haskell Builds at Scale: Comparing Bazel and Buck2"](https://www.youtube.com/watch?v=eA-3Gfr4epU) by Andreas Herrmann
- ["Towards Dream Haskell Build Experience"](https://www.youtube.com/watch?v=WrdXUYFnRv4) by Ian-Woo Kim
Licensed to the public under http://creativecommons.org/licenses/by/4.0