Warpforge is a "Build Anything" tool for the modern era with a deep emphasis on the ability to "factory reset" the build environment at any time -- and not just reset one environment to a checkpoint, but also to be able to effortlessly reproduce all the prior builds and computations that produce the "factory floor" itself, too: on demand, and without fuss.
How much time have you lost to debugging library versioning mismatches? How often has a system update left your machine in a broken state? How often have you installed something on your machine just to try to "make it work"? If you've moved to containers -- how much time have you spent trying to build "the perfect base image", and trying to maintain and wrangle your container build scripts? How much work would still remain in front of you if you needed to reproduce a container image you snapshotted months ago? If the answer to any of these questions troubles you -- we hope Warpforge will present a viable way to help fix all that.
In this talk, we'll introduce Warpforge, describe its objectives, give an overview of its declarative API, briefly cover how the guts work (spoiler: LOTS of hashes)... and ultimately, demonstrate it working, live, because seeing is believing.
Also in this talk, we'll discuss some conventions in package management and dynamic library linking -- and how we can do these differently, to produce systems of more reusable components. We've started pursuing this together with building Warpforge, in a quest to make the most transparent and the most reusable systems possible.
Comparisons to Bazel/Blaze, Nix, Guix, and even to Docker all abound -- we hope you'll find some of the best parts of each visible in Warpforge, plus a few key innovations and user experience improvements of our own, together all in one bundle.
Warpforge's approach to becoming a powerful "Build Anything" tool is to orient itself around providing the user with "computation-addressable" systems -- meaning: the same idea as content-addressable systems, but now extended to also apply to data processing and software compilation. The use of cryptographic hashes to identify data has valuable properties for both security and for simplifying organization; in Warpforge, we apply that same idea to hashing the descriptions of environments and the computations we want to run within them, for similar victories in both security and simplification. What Git did for source code version control, Warpforge wants to do for build instructions and data processing: decentralized, snapshottable, portable, and utterly agnostic to whatever you put in it.
Warpforge is granular: although it uses containers for hermeticity, Warpforge lets you compose the filesystem from as many pieces as you want. Warpforge features a freeform filesystem assembly syntax, which lets you state any content you want shall appear at any path you want. This stands in stark contrast to most other container systems which limit you to monolithic "images" -- which invariably devolve into balls-of-mud, becoming difficult to maintain, difficult to introspect, and almost impossible to compose.
Warpforge is a hackable tool. Everything works via a JSON API. There are both "high" and "low" level variants of this API: the "high" level lets you describe pipelines of computations, wired together with human-readable names of your choosing; the "low" level API always uses only content-addressable hashes for all data input, and always hashes things immediate upon output. The interaction of these two APIs is the secret sauce: it makes a system that's both usable and also highly precise and highly auditable and reproducible.
Warpforge aims to be a developer productivity tool, but at the same time, shift the culture: much like Git taught the world about hash trees, Warpforge wants to teach the world about reproducible builds. Because every output is hashed, computation that fails to reproduce the same output becomes noticed immediately.
Warpforge is a perfect tool for those interested in SBOMs (Software Bill Of Materials). Because Warpforge identifies all inputs explicitly, it can very naturally produce an SBOM. In fact, it does so in standard operation -- and unlike many other sytems, Warpforge doesn't just produce a descriptive, after-the-fact SBOM: Warpforge instructions are actually a load-bearing bill-of-materials by nature.
Warpforge stops slightly short of doing package management. However, Warpforge does offer some suggestions on ways to build package management systems... with more merkle trees, so that things remain reproducible, introspectable, and auditable even as they grow in systemic complexity. (That's enough spoilers; come the talk to see how!)
You can find out more about Warpforge at http://warpforge.io/ and https://github.com/warpfork/warpforge/ .