This is the second in a series of articles on working with Elm in Guix. See the first article for an introduction. The outline is:
- packaging the Elm compiler
- developing Guix infrastructure for building Elm apps, and using this to package the Elm reactor, and the full Elm tool
- deploying an Elm web application on Guix System
In part 1, we succeeded in packaging the Elm compiler. We now have the elm tool available, and can use it to build an Elm application from the command line. I’ll use a small Elm app of mine for demonstration purposes, the web interface to puzzle-draw.
In this article, we’ll develop Guix tooling to allow us to package and build this web app, and to package the full elm tool including the reactor. You can find the code in the last couple of commits of the elm branch here: github.com/robx/guix.
Building the application without Guix
To begin, let’s have a look at what happens when we build the application from the command line. We’ll start by wiping out Elm’s package cache.
$ rm -rf ~/.elm
$ git clone https://github.com/robx/puzzle-draw.git && cd puzzle-draw/web
$ elm make --output=../static/web.js src/Main.elm
Starting downloads...
● elm/time 1.0.0
● elm/json 1.0.0
● elm/url 1.0.0
● elm/browser 1.0.0
● elm/virtual-dom 1.0.0
● elm/html 1.0.0
● elm/core 1.0.0
● elm/http 1.0.0
Dependencies ready!
Success! Compiled 1 module. So elm make can successfully build our application, based on the definition in elm.json:
$ cat elm.json
{
"type": "application",
"source-directories": [
"src"
],
"elm-version": "0.19.0",
"dependencies": {
"direct": {
"elm/browser": "1.0.0",
"elm/core": "1.0.0",
"elm/html": "1.0.0",
"elm/http": "1.0.0",
"elm/json": "1.0.0",
"elm/url": "1.0.0"
},
"indirect": {
"elm/time": "1.0.0",
"elm/virtual-dom": "1.0.0"
}
},
"test-dependencies": {
"direct": {},
"indirect": {}
}
}With an empty package database and missing ./elm-stuff, elm make goes through:
Pull a database of available package versions from
package.elm-lang.org, and write it to~/.elm/0.19.0/package/versions.dat.Download the dependency source packages from Github, and unpack them in
~/.elmand update the package database:$ find ~/.elm/ | grep elm/url ~/.elm/0.19.0/package/elm/url ~/.elm/0.19.0/package/elm/url/1.0.0 ~/.elm/0.19.0/package/elm/url/1.0.0/src ~/.elm/0.19.0/package/elm/url/1.0.0/src/Url.elm [...]Compile some auxiliary files from the dependency sources within
~/.elm:$ find ~/.elm/ | grep 'elm/url.*\.dat$' ~/.elm/0.19.0/package/elm/url/1.0.0/ifaces.dat ~/.elm/0.19.0/package/elm/url/1.0.0/cached.dat ~/.elm/0.19.0/package/elm/url/1.0.0/objs.datCompile the application sources into
elm-stuff:$ find elm-stuff/ elm-stuff/ elm-stuff/0.19.0 elm-stuff/0.19.0/summary.dat elm-stuff/0.19.0/Main.elmo elm-stuff/0.19.0/Main.elmiGenerate output Javascript.
It’s only really steps 1 and 2 that are interesting for us here. We can happily let elm make do its stuff in steps 3-5 within the Guix sandbox.
However, building for Guix, we can’t allow steps 1 and 2 as is both since they require network access, and since we need to explicitly provide all dependencies with hashes for a Guix package. Luckily, it turns out that elm make does work absent network access if ~/.elm is in the right state. It needs:
- A valid
versions.datfile that contains at least all the dependencies listed in our application’selm.json. - Unpacked source archives for the dependencies as listed above.
- An (empty)
elm-stuffdirectory.
Build systems
Every Guix package specifies a build system, which determines in pretty full generality how the package definition is converted into an output in the Guix store.
In the Elm case, it makes sense to treat (Elm) packages and applications separately:
We only need Elm packages to provide source files for application builds. Building an Elm package will simply amount to downloading and unpacking a source archive.
For Elm applications, we want to call
elm maketo compile the Elm source files to Javascript.
Build systems consist of two parts: A core build script, which resides within guix/build. This is build-side code, executed within the Guix sandbox. In our case, this will be two functions elm-package-build and elm-application-build, defined in guix/build/elm-build-system.scm.
Then, there’s a bunch of user-side scaffolding, assembling build ingredients and passing them to the build side. In our case, this will primarily be two build-system records elm-package-build-system and elm-application-build-system, defined in guix/build-system/elm.scm.
Here’s what an Elm-package-package might look like. (build-system elm-package-build-system) here tells Guix to use that build system.
(define-public elm-virtual-dom
(package
(name "elm-virtual-dom")
(version "1.0.0")
(source
(origin
(method url-fetch)
(uri (elm-package-uri "elm/virtual-dom" version))
(sha256
(base32
"0hm8g92h7z39km325dlnhk8n00nlyjkqp3r3jppr37k2k13md6aq"))))
(build-system elm-package-build-system)
(synopsis
"Core virtual DOM implementation, basis for HTML and SVG libraries")
(description
"Core virtual DOM implementation, basis for HTML and SVG libraries")
(home-page #f)
(license license:bsd-3)))Below, we’ll go over the build side code in some detail, and skim the build system definition as such.
elm-package-build
The core of the package build system is the function elm-package-build, which simply unpacks the source archive, and copies it to the output path.
The function is called with three arguments:
sourceA path to the source, as specified in the package definition.
inputsAn association list mapping input names to paths. This includes the various explicit dependencies we could have listed in the package definition’s
inputfields, as well as dependencies specified by the build-system, such as thetartool.outputA store path that we’re meant to write our result to.
(define (elm-package-build source inputs output)
(let ((tar (assoc-ref inputs "tar"))
(gzip (assoc-ref inputs "gzip")))
(setenv "PATH" (string-append (getenv "PATH") ":"
(string-append tar "/bin") ":"
(string-append gzip "/bin")))
(mkdir "source")
(with-directory-excursion "source"
(invoke "tar" "xzf" source)
(let ((dir (car (scandir "." (lambda (f) (> (string-length f) 2))))))
(copy-recursively dir output)))))build-versions.dat
A first ingredient that our application build system needs is way to generate the Elm package database versions.dat.
To interact with it, elm make uses the Haskell binary package. The core of the relevant code can be found in Elm/Package.hs. It’s a pretty straightforward encoding, we mostly have to ensure we order entries correctly matching the Haskell encoding of maps. Using the module (ice-9 binary-ports), here’s an implementation.
(define* (put-packages port pkgs)
"Writes an elm package database to PORT based on the
list PKGS of packages. Each package should be a list
of the form '(\"elm\" \"core\" (1 0 0))."
(define (put-int64 port x)
(let ((vec (make-bytevector 8)))
(bytevector-s64-set! vec 0 x (endianness big))
(put-bytevector port vec)))
(define (put-text port s)
(put-int64 port (string-length s)) ; this should be utf8 length, utf8
(put-string port s))
(define (put-version port v)
(map (cut put-u8 port <>) v)) ; there's a different encoding for very large versions
(define (put-package port pkg)
(match pkg
((author project version)
(begin
(put-text port author)
(put-text port project)
(put-int64 port 1)
(put-version port version)))))
(put-int64 port (length pkgs)) ; total number of versions
(put-int64 port (length pkgs)) ; number of packages
(for-each
(cut put-package port <>)
(sort pkgs (match-lambda*
(((auth1 proj1 _) (auth2 proj2 _))
(or (string<? auth1 auth2)
(and (string=? auth1 auth2)
(string<? proj1 proj2))))))))Using this, we can define a (hacky) function to build a versions.dat file that matches a package tree:
(define* (parse-version version)
"Parse an elm package version from string to a list of
integer components."
(map string->number (string-split version #\.)))
(define* (build-versions.dat)
"Build an elm package database in the file versions.dat
in the current directory to match the existing unpacked elm
modules."
(format #t "building versions.dat in ~a~%" (getcwd))
(let ((packages (string-split
(get-line (open-input-pipe "echo */*/*"))
#\ ))
(out (open-output-file "versions.dat")))
(format #t "packages: ~a~%" packages)
(put-packages
out
(map
(lambda (path)
(match (string-split path #\/)
((author project version)
(list author project (parse-version version)))))
packages))
(close out)))elm-application-build
The core of the Elm application build system is the function elm-application-build. We’ll walk through it now.
(define (elm-application-build source inputs output elm-modules)The arguments to elm-application-build are like those for elm-package-build above, with two modifications: We’ll find that the source archive is redundantly available as an input with key “source” – I’m not sure if this is generally the case, and why we should be using this and not the source argument below, but there must have been a Good Reason.
Then, we pass an extra argument elm-modules to determine how to call elm make. The default is ((("Main.elm") . "main.js")), stating that we want to generate one Javascript file main.js by compiling the Elm module Main.elm.
(let ((elm (string-append (assoc-ref inputs "elm-compiler") "/bin/elm"))
(deps ".elm/0.19.0/package")
(tar (assoc-ref inputs "tar"))
(xz (assoc-ref inputs "xz")))
(setenv "PATH" (string-append (getenv "PATH") ":"
(string-append tar "/bin") ":"
(string-append xz "/bin")))We start by setting some variables. In particular, we set up $PATH to allow the unpack function we copied from some other build system to work later on.
(format #t "collecting dependencies~%")
(mkdir-p deps)
(for-each
(match-lambda
((n . pkg)
(when (and (string-prefix? "elm-" n)
(not (equal? "elm-compiler" n)))
(match (elm-package-and-version pkg)
((name . version)
(copy-recursively pkg
(string-append deps "/" name "/" version)))))))
inputs)Next we start setting up our package directory. We loop over all inputs, filtering for those that seem to be Elm packages. This is a terrible way to do things, but seems to be par for the course when comparing other build systems. A more reliable way might be to check whether there’s an elm.json file in the root directory of an input.
By assumption, these inputs are unpacked source archives, which we copy to the correct place in ~/.elm/0.19.0/package.
(format #t "generating versions.dat~%")
(with-directory-excursion deps (build-versions.dat))With this done, we call out to our helper to generate a package database fitting the set of unpacked packages.
(format #t "setting up elm env: cwd=~a~%" (getcwd))
(setenv "HOME" (getcwd))
(setenv "HTTP_PROXY" ".")
(mkdir "src") ; extra level of src directory because unpack uses the first subdir...
(with-directory-excursion "src"
(format #t "extracting source~%")
(unpack #:source (assoc-ref inputs "source")) ; changes directory
(mkdir "elm-stuff")
(chmod "elm-stuff" #o775)Now we set up things to trick elm make into happiness: We set $HOME to point at the package database we generated, set $HTTP_PROXY to cleanly break network access, unpack the source archive (in a bit of a messy way because that’s how unpack works, and finally generate an empty elm-stuff directory.
(for-each
(match-lambda
((srcs . dst)
(begin
(format #t "building ~a from ~a~%" dst srcs)
(apply invoke elm "make" (string-append "--output=" output "/" dst)
(map (lambda (src) (string-append "src/" src)) srcs)))))
elm-modules))))Finally, we loop over the elm-modules argument, calling elm make with target in the output directory.
The build systems proper
We define the two Elm build systems in guix/build-system/elm.scm:
(define elm-package-build-system
(build-system
(name 'elm-package)
(description
"Elm package build system, merely unpacking the source archive.")
(lower lower-package)))
(define elm-application-build-system
(build-system
(name 'elm-application)
(description
"Elm application build system.")
(lower lower-application)))The rest of the file is more or less boiler plate, to pass between the user-side and the build-side code. There’s some hope this will become cleaner once the build systems are converted to use G-expressions. For the moment it didn’t seem worthwhile really trying to understand this part; I was content to get it to work.
Here’s the implementation of lower-application:
(define* (lower-application name
#:key source inputs native-inputs outputs system target
guile builder modules allowed-references
(elm-modules '((("Main.elm") . "main.js"))))
"Return a bag for NAME."
(bag
(name name)
(system system)
(target target)
(host-inputs `(,@(if source
`(("source" ,source))
'())
("elm-compiler" ,(elm-compiler))
,@inputs
,@(gnu:standard-packages)))
(build-inputs `(,@native-inputs))
(outputs outputs)
(build elm-application-build)
(arguments `(#:guile ,guile
#:modules ,modules
#:elm-modules ,elm-modules
#:allowed-references ,allowed-references))))I don’t understand this well enough to explain everything; important parts are:
We pass in the Elm compiler itself as an input, so it’ll be available to
elm-application-build.We pass through the
elm-modulesargument, allowing it to be specified as a package argument like this:(package ... (build-system elm-application-build-system) (arguments `(#:elm-modules ((("Main.elm") . "main.js")))) ...)
An Elm application importer
At this point we could write Elm package definitions by hand. However, that’s a bit unwieldy, so I went on and implemented guix import elm, which takes an elm.json file and generates a package definition.
The core of the importer is defined in guix/import/elm.scm, with some helper code in guix/scripts/import/elm.scm to deal with the command line interface. The gist of the code is:
- Parse an
elm.jsonfile (elm.json->guix-package). - If it’s an Elm package, just assemble (
make-elm-package-sexp) theelm.jsonfields into a Guix package, together with URL and hash of the source archive. - Else, collect the list of dependencies, and recurse to assemble each of those into a local package definition as above, and then assemble it into a Guix package definition.
Here’s an extract of the code:
(define (make-elm-package-sexp name version summary license)
`(package
(name ,(elm-package-name name))
(version ,version)
(source (origin
(method url-fetch)
(uri (elm-package-uri ,name version))
(sha256
(base32
,(guix-hash-url (elm-package-uri name version))))))
(build-system elm-package-build-system)
(synopsis ,summary)
(description ,summary)
(home-page #f)
(license ,(spdx-string->license license))))
(define (elm.json->guix-package elm.json)
"Read package metadata from the given ELM.JSON file, and return
the `package' s-expression corresponding to that package."
(let ((type (assoc-ref elm.json "type")))
(cond
((equal? type "package")
(let* ((name (assoc-ref elm.json "name"))
(version (assoc-ref elm.json "version"))
(license (assoc-ref elm.json "license"))
(summary (assoc-ref elm.json "summary")))
(make-elm-package-sexp name version summary license)))
((equal? type "application")
(make-elm-app-sexp
(map (match-lambda ((name . version)
`(,(elm-package-name name) . ,(elm->guix-package name version))))
(get-dependencies elm.json))))
(else
(error "unsupported elm package type: " type)))))Running this on elm.json within the puzzle-draw sources now gives us an almost-ready package:
(let ((elm-url
(package
(name "elm-url")
(version "1.0.0")
(source
(origin
(method url-fetch)
(uri (elm-package-uri "elm/url" version))
(sha256
(base32
"0av8x5syid40sgpl5vd7pry2rq0q4pga28b4yykn9gd9v12rs3l4"))))
(build-system elm-package-build-system)
(synopsis
"Create and parse URLs. Use for HTTP and \"routing\" in single-page apps (SPAs)")
(description
"Create and parse URLs. Use for HTTP and \"routing\" in single-page apps (SPAs)")
(home-page #f)
(license license:bsd-3)))
(elm-json
...)
...)
(package
(name #f)
(version #f)
(source #f)
(build-system elm-application-build-system)
(native-inputs
`(("elm-url" ,elm-url)
("elm-json" ,elm-json)
("elm-http" ,elm-http)
("elm-html" ,elm-html)
("elm-core" ,elm-core)
("elm-browser" ,elm-browser)
("elm-virtual-dom" ,elm-virtual-dom)
("elm-time" ,elm-time)))
(synopsis #f)
(description #f)
(home-page #f)
(license #f)))As you see, some fields have been left out, because elm.json doesn’t specify them for applications. Most importantly, we need to fill the source field, such as this:
(define-public puzzle-draw-elm
(let ((elm-url ...) ...)
(package
(name "puzzle-draw-elm")
(version "current")
(source "/home/rob/puzzle-draw/web")
(build-system elm-application-build-system)
(arguments
`(#:elm-modules ((("Main.elm") . "web.js"))))
...)))Here, we’re not defining a “proper” origin for the source, but just telling Guix to use a local checkout.
It works!
$ guix build -L . puzzle-draw-elm
[...]
/gnu/store/dd3i77nimvp17vgrak14yshd10mfm66c-puzzle-draw-elm-current
$ ls /gnu/store/dd3i77nimvp17vgrak14yshd10mfm66c-puzzle-draw-elm-current
web.jsPackaging the Elm reactor
Finally, let’s use our new build systems to package Elm more fully. We can package the reactor web app by calling
$ guix import elm ui/browser/elm.jsonwithin the Elm compiler repository. With some tweaks, we get
(define elm-reactor
(let
((elm-virtual-dom ...)
...)
(package
(name "elm-reactor")
(version "0.19.0")
(source
(origin
(method url-fetch)
(file-name "elm-0.19.0.tar.gz")
(uri "https://github.com/elm/compiler/archive/0.19.0.tar.gz")
(sha256
(base32 "0g4risrjrvngz3j4wf432j82gjcc8i1b7l5lwbb0fhr24hvz6ka9"))
; FIXME: extract reactor subdirectory, there must be a better way to do this
(snippet #~(begin
(use-modules (guix build utils) (ice-9 ftw))
(let ((files (scandir "." (lambda (f) (not (or (equal? f ".")
(equal? f "..")
(equal? f "ui")))))))
(for-each delete-file-recursively files)
(copy-recursively "ui/browser" ".")
(delete-file-recursively "ui"))))))
(build-system elm-application-build-system)
(arguments '(#:elm-modules ((("Errors.elm" "Index.elm" "NotFound.elm") . "elm.js"))))
(native-inputs
`(("elm-virtual-dom" ,elm-virtual-dom)
("elm-parser" ,elm-parser)
("elm-time" ,elm-time)
("elm-url" ,elm-url)
("elm-svg" ,elm-svg)
("elm-core" ,elm-core)
("elm-http" ,elm-http)
("elm-html" ,elm-html)
("elm-json" ,elm-json)
("elm-markdown" ,elm-markdown)
("elm-project-metadata-utils"
,elm-project-metadata-utils)
("elm-browser" ,elm-browser)))
(synopsis "Elm's reactor, internal to elm")
(description "Elm's reactor")
(home-page "https://elm-lang.org")
(license bsd-3))))The interesting bit is the source manipulation snippet, which succeeds in manipulating the source tree to consist of the reactor sources only, in a horrendously messy way. Additionally, we list the Elm source files as the elm-modules argument.
With our new elm-reactor package, we can build the full Elm tool, by modifying the packaging from the previous article slightly:
(define-public elm
(package
(name "elm")
(version "0.19.0")
(source
(origin
(method git-fetch)
(file-name (git-file-name name version))
(uri (git-reference
(url "https://github.com/elm/compiler/")
(commit version)))
(sha256
(base32 "0s93z9vr0vp5w894ghc5s34nsq09sg1msf59zfiba87sid5vgjqy"))
(patches
(search-patches "elm-include-reactor.patch"
"elm-compiler-relax-glsl-bound.patch"
"elm-compiler-fix-map-key.patch"))))The first difference is that instead of patching out the reactor, we patch the build to read a Javascript file from disk instead of calling out to itself:
$ cat elm-include-reactor.patch
diff --git a/ui/terminal/src/Develop/StaticFiles.hs b/ui/terminal/src/Develop/StaticFiles.hs
index 3659d112..f77635c7 100644
--- a/ui/terminal/src/Develop/StaticFiles.hs
+++ b/ui/terminal/src/Develop/StaticFiles.hs
@@ -89,7 +89,7 @@ sansFontPath =
elm :: BS.ByteString
elm =
- $(bsToExp =<< runIO Build.compile)
+ $(bsToExp =<< runIO (Build.readAsset "elm.js"))
Continuing,
(build-system haskell-build-system)
(arguments
`(#:phases
(modify-phases %standard-phases
(add-after 'unpack 'unpack-reactor
(lambda* (#:key inputs #:allow-other-keys)
(copy-file
(string-append (assoc-ref inputs "elm-reactor") "/elm.js")
"ui/browser/assets/elm.js"))))))we add a phase to the build to copy over that Javascript file from elm-reactor after unpacking the Elm sources.
(inputs
`(("elm-reactor" ,elm-reactor)
("ghc-ansi-terminal" ,ghc-ansi-terminal)
...))
(home-page "https://elm-lang.org")
(synopsis "The `elm` command line interface, including `elm reactor`.")
(description
"This package provides Elm, a statically-typed functional programming
language for the browser. It includes commands for developers such as
@command{elm make} and @command{elm repl}.")
(license license:bsd-3)))We add elm-reactor to the inputs, and it’s done.
$ guix package -i elm
[...]
The following package will be installed:
elm 0.19.0 /gnu/store/sfj3rdx3rlc6mgnwgs7y7vhkvpbvdxpn-elm-0.19.0
[...]
$ elm reactor
Go to <http://localhost:8000> to see your project dashboard.That’s it for part 2. The code is available on branch elm of github.com/robx/guix. The elm package itself in gnu/packages/elm.scm. At this point, none of the content of this article has made it to Guix upstream, because I haven’t submitted patches. While all of this works well enough, there’s some work to be done before I’d want to submit this upstream, which I currently don’t plan on doing:
- Improve some of the hacky parts, such as
- unpacking source archives in the build system,
- detecting Elm package inputs.
- Find a better way to specify the
elm makecommand line, such as callingmake buildif there’s aMakefilearound. - Generally clean up the code a bit.
- Split the work into patches that satisfy the somewhat arcane commit message requirements of upstream.
- Decide on how to package Elm package dependencies. I suspect keeping them application-local as we did here is not how the Guix project would prefer to handle this.
- Update to Elm 0.19.1.
I’d be thrilled if someone chose to take care of this, and would be happy to assist!
Tune in again for part 3, where we’ll wrap this up by figuring out how to serve an Elm web application on Guix.