Amalgamation

This page contains all posts amalgamated into a single page SQL-style. You can use this if you want to doomscroll all my posts, I guess. For me, it was a nice way to learn about how to improve load times using content-visibility.

Documenting open source code

Today a friend of mine mentioned an issue they’d had with Radarr, the movie organizer for torrent users. Since it’s an open source project, I figured I could just fix the issue myself – easy peasy. But man, the code for radarr is obtuse… Everything is encapsulated in a subclass of an abstract base class implementing an interface for a service (sic) and there’s not a single comment to be found.

Documentation of open source projects is yet another instance of the 80/20 rule: relatively little documentation goes a long way in helping new contributors find their footing in a project. It can be as simple as writing a couple of lines at the top of every file describing what the module does, or throwing a couple of readme.txts in the folder for each major component.

Wren is of course the ideal. Its source code reads more like a book than a program, though such a comparison is hardly fair; Wren is as much a tutorial as it is an actual live programming language¹. Obviously, not every project can dedicate that much time to documentation.

The Neovim project strikes a much more reasonable balance. Let’s take a look at a src/nvim/undo.c, it contains some rather tricky code for managing the editor’s multi-level undo tree. The reason I know this isn’t because I’m some kind of Neovim expert, I just read comment at the top of the file!

// undo.c: multi level undo facility

That comment is followed by a longer comment explaining how the main data structure of the module works. I think that’s a much better use of the space at the top of the file than the ever-present license comment.

If the Radarr maintainers had been better about documenting their project, I would’ve already finished making my change. Instead, I’m stuck doing detective work trying to figure out what part of the code I’m even looking for. I do of course understand that writing documentation takes time but I think it is well worth it because it supercharges one-time contributors.

As of c2a75f1e there are 3811 semicolons and 2665 comment lines. That’s a ratio of approximately 0.70 comments for every statement! ↩

A surprisingly simple way to package Deno applications for Nix

Introduction

Recently, I was working on a Deno project which I wanted to package for Nix. Usually, when packaging a piece of software for Nix, there exists a language-specific stdenv.mkDerivation derivative which works to bridge the gap between between the langauge-specific package managers and Nix. These are functions like buildNpmPackage and buildPythonPackage but, alas, there is no buildDenoPackage.

Deno is particularly tricky (as compared to for example TCL), because it uses “URL imports” to import directly from URLs as runtime. Doing so is obviously not deterministic which means that bundling Deno applications becomes a bit of a challenge.

In this post, I will go over why none of the existing, community-driven solutions worked for me, what I did instead, and some of the potential drawbacks of my solution.

Existing solution

During my initial research, I found this thread disussing my exact issue: wrapping Deno applications in Nix. The thread settles on using deno2nix. deno2nix works by parsing the lockfiles that Deno generates¹ and generating a matching Nix derivation.

There’s a lot of work involved in what deno2nix does; it has to parse Deno’s lockfile format, clean it up, then generate a matching Nix derivation. All of this code has potential for bugs. Nothing illustrates this better than this issue. It essentially boils down to Deno’s resolution algorithm setting a different User-Agent header than what the Nix builder did. esm.sh was using the user-agent to send different content to Deno than to the browser

The underlying issue here is that deno2nix is trying to replicate the exact behavior of Deno, which is a hard task.

deno2nix also does not support NPM modules (i.e. imports using an npm: specifier) at the time of writing. Doing so will likely cause the amount of code in the repo to double, since NPM packages are handled entirely differently both in the lockfile format and Deno’s resolution algorithm.

My solution

After fighting with deno2nix for a while, I decided to take a different approach.

Deno supports a pretty niche subcommand: deno vendor. This command downloads all dependencies of the given file into a folder. This is called vendoring, hence the name of the command. It also generates an import map² which can be used to make Deno use these local dependencies, rather than fetching from online.

This command is very convenient for us because we can use it to download and fix bundles ahead of time. To make evaluation pure, we can fix the hash of the output (i.e. a fixed output derivation).

In case this sounds too abstract, here’s an example. Suppose we have a simple program which just prints a random string. main.ts just contains:

import { bgMagenta } from "https://deno.land/[email protected]/fmt/colors.ts";
import { generate } from "https://esm.sh/randomstring";

const s = generate();
console.log("Here is your random string: " + bgMagenta(s));

First, we’ll build the vendor directory. We pull out the src attribute into a separate variable, as it is shared between both derivations. The fact that we specify the outputHash attribute means that this is going to be a fixed-output derivation. As such, the builder will be allowed network access in return to guaranteeing that the output has a specific hash.

# This could of course be anywhere, like a GitHub repository.
src = ./.;

# Here we build the vendor directory as a separate derivation.
random-string-vendor = stdenv.mkDerivation {
  name = "random-string-vendor";

  nativeBuildInputs = [ deno ];

  inherit src;
  buildCommand = ''
    # Deno wants to create cache directories.
    # By default $HOME points to /homeless-shelter, which isn't writable.
    HOME="$(mktemp -d)"

    # Build vendor directory
    deno vendor --output=$out $src/main.ts
  '';

  # Here we specify the hash, which makes this a fixed-output derivation.
  # When inputs have changed, outputHash should be set to empty, to recalculate the new hash.
  outputHashAlgo = "sha256";
  outputHashMode = "recursive";
  outputHash = "sha256-a4jEqwyp5LoORLYvfYQmymzu9448BoBV5luHnt4BbMg=";
};

Let’s try building this and taking a peek inside. In the transcript below, you will see that the output contains a directory hierarchy corresponding to our dependencies. It also contains import_map.json at the top level.

$ nix-build vendor.nix
/nix/store/…-random-string-vendor
$ tree /nix/store/…-random-string-vendor
/nix/store/…-random-string-vendor
├── deno.land
│   └── [email protected]
│       └── fmt
│           └── colors.ts
├── esm.sh
│   ├── v135
│   │   ├── @types
│   │   │   └── [email protected]
│   │   │       └── index.d.ts
│   │   ├── [email protected]
│   │   │   └── denonext
│   │   │       └── randombytes.mjs
│   │   └── [email protected]
│   │       └── denonext
│   │           └── randomstring.mjs
│   ├── [email protected]
│   └── [email protected]
└── import_map.json

Now we can build the actual application. We are going to create a little wrapper script which will invoke Deno with the right arguments. We use --import-map to have Deno use our local dependencies and --no-remote to force Deno not to fetch dependencies at run-time, in case random-string-vendor is outdated (i.e. doesn’t include all dependencies imported by the script).

random-string = writeShellScript "random-string" ''
  ${deno}/bin/deno run \
    --import-map=${random-string-vendor}/import_map.json \
    --no-remote \
    ${src}/main.ts -- "$@"
'';

That’s basically all there is to it! The great thing about this approach is that it (by definition) uses Deno’s exact resolution algorithm. We don’t run into trouble with esm.sh because Deno sets the correct UA. That’s an entire class of bugs eliminated!

Shortcomings

It’s not all sunshine and rainbows, though. There are some significant drawbacks to this approach which I will go over in this section.

First of all, the vendor subcommand is woefully undercooked. npm: specifiers are just silently ignored. It is outlined in this issue, which has been open for quite some time. In general, it doesn’t seem like this command has been getting a whole lot of love since its introduction, probably on account of being so niche.

Nevertheless, when Deno does finally get support for vendoring NPM modules, this module will automatically also support them. This is in stark contrast with deno2nix which would require a lot of work to support npm: specifiers.

The second major issue is that this approach doesn’t make good use of caching. The random-string-vendor-derivation we constructed above is essentially a huge blob; if we change a single dependency, the entire derivation is invalidated. If I understand deno2nix correctly, it actually makes a derivation for each dependency and then uses something akin to symlinkJoin to combine them. Such an approach allows individual dependencies to be cached and shared in the Nix store.

The issue of caching is tangentially related to some of the issues outlined by @volth’s Status of lang2nix approaches. A lot of their criticism also applies here.

Conclusion

In this post I have described a simple approach to packaging Deno applications for Nix. I much prefer it to deno2nix simply because I understand exactly how it works. Even then, there are some major drawbacks to using this method. Before implementing this approach in your project, you should consider if those trade-offs make sense for you.

For the uninitiated, I suggest reading the official introduction to Deno’s lockfiles. In essence, lockfiles are just a mapping from URLs to their expected hashes. Their purpose is for locking dependencies to specific versions. Since this sounds a lot like what Nix is trying to do (though admittedly at a much smaller scale) they are usually the input for the various mkDerivation derivatives. ↩
Import maps allow you to tell Deno “when you see an import statement for A, you should actually import B.” The actual file is just a JSON object where the keys are A and the values are B. If this is new to you, you might want to check out the official documentation. ↩

Common font fallbacks

I quite like @yesiamrocks’s CSS fallback fonts repository. It contains a lot of common CSS fallback chains. My only gripe is that I can’t see the fonts in use. Here, I’ve taken the liberty of converting the Markdown to some HTML with examples of each CSS chunk.

Keep in mind that if you don’t have the fonts installed on your system, you will see the first fallback that is installed. Any browser worth its salt will let you see this using its development tools. For example, here’s how to do it in Firefox.

Regrettably, it is not possible to highlight failing fonts, as doing so would allow for easy fingerprinting. This exact issue has been discussed by the CSS Working Group.

This document is split into 3 sections:

Sans-serif fonts
Serif fonts
Monospace fonts