Amalgamation

This page contains all posts amalgamated into a single page SQL-style. You can use this if you want to doomscroll all my posts, I guess. For me, it was a nice way to learn about how to improve load times using content-visibility.

A surprisingly simple way to package Deno applications for Nix

Introduction

Recently, I was working on a Deno project which I wanted to package for Nix. Usually, when packaging a piece of software for Nix, there exists a language-specific stdenv.mkDerivation derivative which works to bridge the gap between between the langauge-specific package managers and Nix. These are functions like buildNpmPackage and buildPythonPackage but, alas, there is no buildDenoPackage.

Deno is particularly tricky (as compared to for example TCL), because it uses “URL imports” to import directly from URLs as runtime. Doing so is obviously not deterministic which means that bundling Deno applications becomes a bit of a challenge.

In this post, I will go over why none of the existing, community-driven solutions worked for me, what I did instead, and some of the potential drawbacks of my solution.

Existing solution

During my initial research, I found this thread disussing my exact issue: wrapping Deno applications in Nix. The thread settles on using deno2nix. deno2nix works by parsing the lockfiles that Deno generates¹ and generating a matching Nix derivation.

There’s a lot of work involved in what deno2nix does; it has to parse Deno’s lockfile format, clean it up, then generate a matching Nix derivation. All of this code has potential for bugs. Nothing illustrates this better than this issue. It essentially boils down to Deno’s resolution algorithm setting a different User-Agent header than what the Nix builder did. esm.sh was using the user-agent to send different content to Deno than to the browser

The underlying issue here is that deno2nix is trying to replicate the exact behavior of Deno, which is a hard task.

deno2nix also does not support NPM modules (i.e. imports using an npm: specifier) at the time of writing. Doing so will likely cause the amount of code in the repo to double, since NPM packages are handled entirely differently both in the lockfile format and Deno’s resolution algorithm.

My solution

After fighting with deno2nix for a while, I decided to take a different approach.

Deno supports a pretty niche subcommand: deno vendor. This command downloads all dependencies of the given file into a folder. This is called vendoring, hence the name of the command. It also generates an import map² which can be used to make Deno use these local dependencies, rather than fetching from online.

This command is very convenient for us because we can use it to download and fix bundles ahead of time. To make evaluation pure, we can fix the hash of the output (i.e. a fixed output derivation).

In case this sounds too abstract, here’s an example. Suppose we have a simple program which just prints a random string. main.ts just contains:

import { bgMagenta } from "https://deno.land/[email protected]/fmt/colors.ts";
import { generate } from "https://esm.sh/randomstring";

const s = generate();
console.log("Here is your random string: " + bgMagenta(s));

First, we’ll build the vendor directory. We pull out the src attribute into a separate variable, as it is shared between both derivations. The fact that we specify the outputHash attribute means that this is going to be a fixed-output derivation. As such, the builder will be allowed network access in return to guaranteeing that the output has a specific hash.

# This could of course be anywhere, like a GitHub repository.
src = ./.;

# Here we build the vendor directory as a separate derivation.
random-string-vendor = stdenv.mkDerivation {
  name = "random-string-vendor";

  nativeBuildInputs = [ deno ];

  inherit src;
  buildCommand = ''
    # Deno wants to create cache directories.
    # By default $HOME points to /homeless-shelter, which isn't writable.
    HOME="$(mktemp -d)"

    # Build vendor directory
    deno vendor --output=$out $src/main.ts
  '';

  # Here we specify the hash, which makes this a fixed-output derivation.
  # When inputs have changed, outputHash should be set to empty, to recalculate the new hash.
  outputHashAlgo = "sha256";
  outputHashMode = "recursive";
  outputHash = "sha256-a4jEqwyp5LoORLYvfYQmymzu9448BoBV5luHnt4BbMg=";
};

Let’s try building this and taking a peek inside. In the transcript below, you will see that the output contains a directory hierarchy corresponding to our dependencies. It also contains import_map.json at the top level.

$ nix-build vendor.nix
/nix/store/…-random-string-vendor
$ tree /nix/store/…-random-string-vendor
/nix/store/…-random-string-vendor
├── deno.land
│   └── [email protected]
│       └── fmt
│           └── colors.ts
├── esm.sh
│   ├── v135
│   │   ├── @types
│   │   │   └── [email protected]
│   │   │       └── index.d.ts
│   │   ├── [email protected]
│   │   │   └── denonext
│   │   │       └── randombytes.mjs
│   │   └── [email protected]
│   │       └── denonext
│   │           └── randomstring.mjs
│   ├── [email protected]
│   └── [email protected]
└── import_map.json

Now we can build the actual application. We are going to create a little wrapper script which will invoke Deno with the right arguments. We use --import-map to have Deno use our local dependencies and --no-remote to force Deno not to fetch dependencies at run-time, in case random-string-vendor is outdated (i.e. doesn’t include all dependencies imported by the script).

random-string = writeShellScript "random-string" ''
  ${deno}/bin/deno run \
    --import-map=${random-string-vendor}/import_map.json \
    --no-remote \
    ${src}/main.ts -- "$@"
'';

That’s basically all there is to it! The great thing about this approach is that it (by definition) uses Deno’s exact resolution algorithm. We don’t run into trouble with esm.sh because Deno sets the correct UA. That’s an entire class of bugs eliminated!

Shortcomings

It’s not all sunshine and rainbows, though. There are some significant drawbacks to this approach which I will go over in this section.

First of all, the vendor subcommand is woefully undercooked. npm: specifiers are just silently ignored. It is outlined in this issue, which has been open for quite some time. In general, it doesn’t seem like this command has been getting a whole lot of love since its introduction, probably on account of being so niche.

Nevertheless, when Deno does finally get support for vendoring NPM modules, this module will automatically also support them. This is in stark contrast with deno2nix which would require a lot of work to support npm: specifiers.

The second major issue is that this approach doesn’t make good use of caching. The random-string-vendor-derivation we constructed above is essentially a huge blob; if we change a single dependency, the entire derivation is invalidated. If I understand deno2nix correctly, it actually makes a derivation for each dependency and then uses something akin to symlinkJoin to combine them. Such an approach allows individual dependencies to be cached and shared in the Nix store.

The issue of caching is tangentially related to some of the issues outlined by @volth’s Status of lang2nix approaches. A lot of their criticism also applies here.

Conclusion

In this post I have described a simple approach to packaging Deno applications for Nix. I much prefer it to deno2nix simply because I understand exactly how it works. Even then, there are some major drawbacks to using this method. Before implementing this approach in your project, you should consider if those trade-offs make sense for you.

For the uninitiated, I suggest reading the official introduction to Deno’s lockfiles. In essence, lockfiles are just a mapping from URLs to their expected hashes. Their purpose is for locking dependencies to specific versions. Since this sounds a lot like what Nix is trying to do (though admittedly at a much smaller scale) they are usually the input for the various mkDerivation derivatives. ↩
Import maps allow you to tell Deno “when you see an import statement for A, you should actually import B.” The actual file is just a JSON object where the keys are A and the values are B. If this is new to you, you might want to check out the official documentation. ↩

I made a thing and it sucks ass

It is not a big thing. It’s just a little utility that shows a hover preview when the mouse hovers a link. I think it’s kind of useful for checking where a link goes when I see text like “You can read more about that here”.

A screenshot of a blog post. The mouse hovers a link. A popup above the link shows the title and description of the page

The thing is, it didn’t take me very long to make this. An hour or two including setting up the user script manager and whatnot. However, the issue is: I just keep finding edge cases. Here’s a couple of examples:

Most sites don’t actually implement the OpenGraph protocol so the popups are kind of useless unless I also implement metadata extraction myself. For now I just extract a <title> if one is present but realistically I also need to generate a description/summary.
Popups tend to ‘leak’ on SPAs because they simply remove the <a> tag from the DOM so the mouseleave event doesn’t fire. That’s pretty easy to handle; just treat element destruction like mouseleave.
I have to use a proxy to get around CORS issues, but fetch doesn’t support actually using proxies and the proxy I use doesn’t handle redirects, so if a link leads to a redirect, I just get a generic NetworkError.

Each of these issues on their own isn’t the end of the world, but the cumulative time spent fixing bugs just isn’t worth it for a small, semi-useful utility.

This whole ordeal reminded me a lot of my uwuifier extension; a silly idea which ended up taking multiple iterations over the span of two years, APIs spanning the entire history of the DOM, and a way too intimate understanding of the DOM tree’s structure. And it’s still not finished! I stuck with that project, because I thought it was funny and some of my friends were using it, but I just don’t have that motivation this time.

The tl;dr is that I made a kind of useful utility, but to implement it fully would require a disproportionate amount of work. So much work that I don’t think I’ll finish the thing. And that’s why I hate making stuff for the web.

Link rot and the innevitable heat death of the internet

Introduction

Yesterday I was reading the slides for Maciej Ceglowski’s talk on The Website Obesity Crisis. It’s a really good talk. I highly recommend giving it a read (or a watch). You might be a bit skeptical since it was written in 2015, but I think it is still highly relevant, even to HTML kiddies¹ such as myself. Much like all good dystopian works, it identified the beginnings of a trend which has now become such a huge issue that the original work seems almost prophetic.

Anyway, in one of the slides he talks about an experiment by some Adam Drake guy.

Adam Drake wrote an engaging blog post about analyzing 2 million chess games. Rather than using a Hadoop cluster, he just piped together some Unix utilities on a laptop, and got a 235-fold performance improvement over the ‘Big Data’ approach.

It seemed pretty interesting, so I clicked the link and… nothing? The link just took me to his homepage². After a bit of detective work I figured out that the server was just redirecting me to the homepage instead of showing a 404. Specifically, the Curl output below shows that any request made to his old domain (aadrake.com) is met with a 301 “permanently moved” pointing to the index page of his new domain (adamdrake.com), completely disregarding the actual resource being requested.

Curl output

Emphasis mine.

$ curl -v http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
*   Trying 192.64.119.137:80...
* Connected to aadrake.com (192.64.119.137) port 80 (#0)
> GET /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html HTTP/1.1
> Host: aadrake.com
> User-Agent: curl/8.1.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Sun, 22 Oct 2023 21:12:20 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 56
< Connection: keep-alive
< Location: https://adamdrake.com
< X-Served-By: Namecheap URL Forward
< Server: namecheap-nginx
<
<a href='https://adamdrake.com'>Moved Permanently</a>.
* Connection #0 to host aadrake.com left intact

He is not the only one doing this. I’ve encountered multiple websites with this 404-is-index strategy and I hate it. It’s the HTTP equivalent of going “what? nah bro i never said that”. Why is your server gaslighting me?

Link rot

Even then, it is not the end of the world. It is just mildly annoying. I do, however, think that it is part of the much larger issue of link rot. The term “link rot” refers to the tendency for hyperlinks to ‘go bad’ over time, as the resources are relocated or taken offline. It is a kind of digital entropy that is slowly engulfing much of the older/indie web.

The 404-is-index strategy is a particularly bad case of link rot. Unlike a usual 404 which presents the reader with some kind of error page, a 301 happens quietly. For example many bookmark managers will quietly update references when it encounters a 301. This means that your bookmarks just disappear because they’ve been ‘moved’ to the website’s index page.

Maciej Ceglowski’s talk is yet another example. His link to aadrake.com was quietly broken and with that another edge in the huge directed graph that is the open web.

What can I do about it?

Okay so link rot is bad. How can I avoid further contributing to link entropy? Just to be clear the “I” in the section title does not refer to you, the reader, and that question before wasn’t rhetorical. I’m not going to pretend to have all the answers and this most certainly isn’t a how-to guide. Think of it more like a kind of public diary of my feeble attempts to fight my own innevitable breaking of this site.

I have tried to be very careful about the layout of the site, URI-wise. For example, all posts are located at /posts/<glob>.html and I never change the glob. I also try to avoid leaking implementatin details in the URL. This is rather easy since this is a static site, but in the future I might add dynamic elements. In that case I’ll try not to introduce extensions like .cgi or .php that couple the URL to the underlying tech stack.

Still, I fear it might not be enough. After all, perfection is the enemy of progress. As I learn and become a better web developer I will surely realise fallacies in my original layout. Already, I am growing a tad annoyed at the fact that images and CSS are grouped under /assets/. It’s rather arbitrary to decide that images and CSS are “assets” but HTML isn’t.

Maybe then I can use 301 for good; as resources are relocated I can maintain a list of rewrite rules. Perhaps I could model it as database migrations. In the database world, when one modifies a schema (e.g. adding or removing a column) that change is associated with a bit of code specifying how to go from one schema to another. That way, all the data (existing hyperlinks) remains valid as the schema (site layout) changes.

There are also more radical approached like IPFS. In this protocol resources are addressed by a content ID computed from their content rather than their location. That way, multiple peers can host it, reducing the likelyhood of the website ever disappearing or going down. It’s all very smart, but I doubt we can convince everyone to switch protocols just like that.

Conclusion

The tl;dr is this: There’s an issue on the web where links tend to ‘go bad’. That is, the location of the underlying resources change and hyperlinks pointing to the old location are broken. This problem is especially prevalent in the IndieWeb community because we don’t have teams of engineers managing our sites and checking for backwards compatibility.

So far my only solution is “be very careful” which isn’t a very inspiring conclusion. In the coming weeks I’ll look into making some automated testing, so I can be notified when I accidentally change the site layout in such a way that it breaks old links.

For lack of a better terminology I’m just going to reuse that of script kiddies. ↩
A home that included the phrase “While I specialize in executive advising on leadership and process, I can also dive into deep technical problems with Data Science”, a sentence so douchebag-techbro-y it made me reconsider whether tech was really the industry for me. ↩

How to treat libgit2 blobs as file handles

I figured I’d document the solution to my hyper-specific problem in case anyone in the future has the same issue.

So here’s the setup: I am using libgit2 to operate on the contents of some files stored in a repository and I would like to pass the contents of a blob (i.e. a piece of data in the Git store) to foo, but libgit2 only lets me access the content of the blob through git_blob_rawcontent which returns a char * and foo only operates on FILE *s.

The POSIX standard comes to the rescue! It defines the special function fmemopen which allows one to construct a file handle from a piece of memory. Here’s an example from the docs:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

static char buffer[] = "foobar";

int main (void)
{
    int ch;
    FILE *stream;

    stream = fmemopen(buffer, strlen (buffer), "r");
    if (stream == NULL) {
        perror("failed to fmemopen buffer");
        exit(EXIT_FAILURE);
    }

    while ((ch = fgetc(stream)) != EOF) {
        printf("Got %c\n", ch);
    }

    fclose(stream);
    return EXIT_SUCCESS;
}

It produces the following output.

Got f
Got o
Got o
Got b
Got a
Got r

Just what I was looking for! With this I can wrap the result of git_blob_rawcontent in a FILE * and pass it to foo.

#include <git2.h>
#include <stdio.h>
git_blob *blob = /* ... */;
FILE *fp = fmemopen(git_blob_rawcontent(blob), git_blob_rawsize(blob), "rb");
foo(fp);

Hopefully this is the solution to your hyper-specific problem as well :^)

Common font fallbacks

I quite like @yesiamrocks’s CSS fallback fonts repository. It contains a lot of common CSS fallback chains. My only gripe is that I can’t see the fonts in use. Here, I’ve taken the liberty of converting the Markdown to some HTML with examples of each CSS chunk.

Keep in mind that if you don’t have the fonts installed on your system, you will see the first fallback that is installed. Any browser worth its salt will let you see this using its development tools. For example, here’s how to do it in Firefox.

Regrettably, it is not possible to highlight failing fonts, as doing so would allow for easy fingerprinting. This exact issue has been discussed by the CSS Working Group.

This document is split into 3 sections:

Sans-serif fonts
Serif fonts
Monospace fonts